Hard-label Manifolds: Unexpected advantages of query efficiency for finding on-manifold adversarial examples
Authors: Washington Garcia
Pin-Yu Chen
Somesh Jha
Hamilton Scott Clouse
Kevin Butler
Venue:  (2021)
Keywords: hard-label attacks
adversarial machine learning
generalization

Abstract: Designing deep networks robust to adversarial examples remains an open problem. Likewise, recent zeroth order hard-label attacks on image classification tasks have shown comparable performance to their first-order alternatives. It is well known that in this setting, the adversary must search for the nearest decision boundary in a query-efficient manner. State-of-the-art (SotA) attacks rely on the concept of pixel grouping, or super-pixels, to perform efficient boundary search. It was recently shown in the first-order setting, that regular adversarial examples leave the data manifold, and on-manifold examples are generalization errors. In this paper, we argue that query efficiency in the zeroth-order setting is connected to the adversary's traversal through the data manifold. In particular, query-efficient hard-label attacks have the unexpected advantage of finding adversarial examples close to the data manifold. We empirically demonstrate that against both natural and robustly trained models, an efficient zeroth-order attack produces samples with a progressively smaller manifold distance measure. Further, when a normal zeroth-order attack is made query-efficient through the use of pixel grouping, it can make up to a two-fold increase in query efficiency, and in some cases, reduce a sample's distance to the manifold by an order of magnitude.

Under review as a conference paper at ICLR 2021
HARD-LABEL MANIFOLDS: UNEXPECTED ADVAN- TAGES OF QUERY EFFICIENCY FOR FINDING ON- MANIFOLD ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Designing deep networks robust to adversarial examples remains an open problem. Likewise, recent zeroth order hard-label attacks on image classification tasks have shown comparable performance to their first-order alternatives. It well known that in this setting, the adversary must search for the nearest decision boundary in a query-efficient manner. State-of-the-er (SotA) attacks rely on the concept of pixel grouping, or super-pixels to perform efficient boundary search. It was recently shown in the first-order setting, that regular adversarial examples leave the data manifold, and on- ma-imooifl examples In this paper, we argue that query efficiency in the zeroth-order setting is connected the adversary's traversal through the data manifold. In particular, query-efficient hard-label attacks have the unexpected advantage of finding adversarial examples close to the data manifold. We empirically demonstrate that against both natural and robustly trained models, an efficient zeroth-order attack produces samples with a progressively smaller manifold distance measure. Further, when a normal zeroth-orde attack is made query-efficient through the use of pixel grouping, it can make up to a two-fold increase in query efficiency, and some cases, reduce a sample's distance to the manifold by order of magnitude.
generalization errors (Stutz al. 2019)
INTRODUCTION
Adversarial examples in the context of deep learning models were originally investigated as blind spots in classification (Szegedy et al. 2013 Goodfellow et al., 2014). Formalized methods for discovering these blind spots emerged, referred to as gradient- level attacks, and became the first style reach widespread attention by the deep learning community (Papernot et al. 2016 Moosavi Dezfooli et al., 2015 Carlini & Wagner 2016 2017). In order to compute the necessary gradient information, such techniques required access to model parameters and a sizeable query budget, needing surrogate information to be competitive (Papernot et al., (017). This naturally led to the creation of score-level attacks, which only require the confidence values output by machine learning models (Fredrikson et al. 2015)] Tramer et al. 2016 Chen et al. 2017: Ilyas et al., 2018). However, the grand prize of adversarial ML (AML), hard-label attacks, have been proposed in very recent years. These methods, which originated from random-wai walk on the decision boundary (Brendel al. 2017), have been carefully refined to offer convergence guarantees (Cheng al., 2019), query efficiency (Chen et al.) 2019 Cheng et al.) 2020). and simplicity through super-pixel grouping (Chen Gu 2020), without ever sacrificing earlier advances.
Despite the steady improvements of hard-label attacks, open questions persist about their behavior, and AMI. attacks at large. The existence of adversarial samples were originally assumed to lie rare pockets of the input space (Goodfellow al. (2014), but this was later challenged by the boundary tilting assumption (Tanay & Griffin 2016) Gilmer et al.) 2018), which adopts a "data- tdata-geometric" view of the input space living manifold. This is supported by [Stutz et al. (2019), who suggest that regular adversarial examples leave the data manifold, while on-manifold adversarial examples generalization errors.
In this paper, we adopt the boundary- tilting assumption and demonstrate an unexpected benefit of query-efficient zeroth order attacks; such attacks are more likely discover on-manifold examples,

---

Under review as a conference paper at ICLR 2021
and it is primarily enabled by the use of down-scaling techniques, such super-pixel grouping. This is initially counter-intuitive since down-scaling techniques reduce the search dimension, which artificially limits the search space, and can lead to worse (farther-away) adversarial examples. Our results suggest, however, that super-pixels help eliminate the search space of off-manifold adversarial examples, leading examples which are truly generalization errors. With this knowledge, it is possible to rethink the design of hard-label attacks, towards those which resemble attack (b) in Figure rather than (a) or (c).
Our specific contributions are as follows:
Reveal new insights of manifold feedback during query-efficient zeroth-order search. We describe an approach for extending dinensioonrrduuction technqi- ues in the score-level setting (Tu et al., 2019) to hard- label attacks. Afterwards, score (Heusel et al. 2018) as an agngossi means for estimating distance the sampled submanifold. This measure allows to empirically demonstrate the connec- tion between query efficiency and manifold feedback from the model, beyond the known convergence rates tied to dimensionality (Nesterov Spokoiny 2017). We later tie this result to known behavior in the gradient- level setting (Engstrom al. a 2019). which shows that manifold information is leaked from models in the hard- label setting, regardless of their robustness.
b
propose the use of FID
Sampled
Image space
submanifold
a. off- off-manifold, low-similarity b: on on-manifold, high-similarity c: on-manifold, low-similarity search space
Figure 1: Our interpretation of zeroth- order attack behavior in the context of boundary tilting (Tanay & Griffin 2016): a) zeroth-order attack targeting low-level features, leaving the manifold, b) an efficient zeroth-order attack targeting mostly high-level features, floating along the manifold, and c) manifold- -ased zeroth-order attack next to the manifold, but sacrificing similarity.
A Attack-age agnostic method for super-pixel grouping. We show that bilinear down-scaling of the input space act as a form of super-pixel grouping, which yields up to 140% and 210% query efficiency gain for previously- proposed HSJA (Chen et al.) 2019) and Sign-OPT at- tack (Cheng al. 2020), respectively. These results align with previously observed behavior in the score- level setting (Tu et al. (2019).
Introduction of manifold distance oracle. Our anal- ysis covers comprehensive array of datasets, model regularization methods, and Ly-norm settings from the literature. Regardless of the setting, we observe consistent behavior of leveraging manifold information during query-efficient attacks. Thus we propose an information coormation-thhorttiaall formulation and interpretation of the noisy manifold distance oracle, which enables zeroth-order attacks to craft on- manifold examples. Studying this problem may assist understanding the fundamental limits and utility of hard-label attacks.
RELATED WORK
Since the original discovery of adversarial samples (Szegedy al. 2013 Goodfellow al.] (014)) and later formulations based on optimization (Carlini & Wagner 2016 Moosavi-Dezoool et al. 2015). the prevailing question was why such examples existed. The original assumption was that adversarial examples lived in low-probability pockets of the input space, and were thus never encountered during parameter optimization (Szegedy et al. 2013). This effect was believed to be amplified by the linearity of weight activations in the presence of small perturbations (Goodfellow et al., 2014). These assumptions were later challenged by the manifold assumption, which in summary claims that 1) the train and test sets of model only occupy sub-manifold of the true data, while the decision boundary lies close to samples on and beyond the sub-manifold (Tanay & Griffin| 2016), and 2) the "data geometric" view, where high-dimensional geometry of the true data manifold enables low-probability error set to exist (Gimerr et al. 2018). Likewise the manifold assumption describes adversarial samples as leaving the manifold, which has inspired many defenses based on projecting such samples back to the data manifold (Jalal et al. 2019 Samangouui et al. 2018a), and adaptive attacks for foiling these defenses (Carlini al. 2019 Carlini & Wagner 2017 Tramer et al. 2020).
2

---

Under review as a conference paper at ICLR 2021
We investigate the scenario where adversary uses zeroth-order information to either estimate the gradient direction, or find the closest decision boundary, related to previous work in the gradient-level setting by (Stutz et al. (2019). In our setting, the adversary uses the top-1 label feedback from the model to reach their goal. They can also use a low-dimensional approximation of the data manifold encourage query-efficiency. However, date it is not completely understood how this affects the traversal through the data manifold, particularly in the zeroth-order setting.
3 ZEROTH-ORDER SEARCH THROUGH THE MANIFOLD
Our primary motivation is to characterize recent zeroth-order attacks as they relate to ideas of manifold traversal (Chen et al., 2019 @019/Cheng et al.) 2020) Chen & Gu 2020). In the most common problem setting, the adversary is interested in attacking a K K-way multi-class classification model T: {1,.... K} J. Given
original example Xo. the goal is to generate adversarial example such that x is close to Xg and f (x) * (xa); where closeness is often approximated by the Ly norm of x Xg. The value of this approximation is debated in the literature (Heusel et al. 2018: Tsipras et al. 2018 Engstrom et al. 2019). Likewise we turn to alternative methods shown later for measuring closeness. First we step through the formulation for contemporary hard-label attacks, then show how dimension-reducee score-level attacks are extended to the hard-label setting, which enables analysis of zeroth-order decision boundary search.
3.1 GRADIENT-LEVER FORMULATION
For gradient-level attacks, the goal is satisfied by first assuming that f(x) = argmax; (Z()),, where (x) E K is the final (logit) layer output, and Z(x); is the prediction score for the i-th class, the stated goal is satisfied by the optimization problem.
lxx) argmin {||x xolp +c( cl(Z(x)))
for the Euclidean Lp norm and is regularization parameter. A popular choice of loss function is the Carlini & Wagner (0016)
.ppp C(-) is the loss function corresponding to the goal of the attack,
loss function.
3.2 SCORE- SCORE-LEVEL AND HARD-LABEL ATTACKS
In the gradient-level setting, we require the gradient V)(-). However, in the score-level setting af(x)
are forced to estimate Hx without access to Vf((), only evaluations of Z(-). Tu et al. (2019) version relying instead on the ranking of class predictions trom Z. In practical scenarios, the estimate is found using random gradient-free method (RGF), a scaled random full gradient estimator of V (x(),, over q random directions {u:)²_, The score-level setting was extended to several renditions of the hard-label setting, which we clarify below. In each case the goal is to approximate the gradient by g.
reformulate the previous problem
OPT-Attack For given example Xp. true label yo, and hard-label black-box function f Rd {1...., K). Cheng al. (0199) define the objective function g Rd + R as a function of search direction 0, where g(0 is the minimum distance from Xy to the nearest adversarial example along the direction 0. For the untargeted attack, g(0) corresponds to the distance to the decision boundary along direction 0, and allows for estimating the gradient as,
( + Bu.) (())
where 3 is small smoothing parameter. Notably, g()) is continuous even if f is a non-continuous step function.
Sign-OPT Cheng et al. (2020) later improved the query efficiency by only considering the sign of the gradient estimate,
3

---

Under review as a conference paper at ICLR 2021
Vg (0) n
Σ sgn (o(0 +.,))) -(0)) un-
We focus on the Sign-OPT variant, as the findings are more relevant to the current SotA.
HopSkipJJmpAAttakk Similar to Sign-OPT, HopSkip/umppAtack (HSJA) (Chen et al. 2019) uses a zeroth-order sign oracle improve Boundary Attack proposed by [rendee et al. (2017). HSJA lacks the convergence analysis of OPT Attack/Sign-OPT and relies on one-point gradient estimate. Regardless, HSJA competitive with Sign-OPT for SotA in the L2 setting.
RayS Chen & Gu 2020) propose an alternative method which is to search for the minimum decision boundary radius r from a sample Xo. along a ray direction 0. Instead of searching over Fd to minimize ( Chen et al. propose to perform ray search over directions 0 E {-1, resulting in 24 maximum possible directions. This reduction of the search resolution enables SotA query efficiency in the Loc setting with proof of convergence. The search resolution is further reduced by the hierarchical variant of RayS, which performs on-the-fly upscaling of image super-pixels.
3.3 DIMENSION-REDUCEED ZEROTH-ORDER SEARCH
The attacks described so far each represent an improvement in query efficiency under different Lp-norm scenarios. The difficulty in performing holistic analysis of their behavior lies in each attack's 's unique design. In order to characterize query-efficient attacks in a consistent way, we rely on hard-label version of the reduced-dimension scheme proposed by TTu et al. (2019). This scheme can allow to dynamically scale the expected query-efficiency up or down in a controlled manner. The reduced-diimenion search is feasible since it was shown that the intrinsic dimensionality of data is often lower than the true dimension (Amsaleg et al. 2017). Likewise, zeroth -odder attacks can exploit the known convergence rate of zeroth order gradient descent, which tied dimensionality of the vectorized input (Nesterov & Spokoiny 2017] Liu et al.) 2018). In practice this reduction is implemented through an encoding map & IRd + R for reduced dimension d' and decoding map
D Rdd + Rd. In general the adversarial sample is created by
D(O') |D'O'
x = xo x g
where O' E IRd and is optimized depending on the respective attack (e.g., Sign-OPT and HSJA), and as before, g is measure of distance to the decision boundary in direction D(0'). The mapping functions can be initialized with either an autoencoder (AE), or a pair of channel-wise bilinear transform functions (henceforth referred to as BiLN) which simply scales the input up or down depending on a fixed scaling factor. These choices were previously investigated by Tu et al. (2019) as a way to improve query efficiency in score-level attacks, and ultimately performed similarly with respect to query efficiency. For our purposes, these functions represent two distinct methods of synthesizing adversarial samples, which either rely on an approximate description of the manifold (AE), or instead use deterministic rescaling to achieve efficiency (BiLN). The inclusion of BiLN is important, because it allows to measure the scenario where the adversary has no explicit knowledge of the manifold, and only relies on the feedback from the model.
Under the AE scenario, the AE is tuned to minimize reconstruction error of input images. Due to this dependence on labeled data, the output quality of the AE is dependent on the adversary's ability to collect data, which is a realistic consideration. We model the scenario where the adversary only has access to the test set, which is often considerably less informative than the training set. This manifests as an extra distortion in addition to the adversarial noise. Thus the output of the AE-initialized decoder can be used in different ways, which we discuss briefly in Section A.3 of the Appendix. In the case of BiLN, independent of the adversary's 's manifold description (i.e., possible extracted knowledge about test samples). This can manifest lower overall distortion, in the case where only a crude manifold
additional training is required, which means it synthesizes search direction
4

---

Under review as a conference paper at ICLR 2021
description can be extracted from the test set. Next we describe the exact usage of the mapping functions for each attack scheme.
Sign-OPT & HSJA. In general for the attacks which rely on the Cheng et al. (2119) formulation, the update in Equation 2 becomes
g(O' + Bu() - g(0') u/,
g
(4)
1=0
for the reduced-dimension Gaussian vectors {u/ E IR Yi and direction 0' E Rd for integer d' < d. The reduced-din dimeesionn direction 0' is initialized randomly with 0' ~ (0, 1) for the untargeted case, or for the targeted case as 0' = E(x),, where X, is a test sample correctly classified as target class t by the victim model. This scheme also applies to HSJA, since HSJA performs a single-point sign estimate. As in the normal variants, g is used to update 0'.
The intuition behind RayS attack is to perform a discrete search in at most 2 directions. Chen et al. also perform a hierarchical search over progressively larger super- of the image. This has the effect of already upscaling on-the-ffy (Chen & Gu 2020). The inclusion of RayS in our analysis is important, since it has the unique behavior of pertorming a discrete search for the decision boundary, rather than an explicit gradient estimate. To achieve an appropriate Scduced-dimension version of RayS, and test our hypothesis, we modify the calculation of in Algorithm 3 of Chen & Gu (2020), which either speeds up upscaling by a factor a (i.e., 8 = + a), or extends the search through a specific block index by factor b (increase block level at l 2 instead of k -2))
RayS.
3.4 CAPTURING SEROTH-ORDER SEARCH DEVIATION
Our analysis follows in the wake of findings presented by (Stutz et al., 2019). It is shown that "regular" gradient-level adversarial examples leave the data manifold i.e., the sample's s distance to the manifold is larger than with "on-manifold" gradient-level adversarial example. In the score-level and hard-label settings, the manifold can be used to guide the search for the boundary. The benefits of this scenario can be observed in score-level attacks by Tu al. (2019). Similarly, hard-label attacks can leverage the concept of super-pixels to achieve gains in performance (Chen & Gu 2020). However, to date the connection between dimension manipulation and manifold traversal has not been investigated. The reduced search fidelity naturally limits the resolution of search direction. which is the final noise vector applied to the original sample, as evidenced by Equation 3 This introduces our central research question: How does searching over search resolution decreased as side-effect?
increase efficiency, if the
To approach this question, we observe that certain gradient-level and score-level attacks can leverage a prior manifold description to attack more efficiently (Stutz et al. 2019 Tu et al. 2019). We can rely on a sample's 's distance to the manifold as measure of deviation during the search, similar to the work by [Stutz et al. (2019). Hereafter, we refer to this distance w.L.ogg as the manifold distance. Our choice is motivated by the fact that from fata-geomettic perspective, manifold distance describes the amount of semantic features preserved during the attack process (Engstrom et al. 2019). Likewise the manifold distance can communicate information about attack behavior better than Ly distortion measurements, which are the common choice in existing zeroth-order attack literature (Cheng et al.) 2020 Chen et al. 2019 Chen & Gu 2020). Second, the manifold approximation technique by Stutz ct al. 2019) is mainly suited for Lg-norm, whereas hard-label attacks exist for both Lg and L -nomm. Unfortunately, the real data manifold is difficult describe. This is an open problem in the study of Generative Adversarial Networks (GANs), as designers must ensure that generator images are on-manifold using a continuous function. This has motivated the recently proposed Frechet Inception Distance (FID), which acts as surrogate measure of the manifold distance (Heusel et al.) 2018). We can leverage FID by treating the adversarial samples as synthetically generated images. FID is viable as it computes Frechet distance of candidate images with respect to images sampled from the true data mabbmanifold. Since FID uses an Inception-Ve V3 coding layer to encode images, this distance will correlate with distortion of high-level features, which will be result of sampling farther from the data manifold. We do not target the Inception-V V3 network in any of our experiments, so the FID metric will not rely any internal aspects of the victim models.
5

---

Under review as a conference paper at ICLR 2021
4 RESULTS
4.1 METHODOLOGY
Our experimental analysis addresses the following three research questions about zeroth-order attacks:
QI. What is the trade-off between query efficiency and reduced search resolution in the zeroth order setting, against both natural and "robust" models?
Q2. In addition to reducing queries, are there unexpected benefits for performing a query-efficient zeroth order attack?
Q3. Compared to the previous results in the score-level setting (Tu et al. 2019), do dimension- reduced hard-label attacks produce a similar amount of reduction in query usage?
We answer these questions by comparing three hard-label attacks against their dimension-reduced variants. Some variants are not shown due to incompatibility with the base attack. For example, AE+HSJA is not implemented as it relies on only a single-point estimate, thereby only allowing to attack the manifold directly. This is not practical due to induced distortion discussed previously in Section 3. RayS can perform two-point search, but assumes codependence of input features, which may not be the case for well-defined latent space of autoencoder[ Thus for the AE variant, rely on Sign-OPT as it can perform two-point estimate and does not rely on codependence of features. In the BiLN cases, the implementations follow the discussion in Section 3.3
Experimental Highlights. Our experiments show that query-efficient attacks exhibit unexpected behaviors and benefits, with explanations summarized below:
A1. Reduced search resolution increases search efficiency by allowing the discovery of samples not encountered during training. This is especially true for robust models, which we show have tendency overfit on the first-order noise characteristics.
A2. Surprisingly, query-efficient attacks search closer to the manifold than non-efficient attacks, thus are more likely to produce on-manifold examples. This occurs because query-efficied search acts indirect manifold feedback. By reducing the search fidelity, we more likely to modify high-level features. This effect is magnified on robust models, which are already known to leak manifold information in gradient-level settings (Engstrom al. 2019).
A3. Dimension-reduced attacks are capable of SotA query-efficiency gains for HSJA, and two-fold improvement for Sign-OPT.
Setup. All attacks run for 25k queries without early stopping. For brevity, we only show results for the untargeted case. FID score is calculated using the 64-dimensional max pooling layer of the Inception-V3 deep network for coding (denoted as FID-64 in figures), taken from an open- source implementation! ? The choice of the 64-dimensional feature layer allows to calculate full-rank FID without the full 2,048 sample count of original FID, which is prohibitive given the high time complexity of chosen robust models for analysis (in particular, certified smoothing ImageNet). This incurs the cost of losing some high-level features, but due to the position of the chosen coding layer in the network, it is still valuable for direct comparison of manifold distance between attacks. Since the coding layer differs from the original implementation, the magnitudes will differ from those published by Heusel et al. (2018).
Image data consists of the CIFAR-10 (Krizhevsky 2009) and ImageNet (Russakovsky et al. 2015) classification datasets. This selection allows to study attack behavior on both small and high-resolution image data. Original samples are chosen from the test set of each dataset using similar technique as in Chen et al. (2019). On CIFAR- 10, ten random samples are taken from each of ten classes. On NetgeNet, ten random classes are chosen with ten random samples taken from each (i.e., 100 total samples on either dataset). We provide further implementation details in Section A.3 of the Appendix. In addition to natural images, are interested in the attack behavior for model regularized with some variant of first-order noise. For CIFAR-10 10, we choose the representative adversarial training
'Experimentally, codependence hurt the AF+RayS variant more than was practically useful. https: //g http://ttt.....ooooooeeeeeeeeee http:///eeeetteee////trrcoc.oooe

---

Under review as a conference paper at ICLR 2021
Natural
Madry Adv.
Natural
Madry Adv.Tr.
Divoetion (ta))
11
Wilhetops and
Hil Miloee an
Biteeita and
BiAheii and ti
BiLhotup bui
0.2
FID-64
1ii
IV.
HOl
Distotion (Z.))
FID-64
0.1
10
Distotionn (Ca)
33
REN
REA ADdigpoee
0.2
FID-64
ii
311
3000
10000
1900
2000
200e
sei
1800
7300 M00
1300
2000
2500
500
1300
1900
2000
2000
of queries
of queties
# queries
of queries
(a)
(b)
Figure 2: Results across attacks for CIFAR-10 dataset, corresponding a) distortion against query usage (dotted red line denotes the value of c, shaded areas mark standard deviation), and b) FID-64 trajectory against the same query usage.
technique proposed by Madry et al. (2017). For completeness, we also include ablation results on more recent regularization techniques Section[A.5] of the Appendix. For ImageNet, we compare against the SotA at time of writing, randomized smoothing proposed by Cohen et al. (201)). We use the pre-trained Resnet50 weights and implementation provided by Cohen et al., corresponding to smoothing parameter 0.5 and 1.0.
4.2 EXPERIMENTAL DETAILSS
On each dataset, we target the I. -nom which the robust models were regularized or certified under. Likewise we use the Loo versions of each attack for CIFAR-IO and L2 versions for ImageNet.
CIFAR-10 case study (Loz). We start by measuring the distortion against remaining query budget of the adversary in Figure results. The BiL.N variants of RayS each have minimal effect on overall query efficiency (Insets P and This is a result of RayS not relying on explicit gradient estimation. The main improvement is with BiLN+HSJA against the Madry adversarial training model, with average distortion at 4k queries decreasing from 0.09 to 0.07, and corresponding success rate increasing from 15% to 25%. This improvement aligns with the result Tu al. (2019). and is contrary to the minimal effect the natural model (Inset ii.. AE+Sign-OPT outperforms against regular Sign-OPT and BiLN+Sign- OPT on the robust model, since searching along the manifold can grant distortion which was not encountered during first-order regularization (Stutz et al. 2019: Jalal et al. 2019). However, the success with AE+Sign- tends to be situational, whereas HSJA and RayS outperform in either scenario. Mainly the low quality of the adversary's 's AE does not permit fine the-grained adjustment in latent space that RayS and HSJA can provide in image space. We next focus on Figure 2p. which shows the FID score's 's trajectory as the search progresses. Every trajectory will begin at zero value, since there is an expected score of zero for identical images, and then peaks as the attack initialization is performed. In this scenario, our main observation is the similarity of FID trajectory between RayS, HSJA, and dimension-reducee Sign-OPT, despite the method of model regularization, as shown in Insets 2..i and 2biii, respectively. For example, the magnitudes for AE+Sinn-OP and RayS peak to 0.28 and 0.04, then fall (and stay) at values near 0.004. BiLN+HSJA and BiLN+Sign-OPP both exhibit lower FID scores than their normal variants, much as two orders of magnitude less in
general, the normal variants of each attack align with the published
7

---

Under review as a conference paper at ICLR 2021
Natural
Smoothing
Natural
Smoothing
Wilheteps and
MiSokeed and BiLhotup bul
Dotoction
FTD
1.
11
1.
30-
Ditoction
40-
FTD
ii
Siga DIf AEsSge-001
SppcePt NUNiRgS CPFT Alhdiga-ITT
Distorion (f.)
FID-64
3000
1000
1900
3000
1500
500
1010
7300
2500
1300
2000 2500
500
13:0
1900
2000
2000
of queries
of queries
of queries
of queries
(a)
(b)
Figure 3: Results across attacks for ImageNet dataset, corresponding a) distortion against query usage (dotted red line denotes the value of c, shaded areas mark standard deviation), and b) FID-64 trajectory against the same query usage.
the case of Sign-OPT and +iL. For a detailed comparison of distortion and FID values for CIFAR-10, we offer Table in Section of the Appendix. The cimeension-reducee variants of RayS do not have a large variation between them (2.ivv a side-effect of the adaptive super-pixel search, which can automatically scale the super-pixel size the search progresses. At high level, we observe that BiLN+HSJA offers comparable performance to regular RayS, while in the robust case, BiLN and AE variants tend towards lower FID scores than regular variants. This can be viewed as the model leaking manifold information through the decision, as was shown for the first-order gradient by Engstrom et al. (2019)
ImageNet case study (L2). We attack ImageNet in the Ly-norm setting to compare against the certified smoothing technique proposed by [ohhnn et al.] (2019). The label output comes from a smooth classifier, approximated by many rounds of Monte Carlo search, which uses the regular model regularized by Gaussian noise. Thus hard-label attacks are well positioned for attacking the smoothing technique. The distortion results of these attacks are shown in Figure 33. Dimension reduction has larger impact when coupled with the large ImageNet resolution. Particularly the BiLN+HSJA and BiLN+Sign-OPT attacks profit the most. At 8k queries, success rate increases 1.4x and 2 for HSJA and Sign-OPT, respectively. As before with CIFAR-10, the RayS dimension reduction is saturated on ImageNet (Insets Jui and a.ii) Since RayS does not explicitly rely on gradient estimation, it benefits techniques. Apart from RayS, BiLN variants outperform both AE
the least from dimension-reducct and regular variants in every scenario, despite the reduced search fidelity. The improvement is largest on the smoothing technique, particularly with HSJA and Sign-OPT (second column of Figure 3D). This is due to 1) the adversary's 's reduced description of the manifold, by only having access the test set, and 2) BiLN allowing to search closer to the original sample, since it is deterministic function independent of the adversary's knowledge. The FID scores in F paint more comprehensive picture. BiLN variants produce adversarial examples closer to the manifold than either regular or AE variants, highlighted with HSIA+BiLN Inset Bp.i. RayS saw no such improvement, as it does not perform explicit gradient estimate (Inset Bpi)). We interpret this follows: BiLN variants on HSJA and Sign-OPT leverage reduced fidelity to increase the probability of finding the decision boundary, and 1) produce smoother noise distribution, resulting in spatially correlated distortion, which as a result 2) produces adversarial examples closer to the manifold. Since the manifold description of ImageNet is crude, BiLN variants can excel, since they search independent of
8

---

Under review as a conference paper at ICLR 2021
this description. Another key observation is the fluctuation of LID score towards the end of Sign- OPT and AE+Sign-OPP which are not present for HSJA or RayS (first column of Figure Bp). Notably there is direct signal of manifold distance for the normal BiLN variants. This indicates that query-efficient attacks do a better job of capturing implicit manifold distance feedback from the model.
5 DISCUSSION
The noisy manifold distance oracle. In Section | we observed that query-efficient attacks, i.e., those which leverage the concept of super-pixels to reduce search fidelity, are more likely to produce samples close to the manifold. This generates samples that are unseen by robust models during their first-order adversarial training. However, this takes place without any direct feedback about manifold distance. To approach this, we first consider that the model relies on a subsampled manifold of the image space. This manifold can be leaked by the loss landscape of the model, as shown by Engstrom perspective, the zeroth-order adversary observes the
et al. (219)) From an cnermatiion-hhoretiien noisy gradient, which is leaked side information by each model decision. Under this explanation, the decision feedback by the model is viewed as a noisy manifold distance (NMD) oracle. The improvement of AE+Sign-OPT on robust CIFAR-10 10 can be argued as a result of the NMD oracle improving as well. This can be shown using the data processing inequality (Beaudry & Renner (012)): if (M, g) increases, then I(M, g) also merrases, where I is mutual information, M is the manifold, and g is the noisy gradient. In words, the quality of the noisy gradient depends on the quality of the model's loss landscape, which can more closely resemble the manifold under robust regularization. This means higher quality loss landscape leads to higher quality zeroth-order attack. Qualitative evidence of this effect can be observed in Section A.4 of the Appendix.
"Topology" of hard-label settings. We can view zeroth-order attacks as following a topological hierarchy that is a function of the original data dimension. A very simple version of this idea is illustrated in Figure | Each technique illustrated in Figure [ffferss a different traversal distance both along the manifold, and away from it. Efficient attacks represented by (b) can combine elements of staying near the manifold, and traversing it. This is best seen by the results of BiLN variants and regular RayS. RayS succeeds because it assumes spatial correlation, which can be considered a loose description of the manifold, without making further assumptions of the data. Thus we can expect this behavior from future attacks which do not perform explicit gradient estimation. Similarly, purely traversing close an ideal manifold description, as in (c), may not be advantageous, because a crude manifold approximation induces its own error. Further, the nearest boundary on the manifold could be far away. Thus we can consider an attack which leverages an ideal "manifold description" (c.g.. through an autoencoder), but leverages the description as method for selecting progressively smaller super-pixel groupings. For instance, RayS selects super-pixels as blocks, which are bifurcated until the number of super-pixel, equals the original dimension. Notably, RayS displayed little to no benefit from coimension-reductii technique which leverages the manifold description to select super-pixel groups could yield better performance, both in terms of distortion and FID score. To this end, we feel the FID score offers a valuable measure of manifold distance, which can inform the topologiaal behavior, and the quality of future hard-label attacks.
which a byproduct not relying on explicit gradient estimation. A
6 CONCLUSION
Despite the recent progress in zeroth-order attack methods, open questions remain about their precise behavior. We shed light on an unexpected nuance, which is their ability to produce on-manifold adversarial examples. This is despite the absence of manifold distance information, which motivates the proposal of the noisy manifold distance oracle. Future work could create a formal definition of this oracle, and attempt to bound the information revealed by the oracle. On the other hand, with knowledge of the oracle's 's existence, it is possible to further refine hard-labe attacks, so they continue to reveal insights into the weaknesses of learning systems.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
I. Amsaleg, J. Bailey, D. Barbe, S. Erfani, M. E. Houle, V. Nguyen, and M. Radovanović. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In 2017 IEEE Workshop on Information Forensics and Security (WIFS), pp. 1-6, December 2017. doi: D..1109/WIFS
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give False Sense of Security: Circumventing Defenses to Adversarial Examples. 2018. ISSN 1938-7228. doi: 10. 10160 arXiv: 1802.00420
cttappo.2012.99.00 [ttp:t http://aaxvvv...g...... abs 00420
ISBN: 0970-7077.
Normand J. Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv: 0tfl:///77.00000 /07.07400000000 [quant-ph), September 2012. htLL:ttp.///axxiv.or///////////1
arXiv: 1107.0740.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models. arXiv: 04248 [cs, stat], December
2017. /ttp:tt:///xxxv..oooo /abo /ttp///ss////// 1111. C4248
arXiv: 1712.04248.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In Security and Privacy (SP), pp. 582-597, 2016. ISBN 978- -5990-53333--33333 doi: 10. 10.1100/SP:201..499
arXiv: 1608.04644 ISSN: 10816011.
Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security AlSec /7, pp. 3-14, Dallas, Texas, USA, 2017. ACM Press. ISBN 978- 4500-5520-- URL http: //d1 acm. org wwwpgriitaiion....ooeooooeeooooe cfmooidd
4. doi: 10.1145/31 $128572.314042
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On Evaluating Adversarial Robustness. 19XX::9902.06000 [cs, stat), February 2019. URL httpa:////aaav...oo//////22.00.0 arXiv: 1902.06705.
Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright, HopSkip/umpAttack- Query-Efficient 1144.02/44 144 [cs, math, stat), April 2019. URL httt:://a///ii
Basee Attack. /904.02
org/abs/19tt /1904 02144
arXiv: 1904.02144.
Jinghui Chen and Quanquan Gu. RayS: A Ray Searching Method for Hard-label Adversarial Attack. arXiv:2006. /2792 [cs, stat), June 2020. URL
arXiv: 2006. 12792.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In ACM Workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
Minhao Cheng, Thong Le, Pin- Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsich. Query- efficient hard-label black-box attack: An optimization-base approach. International Conference Learning Representations, 2019.
Minhao Cheng, Simranjit Singh, Patrick Chen, Pin-Yu Chen, Sijia Liu, and Cho-Jui Hsich. SIGN- OPT: A SUURRY-EFFICIES HARD-LABEL ADVERSARIAL ATTACK. The International Conference Learning Representations (ICLR), pp- 16, 2020. URL TRRLhttps:/// htttps://pperrvii Sott/forumiid==kk11eee
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified Adversarial Robustness via Ran- domized Smoothing. 100.00192.00218 (cs, stat), February 2019. URL http: httt::://xxxvv.orrg
abs/1902. 02918
arXiv: 1902.02918.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Learning Perceptually-Aligne Representations via Adversarial Robustness arXiv: 19066.090094555 [cs, stat], June 2019. URL http::/aax.vv.or//ab////e 00945
arXiv: 1906.00945.
10

---

Under review as a conference paper at ICLR 2021
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model Inversion Attacks that Exploit Con- fidence Information and Basic Countermeasures. Proceedings of the 22nd ACM SIGSAC Con- ference on Computer and Communications Security CCS '15, pp. 1322-1333, 2015. ISSN f
15437221, doi: 10. 14//28111133.28111111 16677. URL http:://ll...... acm. wwtp/o/ggaatoooooooooooooo doic=2810100 3677
ISBN: 978 1783433333
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and lan Goodfellow. The Relationship Between High- High-Dimensional Geometry and Adversarial Examples. 1001.v18010.2 00774 [cs], September 2018. URL htt::tt:///// http:://aaxivv. org/ /aap///a///// 18001
02774 arXiv: 1801 302774.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and Harnessing Adversarial Examples. 2014. ISSN 0012-7183. URL http:://arxiiv.rg//bbb////////// 6572 arXiv:
1412.6572 ISBN: 1412.6572.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. arXiv: (Tttv:7706.0000 08500 [cs, stat), January 2018. URL httt::///axxv.orrr arXiv: 1706.08500.
/2//170000 1706. 08500
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box Adversarial Attacks with Limited Queries and Information. arXiv: 08598 [cs, stat], July 2018. URL http arXiv: 1804.08598.
//arxiv. a1804 08598
Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G. Dimakis. The Robust Manifold Defense: Adversarial Training using Generative Models. 1rXiv:1712.09190 [cs, stat), July 2019.
htLthtt:///aat org/ abs//a//l.../// 09112.099990 arXiv: 1712.09196.
Kim Jungeum and Xiao Wang. Sensible Adversarial Learning. 2020. URL https: ://
ggenreeveww.ett/dd.. finffiiddrrii.. RVKwr
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. pp. 60, 2009.
Sijia Liu, Jie Chen, Pin-Yu Chen, and Alfred O Hero. Zeroth-Orae Online Alternating Direction Method of Multipliers: Convergence Analysis and Applications. In Proceedings of the 21 International Conference on Artificial Intelligence and Statistics (AISTATS) 2018, volume 84, pp. 10, Lanzarote, Spain, 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learnnng Models Resistant to Adversarial Attacks. arXiv:1706 06083 [cs, stat), June 2017. URL
706. 06083
arXiv: 1706.06083.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: a simple and accurate method to fool deep neural networks. 2015. ISSN 10636919. doi: 10. 10.1100 1009 CVPR.2016.282. 9781467388511.
04599
arXiv: 1511.04599 ISBN:
Yurii Nesterov and Vladimir Spokoiny. Random Gradient-Free Minimization of Convex Functions. Foundations of Computational Mathematics, 17(2):227-566 April 2017. ISSN 1615-3375, 1615- 3383. doi: 10.1000/100208-0155 s1 s10208- 015 015-966-22-996
URL WtRLhhttp://Iii springeeee www.geerr.oommmmmmmm 10mmmm10.100
Nicolas Papernot, Patrick Mcdaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. Proceedings 2016 IEEE European Symposium on Security and Privacy, EURO S and P 2016, pp. 372-387, 2016. doi: 10. 109/ EuroSP.2016.36. URL htll:tttp///xxv..oot 978150901751
07528
arXiv: 1511.07528 ISBN:
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram
Swami. Practical Black-Bor Attacks against Machine Learning. Proceedings of the 2017 ACM
Asia Conference Computer and Communicaatonns Security, pp. 506 519. 2017. doi: 10. URLhttp: arxii.
1145/3052973.305300
02 697
arXiv: 1602.02697
ISBN: 4701499949

---

Under review as a conference paper at ICLR 2021
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-552 2015.
doi: 10.10///s 1263-015-0816-y
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations, 2018a.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN: Protecting Classifiers against Adversarial Attacks using Generative Models, pp. 17, 2018b.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and gener- alization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, pp. 1-10, 2013. ISSN 15499618. doi: 10. 10.1021/c22009288. 1549-9618.
6199
arXiv: 1312.6199 ISBN:
Thomas Tanay and Lewis Griffin. Boundary Tilting Persepective on the Phenomenon of Adversarial Examples. arXiv: (et::0006000000 [cs, stat], August 2016. URL 07690 arXiv: 1608.07690.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. wwXiv:2002.08344 [cs, stat], February 2020. URL http //w.xii.org/abs//00000 08347
arXiv: 2002.08347.
Florian Tramèr, Fan Zhang. Floriantra Er Epfl, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing Machine Learning Models via Prediction APIs. 2016. URL https 7/www. www.uuu.eei.nn..oo.....
aerpeencee/eennixsseurrreyyeeee presentation/tramer ISBN: 978-1-99991999
sessions/
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness May Be at Odds with Accuracy. arXiv:1805. 12152 [cs, stat], May 2018. URL arXiv: 1805.12152.
http://arxis> org
h/g///bb.. wwaaaas1111 1805 12152
Chun-Chen Tu, Paishun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. AutoZOOM: Autoencoder- Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks. Proceedings of the AAAI Conference Artificial Intelligence, 33:742-749, July 2019. ISSN 2374-3468, 2159- 5399, doi: 10. v33300 3301742. URL
Haichao Zhang and Jianyu Wang. Defense Against Adversarial Attacks Using Feature Scattering- based Adversarial Training. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d tetquutesiiee Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 1831-1841. Curran Associates, Inc., 2019.
Haichao Zhang and wei Xu. Adversarial Interpolation Training: A simple approach for improving model robustness. 2020. URL.ht.tss:
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. PMLR, pp. 11, 2019.
12

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
MNIST ABLATION STUDY
Natural
Madry Adv. Tr
TRADES
Robust Manifold
Rt RCCckky tw? Di2oday b-4
Nak
Distoction
B1344aad b-4
BENSHSIA
BENOHSIA
BESHISIA
BESNESIAT
Distotion
Distoricn (Ka)
RENiiia O9T Alidkeg CIP
Kiliii Alrikge CPrT
alriigsOPt Oa?
Airlpp CPr
300
13000
1500
2000
2500
5000
1000
15300
3000
2500
5000
10000
15000
2300
25003
1000
1500
23000
20033
# of queries
# of queries
A of queries
A of queries
Figure 4: Results of image space distortion for the MNIST ablation on the natural model, manifold projection defense (Robust Manifold) and non-projection baselines (Madry adversarial training and TRADES). Dotted red line denotes the value of e, shaded areas mark standard deviation.
Manifold projection was originally proposed by [Samangouei al. (2018b) as scheme defend models in the gradient-level setting, and was partially broken by AAthaly et al. (2018) due to the imperfect projection of the defender decoder. Dalal et al. (2019) later showed it could be circumvented completely, by searching for latent sample pairs which are close to each other in latent space, but far apart on the model loss landscape (Talal et al. 2019 These sample pairs are projected to input space and used for adversarial training, which yields the Robust Manifold defense. In direct comparisons under the first-order setting, [aall al. (2019) show that the Robust Manifold scheme defends better than previous baselines, Madry adversarial training (Madry et al. 2017) and TRADES Zhang & Wang 2019), which do not leverage the manifold projection. We conduct experiments to directly compare this behavior in the hard-label setting, shown in Figure 4 The hard-label attacks consistently outperform against the Robust Manifold defense, despite this defense outperforming both the Madry and TRADES defenses under gradient-level attacks. This be considered result of the Robust Manifold defense overfitting to the manifold projection of the fixed generator (or "spanner" in the vocabulary of Jalal et al.). Further, dimension-reducct variants yield better exploration, shown by consistently better performance from 0-5000 queries. This supports the results of the main paper, which show that these defenses can overfit to first-order information.
The results on the MNIST ablation also support the notion of a dynamic topological hierarchy, which is dependent on the original data dimension d. In the case of HSJA (second row of Figure 4. query-efficient attacks do not benefit the adversary, because MNIST is already sufficiently low- dimension to yield samples near the manifold with regular hard-label attacks. We can validate this finding by comparing a notion of manifold distance between the attacks. FID is not suitable for MNIST as it requires re-training of the Inception-V3 network and calibration for use as a metric, which is outside the scope of this work. Instead we project the adversarial samples back to the manifold, using the Robust Manifold defender's 's variational autoencoder (VAE), and measure the pairwise distances between original and adversarial projections. This distance is described formally as [EDx))) - DxE(x)) for each adversarial sample x and original sample Xg- We use the checkpoint provided by Jalal al. 2019). Notably the defender network has the best description of
13

---

Under review as a conference paper at ICLR 2021
Natural
Madry Adv. Tr
TRADES
Robust Manifold
10
10
PLC
10
40
Racuas Dismscsio
Raciea Dinamce (2))
Disnaco (8c)
Racmes Disnace (2))
Figure 5: Measurement of pairwise reconstruction distance - for the MNIST ablation the natural model, manifold projection defense (Robust Manifold) and non-projection baselines (Madry adversarial training and TRADES), using the original VAE checkpoint provided by Jalal et al. (0099) for & and D.
the manifold, since it has access the full training set. The distribution of pair-wise distances for each attack and defense combination is shown in Figure In these plots, we expect the reconstruction distance to be smaller for adversarial samples closer to the manifold, since the VAE will capture high-level semantic information of the dataset.
Under our hypothesis, successful hard-label attacks should have a smaller distance on natural models than regular white-box attacks. Thus we also compare against samples generated with Projected Gradient Descent (PGD) in red. The results in Figure support this hypothesis, as both Sign-OPT and HSJA have smaller distance than the PGD samples. This is exaggerated with the AE+Sign- OPT variant, due to the manifold approximation of the adversary. On robust models (second to last columns), PGD acts as a best-case for the adversary, since it is known from previous results by |Engstrom et al. 2019)) that gradient-level attacks on robust models leak manifold information. As discussed in the main text, hard-label attacks receive a noisy version of this manifold feedback, which is sufficient to create samples near the manifold. This is supported by the pairwise distances on robust models, since PGD samples are consistently closer to regular samples. Due to the dynamics of the topological hierarchy discussed before, the dimension-reduuctoon is less effective on MNIST. Notably the regular hard-label attack variants (shaded blue) can be closer to the original samples than their dimension-reduced variants. RayS (top row) is consistently farther from the PGD samples, as it does not explicitly rely on gradient estimation like Sign-OPT and HSJA. We further point out that despite the variability of distortion in image space between regular and BiLN variants in Figure 4 these samples still lie near each other when projected to the manifold in Figure S On the Robust Manifold defense, HSJA and Sign-OPT attacks, regardless of variant, will lie closer to the PGD samples. This supports our main result, that query-efficient hard-label attacks leverage manifold information to produce samples, particularly against robust models.
A.2 SUCCESS RATE PLOTS
In Figure 6 provide query vs. success rate to accompany the results in the main text.
14

---

Under review as a conference paper at ICLR 2021
Natural 0.031
Madry Ad,.Tr. 8 0.031
Natural E 3.0
Smoothing E 1.0
8
Ray
Ro5 2 Mil
Rid Bt
Sucoess Rate
Sunson Pate:
2.6
Rey
32.
12
2400
1000 1500
2300
1500
2000
1230
1500
2300
2500
230 1000 1960
2000
25000
230
1000
15301
20.00
1500
Suno0ss Ruae
3.5
Sucoea Rate
BINGISI 12
3.4
181 BLLhiiss 32
30
62
1.3
12
Sande7
Suaccw Rudi
Succcce Rowi:
3.6
20
300 1000 1900
3000
2500
300
1000
1900
3300
2508
500
1000 1300 13:00 2000
330 1000 1990
30000 1500
queries
queries
*f queries
queries
Figure 6: Query vs. success rate plots corresponding to each attack variant in the main text, for a) CIFAR-10 and b) ImageNet.
A.3 IMPLEMENTATION DETAILS
We are primarily interested in the effect of reduced search resolution on attack behavior. Thus in this work, given candidate direction O' and magnitude (or radius) r, the adversarial sample in the AE case is the blending (1 r)xu- + rD (E(x)) 0')
Natural vietim architectures consist of deep convolutional neural network for CIFAR-10, 10, and Resnet50 network for ImageNet. The CIFAR- 10 network is the same implementation open- spurcse by Cheng et al. (2020). while the Resnet50 network is taken from the PyTorch Torchvision library, 4l For AE attack variants, we implement the same architecture described by Tu al. (2019). Specifically it leverages fully convolutional network for the encoder and decoder. ImageNet samples are downsized to 128x before passing to the encoder, and the output of the decoder is scaled back to 224x224, as described by Tu al. (2019). Every AE is trained using the held out test set, as we assume disjoint data between adversary and victim. To avoid ambiguity, we label each BiLN variant with the spatial dimension after performing the bilinear transformation, and keep this variable fixed for simplicity.
including pre-trained weights
A.4 VISUAL RESULTS CIFARI0
We provide visual qualitative results for each attack on CIFAR-10 in Figure 7
P We observed that it is detrimental set D(E(xx) + r0') or D(r0') directly. Despite remaining on the data manifold by attacking it directly, the approximation of the data manifold is crude, which results in large distortion (Stutz et al. 2019).
httttp:://pyttrrhh.oo
htm
15

---

Under review as a conference paper at ICLR 2021
Querios: Original
25409
25067
Lee 00.01
(330)
Original
Lay
(a) Sign-OPT on CIFARIO
(b) Sign-OPT CIFARIO (Madry Adv. Tr.)
Querios: Original
945
13667 00.0
25128 L- -005
Final (X50)
Queries: Original
871
13949 -0.57
25385 0.57
Final (X50)
-
0.64
(c) BiLN OPT on CIFARIO
(d) BiLN OPT CIFARIO (Madry Adv. Tr.)
Queries:
972
14021 La -0.07
25356 Los -000
Final (X50)
Queries: Original
900 Las -1.77
15778 Law -021
25084 -1.20
Final (L50)
Original
w0.32
(e) AE Sign-OPT CIFAR1O
(f) AE Sign-OPT CIFARIO (Madry Adv. Tr.)
9898
25417
Final (X50)
9898
25417
Final (x50)
Original
-0.11
Original
00.46
-0.08
(g) HSJA on CIFAR10
(h) HSJA on CIFARIO (Madry Adv. Tr.)
Original
(x50)
Original
00007
(i) BiLN HSJA on CIFARIO
ii BiLN HSJA on CIFAR1O (Madry Adv. Tr.)
Queries: Originol
11955 Lec 00.01
25001 Lu
Queries: Original
18 La 0.99
11327 Lo 0.00
25005 0.06
Final (x50)
0.98
(x50)
(k) RayS on CIFARI0
(I) RayS CIFAR10 (Madry Adv. Tr.)
Figure 7: Visual selection of attack trajectories on CIFAR-10.
A.5 SUPPLEMENTAL RESULTS
A.5.1 EXTRA EXPERIMENTS ON ROBUST CIFAR-10
provide results four extra model regularization methods from the literature: TRADES (Zhang et al.) 2019), adversarial interpolation learning (Zhang & Xu 2020), feature
In Figures 8 and
scattering (Zhang & Wang 2019), and SENSE (Jungeum & Wang 2020).
16

---

Under review as a conference paper at ICLR 2021
TRADES
Interpolation
Feat. Scattering
SENSE
Disoction (Ka))
03
Reyh
REE
Reyb
Roy
RIK Royk Royd
Rid
Red
63
64
60
Disertion (Za))
63
ISSI BLLN HSSA
USSk BLE HSSt
USSK DENISIA
USIA DENESIA
64
64
Distotion (Za)
6.3
02-
6.1
1960
1998
200
7000
1000
1908 quecies
3000 7900
1000
3000 1901
10000
1903
queries
quesies
queries
Figure 8: Results of distortion for CIFAR1O with four other choices of robust regularization from the literature.
TRADES
Interpolation
+ Feat. Scattering
+ SENSE
BayS
ReyS
RuyB Royd
FID.64
Reyd
60
HELS
HSE RERISIC
PATA RENPIICH
BIX
FID-64
60
Squatir
Spadef BLk OFT
Sppco BEX OFT Al-ka--07
FDD-44
1.5
BILK Squ
Millia cer
10000
2000
2000
15000
2500
13000
15000
2500
1200
19400
J0h0n
15000
queries
queries
queries
queries
Figure 9: Results of FID score for CIFARIO with four other choices of robust regularization from the literature.
17

---

Under review as a conference paper at ICLR 2021
CIFAR-IO 0.031)
Madry Adv.Tr. (e 0.031)
# Queries
Avg. Too
SR
FID-64
Queries
Avg. 1.
FID-64
4,000
0.01
98.0
0.02
4,000
0.05
33.0
0.36
RayS
8,000
0.01
100.0
0.01
8,000
0.04
36.0
0.28
14,000
0.01
100.0
0.01
14,000
0.04
39.0
0.23
4,000
0.01
99.0
0.02
4,000
0.05
32.0
0.44
BiLN+RRys a=2
8,000
0.01
100.0
0.01
8,000
0.05
35.0
0.33
14,000
0.01
100.0
0.01
14,000
0.04
36.0
0.25
4,000
0.01
100.0
0.02
4,000
0.05
33.0
0.37
BiLN+RayS b=2
8,000
0.01
100.0
0.01
8,000
0.04
37.0
0.29
14,000
0.01
100.0
0.01
14,000
0.04
39.0
0.24
4,000
0.01
93.0
0.03
4,000
0.05
32.0
0.44
BiLN+RayS
8,000
0.01
100.0
0.02
8,000
0.05
35.0
0.33
14,000
0.01
100.0
0.01
14,000
0.04
38.0
0.25
4,000
0.01
88.0
0.06
4,000
0.09
15.0
3.57
HSJA
8,000
0.01
100.0
0.02
8,000
0.08
26.0
2.41
14.000
0.01
100.0
0.01
14,000
0.06
27.0
1.69
4,000
0.02
82.0
0.03
4,000
0.68
BiLN+HSJA 16
8,000
0.01
94.0
0.01
8,000
0.06
31.0
0.47
14,000
0.01
97.0
0.01
14,000
0.05
32.0
0.37
4,000
0.06
52.0
0.70
4,000
0.33
2.0
2.28
Oign-OPT
8,000
0.04
71.0
0.89
8,000
0.30
3.0
1.04
14,000
0.02
79.0
0.00
14,000
3.0
0.59
4,000
0.09
21.0
0.01
4,000
0.37
2.0
0.08
BiLN+Sign-OPT 16
8,000
0.06
29.0
0.00
8,000
0.35
3.0
0.04
14,000
0.06
36.0
0.00
14,000
0.33
3.0
0.04
4,000
0.09
20.0
0.01
4,000
0.20
8.0
0.01
AE+Sign-OPT
8,000
0.07
31.0
0.00
8,000
0.17
11.0
0.21
14,000
0.06
35.0
0.00
14,000
0.15
11.0
0.21
Table 1: Comparison at certain query intervals between regular and robust models CIFAR-10.
18

---

Under review as a conference paper at ICLR 2021
Imagenet (c = 3.0)
Smoothing (a = 0.5, = 1.0)
# Queries
Avg. L2
SR
FID-64
Queries
Avg. L2
SR
FID-64
4,000
4.11
47.0
0.22
4,000
12.30
9.6
0.77
RayS
8,000
3.17
8,000
12.23
11.5
0.77
14,000
2.55
69.0
0.09
14,000
12.19
11.5
0.77
4,000
4.55
43.0
0.32
4,000
11.95
7.8
0.61
BiLN+RayS a=2
8,000
3.54
55.0
0.19
8,000
11.86
9.1
0.61
14,000
2.77
63.0
0.11
14,000
11.81
9.1
0.61
4,000
4.07
47.0
0.22
4,000
10.27
9.0
0.39
BiLN+RayS b=2
8,000
3.15
59.0
0.13
8,000
10.20
10.0
0.39
14,000
2.52
67.0
0.08
14,000
10.14
12.0
0.39
4,000
4.51
44.0
0.24
4,000
9.46
13.4
0.39
BiLN+RayS b=4
8,000
3.50
53.0
0.16
8,000
9.41
13.4
0.38
14,000
2.84
63.0
0.10
14,000
9.38
14.6
0.39
4,000
6.27
43.0
0.86
4,000
75.75
0.0
132
HSJA
8,000
3.08
62.0
0.17
8,000
74.65
0.0
130
14,000
1.90
82.0
0.04
14,000
72.92
2.1
125
4,000
2.05
78.0
0.01
4,000
15.73
0.0
1.33
BiLN+HSIA 32
8,000
1.28
92.0
0.00
8,000
13.56
0.0
1.06
14,000
98.0
0.00
14,000
11.78
7.1
0.76
4,000
28.57
26.0
0.59
4,000
71.69
4.0
147
Oign-OPT
8,000
27.04
35.0
0.36
8,000
71.45
4.0
145
14,000
24.97
43.0
0.91
14,000
71.37
4.0
144
4,000
5.48
44.0
0.12
4,000
23.61
6.0
2.95
Bil Bil.N+Siin-OPTS 32
8,000
3.38
74.0
0.02
8,000
23.45
6.0
2.88
14,000
2.41
81.0
0.01
14,000
23.40
6.0
2.86
4,000
34.82
3.0
1.73
4,000
48.97
5.0
10.20
AE+Sign-OPT
8,000
28.01
4.0
1.16
8,000
48.89
5.0
9.92
14,000
23.77
6.0
1.16
14,000
48.89
5.0
9.91
Table 2: Comparison at certain query intervals between regular and robust models on ImageNet.
19


Reviews:
[{"review_id": "wijZvDa4SL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe paper investigates methods that perform hard-label adversarial attacks at the zeroth-order and analyze them from the perspective of generating on-manifold adversarial examples, down-scaling of the input space, and query-efficiency on two datasets, cifar and imagenet.\n\n\nPros\n------\n- The paper is well written.\n- The paper tries to address a very important topic of adversarial attacks.\n- The paper provides some interesting insights on dimension-reduced attacks, query efficiency and reduced resolution for current methods that use zeroth-order hard-label attacks.\n\nCons\n-------\n- Although the insights are interesting, it  is not clear how they can be used to design a new SOTA algorithm that outperforms OPT-Attack,  Sign-OPT, and RayS in terms of identifying an adversarial example with smaller distortion with less queries. This seems to be  a big weakness of this work as the experiments comparing the existing methods are not comprehensive enough to help advance the field of designing better adversarial attacks and ways to counter them.\n\n- It is not clear the claim that down-scaling techniques would lead to worse (farther-away) adversarial examples.\n\n- It is not clear why FID is a good measure for determining whether an adversarial example is an on-manifold example.\n\n- No success rates vs query plots like in [1] which is common in the  literature.\n\n- Code is not provided to verify the results\n\n- It doesn't seem like there were multiple runs of the experiments. It is important to have multiple runs and report the mean and std to get more reliable results.\n\n- Lacks method. The paper does not propose a new method. Instead it compares between few existing methods and makes non-impactful insights on their behavior.\n\nConclusion\n---------------\nAlthough the paper is well written and the insights are interesting, it doesn't seem they are impactful enough to be publishable in ICLR. \n\nHowever, if the authors convince me otherwise about the impact of these insights in the rebuttal period  I will be happy to increase the score.\n\nPost-rebuttal \n------------------\n\nI appreciate the response and the manuscript update, but some of the comments have not been addressed such as the code which has not been provided, and the fact that the results are still very inconsistent like R2 mentioned although the arguments made in this work seem clearer to me. But it is still not clear to me how the results can be used to improve current methods for identifying adversarial examples with smaller distortion and less queries. Thus, I will retain my score as is. \n\n[1] SIGN-OPT: A QUERY-EFFICIENT HARD-LABEL ADVERSARIAL ATTACK. ICLR2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "aYfYRRSO8J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Major Issues:\n1. The biggest problem is, the author claimed \"super-pixels help eliminate the search space of off-manifold adversarial\nexamples, leading to examples which are truly generalization errors\" in the introduction, but never provide convincing proofs for this argument. The analysis in Section 3 is basically based on intuition and is full of holes.\n2. It is very obvious that the authors do not have enough time to polish this paper. This problem is even more obvious in the later parts of this work. To name a few: (a) in the contribution part of Section1, the sub-title is missing for the third part. (b) in Section 3.3, the author first term bilinear transformation as BiLIN but refer it as BiLN in later parts. (c) Some notations are absent or inconsistend. For example, the $u_i$ vector in Equation 2 is first denoted as $u_j$. BTW, what does $\\beta$ represents? (d) The information provided by the images is very unclear. What does the column of \"CIFAR10/ImageNet\" represents? Standard training? This makes the reviewer can not catch the main idea of the work. Even after reading this paper, I still do not understand what method has the author proposed as they claimed in their third contribution.\n3. The author claimed in the third contribution of their work that their method can encourage on-manifold examples. How will this help improving robustness? Will this decrease the generalization errors?\n\n#### Minor Issues:\n1. In Sec 3.4, the author wrote: \"Thus, we are interested in the tradeoff between query\nefficiency and search fidelity. This introduces our central research question: How does searching over\nreduced-dimension increase efficiency, if the search resolution is decreased as a side-effect?\" What is the answer to this question. BTW, the author never mention this \"trade-off\" part in the introduction, which seems very abrupt to me.\n2. The Q&A in Sec 4.1 are redundant. It basically rephrase the content in the three contribution.\n3. Why select FID-64? How about features in other depth?", "extra_scores": {}, "extra_text": {}}, {"review_id": "NEJbzIVdLYu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies hard label adversarial attacks, to specifically examine whether such attacks find adversarial examples that are close to the data manifold. The paper argues this is counter intuitive to our current understanding that such examples leave the data manifold. The paper studies such examples in both regular and robust models on CIFAR and ImageNet datasets. I enjoyed reading the first half of the paper, but the second half and observations from the experiments were quite hard to follow because they are often vague and rely (heavily) on results from recently published work. This makes it challenging to interpret the results clearly. The plots themselves are also quite hard to read in most cases.  \n\nStrengths:\n+ Understanding the manifold traversal for zeroth order attacks is pertinent. The paper reveals an unexpected benefit for such methods, indicating that these adversarial examples may lie on the manifold, indicating that previous notions that rely explicitly on manifold-projection based strategies can fail defend against such attacks.\n+ The paper presents an interesting case study of known attacks on popular benchmark recognition tasks. The central questions in the paper are interesting.\n+ The results show to some degree that the variants considered in the paper (proposed by earlier papers) are closer to the manifold (under the FID measure). This improvement is exaggerated when the models are robust, which is very interesting.\n\n\nWeaknesses:\n* While the paper is well written overall, and easy to read, I found the language used to be imprecise which made some parts of it harder to appreciate. The paper studies an important problem, and demonstrates empirically, the benefit of using a query efficient zeroth order hard label attack, yet it seems the paper does not explain why the result is so? The observation is repeatedly referred to as unexpected, and considering it is one of the central contributions of the paper, I think it would benefit if the paper discussed the main insight into why such an observation occurred in the first place, and more importantly what it tells us about adversarial attacks. \n* Too many details are left out of the paper, which make the exact contributions of this work harder to understand. For example, there is only a brief mention of the mapping functions like an AE or a BiLIN or how they are used in this context. How is the BiLIN used here? The reduced dimension ZO search is already shown to be more query efficient in Tu et al (2019). Why do BiLIN variants exploit reduced resolution better?\n* Is the finding Q3 of this paper or from Tu et al. (2019)? What new insight does the present manuscript bring that Tu et al (2019) do not have?\n* Why is FID the right manifold distance measure? It would greatly help if the observation that query efficient ZO attacks find on-manifold samples, was validated independently outside the FID measure. In other words, does this attack render “manifold projection” based defenses useless? If the sample is already on or close to the manifold, in theory these attacks would be exactly reproduced under a manifold-projection defense?\n* The explanations of CIFAR-10 results could be expanded further, it is not clear why AE+Sign-OPT does worse. What does the similarities of the FID curves imply? In general, I think it would help if the paper made more explicit what were to be inferred from these plots. They are also hard to read (perhaps a log axis would help?)\n* What is the success rate for these attacks? This is only provided in the supplement, but the experiments section refers to them. It would help if they were in the main text as well. \n* Some editorial comments: I would recommend taking some of the explanations for BiLIN outperforming AE variants to motivate its use earlier when it is introduced itself.\n* in Section 5 “the manifold can be leaked by the loss landscape” what does this mean exactly?  \n\n\n\n----------- Update after revision ---------------------\n\nDear Authors, \n\nThank you for the revised manuscript. The expanded sections have helped me understand the paper more deeply, and place the contributions in context to prior work better than before. I especially appreciate the new experiments and explanations on the Robust Manifold Defense. \n\nA few comments on the revised paper: \n\n* First, the success rate plots which were missing are quite revealing: While the biggest benefit of the reduced resolution query search appears to have been in robust models (Madry et al), the benefit is not reflected as clear in terms of the success rates (which is arguably more critical). For example, in CIFAR-10 experiment in Fig 2(a) while in the Natural case, BiLN+HSJA is slightly more query efficient it appears less successful than plain HSJA in Fig 6(a). (Once again, the plots and figures are extremely illegible). \n* For \"Robust\" CIFAR-10 models and \"Natural\" ImageNet , there appears to be a considerable gain in both $L_\\infty $ distortion *and* success-rate.  For the \"Robust\" ImageNet case, while there are *huge* gains in terms of $L_\\infty $ distortion, there is essentially no discernible difference in the success rate between reduced-resolution and the original attacks. \n* by the way there appears to be a typo in Fig 6(a): both BiLN+Sign-OPT and AE+SignOPT are the same color in the plot\n* In the new Figure 4 with MNIST experiments, in all the experiments, the base attack appears to be more query efficient than the resolution reduced attack. For a few it appears that before 5000 queries the reduced-resolution are better, but claiming this supports the earlier results isn't as convincing.  \n* The observations from these plots are confusing: \n>\"The hard-label attacks consistently outperform against the Robust Manifold defense\": \nAgain not clear how it is claimed that the attack is outperforming the defense without mentioning success rates for this experiment. This also does not clearly outline why the reduced-resolution variants to not match up to the base attacks?\n* The reconstruction error plots are also very hard to see -- once again all the reduced-resolution suffer in these experiments. The explanation given is that:\n> \"Due to the dynamics of the topological hierarchy discussed before, the dimension-reduction is less effective on MNIST\" \nThis is another example of very heavy terminology that makes it hard to understand what the argument is exactly. \n\nOverall, I think the paper needs more clarity in its experiments. The story is muddled by several inconsistent results, some showing improvements with reduced resolution (the finding is surprising and genuinely interesting.. but unfortunately doesn't seem to be consistent; especially when looking at it with the success rate metric)\n\nI think the authors have a convincing motivation and argument that query-efficient hard label attacks achieved with reduced-resolution can be closer to the manifold, this is clear in Robust CIFAR-10, Natural ImageNet models. The paper does a good job of explaining why these are the case, unfortunately the hypothesis doesn't hold up just as well in the other scenarios, which raises the question as to how these can be used in a more realistic scenario -- and more importantly as one of the other reviewers also pointed out -- what the reader must take out of these results to appreciate query efficient attacks better. I think in its current state the experiments portray a more confusing picture than the arguments in the paper suggest. As a result, I will retain my original score. \n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates several properties of adversarial examples obtained by hard-label attacks. There are some interesting findings in this paper, such as the connection between query efficiency and distance to the image manifold. However, all the reviewers think the paper is below the acceptance threshold due to several weaknesses, including insufficient experiments, clarity, and whether the observations made in the paper can benefit query efficiency or quality of hard-label attacks. 
Decision: Reject