Towards Lightweight Black-Box Attack Against Deep Neural Networks
Authors: Chenghao Sun
Yonggang Zhang
Wan Chaoqun
Qizhou Wang
Ya Li
Tongliang Liu
Bo Han
Xinmei Tian
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial examples
Adversarial attacks
Black-box attack.

Abstract: Black-box attacks can generate adversarial examples without accessing the parameters of target model, largely exacerbating the threats of deployed deep neural networks (DNNs). However, previous works state that black-box attacks fail to mislead target models when their training data and outputs are inaccessible. In this work, we argue that black-box attacks can pose practical attacks in this extremely restrictive scenario where only several test samples are available.  Specifically, we find that attacking the shallow layers of DNNs trained on a few test samples can generate powerful adversarial examples. As only a few samples are required, we refer to these attacks as lightweight black-box attacks. The main challenge to promoting lightweight attacks is to mitigate the adverse impact caused by the approximation error of shallow layers. As it is hard to mitigate the approximation error with few available samples, we propose Error TransFormer (ETF) for lightweight attacks. Namely, ETF transforms the approximation error in the parameter space into a perturbation in the feature space and alleviates the error by disturbing features. In experiments, lightweight black-box attacks with the proposed ETF achieve surprising results. For example, even if only 1 sample per category available, the attack success rate in lightweight black-box attacks is only about 3% lower than that of the black-box attacks with complete training data. 

Towards Lightweight Black-Box Attacks Against Deep Neural Networks
Chenghao Sun'
Yonggang Zhang?
Wan Chaoqun
Qizhou Wang?
Ya Li Tongliang Liu
Bo Han
Xinmei Tian" 1"
University of Science and Technology China
"Hong Kong Baptist University
3 Alibaba Cloud Computing Ltd
*iFlytek Research
TThe University of Sydney
Abstract
Black-box attacks can generate adversarial examples without accessing the param- eters of deep neural networks (DNNs), largely exacerbating the threats of deployed models. However, previous works state that black-box attacks fail to mislead DNNs when their training data and outputs are inaccessible. In this work, we argue that black-box attacks can pose practical attacks this highly restrictive scenario where only several test samples are available. Specifically, we find that attacking the shallow layers DNNs trained on few test samples can generate refer
powerful adversarial examples. As only few samples are required, these attacks as lightweight black-box attacks. The main challenge to promoting lightweight attacks is mitigate the adverse impact caused by the approximation error of shallow layers. As it is hard to mitigate the approximation error with few available samples, we propose Error TransFormer (ETF) for lightweight attacks. Namely, ETF transforms the approximation eтrоr in the parameter space into a perturbation in the feature space and alleviates the error by disturbing features. our experiments, lightweight black-box attacks with the proposed ETF achieve surprising results. For example, even if only sample per category is available, the attack success rate achieved by lightweight black-box attacks is only about 3% lower than that of the black-box attacks using complete training data
Introduction
Black-box attack methods 12 can mount successful attacks without accessing the parameters of deep neural networks (DNNs), posing great challenges to deep learning in safety-critical situations. Existing black-box attack methods implicitly assume that the training data and/or the outputs of target models are available to adversaries. For scenarios where training data are available, adversarial examples can be generated by attacking surrogate models constructed for approximate target models. Using the target models' outputs for estimating gradients is another practical approach to crafting adversarial examples.
general, due to the low dependency on target model information, the black-box attacks raise realistic threats to real- world deep learning systems, including semantic
segmentation 2 36). object detection [18 45 44], and automatic driving [54]
However, these assumptions can be violated in many practical scenarios 3 46), where both the training data and the outputs target models are inaccessible. This realistic attack scenario introduced in 3 generally termed as the no-box setting. Under such restrictive setting, Li et
"Corresponding author (xinmei@ustc.edu...
'Code is available at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

al. [34] stated that existing black-box attacks fail to mislead target models because it is now forbidden to access previously essential information in mounting black-box attacks. Therein, the failure of existing black-box attacks is not surprising. The reason is that constructing reliable surrogate models typically relies on accessing the complete training data of target models, which is forbidden in such a restrictive setting with only a few test samples available.
In this work, we aim to reveal the potential threats of black-box attacks in the no-box setting, as mounting black-box attacks with highly limited accessible information leaves many models exposed to attack. Specifically, we challenge the previous believes by raising the following question: Can black-box attacks success in the no-box setting? ? The doubts about the potential threats of the black- box attack are not groundless. Specifically, the potential threats of black-box attacks stem from two facts: a) adversarial examples can be generated by perturbing representations at shallow layers of DNNs 28 27 48 ; b) regarding the representation of shallow layers, there do not exist critical differences between those models learned from a few data and that of the whole training data. [
Building upon the above facts, it is actually possible that an adversary can successfully mount attacks using limited samples, as powerful adversarial examples can be generated by attacking the shallow layers of DNNs trained on few samples. Concretely, the adversary can leverage available samples to construct a surrogate model to approximate the shallow layers of target models within acceptable errors. Consequently, adversarial examples can be generated by perturbing the features obtained from the shallow layers of the constructed surrogate model. As merely a few samples are required for attacking, we refer to black-box attacks in the no-box setting as lightweight black-box attacks. Intuitively, the closer the surrogate model is to the target model, the higher the attack success rate the crafted adversarial examples have 31 the approximation error of shallow layers is alleviated, the lightweight black-box attack can be as powerful as black-box attacks with complete training data. Hence, the main challenge to mounting a lightweight attack is to mitigate the adverse impact caused by the approximation error of shallow layers.
However, it is challenging to mitigate the approximation error, especially when the number of available samples is limited. Fortunately, it is straightforward to identify which kind of perturbations are preferred: if perturbations applied to a feature contribute to fooling the surrogate model, the perturbation is preferred. Therefore, bridging the connection between the parameter space and the feature space is the key to mitigating the approximation error, Accordingly, we propose transforming the approximation error in the parameter space as the perturbation in the feature space, dubbed Error TransFormer (ETF). Namely, ETF transforms the worst-case approximation error to the worst-case feature perturbation, leading to a min-max scheme that generates adversarial examples under the worst feature perturbations. We verify the attack success rate of lightweight black-box attacks using 7 models trained on the ImageNet dataset [47) and find that existing attack methods are much more potent than previously claimed. Moreover, the performance of lightweight black-box attacks can be further promoted by the proposed ETF, i.e., the attack success rate is only 3% lower than that achieved by black-box attacks having complete training data of target models.
2 Related works
2.1 Adversarial attack
According to the amount of accessible information, existing adversarial attacks can be roughly divided into two categories: white-box attacks and black-box attacks. White-box attacks [19 41 [5| [6] assume that the target model is transparent to adversaries, i.e., adversaries can access all information about the target model. Nevertheless, the assumption of transparent target models can be violated in many practical scenarios [T. Hence, black-box attacks [4] 37 64 7 are proposed and applied to the scenario where relatively limited information about the target model is accessible, i.e., only certain number of model queries or the training data of target tasks are available.
are widely explored to improve the adversarial transferabiliiy. The core idea of these attacks is based on the empirical observation that well-trained models' intermediate features are transferable [58]. This is consistent with the recent works showing that adversarial examples comprise spurious features having the transferable property 63 G. Hence, Inkawhich et al. (28) propose to perturb the feature space of neural networks to create more transferable adversarial examples. These methods provide an interesting empirical conclusion that disturbing the shallow layers of DNNs can also generate adversarial examples.
Among these black- box attacks, intermediate-Iever attacks 48 28 24

---

Existing black-box box attacks assume that the large-scale training data of target models and the feedback from querying target models are accessible, but these assumptions can be violated in many scenarios, e.g., the no-box setting (3. Li et al.
<4 stated that existing black-hox attacks cannot be successfully
mounted because existing DNNs require large-scale training data for generalization. Given only small- size data available, obtaining a surrogate model with strong generalization is challenging. Hence, Li et al. propose replacing black-box attacks with their proposed no-box attack [34], where they train classical auto-encoder model instead of the supervised classification model due to the constraint of a small-scale dataset. Concretely, they train 20 auto-encoders for each category and generate adversarial examples by attacking these auto-encoders, which is time- and computational-conseenn
Different from existing black-box attacks. lightweight black-box attacks aim reveal the potential risk of black-box attacks under the no-box setting. Because one main challenge to perform lightweight black-box attack is to mitigate the adverse impact caused by the approximation error, we propose error transformer that is a min-max strategy transforming the approximation error in the weight space to the feature space. The min-max strategy is different from that introduced in (55), where min-max strategy is proposed to flatten the loss landscape in the weight space, see Appendix
2.2 Approximation of Shallow Networks
A recent study [58 confirms the intuition that shallow layers in DNNs can be as low-level feature extractors, provided that strong data augmentation is used []. Motivated by this observation, Asano al. (1] design method to explore the information in every layer of DNNs. Asano et al. [ train a supervision model on ImageNet I LSVRC-12 12, and a self-supervision model on a small-scale dataset. Then, they apply linear probes 61 to all intermediate layers of networks, where linear classifier is trained on the top of pre-trained and fixed feature representations. In this way, they evaluate the quality of the representation learned different depths of the networks. The results show that, given heavily synthetic transformations, the shallow layers of DNNs learned from few images can approximate that of DNNs trained on millions of images. According to the intriguing empirical observation, it is possible to construct a surrogate model with limited samples, where its shallow layers are similar that of the target models. Besides, it known that the neural networks prefer to make the decision through the spurious feature [63] on which are focused by models to correlate to the true label. Since shallow layers based on the small-scale or large-scale data set can acquire the similar spurious feature [ it is an effective way to leverage the shallow model to mount an attack in the spurious features when the number the data is limited.
3 Preliminaries
3.1 No-Box Threat Model
The no-box setting 3 denotes threat model where available information about the target model is extremely limited, making it challenging to generate adversarial examples. According to the description in B], the no-box threat model can be defined as follows.
Definition (Nx-Box Threat Model) A threat model is called the no-box threut model if the adver- sary in this scenario is not allowed to access the targel model's 's training data and outputs, so it only has some samples that can be correctly predicted with high probability by the target model.
The definition is consistent with the no-box setting introduced in B where the target model's training data and outputs are inaccessible to the adversary. Moreover, the available samples used for generating adversarial examples are expected to be limited. This is because if the number of samples is similar adversary can leverage these samples to construct a surrogate model similar to target models, making the condition of inaccessible training data meaningless. is intuitive that target models can correctly predict the labels of these available samples with high probability. For example, it is meaningless to perform attacks using samples misclassified by the target model.
even larger than the number of samples used for training target models, the
3.2 Lightweight Black-Box Attack
Definition (Lightweight Black-Box Attack) An attack is called the lighweeight black-box attack an adversary aims to perform attack in the context of a no-box threat model.
3

---

The definition of the lightweight black-box attack shows that lightweight attack is special kind of black-box attacks. The extremely limited accessible information distinguishes it from other black- box attacks. For example, query-efficient black-box attacks require the feedback information of target models [2] 26), while lightweight attacks do not. To our best knowledge, how to perform lightweight black-box attack is still lacking in the literature. For example, Li et al. [34] claim that black-box attacks fail to fool target models when their training data and outputs are inaccessible.
Approach
This section gives detailed description of how to perform lightweight black-box attacks and the proposed Error TransFormer (ETF) to alleviate to adverse impact caused by approximation error.
4.1 Lightweight Surrogate Model
Surrogate models are widely used in black-box attacks since adversarial examples are usually generated by attacking surrogate models when querying target models is forbidden. Since the lightweight black box attack is a special kind of black-box attack, performing the attack also requires constructing surrogate models. We call surrogate models employed lightweight black-box attacks the lighteeight surrogate model, building upon the fact that the number of available samples in the no-box threat model typically limited.
Similar to existing black-box attacks, we assume that the label information of samples used for generating adversarial examples are accessible. In this scenario, the lightweight surrogate model is trained for a classification task. Specifically, the lightweight model is realized by training a classification DNN equipped via conventional supervised learning, using cross-entropy loss ((;')
min B (aW)>D (() 2t
where stands for the parameters of the surrogate model J, and (x.,)) denotes the sample label pair of random variables sampled from the distribution of natural data D.
In black-box attacks, adversary only leverages the shallow layers of DNNs, leading to a straightfor- ward approach to constructing lightweight surrogate models. Specifically, can train the lightweight surrogate model in a contrastive manner, where the supervised information is no longer necessary:
w)) Ex-Dri of w) (()
min
log XED
D
where is specific similarity metric, e.g., cosine similarity, T denotes the data transformation operation, and D means that x is removed from the dataset D. The contrastive strategy in Eq 2 makes it possible to perform attacks when the label information is unavailable. Unless otherwise specified, we mainly uses Eq. to train the surrogate models.
4.2 Feature Space Perturbation
Built upon the empirical observation 58 using limited samples can approximate the shallow layers of DNNs within acceptable errors. This suggests that deep layers other than shallow layers can cause large approximation error. Accordingly, merely the shallow layers of the learned lightweight surrogate models are used for generating adversarial examples, which is similar to the feature space attacks 488). For brevity, we define the shallow layers as &p and denote the feature as p(r; w).
In feature space attacks, guide images are usually required for generating adversarial examples [48] natural image used for generating adversarial example is usually called source image. Given a souree image Zg. generate adversarial example Fadv by perturbing x, such that its feature is similar to that of guide image Zg. where the labels of these two images are different. Specifically, the adversarial examples are generated follows:
Fadu = arg min
|| -xll, d(p((gg w), ;
where d stands for a specific distance, e controls the strength of adversarial perturbation, and denotes the perturbed version of the source image.

---

4.3 Error Transformer
Although applying feature space attacks to lightweight surrogate models can generate transferable adversarial examples, the approximation error will cause adverse impact to mounting attacks. The reason is that large dissimilarity between the surrogate models and the target models reduces the attack success rate. Therefore, the main challenge to mounting lightweight attacks is to mitigate the adverse impact caused by the approximation error. However, available samples are usually limited in the no-box threat model, making it hard to alleviate the approximation error.
To address the challenge, we propose transforming the parameter space's approximation error as the feature space's perturbation. Specifically, we seldom know which perturbations can point (from the surrogate model) to the target model, making it challenging to alleviate the approximation error in the weight space. In contrast, we have the prior that samples with different labels should have distinguishable representations. Thus. we can leverage such prior knowledge to select preferred perturbations in the feature space, i.e., we prefer perturbations that can make representations of samples with different labels indistinguishable. Similarly, it also straightforward perturbations, leading to the design of the min-max optimization to identify the "worst" model. Thus, connecting the parameter space and feature space makes it possible to mitigate the adverse impact caused by approximation error. To bridge the connection, we introduce the following key identity, which is also used in [33]
define the "bad"
<p(x; {wo + + A) Uf w (w\"")) = s((w" +wo A)z;w)ww = (x Ar); w/ w/w")) = pp(x +o; Am; u)) (4)
where w stands for the first layer parameters of model yp, A is a transformation matrix used for perturbing w², (w' + A' U { {ww)uu [w'w'} means that the first layer's parameters are perturbed, the other layers' parameters keep unchanged. and g is the function parameterized with w w'ww used for processing the first layer's outputs. According to Eq. 4 we can transform a perturbation in the parameter space as the perturbation in the data space, i.e., sox;; { (w" + w } U (w ww)""")) = sp(x + Ax; w) Similarly, we can apply the transformation operation to the hidden layers, see Appendix A for more details.
Built upon the connection in Eq. the target model and the surrogate model. Taking the first layer as an example, let w, be the parameters of the target model and w/ be its first layer parameters, where the other parameters are the same as the surrogate model. Assume that there exists a transformation matrix A applied to the first layer parameters of the surrogate model such that !! w w A, we have:
transform the first layer parameters' discrepancy between
(fx;; wi U (w /w) }} v0(;; [w- + w A} J {ws) ws) \w: }) = ((x + Aa; {w" fu (wx }) = <(x + Ax; w))
(5)
The identity in Eq. shows that we can find a perturbation in the data space to alleviate the difference in the first layer parameters between these two models. Built upon the transformation, we can transform the worst approximation error to the worst perturbation in the data space. Therefore, we can employ min-max max strategy to mitigate the adverse impact of approximation error through generating adversarial examples under the worst perturbation. Thus, lightweight black-box attacks with ETF generate adversarial examples as follows:
Fadu
arg || - , Se II,, SI
min
d(p(11 + w), sple + A);w)),
(6)
where 1-11, is the l, no-m, and 7 control the strength of perturbations, 4, and A, are the data space perturbations for mitigating the approximation eиror in parameter space, and x' denotes the perturbed version of source image. In this way, we reduce the error in parameter space to approximate the target model. We solve the inner maximization problem by generating perturbations in the feature space, given a perturbed adversarial example. This step aims to mitigate the approximation error. The outer minimization problem is solved by finding adversarial perturbations in the input space, the same as the adversarial example generation. After iterative generation of perturbations, adversarial examples generated by attacking a model with reduced approximation error.
5 Experiments
In this section, we conduct extensive experiments to verify the power of lightweight black-box attacks augmenting with ETF.
Discussiin about the assumption be found in Appendix E
5

---

Table 1: Performance of the different lightweight black-box and black-box adversarial attacks
Model
VGG19 51
Inception v3
RN152 21
DenseNet
SENet 22
WRN
MobileNet
Average
23
59
v3
Clean
67.43
64.36
74.21
73.34
51.28
73.22
65.06
66.99
Autoattack 9
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Deep-PGD
49.01=0.23
52.26+00255
60.71+0,74
57.92+0.37
27. 17
60.18:0.64
44.20=0.63
50.31
Deep-MI
38.9240.43
42.37=0.37
49.53+0.49
49.06*0.89
19.4440.75
49.11
33.46+0.80
40.69a0.96
Deep-DI
43.34=0.40
43.13=0.52
53.78+0.38
55.41=0.53
23.5340.52
51.77=0.48
38.14=0.74
44.15=0.60
Deep-TI
49.46=0.52
49. 64:0.27
58.8940.71
58,75+0.30
26.1940.16
56.31+0.58
44,02=0.46
49.03+0.51
Shallow-PGD
22.93=0.33
31.07=0.58
34.71+0.67
36.20=0.87
13.0840.36
32.1640.66
16.65=0.54
26.69:0.49
Shallow-MI
22.62:40.25
30.83+0.48
34.05+0.27
35.7440.76
12.31+0.41
29.98:4066
17.72+0.31
26. 1740.56
Shallow-DI
22.14=0.39
29.78+0.17
35.51+0.33
35.79#0.61
8.99+0.42
30.61=0.88
16.88=0.47
25.67=0.55
Shallow-TI
21.82=0045
28. 04
34.78+0.15
34.71+0.39
7.96#0.48
30.14
15.77=0.51
24.81=0.37
ETF-PGD
14.11=0.24
20.22+0029
24.20+0.34
24.74
6.96+0.44
20.73++0.28
10.6640.31
17.37=0.35
ETF-MI
15.32=0.52
19.9 19.97=0.28
26.2550.14 14
28. 10+0.5
7.02+0.43
22.21=0066
12.23=0.32
18.72=0045
ETF-DI
14.77=0.35
20.6 03
23.71+0.83
7.23+0.37
20.22:4064
11.53+0.50
17 17.68400.77
ETF-TI
15.45=0.37
18.03=0.34
22.63+0.45
24.20+0.68
6,94+0.41
21.533+0.25
12.88+0.34
17.38+0.71
Decp*-PGD
12.43=0.51
28.15 15:0.3
16.5420.49
12.61#0.22
7.09+0.32
13.33=0.54
9.6420.28
14.25=0037
Deep*-MI
11.77=0.75
25. 14
18.10:0.64
13.72+0.34
4.26:40.5
14.61#0.37
8.30=a.37
03.7020.68
DDe
7.61#0.41
45:0.5
8.23+0.33
9,90+0.57
6.6620.34
9.72+0.42
7.91#0.66
9.74+0.55
Deep*-11
9.55#0.48
23.48=0.86
10.6320.64 13.51=0.46
6.4640.26
10.92=0.61
9.55=0.35
12.01=0.43
Deep* refers to the attacks mounted in the model trained on the large-scale training data.
5.1 Experimental Setup
Models Architectures. All surrogate models are based on the ResNet-18 (21.. In the ordinary settings of the black-box attack, the general surrogate models are trained through the whole training set of ImageNet [47]. On the contrary, the lightweight surrogate models adopt only 1, 000 images randomly sampled from the validation set of ImageNet, considering that images in the training set are probably inaccessible in practice [3]. Results evaluated on the CIFAR10 dataset can be found in Appendix D Please refer to Table|5iin i the Appendix [B for the detailed information on the lightweight surrogate model and the general surrogate model. Regarding the target models, various model architectures are selected for full comparison, including, VGG-19 (51). Inception v3 (53).
ResNet- 52 21]. DenseNet (23) SENet (22). wide ResNet (WRN) (59 and MobileNet v2 (49 All these target models are well-trained on ImageNet.
Implement Details. The lightweight surrogate models are randomly initialized. The batch size is 128, the epoch is 500, and the initial learning rate 0.4, linearly decreasing to 0.008. To better approximate the low-level feature of shallow layers, we refer to the self-supervision work (5) and apply various random augmentations to the data in each epoch for training. To mount attacks, the classic methods, e.g., PGD [40), MI [I]] DI (56), and TI [12), are applied to all the surrogate models. Unless otherwise specified, our attack is mounted on the first layer of ResNet-18 18 for lightweight surrogate models. Following previous works on intermediate-leveel attacks, the metric d in Eq. of Shallow- in Table instantiated as the formo Meanwhile, to maximize the potential of ETFs, apply contrastive loss to ETF-(-) in Table The Top-1 prediction accuracy is adopted as the evaluation metric, i.e., lower the classification accuracy means better attack success rate. All the experiments are run for 5 individual trials with different random seeds. In our experiments, target models are evaluated using fe -orm adversarial examples with maximum distortion 0.1, results evaluated under mores settings, e.g., more strict constraints 0.05, fy-norm adversarial examples, and more test images 5,000 images, can be found in Appendix B.4
5.2 Main Results
Results on normally trained models. The results of classification accuracy are shown in Table I which exhibits the evaluations for both lightweight and general surrogate models. The last four rows show the ordinary results where the general surrogate models are trained through all data in the training set. The black-box attacks are mounted based on the whole model, termed "Deep/Deep"" As the size of the training data is reduced to 1, 000, this conventional method obviously loses its aggressiveness. Specifically, the averaged accuracies of 3-6 rows are among 40-50%, revealing limited attacking capability compared to 66.99% of the clean accuracy. It demonstrates the necessity

---

Table 2: The accuracy (%) of 7 normally trained target models evaluated 1000 adversarial examples generated by no-box attacks and lightweight black-box attacks with different intermediate- level attacks. The best results are in bold. (The lower, the better)
Model
VGG19
Inception
RN152
DenseNet
SENct
WRN
MobileNet v3 49
Average
51
v3
21
23
22
59
Clean
67.43
64.36
74.21
73.34
51.28
73.22
65.06
66.99
No-box
34
18.74
33.68
34.72
26.06
42.36
33.16
16.34
29.29
ILA
20.13
28.01
35.72
35.14
9.16
29.97
14.31
24.63
AA
22.76
31.21
36.67
37.94
7.72
33.16
18.17
26.81
ETF
14.11
20.22
24.20
24.74
6.96
20.73
10.66
17.37
of large amount of data for classic black-box attacks. In rows 7-10, shallow layers without ETF are used generate adversarial examples. The averaged accuracies of around 25% illustrates the less dependence of shallow networks on the amount of data. In rows 11 11-14, attacks on shallow layers are further enhanced by ETF, decreasing the averaged accuracies about 17%. The narrow gap between the proposed ETF and the general surrogate model indicates the potential capability of the black-box attack in the no-box settings.
Results on Adversarially Trained Models. Here, we consider the models that are trained through adversarial training [40], which is more persuasive evaluate the effectiveness of attacking methods. Following 40). the target models are trained on the whole training set of ImageNet. During adversarial training, the adversarial examples are generated under e 0, 4/ /2555 8/ 8/255, where means the 6xo- norm constraint. Table 3 exhibits the attacking results mounted by ETF, black-box attacks (general surrogate models with all training data avaliable). and no-box attacks. For simplicity, only the results of PGD attacks are reported. As analysed above, the black-box attacks shows the best attacking results, while ETF only has a narrow gap with it. However, when e = 4/255 or 8/255 ETF exceeds the black-box attacks (29.13% 11t.13%/6661.. 144% vs. 48.11%/38. 24%). It is interesting that black-box attacks are even worse than lightweight attacks. We believe that exploiting the phenomenon can bring something new, but we leave it as our future work due to the limited space.
5.3 Further Analysis and Ablation Study
Intermediate-Level Perturba- tions. Since the lightweight black-box attacks are imple- mented mainly on intermedi- ate features, two classic attacks mounted on intermediate features,
Table 3: The performance of different attacks the adversarial trained ResNet-50 [13]. Therein, e refers to the constraint foo in
adversarial examples for adversarial training. The accuracy (%)
is evaluated on 1000 adversarial examples. E = 0.1 (the lower
i.c. ILA (24) and AA (28), are selected for experiments. Com- paring the ILA and AA in Ta- bleeland Shallow-M/DI/TIPPe in Table D the difference be- tween their average accuracies is marginal. It indicates those intermediate-Ievel feature pertur-
the better). White-box refers to Auto-Attack [9].
Adv_modes Clean
ETF
Black-box
No-box White-box 34]
40
[
0/ 0/255 69.43
16.97
8.20
24.53
0.00
= 1/255
55.62
29.13
48.11
39.62
0.00
3/255
41.68
26.14
38.24
35.87
0.48
bation has limited benefit to lightweight black-box attacks. The reason is that the intermediate- attacks highly rely on the low-level features extracted by the lightweight model to generate adversarial examples and it can weaken the role of high-level information when attacking. The proposed ETF achieves better performance. On one hand, ETF does not rely on the features from high-level layers. On the other hand, ETF introduces min-max scheme to mitigate the adverse impact induced by the approximation error from the shallow layers.
Comparison with No-box Attacks. No-box attack is the first work to explore how mount attacks in the no-box setting. Since Li et. al (34 stated that black-box attacks do not work this setting, surrogate models instantiated auto-encoders rather than models for approximating target models.
7

---

60-
Irceptionns
35
Danneher
Nobos
50
StiNet
WINN
Average accurncy
30
40
24
20
Black hox
Avcrage
10
10
2003
1060
3540
axppoiig
Tayal
1gye2
layad
PC
Number of training images
Layer index
(a)
(b)
Figure 1: (a) How the lightweight attack performance of our approach varies with the number of
images used for training the surrogate model. (b) The influence of low-level feature extraction
different layers of ResNet-18 on lightweight black-box attack performance. (The lower, the better)
Table 4: Model accuracy (%) under lightweight black-box attacks under challenging scenarios, where supervision information the in-distribution data are unavailable, named Unsupervised and OOD.
Model
VGG19
Inception
RN152
DenseNet
SENet
WRN
MobileNet
Average
11
v3
21
23
22
59
v3 49
Clean
67.43
64.36
74.21
73.34
51,28
73.22
65.06
66.99
Supervised
14.11
20.22
24.20
24.74
6.96
20.73
10.66
17.37
Unsupervised
15.54
19.16
26.27
23.75
7,66
22.79
11.43
18.08
OOD
6.13
21.72
25.44
21.89
5.02
24.33
7.16
15.96
Namely, they design the pretext task to train 20 auto-encoders per category to generate adversarial examples. However, the difference between the pretext tasks and the target tasks may significantly reduce the transferability of adversarial examples. In contrast, surrogate model for tasks similar to or same as the task of target models. Hence, though only one sample per category is available, we achieve better performance than no-box attacks, shown in Table 2
method trains the lightweight
Number of Training Samples. One key difference between the lightweight black-box attack and existing black-box attacks mainly lies in the number of samples n used for training surrogate models. Therefore, we further explore how lightweight attack performance varies with different number of samples, i.e., n. We conduct experiments on randomly selected n. images from the validation set of ImageNet. The experimental results shown in Figure ! The average accuracy decreases as more samples are available, thus achieving better attacking results. More data makes it possible for the lightweight black-box attack approximate the shallow layers of the target model more accurately. Especially the lightweight black-box attacks with 10, 000 test images can achieve the same performance general black-box attacks which train the surrogate model 1, 200, 000 training images. Also, our method can still mount attacks even when only 2 or 10 images are available.
Surrogate Model. As analyzed above, we utilize the first few layers of the lightweight surrogate model to approximate the shallow layers of the target model to mount attacks. Hence, would be interesting to study how the layers selection for lightweight models impacts the attack performance. We adopt different layers of ResNet-18 to generate adversarial examples utilizing ETF and attack the different target models. The results are summarized in Fig. It can be seen that low-level information the first block of the model is sufficient for achieving promising attack performance. With more block information the model, the performance of the attack does not improve. In particular, the transferability of adversarial examples significantly decreases when we attack the fully-connected layer, verifying analysis that the high-level layer of the lightweight model cannot approximate the target model well. More results for architecture selection can be found in Appendix [B.4]
Adversarial Example Visualization. To verify that adversarial examples are truly imperceptible, we provide visualization in Figure 2 where Deep*-PGD attack (using training images), Deep-PGI attack (using test images), and lightweight black-box attack are considered.

---

(T)
(b) Deep
(c) Deep*
Figure 2: Adversarial examples crafted by: a) ETF, b) Deep, and c) Deep* attacks.
6 Discussion
Though the no-box threat model provides a weak assumption, many realistic scenarios may be more complex. For example, adversaries have access to only data without supervision information due to security 33 privacy [66 issues. In more extreme cases, c.g., autonomous vehicle [54], adversaries even have no access to the test samples from the distribution over which the model is trained. Instead, the adversaries have access to samples from a similar (but not exact) distribution (because the adversary knows the deployment environment). Hence, we consider two more challenging scenarios which we provide solutions to mount lightweight black-box attacks.
Insufficient supervision information. We design an experiment for mounting attacks using 1, 000 unlabeled samples randomly selected from the ImageNet validation set to explore the power of ETF under this challenging setting with unlabeled data. Thanks to the framework of lightweight attack, feasible train the unsupervised model on unlabeled data for attacks using Eq. 2 Specifically, we adopt contrastive learning [5] to train lightweight surrogate model on the small- scale unlabeled dataset and generate adversarial examples by attacking the trained lightweight surrogate model. The performance is shown in Table where ETF still achieves similar performance without supervision information as that of the results under supervision. Intuitively, we can also employ other self-supervised learning method, e.g., rotation prediction, to train shallow layers, see details of experimental settings and results in Appendix G
Consideration of Out-of-distribuutinn We further study the possibility for mounting attacks under a more strict scenario. where only OOD -iiir--f-diitiitiinnn OOD) data are available. Specifically, we utilize the model pre-trained on OOD samples to mount lightweight black-box box attacks. The rationality lies in that only low-level information required mount lightweight black-box attacks and the acquisition of low-level information. We load pre-trained ResNet-50 on STL-10 (38] dataset as the surrogate model. Without further fine-tuning, we experiment with two test in indiisrriuttion samples, and the results are recorded in Table named OOD. As we can see, the lightweight black-box attack with models trained on OOD data can achieve surprising attack success rate, showing that black-bor box attacks enhanced with ETF can pose practical threats to deployed models.
7 Conclusion
In this paper, we propose the conception of the lightweight black-box attack reveal the potential risk of black-box attacks under the no-box threat model. To mount effective lightweight attacks, we find it erucial leverage DNNs shallow layers because they can learn similar features regardless of the number of samples. This is based on the empirical observations that the model trained with limited samples can approximate the shallow layers the target models within acceptable errors, which can be used for crafting adversarial examples. Therefore, to further enhance the performance, we propose Error TransFormer (EFT) to decrease the approximation error by transforming the approximation error in the parameter space into the feature space. The experiments show that the proposed method

---

achieves a surprising attack success rate under the no-box threat model using only one image per category, i.c., only 3% lower than black-box attacks with complete training data of the target model.
Acknowledgements
This work was supported by NSFC No. 61872329 and No. 62222117, and the Fundamental Research Funds for the Central Universities under contract WK3490000005, YZ and BH was supported by NSFC Young Scientists Fund No. 62006202 and Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652. TL was partially supported by Australian Research Council Projects DP1 0034444,,,, DE- 190101014,7 IC- IC-190100031, DP-220102121, and FT-220100318.
References
[1] Y. M. Asano, C. Rupprecht, and A. Vedaldi. A critical analysis of self-supervision, what can learn from a single image. In ICLR, 2020.
[2] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 40(())834-8) -44888844888 2018.
[3] P. Chen, H. Zhang, Y. Sharma, J. Yi, and C. Hsich. ZOO: zeroth order optimization based black- box attacks to deep neural networks without training substitute models. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15- 15-26, 2017.
[4] P.-Y. Chen, Y. Sharma, H. Zhang, J. Yi, and C.-J. Hsieh. Ead: elastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
[5] Chen, S. Kornblith, M. Norouzi, and E. Hinton. A simple framework for contrastive learning of visual representations, In ICML, 2020.
[6] Y. Chen, C. Wei, A. Kumar, and T. Ma. Self-training avoids using spurious features under domain shift. In NeurlPS, 2020.
[7] Y. Chen, H. Yang. Y. Zhang. K. Ma, Liu, B. Han, and J. Cheng. Understanding and improving graph injection attack by promoting unnoticeability. In ICLR, 2022.
[8] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, Mittal, and M. Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[9] Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML., 2020.
[10] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NeurlPS, 2014.
[11] Y. Dong, F. Liao, Т. Pang, H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum. In CVPR, 2018.
[12] Y. Dong, T. Pang, Su, and J. Zhu. Evading defenses transferable adversarial examples by translation-invariian attacks. In CVPR, 2019.
[13] L. Engstrom, A. Ilyas, H. Salman, S. Santurkar, and D. Tsipras. Robustness (python library). 2019.
[14] L. Gao. Q. Zhang. J. Song, X. Liu, and T. Shen. Patch-wise attack for fooling deep neural network. In ECCV, 2020.
[15] R. Gao, F. Liu, J. Zhang, B. Han, T. Liu, G. Niu, and M. Sugiyama. Maximum mean discrepancy test is aware of adversarial attacks. ICML, 2021.
[16] R. Gao, Wang, K. Zhou, F. Liu, B. Xie, G. Niu, B. Han, and J. Cheng. Fast and reliable evaluation of adversarial robustness with minimum-mann attack. In ICML, 2022.
10

---

[17] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
[18] R. B. Girshick. Fast R-CNN. In ICCV, 2015.
[19] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[20] S. Gowal, S. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. In NeurIPS, 2021.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] J. Hu, L.. Shen, and G. Sun. Squeere-and--ecctatioo networks. In CVPR, 2018.
[23] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[24] Q. Huang. I. Katsman, Z. Gu, H. He, S. J. Belongie, and S. Lim. Enhancing adversarial example transferability with an intermediate level attack. In ICCV, 2019.
[25] A. Ilyas, L. Engstrom, A. Athalye, and Lin, Black-box adversarial attacks with limited queries and information. In ICML., 2018.
[26] A. Ilyas, L. Engstrom, and A. Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In ICLR, 2018.
[27] N. Inkawhich, K. Liang, B. Wang, M. Inkawhich, L. Carin, and Y. Chen. Perturbing across the feature hierarchy to improve standard and striet blackbox attack transferability. In NeurIPS, 2020.
[28] N. Inkawhich, W. Wen, H. H. Li, and Y. Chen. Feature space perturbations yield more transferable adversarial examples. In CVPR, 2019.
[29] Q. Kang, Y. Song, Q. Ding, and W. P. Tay. Stable neural ODE with lyapunov-stable equilibrium points for defending against adversarial attacks. In ,
[30] Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In ICLR 2016.
[31] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In ICLR, 2017.
[32] A. Kurakin, I. J. Goodfellow, and S. Bengio. Adversarial examples in the physical world. In ICLR, 2017.
[33] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting -utto--distributtin samples and adversarial attacks. In NeurIPS, 2018
[34] Q Li, Guo, and H. Chen. Practical no-box adversarial attacks against dnns, In NeurIPS, 2020.
[35] K. J. Liing, G. Heilmann, C. Gregory, S. O. Diallo, D. Carlson, G. P. Spell, J. B. Sigman, K. Roe, and L. Carin. Automatic threat recognition of prohibited items at aviation checkpoint with x-ray imaging: deep learning approach. In Anomaly Detection and Imaging with X Rays (ADIX) III. volume 10632, page 1063203, International Suciety for Optics and Photonics, 2018
[36] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In 2015.
[37] Y. Long, Q. Zhang, B. Zeng, L. Gao, X. Liu, J. Zhang, and J. Song. Frequency domain model augmentation for adversarial attack. CoRR, abs/2207.05388 2022.
[38] D. G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput, Vis., 60(2):91-110, 2004.

---

[39] X. Ma, B. Li, Y. Wang, S. M. Erfani. S. N. R. Wijewickrema, G. Schoenebeck, D. Song, M. E. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
[40] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[41] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In CVPR, 2016.
[42] T. Pang, M. Lin, X. Yang, J. Zhu, and S. Yan. Robustness and accuracy could be reconcilable by (proper) definition. In ICML, 2022.
[43] H. Petzka, M. Kamp. L. Adilova, C. Sminchisescu, and M. Boley. Relative flatness and generalization. In NeurIPS, 2021.
[44] Redmon, S. K. Divvala, B. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[45] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[46] M. Rigaki and S. Garcia. A survey of privacy attacks in machine learning. CoRR, abs/2007.07644 2020.
[47] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and I. Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., (15(3)211-2522 2015.
[48] S. Sabour, Y. Cao, Faghri, and D. J. Fleet. Adversarial manipulation of deep representations. In ICLR, 2016.
[49] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
[50] V. Schwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, and P. Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In ICLR, 2022.
[51] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[52] Su, D. V. Vargas, and K. Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828-88) 841, 2019.
[53] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In CVPR, 2016.
[54] C. E. Tuncali, G. Fainekos, H. Ito, and Kapinski. Simulation- based adversarial test generation for autonomous vehicles with machine learning components. In 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018
[55] D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[56] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
[57] K. Yang, T. Zhou, Y. Zhang, X. Tian, and D. Tao. Clas-ddisentagllement and applications in adversarial detection and defense. NeurIPS. 2021.
[58] Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
[59] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, 2016.
12

---

[60] Q. Zhang, X. Li, Y. Chen, J. Song, L. Gao, Y. He, and H. Xue'. Beyond imagenet attack: Towards crafting adversarial examples for black-box domains. In ICLR, 2022.
[61] R. Zhang, P. Isola, and A. A. Efros Split-brain autoencoders: Unsupervised learning by
cross-channel prediction. In CVPR, 2017.
[62] X. Zhang, J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell., 38( (8(10):1943-193)) 2015.
[63] Y. Zhang, M. Gong, T. Liu, G. Niu, X. Tian, B. Han, B. Schölkopf, and K. Zhang. Adversarial robustness through the lens of causality. In ICLR, 2022.
[64] Y. Zhang, X. Tian, Y. Li, X. Wang, and D. Tao. Principal component adversarial example. IEEE Trans. Image Process., 29:4804-485 4815, 2020.
13


Reviews:
[{"review_id": "tp0TnbL1DHj", "reviewer": "Reviewer_RUwr", "summary": "The authors propose a \"lightweight\" black-box attack method that uses limited data called Error TransFormer (ETF). Experimental results show that the proposed method achieves decent performance compared to other black-box attacks with few available examples.", "questions": "I have no questions for now.", "limitations": "I didn't find any potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The method is novel to me, and technically sound. \n+ The performance is good. It is interesting that only 2 training images can still drop large accuracy.\n\nWeakness:\n- The writing should be carefully checked. I notice there are many errors that may lead the reader to significant confusion, for instance, the distance should be *maximize* instead of *minimize* for generating adversarial examples in line 173, the similarity should be *decrease* in line 234, the first row in Table 1 has some error, and it should be $\\epsilon=0.1$ in the caption of Table 1. Hope the authors check the paper carefully and fix these errors.\n- The authors only test under $\\epsilon = 0.1$, it is more convincing that perform some experiments under smaller $\\epsilon$, such as $\\epsilon=0.05$.\n- It is more convincing that perform experiments on more examples since the size of only 1,000 examples is too small.\n- It is better bold the best results in *all tables*.\n- The experiment setup should be more detailed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kALk0MdYBA", "reviewer": "Reviewer_kZtY", "summary": "This paper studies lightweight black-box attacks with limited knowledge of the target model and its output.\nThe author proposed an Error Transformer(ETF) to alleviate the approximation error between the surrogate model and the target model.\nThe experiment results show that using ETF + limited samples is only 3% attack success rate lower than the black-box attack with full access of the training data.", "questions": "1.  According to eq. 4, is the Error Transformer simply learning one more transformation matrix $A$? If it is, then is $\\Delta_s = Ax$? Or does it shows that the feature space perturbation at the first layer is equivalent to data space perturbation(when using $Ax$ as noise)?\n2. In Section 4.3, line 185, the author said, \"Therefore, connecting the parameter space ...\" is there a more intuitive explanation for it? It is clear that eq. 4 shows it, but I don't think it is intuitive with the description in lines 183~184.\n3. In section 4.3, lines 199~200, the author assumes there exists a transformation matrix $A$, $s.t.$ $w_t^1 = w^1 + w^1A$. How strong is the assumption here? What if the target and surrogate models have different dimensions at the first layer?\n4. In the eq. 6 and 9, what does $\\Delta_s$ means? I am unsure if there is a difference between $\\Delta_t$ and $\\Delta_s$.\n5. Where is the $x^{\\prime}$ defined in the paper? It looks like the perturbed version of $x$ in eq. 3, and it looks like source $x$ in eq. 6.", "limitations": "The author addressed the social impact in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n 1. The experiment results are compelling; this paper shows some interesting results that a black-box attack can be performed without using the training set.\n 2. To my best understanding, using the shallow layer & ETF for approximation of the target model is novel to the ML security field.\n 3. The motivation of the proposed method and the lightweight black-box attack are well illustrated.\n 4. The author conducted a thorough analysis and ablation study in section 5.\n\nWeaknesses:\n  1.  I suggest the authors can show at least one visual example/noise generated by their method.\n  2. Please refer to the questions.\n  \nI suggest the authors add more explanations to their equations to make them easier to follow. Overall, I think the experiment results are good and the motivation is clear. Therefore, I tend to vote for acceptance of this paper and am willing to change my score after the author clarifies my questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "Hdlih3dePeW", "reviewer": "Reviewer_LA1D", "summary": "The paper looks at the interesting task of no-box attacks. This is a setting that is even harder than black-box attacks, where typically the attacker has access to either the training data and can construct a surrogate model or the model outputs to estimate gradients. In the no-box setting considered in this paper, the attacker only has access to a small number of samples (1k on ImageNet) that are correctly classified by the target model. The attack is based on feature space perturbations, as only the shallow layers of DNNs can be approximated with this little data and the authors propose the Error TransFormer to alleviate issues caused by approximations errors. \n\nThey first train a Shallow DNN on the limited data pool. As the shallow layers do not yield a label output, instead of the standard adversarial objective used for FGSM/PGD attacks, their attack objective transforms the feature representation of an image into that of a given image that comes from a different class. To mitigate the difference between the target model and the surrogate model, they employ a min-max strategy that combines feature and parameter space. \nThey evaluate their approach on various ImageNet source models. \n\n", "questions": "- The paper only considers L-inf perturbations. While they are the most common, does the method work for L0, L1 and L2 too? \n\n- In 4.1 you say that the model has access to label information on the limited data pool. From the insufficient supervision information ablation, I also take that this is used during training. However 4.1 also motivates that for shallow attacks we only need to train shallow nets with an unlabeled contrastive loss. Which loss (Eq (1) vs (2)) is used for the actual training of the surrogate model? \n\n- You use heavy data augmentation, what is the impact of that and did you experiment with different ones? ", "limitations": "Obviously, attack methods can in general be used in a malicious way, especially if we go to more realistic scenarios. I think adding a sentence about that in C would make sense. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to understand. \n\n- The setting is very interesting. Assumptions from the standard black-box setting are often not realistic and the no-box setting is more applicable in real world scenarios. \n\n- Their method is simple and easy to implement.\n\n- The evaluation is clear. Deep models can not be trained in the no-box setting, shallow models work much better but can be improved by their ETF approach.\n\nWeakenesses:\n\n- The novelty is somewhat limited. A similar min-max objective similar to ETF in weight space has for example been explored already in \"Adversarial Weight Perturbation Helps Robust Generalization\" by Wu et al. Most other ideas like surrogate training of a shallow model also appeared in previous papers. \n\n- The evaluation focuses on Imagenet only. While this is certainly an interesting setting, it might be worth to also add some experiments on other datasets. For example, while CIFAR10 is a much easier dataset, there also exist much more robust models and it might be interesting to evaluate the performance of the method to the most robust models for example from RobustBench.\n\n\n- In Table 1-3, their should be another row for the ground truth robust accuracy in a white-box setting. While this is not computable exactly, AutoAttack should yield a fairly accurate estimate. \n\n- Table 1 is somewhat hard to read. I understand that boldfacing the single best result per column is not optimal as it does not make sense to compare different attacks, but it would be easier to understand the general message of the table, which clearly is that ETF is the best method in the no-box setting, by some form of highlighting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-Jjj-KeaDj8", "reviewer": "Reviewer_SEqv", "summary": "Recently, there has been a lot of focus on techniques for the black-box generation of adversarial attacks. However, in practice, an adversary might not have unrestricted access to the predictive interface of the target model or unlimited data to generate adversarial examples. In this paper, the authors propose Error TransFormer (ETF) for generating adversarial examples, which works even when the adversary doesn't have access to training data and model outputs. ETF works by constructing a surrogate model with the limited available data and then perturbing the features obtained from its shallow layers.", "questions": "- How imperceptible are the examples generated by the proposed technique? For instance, even though most black-box adversarial techniques have higher success rates, they generally forgo imperceptibility. \n- How good are examples generated by the proposed technique in evading the recent class of adversarial example detection methods?\n- Does surrogate model architecture impact the success rate of the proposed technique?\n- Does the technique also work in other domains like NLP?\n- The surrogate model is trained in a contrastive manner. Can other self-supervision tasks like rotation be used to train it? (ref. Unsupervised Representation Learning by Predicting Image Rotations)\n- For table 1, were the surrogate model trained using labels or in a contrastive manner?\n- How useful are the examples generated by ETF in improving the robustness of the models?\n- In addition, as the performance of the surrogate is tied to the samples used for training it, I would encourage the authors to run the experiments on different sample sets and report the variance.", "limitations": "I would encourage the authors to list the limitations of the proposed approach.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- There are relatively few techniques that can work under the threat model considered in the paper.\n- Even though the technique appears simple, it performs pretty well in practice.\n\n\nWeaknesses:\n- The paper is a bit hard to follow, and the writing needs improvement.\n- I found the experiments a bit lacking, and please refer to the next section for a list of questions I would like to be answered.\n\nOne vital information authors leverage is that we can generate powerful adversarial examples by attacking shallow layers. I would encourage the authors to add appropriate references. \nA vital piece of the technique is missing, like how is the optimization problem in Eq. 6 solved. I would encourage the authors to include such details in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept