Luring of transferable adversarial perturbations in the black-box paradigm
Authors: Rémi Bernhard
Pierre-Alain Moëllic
Jean-Max Dutertre
Venue:  (2021)
Keywords: Neural Networks
Adversarial Machine Learning
Security

Abstract: The growing interest for adversarial examples, i.e. maliciously modified examples which fool a classifier, has resulted in many defenses intended to detect them, render them inoffensive or make the model more robust against them. In this paper, we pave the way towards a new approach to improve the robustness of a model against black-box transfer attacks. A removable additional neural network is included in the target model and is designed to induce the "luring effect", which tricks the adversary into choosing false directions to fool the target model. Training the additional model is achieved thanks to a loss function acting on the logits sequence order. Our deception-based method only needs to have access to the predictions of the target model and does not require a labeled data set. We explain the luring effect thanks to the notion of robust and non-robust useful features and perform experiments on MNIST, SVHN and CIFAR10 to characterize and evaluate this phenomenon. Additionally, we discuss two simple prediction schemes, and verify experimentally that our approach can be used as a defense to efficiently thwart an adversary using state-of-the-art attacks and allowed to perform large perturbations.

Under review as a conference paper at ICLR 2021
LURING OF TRANSFERABLE ADVERSARIAL PERTUR- BATIONS IN THE BLACK-BOX PARADIGM
Anonymous authors Paper under double-blind review
ABSTRACT
The growing interest for adversarial examples, i.e. maliciously modified exam- ples which fool a classifier, has resulted in many defenses intended to detect them, render them inoffensive or make the model more robust against them. In this paper, we pave the way towards a new approach to improve the robustness of a model against black-box transfer attacks. removable additional neural network is included in the target model, and is designed to induce the luring effect, which tricks the adversary into choosing false directions to fool the target model. Train- ing the additional model is achieved thanks to a loss function acting the logits sequence order. Our deception-basee predictions of the target model and does not require labeled data set. We explain the luring effect thanks to the notion of robust and non-robust useful features and perform experiments MNIST, SVHN and CIFAR1O to characterize and eval- uate this phenomenon. Additionally, we discuss two simple prediction schemes, and verify experimentally that our approach can be used as defense to efficiently thwart an adversary using state-of-the-art attacks and allowed to perform large perturbations.
method only needs to have access to the
INTRODUCTION
Neural networks based systems have been shown to be vulnerable to adversarial examples (Szegedy al. 2014). i.c. maliciously modified imputs that fool a model inference time. Many directions have been explored to explain and characterize this phenomenon (Schmidt et al.] [018 Ford et al., 2019 Ilyas et al. 2019; Shafahi et al.) 2019) that became growing concern and a major brake on the deployment of Machine Learning (ML) models. In response, many defenses have been proposed to protect the integrity of ML systems, predominantly focused on an adversary in the white-box set- ting (Madry et al. 2018 Zhang et al. 2019] Cohen et al., 2019: Hendrycks et al. 2019; Carmon jet al. 2019). In this work, we design an innovative way to limit the transferability of adversarial perturbation towards model, opening new direction for robustness in the realistic black-box set- ting (Papernot et al.] 2017). As ML-based online API are likely to become increasingly widespread, and regarding the massive deployment of edge models in large variety of devices, several instances of a model may be deployed in systems with different environment and security properties. Thus, the black-box paradigm needs to be extensively studied efficiently protect systems in many critical domains.
Considering a target model M that defender aims protecting against adversarial examples, we propose a method which allows to build the model T, an augmented version of M, such that adver- sarial examples do not transfer from T to M. Importantly, training T only requires to have access to M. meaning that no labeled data set is required, so that our approach can be implemented at low cost for any already trained model. P (with T = M P) taking the form of a neural network trained with specific loss function with logit-based constraints. From the observation that transferability of adversarial perturbations between two models occurs because they rely similar non-robust features (Ilyas et al. 2019), we design P such that (1) the augmented network exploits useful features of M and that (2) non non-robuse robust features of T and M are either different or require different perturbations to reach misclassification towards the same class. Our deception-bart based method is conceptually new as it does not aim at mak- ing M relying more on robust-featur
built by augmenting M with an additional component
with proactive schemes (Madry al. 2018 Zhang al.

---

Under review as a conference paper at ICLR 2021
2019), nor tries to anticipate perturbations which directly target the non-robust features of M with reactive strategies (Meng & Chen 2017 Hwang et al. 2019).
Our contributions as follows:
We present an innovative approach to thwart transferability between two models, which we name the luring effect. This phenomenon, conceptually novel, opens new direction for adversarial research.
We propose an implementation of the luring effect which fits any pre-trained model and does not require label data set. An additional neural network is pasted to the target model and trained with specific loss function that acts on the logits sequence order.
We experimentally characterize the luring effect and discuss its potentiality for black-box defense strategies on MNIST, SVHN and CIFARIO, and analyze the scalability on Ima- geNet (ILSVRC2012).
For reproducibility purposes, the code is available https::/anonymolll 4open. science/x/3c6de// 927d- 4f51 b187 583e64536ff6)
LURING ADVERSARIAL PERTURBATIONS
2.1 NOTATIONS
We consider classification task where input-label pairs (z. v) E X x y are sampled from a distri- bution D. | = is the cardinality of the labels space. A neural network model M3 : X + V, with parameters 6, classifies input X' to a label (x) V. The pre-softmax output function of Ma (the logits) is denoted hdd X' - RRC For the sake of readability, the model Mg is simply noted as M, except when necessary.
2.2 CONTEXT: ADVERSARIAL EXAMPLES IN THE BLACK -BO SETTING
Black-box settings are realistic use-cases since many models are deployed (in the cloud or embed- ded in mobile devices) within secure environments and accessible through open restrictive API. Contrary to the white-box paradigm where the adversary allowed to use existing gradient-based
attacks (Goodfellow et al. 2015 Carlini & Wagner 2017 Chen et al.) 2018 Dong et al., 2018 Madry et al.) 2018 Wang et al.) 2019), an attacker in a black-box setting only accesses the output label, confidence scores logits from the target model. He can still take advantage of gradient free methods (Uesato et al. 2018 Guo et al. 2019 Su et al. 2019 Brendel et al.) 2018: Ilyas et al.) 20188] Chen 2020) but, practically, the number of queries requires mount the attack is prohibitive and may be flagged as suspicious (Chen et al.] 2019] Li et al.] (020). In that case, the adversary may take advantage of the transferability property (Papernot al. (217)) by crafting adversarial examples on a substitute model and then transfering them to the target model.
2.3 OBJECTIVES AND DESIGN
Our objective to find novel way to make models more robust against transferable black-box adversarial perturbation without expensive (and sometimes prohibitive) training cost required by many white-box defense methods. Our main idea is based on classical deception-based approaches for network security (e.g. honeypots) and can be summarized as follow: rather than try to prevent an attack, let's fool the attacker. Our approach relies on a network P : X + X, pasted to the already trained target network M before the input layer, such as the resulting augmented model will answer T(x) = M o P(x) when fed with input æ. The additional component P is designed and trained reach a twofold objective:
Prediction neutrality: adding P does not alter the decision for clean example i.e. T(x) = M o P(x) (();;
Adversarial luring: according to adversarial example x' crafted to fool T. M does not output the same label as (i.e. M P(x') # M (x')) and, in the best case, x' inefficient (i.e. M(x)) y).
2

---

Under review as a conference paper at ICLR 2021
To explain the intuition of our method, we follow the feature-based framework proposed in Iyas et al. (2019) where a feature f is a function from X to R, that M has learned to perform its pre- dictions. Considering binary classification task, f is said to be p-useful (p 0) if it satisfies Equation a and is y-useful robust if under the worst perturbation 5 chosen in a predefined set of allowed perturbations A, stays y-useful under this perturbation (Equation [p). A p-useful feature f is said to be a non-robust feature if f is not robust for any y 2 0.
(a) E( ly [(x)] P
(b) (a inf V- f(x (1x)) >7 6EA
(I)
An adversary which aims fooling a model will thus perform perturbations the inputs to influence the useful features which are not robust with respect to the perturbation he is allowed to apply. We denote Fis the set of p-useful features learned by M. We consider set of allowed perturbations A and 7 > such that we note FM and FM F*,NR respectively the set of "y-robust and non non-robust features learned by M (relatively to A). An adversary that aims at fooling M o P will alter function compositions of the form f o P with f E FM- These function compositions are the non-robust useful features of M o P. whose set is denoted F F+.NR
MoP
Based on the observations that transferability of adversarial perturbations between two mod- els occurs because these models rely on simi- lar non-robust features (Ilyas et al.) 2019). we consider o P E Fuio and derive two possibilities which ensure the lowest transfer- ability between M o P and M, regarding the robustness of f. If f is robust, means the adversarial perturbations from A is sufficient to flip the augmented feature f o P but is not efficient to directly impact ʃ. This case the optimal one, since the adversarial example is unsuccessful on the target model (M P(x') * and (xx) y). On the other hand, if f E Fir NR (i.e. both f and f o P are non-robust), restraining the transferability means that the additional model P impacts the way useful features vary with respect to input alterations so that the adversarial perturbation lead to two different labels. We encompass these two cases (illustrated in Figure T within what we name the luring effect. The adversary is tricked into modifying input values in some way to flip use- ful and non-robust features of M o P and these modifications are either without effect on the useful features of M, flip the non-robust features of M in different way (and therefore detectable, as presented in Section
MoP P
M F
E
T
NR SOPET
MoP E
F., that
x²
i E NR
M
Figure 1: Luring effect. (left) x' fools M o P by flipping non-robust feature f o P (toward the green class). (right) However, f can be a robust feature, then M is not fooled (still in the blue class), or a non-robust feature but switched dif- ferently than o P (towards the orange class).
2.4 TRAINING THE LURING COMPONENT
To reach our two objectives (prediction neutrality and adversarial luring), we propose train P with constraints based on the predicted labels order. For x E X, let a and 33 be the labels corre- sponding respectively the first and second highest confidence score given to æ by M. The training of P is achieved with a new loss function that constraints a to (still) be the first class predicted by M P (prediction neutrality) and that makes the logits gap between a and / the highest as possible for M o P (adversarial luring).
To understand the intuition behind this loss function, let's formalize the concepts learned by M and M o P. The prediction given by M corresponds to "class a is predicted, class B is the second possible class". Once P has been trained following this loss function, the prediction given by M P corresponds to "class is predicted, the higher confidence given to class a, the smaller confidence given to class 3". Concepts learned by M and M o share the same goal of prediction, i.e. "class a" predicted, but the relation between class and class 3 is forced to be the most different as possible. As learned concepts are essentially different, then useful features learned by M and M P to reach these concepts necessarily different, and consequently display different types of sensitivity to
3

---

Under review as a conference paper at ICLR 2021
the same imput pixel modifications. In other words, the direction of confidence towards classes is forced to be structurally different for M o P and M, we hypothesize that useful features of the two classifiers should be different and behave differently to adversarial perturbations.
The luring loss, designed to induce this behavior, is given in Equation 2 and the complete training procedure detailed in Algorithm [ The parameters of P are denoted by 0, and M is the target model. M has already been trained and its parameters are frozen during the process. It" (a:) and Mo (x) denote respectively the logits of M and M o P for input x. h4 (x)
X is input
and h Mof (x) correspond respectively to the values of h (x) and h MoM
(xx) for class i. The classes and b correspond to the second maximum value of MM and h MoP respectively.
M(x) = ((H (a) 1
(x)) max (0, lib Mo (x)) hMM()) (1))
The first term of Equation optimizes the gap between the logits of M P corresponding to the first and second biggest unscaled confidence score (logits) given by M (i.e. M(x) and a). This part formalizes the goal of changing the direction of confidence between M P and M. The second is compulsory to reach a good classification since the first part alone does not ensure that h MoP (Mw) (x)
is the highest logit value (prediction neutrality). The parameter À > O, called the luring coefficient, allows to control the trade-off between ensuring good accuracy and shifting confidence direction.
Algorithm Training of the luring component
Input: trained model M, training steps K, learning rate 77, batch size B, luring coefficient À Output: luring component P. with parameters 0
h. MoM)P(x) denotes the logits of M P for input x and class
(x)
2: Randomly initialize the parameters 0 of P
3: for step = 1... ..... do
4: {x1, 22, .2.......... ... .B batch of training set examples
5: for m-ri1 1... B: do
6: (a,b) + (class of the 2nd max value of h M (I;), class of the 2nd max value of h MoP(P (z:)) MoP
[(r,, ((x)) t- - X(h M(2 MoP (xx)) + max(0, Mo (x:) h M(. (x:))
MoP
7:
8: end for
9: 8-8 - E 1=1 VoL(E,
end for
11: return P
3 CHARACTERIZATION OF THE LURING EFFECT
3.1 OBJECTIVE
Our first experiments are dedicated to the characterization of the luring effect by (I) evaluating objectives in term of transferability and (2) isolating it from other factors that may influence transferability. For that purpose, we compare our approach (Luring) to the following approaches, which differ from ours in the training procedures and loss functions:
Stack model: M P is retrained as a whole with the cross-entropy loss. Stack serves as first baseline measure the transferability between the two architectures M P and M.
Auto model: P is auto-encoder trained separately with binary cross-entropy loss. Auto serves as a second baseline of a component resulting in a neutral mapping from Rd to Rd.
C.E model: P is trained with the cross-entropy loss between the confidence score vec- tors M P(x) and (xx) in order to mimic the decision of the target model M. This model serves as a comparison between our loss and a loss function which does not aim at maximizing the gap between the confidence scores.
We perform experiments on MNIST (Lecun et al. 1998). SVHN Netzer et al. 2011) and CI- FAR10 (Krizhevsky) 2009). For MNIST, M has the same architecture in Madry al. (018).
4

---

Under review as a conference paper at ICLR 2021
MNIST
FGSM
PGD
MIM
MIML2
DROX LARGX
0.
0.3
6.3
0.4
02
23
radiue
0.4
02
ball
2.
0.2
0.5
0.4
SVHN
FGSM
PGD
MIM
MIM12
DRIX TAROX X
Adto
ICE
Luring
0.2
022
0.2
radiue of ball
CIFAR10
PGSM
PGD
MIM
MIML2
DR(XO, TAR(X)
1.0
3.0
1.0
04
0.02
0.0
0.04
0.02
0.03
0.04
0.02
0.03
0.09
0.02
3.53
0.01
rradius of lac
ball
Figure 2: Disagreement Rate (solid line) and Inefficient Adversarial examples Rate (dashed line) for different attacks.
For SVHN and CIFARIO, we follow an architecture inspired from VGG (Simonyan & Zisserman 2013). Architectures and training setup for M and P are detailed in Appendix A and B Table in Appendix C gathers the test set accuracy and agreement rate (on the ground-truth label) between each augmented model and M. We observe that approach has limited impact on the test accuracy with a relative decrease of 1.71%, 4.26% and 4.48% for MNIST, SVHN and CIFARIO respectively.
3.2 ATTACK SETUP AND METRICS
For the characterization of the luring effect, we attack the model M o P the four approaches and transfer to M only the adversarial examples that are successful for these four models. We define the disagreement rate, noted D R (X'), that represents the rate of successful adversarial examples crafted on M P for which M and M P do not agree. To measure the best
where the luring effect leads to unsuccessful adversarial examples when transferred to M, we note I TAR(X') an inefficient adversarial examples rate that represents the proportion of successful adversarial examples MoP but not on M. For both metrics (see Equation 3). the higher, the better the luring effect to limit transferable attacks on the target model M.
DR(X') = Ex Masollleee LAR(X') Ex IMOP(r))uuMM
(3)
Ex IMOP(H)FT
Ex IMOP(E))
We the gradient-hee based attacks FGSM (Goodfellow et al. 2015). PGD (Madry et al. 2018). and MIM (Dong et al.) (2018) in its loo (MIM) and l2 (MIML2) versions. We used three loo perturbation budgets values): 0.2, 0.3 and 0.4 for MNIST, 0.03, 0.06 and 0.08 for SVHN, and 0.02, 0.03 and 0.04 for CIFAR10. We detail the parameters used to run these attacks in Appendix D.I
3.3 RESULTS AND COMPLEMENTARY ANALYSIS
We report results for the DR and IAR in Figure The highest performances reached with loss (for every e) show that our training method is efficient at inducing the luring effect. More precisely, claim that the fact that both metrics decrease much slower increases compared to the other
5

---

Under review as a conference paper at ICLR 2021
Auto C_E. Luring
Freganny
Broquaacy
Luning
distortion
distection
Auto
C.E
Laring
adversarial distortion for MNIST (left) and CIFARIO (middle). Saliency maps for MNIST (right): (top) clean image and gradient of the cross-entropy loss with respect to input; (bot- tom) mapping gradients P(x)) for augmented models.
Figure 3:
architectures, brings additional confirmation that non-robust features of M and M o tend behave more differently than with the three other considered approaches.
As complementary analysis, we investigate the magnitude of the adversarial perturbations. Results show that approach leads to similar 4o0 and l2 distortion scales than the other methods. This indicates that our approach truly impacts the underlying useful features and does not only imply different distortions scales needed to fool both M and M P. The complete analysis is presented in Appendix D.2
For MNIST, we notice that the in distortion (i.e. the cardinality of the pixels impacted by the adver- sarial attack) is significantly higher with our method, as presented in Figure 3 (left). This points out that P leverages more different useful features, which are easily identifiable for MNIST because of the basic structure of the images. This effect can also be demonstrated thanks to saliency maps (i.e. analyzing Pxx())), as illustrated in Figure 3 (right): for the Luring model, P is sensitive to many additional useless background pixels compared to M. Note that for Auto, modifying P's mapping consists almost exclusively on modifying pixels correctly correlated with the true label.
For more complex inputs from SVHN or CIFAR1O, without uniform background as for MNIST, we observe that the lo distortion is approximately the same for all architectures (Figure 3 middle). However, for these two data sets, we analyze the influence of P's mapping thanks to the logits variations. The detailed analysis in presented in Appendix D.3] We find that logits between M and M P vary more differently with respect to input modifications when P is trained with our approach, which confirms that our luring loss enables to reach Adversarial Luring.
USING THE LURING EFFECT AS A DEFENSE
By fooling an adversary on the way to target a black-box model, the phenomenon that we character- ized in the previous section can be seen defense mechanism its state-of-the-art approaches. In this section we discuss the way to use our approach as protection.
complement of
4.1 THREAT MODEL
Attacker Our work sets in the black-box paradigm, i.e. we assume that the adversary has no access to the inner parameters and architecture neither of the target model M nor the augmented model M This setting is classical when dealing with cloud-based API or edge neural networks in secure embedded systems where users have only access to input/output information with different querying abilities and output precision (i.c. scores or labels). More precisely:
The adversary goal is to craft (untargeted) adversarial examples on the model he has access to, i.e. the augmented model T = M P. and rely transferability in order to fool M.
The adversarial knowledge corresponds to an adversary having a black-box box access to a ML system SA containing the protected model T M P. while M stays completely hidden. He can query T (without any limit on the number of queries) and we assume that he is able get the logits outputs. Moreover, for even more strict evaluation, also consider

---

Under review as a conference paper at ICLR 2021
stronger adversaries allowed to use SOTA transferability designed gradient- based methods to attack Tsee 4.2).
Classically, the adversarial capability an upper bound с of the perturbation ||~ - xllocc
Defender For each query, the defender has access to two outputs: (x) and M P(x). Therefore, two different inference schemes (illustrated in Appendix B are viable according the goals and characteristics of the system to defend, noted Sp:
(detection scenario) p and 1 are the same, the goal of the defender take advantage of the luring effect to detect an adversarial example by comparing the two inference outputs (x) and M P(r)). appropriate metric is the rate of adversarial examples which are either detected well-predicted by M, expressed in Equation 4 and noted DAC for Detection Adversarial Accuracy.
DAC = EnEx' I M(r").
(4)
X"
(transferability scenario) If S is a secure black-box system with limited access, that only contains the target model M, the goal of the defender is to thwart attacks crafted from S.. For this specific scenario, the defender may only rely on the fact that the luring effect likely leads to unsuccessful adversarial examples (M(x') y). The appropriate metric is the classical adversarial accuracy (AC), which is the standard accuracy measured on the adversarial set X' and noted ACMpP and ACM respectively for M Pndd M.
Related real-world transferability scenarios may be linked to the increasingly widespread deploy- ment of models in a large variety of devices (edge A/) or services (cloud A/) as an integral part of complex systems with different security requirements. For example, it is classically the case with the IoT domain where a model may be the part of a critical system and, simultaneously, be deployed and used in more mainstream connected objects with looser security requirements. The model M is deployed for the critical system with strong security access, not accessible to an attacker, and the augmented protected model will be deployed in the others systems so that attacks that may be crafted more easily will not transfer to M.
4.2 ATTACKS
In order to evaluate our defense with respect to the threat model, we attack M o P with strong gradient-free attacks:
Table 1: Adversarial accuracy for M (ACMop), M (ACM), and Detection Adversarial Accuracy (DAC) for different architectures. SVHN (top), CIFARIO (down).
SVHN
STACK
AUTO
C.F
LURING
M.P
AC
DAC
ACMrP
AC
AC
SPSA
0.03
0.10
0.54
0.56
0.06
0.37
0.38
0.06
0.67
0.68
0.0
0.96
0.97
0.06
0.01
0.21
0.24
0.0
0.11
0.96
0.96
0.08
0.0
0.13
0.15
0.0
0.06
0.06
0.0
0.23
0.28
0.0
0.94
0.96
ECO
0.03
0.06
0.42
0.44
0.14
0.49
0.97
0.98
0.06
0.0
0.11
0.12
0.06
0.09
0.11
0.1
0.35
0.39
0.1
0.86
0.88
0.08
0.0
0.03
0.07
0.06
0.09
0.09
0.08
0.29
0.32
0.09
0.84
0.56
MIM-W
0.03
0.04
0.32
0.35
0.01
0.20
0.21
0.03
0.41
0.45
0.11
0.81
0.87
0.06
0.0
0,06
0.09
0.0
0.03
0.05
0.0
0.10
0.18
0.58
0.71
0.08
0.0
0.03
0.06
0.0
0.01
0.02
0.0
0.06
0.13
0.48
0.67
CIFARTO
STACK
C.E
LURING
ACM
AC
ACMsP
AC
DAC
MaP
DAC
SPSA
0.02
0.01
0.75
0.78
0.06
0.68
0.71
0.12
0.78
0.82
0.03
0.0
0.57
0.62
0.01
0.45
0.49
0.02
0.59
0.64
0.38
ECO
0.02
0.15
0.7
0.71
0.14
0.6
0.63
0.16
0.81
0.87
0.03
0.09
0.44
0.59
0.09
0.36
0.42
0.2
0.65
0.67
0.05
0.29
0.04
0.34
0.09
0.35
MIM-W
0.0
0.52
0.43
0.03
0,58
0.64
0.03
0.0
0.24
0.28
0.0
0.15
0.19
0.0
0,30
0.39
0.04
0.0
0.13
0.16
0.0
0.05
0.1
0.0
0.18

---

Here best is from the adversary point of view. Under review as a conference paper at ICLR 2021
SPSA attack (Uesato et al. 2018 ensures to consider the strongest adversary in our black- box setting as it allows the adversary to get the logits of M P.
Coherently with an adversary that has no querying limitation, ECO (Moon el al. 2019) is a strong acore-based gradient-estimation free attack.
To perform an even more strict evaluation, and to anticipate future gradient-free attacks, we report the best results" obtained with the state-of-the-ann transferability designed gradient-bast attacks MIM. DIM, MIM-TI and DIM-ti (Dong et al. [2018] Xie et al., (0188) Dong et al. (2019), under the name MIM-W. The parameters used to run these attacks are presented Appendix
4.3 RESULTS
The results presented in Table I for SVHN and CIFARIO (results for MNIST are close to SVHN and are presented in Appendix G. For CIFARIO, since no autoencoder we tried allows to reach a correct test set accuracy for the Auto approach, we do not consider it. First, notice that ACmor almost always equal O with gradient-base iterative attacks This is an indication that P does not induce form of gradient masking (Athalye et al. 2018). Remarkably on SVHN, for versarial examples tuned to achieve the best transferability only reduce ACM to 0.48 against our approach, compared to almost O for the other architectures. The robustness benefits are more ob- servable on SVHN and MNIST but the results on CIFARIO are particularly promising in the scope of defense scheme that only requires pre-trained model. Indeed, for the common loo perturbation value of 0.03, the worst DAC value for the CLE and Luring approaches are respectively 0.19 (against DIM attack) and 0.39 (against DIM-TI attack).
0.08 (the largest perturbation), and for the worst-case attacks, ad-
4.4 COMPATIBILITY WITH IMAGENET AND ADVERSARIAL TRAINING
We scale our approach to ImageNet (ILSVRC2012). For our experiments, the target model M con- sists of a MobileNetV2 model (Sandler al., 2018), reaching 71.3% of accuracy on the validation set. For a common Lo perturbation budget of 4/255, the smaller ACM and DAC' observed against the strong MIM-W attack with our approach equal 0.4 and 0.55, while they equal 0.23 and 0.35 with the C_E approach. Following the results previously observed on the benchmarks used for charac- terization, these results validate the scalability of our approach large-scale data sets. Details on the ImageNet experiments and results are presented in Appendix H.I
Even if very few efforts tackles the issue of thwarting adversarial perturbations transferred from a source model to target model (transfer black-box attacks), there exists numerous detection and defense schemes intended to protect a target model in a white-box or gray-box setting against ad- versarial perturbations. As the luring effect is conceptually new, it can be combined to any of these existing methods, to provide even more protection. Indeed, our deceiving- biaiiiiing--eedddd approach acts on the way M P performs inference relatively to M. As the effort is focused on the design of P, another defense method, this time focusing on M and designed to protect M, can then be combined with our scheme. We choose here consider the combination with adversarial training (Madry al. 2018), a state-of-the-art approach for robustness in the white-box setting. The model M/ is already trained with adversarial training. Interestingly, we note that the joint use of these defenses clearly improves the detection performance, with DAC values superior to 0.8 for the three data sets as well a strong improvement of the ACM metric for MNIST (0.97) and CIFARIO 10 (0.85). The detailed experiments are presented in Appendix H.2
5 DISCUSSION AND RELATED WORK
A good practice in the field of adversarial robustness is to consider an adaptive adversary (Carlini jet al. 2019): an attacker using some knowledge of the defense method to bypass more efficiently. In our case, the only information that could be given to an adversary without breaking the black-box
Parameters tuned relatively to the adversarial goal: lowering our defense efficiency. Thus, the lowest nerformances sometimes do correspond to the lowest accuracy of M P on the adversarial examples.
8

---

Under review as a conference paper at ICLR 2021
threat model is the supplementary knowledge that the augmented model has been trained with the luring loss. Indeed, M has to stay hidden, otherwise the adversary could use a decision based attack to thwart the defense (Brendel et al. 2018 Cheng al. 2020). Moreover, with more permissive access to T. for example a white-box access to 7, the adversary could simply extract M from the augmented model, and then defeat the defense. The additional information of the luring loss, would not bring any advantage to the adversary in order to craft adversarial perturbations on '' which transfer M. Indeed, would imply for the attacker inverse the effect of the luring loss on the prediction output vector of T, which appears as a prohibitive effort. Therefore, respecting the threat model, an adaptive adversary would not be more efficient than the adversary considered throughout this work.
Our characterization and first results pave the way for a possible extension of the luring effect to more permissive threat models. For now, in a white-box setting, the defense scheme would be defeated since the adversary could use a gradient- solution to exploit the luring effect in gray setting would be to cause the luring effect between two models M and M' with different architectures. A model M' trained with the luring loss would be publicly released. The adversary could then be allowed to mount gradient- based attacks M' without being able to access M. The possible efficiency of such future work is backed by results on the MIM-W attacks, which corresponds to T being attacked by gradient-based attacks.
attack to fool both M and M P. Actually, the better
Defenses in the black-box context are weakly covered the literature as compared to the numer- approaches focused white-box attacks. More particularly, very few approaches deal with query-based black-box attacks, such as the recent BlackLight (Li et al., 2020). To our knowledge, our work is the first exploit the idea of luring an adversary taking advantage of the transferabil- ity property. However, a honeypot-based approach has been recently proposed (Shan et al., 2019) with a trapdoored model detect adversarial examples. Even if the threat model (white-box) is different from ours, this approach is conceptually close to our deception approach, and it is inter- estingly mentioned that transferability between the target and the hapdoor-protectee model is very poor. By fitting the trapdoor approach in our black-box threat model we highlight complementary performances in terms of AC's and DAC metrics, meaning that an overall deception-based strategy a promising direction for future work. The detailed analysis is presented in Appendix
6 CONCLUSION
We propose conceptually innovative approach to improve the robustness of model against trans- fer box adversarial perturbations, which basically relies on a deception strategy. Inspired by the notion of robust and non-robust features, derive and characterize the luring effect, which is implemented via decoy network built upon the target model, and a loss designed to fool the adver- sary into targeting different non-robust features than the ones of the target model. Importantly, this approach only relies on the logits of target model, does not require a labeled data set and therefore can by applied to any pre-trained model. We show that our approach can be used as a defense and that a defender may exploit two predic- tion schemes to detect adversarial examples or enhance the adversarial robustness. Experiments on MNIST, SVHN, CIFAR1O 10 and ImageNet demonstrate that exploiting the luring effect enables to successfully thwart an adversary using state-of the-ra optimized attacks even with large adversarial perturbations.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 2018.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference Learning Representations, 2018
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 190..0679 2019.
Yair Carmon, Aditi Raghunathan. Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, 2019.
Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. Hopskipjumpattack: A query-efficient decision- based attack. 2020 IEEE Symposium on Security and Privacy (SP), 2020.
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: elastic-net net attacks to deep neural networks via adversarial examples. In The Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial at- tacks. arXiv preprint arXiv: 1907.05587, 2019.
Minhao Cheng, Simranjit Singh, Patrick Chen, Pin-Yo Yu Chen, Sijia Liu, and Cho-Jui Hsieh. Sign- opt: query-efficient hard-label adversarial attack. In International Conference on Learning Representations, 2020.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. Proceedings of the 36th International Conference on Machine Learning. ICML 2019, pp. 1310-1323 1320, 2019.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp- 248 255, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by iosslatioo-invarialn attacks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Nicolas Ford, Justin Gilmer, and Ekin D. Cubuk. Adversarial examples are natural consequence of test error in noise. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, pp. 2280-2289, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations. Wiley Online Library, 2015.
Chuan Guo, Jacob R. Gardner, Yurong You, Andrew G. Wilson, and Kilian Q Weinberger. Simple black-box adversarial attacks. In Proceedings of the 36th International Conference on Machine Learming, ICML 2019, 2019.
Dan Hendrycks. Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 2019.
10

---

Under review as a conference paper at ICLR 2021
Uiwon Hwang, Jaewoo Park, Hyemi Jang, Sungroh Yoon, and Nam Ik Cho. Puvae: A variational autoencoder to purify adversarial examples. IEEE Access, 11126582-126599, 2019.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, 2018.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, (6(11))2278-22) 2324, 1998.
Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y. Zhao. Black- light: Defending black-box adversarial attacks on deep neural networks. arXiv preprint arXiv:2006. 14042, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Dongyu Meng and Hao Chen. Magnet: two- two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147, ACM, 2017.
Seungyong Moon, Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via efficient combinatorial optimization. In Proceedings of the 36th International Conference on Machine Learning, pp. 6636-4645 2019.
Yuval Netzer, Tiejie Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference Computer and Communications Security, pp.
ACM, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations, 2019.
Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y. Zhao. Using honeypots to catch adversarial attacks on neural networks. In Proceedings of ACM Conference on Computer and Communications Security (CCS), 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
11

---

Under review as a conference paper at ICLR 2021
Jonathan Uesato, Brendan O'Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference Machine Learning. ICML 2018, pp. 5025-5034, 2018.
Shiqi Wang, Yizheng Chen, Ahmed Abdou, and Suman Jana. Enhancing gradient- attacks with symbolic intervals. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 2019.
Cihang Xie, Zhishuai Zhang, Jianyu Wang, Yuyin Zhou, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and aceuracy. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 2019.
12

---

Under review as a conference paper at ICLR 2021
A SETUP FOR BASE CLASSIFIERS
The architectures of the models trained on MNIST, SVHN and CIFARIO are detailed respectively in tables 2 3 and 4 BN, MaxPool(u.v),, UpSampling(u.v) and Conv((.k.k) denote respectively batch normalization, max pooling with window size (u,v), upsampling with sampling factor (u.v) and 2D convolution with filters and kernel of size (k.k).
For MNIST, we used 5 epochs, a batch size of 28 and the Adam optimizer with learning rate of 0.01. For SVHN, we used 50 epochs. a batch size of 28 and the Adam optimizer with a learning rate of 0.01. For CIFARIO. 10, we used 200 epochs, a batch size of 32, and the Adam optimizer with piecewise learning rate of 0.1, 0.01 and 0.001 after respectively 80 and 120 epochs.
Table 2: MNIST base classifier architecture. Epochs: Batch size: 28. Optimizer: Adam with learning rate 0.01.
ARCHITECTURE CoNY(32,5,3) RELU MAXPOOL(2.2) CoNv(64,5,5) RELU MAXPOOL(2.22 DENSE(1024) + RELU
DENSB(I0) + SOFTMAX
Table 3: SVHN base classifier architecture. Epochs: 5. Batch size: 28. Optimizer: Adam with learning rate 0.01.
ARCHITECTURE CoNv(64,3,3) BN RELU CoNv(64,3.33) MAXPOOL. BN + RELU
CONV(128,3,3) + BN + RELU
CoNv(128,3.3) MAXPOOL(2,2) + BN RELU
CONv(256.3.3) BN RELU
CoNv(256,3,3) MAXPO01,2222) + BN DENSE( 1024) + RELU
DENSE( 1024) BN + RELU
DENSE(1O) 10) + SOFTMAX
Table 4: CIFARIO base classifier architecture. Epochs: 200. Batch size: 32, Optimizer: Adam with learning rate starting at 0.1, decreasing 0.01 and 0.001 respectively after 80 and 120 epochs.
ARCHITECTURE CONv(128,3,3) BN RELU
CoNv(128,3,3) MAXPOOL + BN
CoNv(256.3.3) RELU CoNv(256,3,3) + MAXPOOL(2,2) + BN + RELU
CoNv(512.3.3) BN RELU
CoNv(512.3.3) MAXPOOL(2.2) + BN RELU
DENSE(1024) + RELU
DENSE( 1024) BN + RELU DENSE( 10) + SOFTMAX
B TRAINING SETUP FOR DEFENSE COMPONENTS
The architectures for the defense component P MNIST, SVHN and CIFARIO are detailed respec- tively in Tables 6 and 7 BN, Mar (ool(u, v), U UpSampling(au v) and Cono k, k) denote respectively batch normalization, max pooling with window size (u, v), upsampling with sampling factor (u, 2 and 2D convolution with f filters and kernel of size (k, k). The detailed parameters used perform training are also reported.
13

---

Under review as a conference paper at ICLR 2021
Table 5: Defense component architecture for MNIST.
ARCHITECTURE
CoNv(16,3,3) RELU MAXPOOL(2,2)
CONv(8.3,3) + RELU
MAXPOOL(2,2)
CONV(8,3,3) + RELU
CONV(8,3,3) RELU
(PSAMPLING(2.2))
CONV(8,3,3) + RELU
UPSAMPLING(2.22
CONv(16.3.3)
UPSAMPLING(2.22) CONv(1,3,3) SIGMOID
MNIST:
Stack Epochs: 5. Batch size: 28. Optimizer: Adam with learning rate 0.001
Auto Epochs: 50. Batch size: 128. Optimizer: Adam with learning rate 0.001
C.E Epochs: 64. Batch size: 64. Optimizer: Adam with learning rate starting at 0.001, decreasing to 0.0002 and 0.0004 respectively after 45 and 58 epochs
Luring Epochs: 64. Batch size: 64. Optimizer: Adam with learning rate starting at 0.001, decreasing to 0.0002 and 0.0004 respectively after 45 and 58 epochs
Table 6: Defense component architecture for SVHN.
ARCHITECTURE CoNv(128,3.3) + BN + RELU
MAXPOOL(2.2) CONV(256,3.3) BN RELU
MAXPOOL(2.2) CoNV(512,3,3) + RELU
CONv(1024,3,3) BN + RELU
(2X))
CONv(512,3,3) BN RELU
CONv(512.3,3) RELU
UPSAMPLING(2.22
CONV(256.3,3) BN RELU
UPSAMPLING(222) CONV(128.3,3) BN RELU
(PSAMPLING(2.22 ONV(64,3,3) + BN RELU
CoNv(3,3.3) + BN SIGMOID
SVHN:
Stack Epochs: 20. Batch size: 256. Optimizer: Adam with learning rate 0.001
Auto Epochs: 5. Batch size: 128. Optimizer: Adam with learning rate 0.001
C Epochs: 210. Batch size: 256. Optimizer: Adam with learning rate starting at 0.0001, decreasing to 0.00001 and 0.000008 respectively after 126 and 168 epochs
Luring Epochs: 210. Batch size: 256. Optimizer: Adam with learning rate starting
at 0.0001, decreasing to 0.00001 and 0.000008 respectively after 126 and 168 epochs. Dropout is used
14

---

Under review as a conference paper at ICLR 2021
Table Defense component architecture for CIFARIO.
ARCHITECTURE CoNv(128,3.3) BN RELU
CoNv(256,3.3) BN RELU
MAXPOOL((2.2) CoNv(512.3,3) + BN + RELU
C3Nv(1024.3,3) BN + RELU
CONv(512.3,3) BN RELU
C0Nv(512,3,33 BN RELU
CoNv(255,,,33 + + RELU
UPSAMPLING(2.2) CONv(128,3.3) BN RELU
C0,v(64.3.3) BN RELU
CONY(3,3.3) + BN SIGMOID
CIFAR1O:
Stack Epochs: 200. Batch size: 32. Optimizer: Adam with learning rate starting at 0.1, decreasing to 0.01 and 0.001 respectively after 80 and 120 epochs
C.E Epochs: 216. Batch size: 256. Optimizer: Adam with learning rate starting at 0.00001,
decreasing to 0.000005 and 0.0000008 respectively after 154 and 185 epochs. Dropout is used
Luring Epochs: 216. Batch size: 256. Optimizer: Adam with learning rate starting
0.00001, decreasing to 0.000005 and 0.0000008 respectively after 154 and 185 epochs. Dropout is used
C TEST SET ACCURACY AND AGREEMENT RATES
For the luring parameter, we set = 1 for MNIST and SVHN, and À = 0.15 for CIFARIO. For CIFARIO, since autoencoder tried reaches correct set accuracy, we exclude the Auto model.
Table 8: Test set accuracy and agreement (augmented and target model agree on the ground-truth label) between each augmented model and the target model M, noted BASE.
MODEL
MNIST
SVHN
CIFARTO
TEST
AGREE
TEST
AGREE
TEST
AGREE
BASE
0.991
0.961
0.893
STACK
0.98
0.976
0.925
0.913
0.902
0.842
AUTO
0.971
0.969
0.95
0.943
C.
0.982
0.977
0.919
0.907
0.860
0.834
LURING
0.974
0.969
0.920
0.917
0.853
0.822
D LURING EFFECT ANALYSIS
D.1 ATTACK PARAMETERS
For MIML2, we report results when adversarial examples are clipped to respect the threat model with regards to An illustration of a clean image and its adversarial counterpart for the maximum perturbation allowed presented in Figure 4 We note that the ground-truth label still clearly recognizable. For PGD, MIM and MIML2, the number of iterations is set to 1000, the step size to 0.01 and to 1.0 (MIM and MIMI2). For MIMI.2, the 1z bound is set to 30 on MNIST and on SVHN and CIFAR10.
D.2 ANALYSTS OF THE l2 AND 100 DISTORTIONS
We investigate the impact our additional component on the distance between clean and an adversarial example, compared the other approaches. We measure the l means of the adversarial
15

---

Under review as a conference paper at ICLR 2021
4
4
Figure 4: (top) Clean image, (bottom) adversarial example for the maximum perturbation allowed (left to right: 0.4, 0.08, 0.04).
perturbations needed find an adversarial example, and verify that our approach is globally the same level than the other approaches.
For that purpose, an appropriate and recommended attack the Carlini Wagner l2 attack (here- after, CWL2) (Carlimi & Wagner (2117) that searches for the closest adversarial examples in terms of the le distortion. We run CWL2 with the following parameters: 10 binary search steps, learning rate of 0.1, initial constant of 0.5 and 500 iterations for MNIST, SVHN and CIFAR1O. These pa- rameters have been chosen such that increasing the number of iterations does not lower the final l2 distortion. For each data set, we run the attack on 1, 000 test set examples correctly classified for the four approaches for both M and M o P.
Table 9: Average /2 and l.c distortions between clean and adversarial examples generated with the CWL2 attack for the target model M and the augmented models M o P
MNIST
SVHN
CI
MODEL
MEAN l
MEAN Lo0
MEAN l
MEAN Too
MEAN
MEAN Tc
TARGET
0.96
0.27
0.32
0.04
0.42
0.04
STACK
0.87
0.25
0.34
0.05
0.4
0.03
AUTO
0.94
0.35
0.29
0.04
C.E
0.92
0.34
0.29
0.04
0.41
0.04
LURING 0.94
0.31
0.31
0.04
0.39
0.04
The average la and loo distortions are reported in Table 9 and show that there is no significant dif- ference between our approach and the other approaches. This is an additional observation that our luring loss allows train an augmented model which causes the luring effect by predominantly targeting different useful non-robust features rather than artificially modifying the scale of the ad- versarial distortion needed to cause misclassification.
D.3 LOGITS ANALYSIS
As complementary analysis the impact of the mapping induced by the additional compoment P. we analyse how P acts on the variation of the logits with respect to the input features (V {(x)) For that purpose, we first pick 1000 test set examples correctly classified by all the models (the base model M and the augmented models of the four approaches: Auto, Stack, C_E and Luring). Then, for each augmented model and each class c, we compute the number Q. of pixels Fa that lead to opposite effect on prediction between M and M P (see Equation 5):
fd:
dee
dePe
< of
Ω -
(5)
dra
Ora
Next, look at the proportion of these examples for which Ω. is higher for the Luring approach than for the other approaches. Results for each class are presented Table I0, For both SVHN and CIFAR10 10, these proportions are strictly higher than 80% whatever the class. This is supplementary
16

---

Under review as a conference paper at ICLR 2021
indication toward the fact that with respect the allowed adversarial perturbation altering x approach more relevant than other approaches (that also perform a mapping within X) to flpp useful and non-robust features of M P and M towards different labels.
Table 10:
Rate of examples (over 1000) for which logits vary more differently between M and M P. for the Luring approach against the other approaches.
2
8
10
SVHN
0.81
0.85
0.86
0.89
0.86
0.82
0.83
0.84
0.83
0.82
CIFAR 0.86
0.88
0.88
0.87
0.88
0.87
0.89
0.86
0.88
0.85
E THREAT MODELS ILLUSTRATION
SA
SD
detection stenario
LURING MODEL T-MoP
P
M
&
cut
+
M : X' -
ostrich
cat or dog
x x ostrich
transferabilityy scenario
LURING MODEL
T-MoP
TARGET
cat
cat osstric x
M
M
cat
P:X-A
Ms X
M : X
SA A
SD
ostrich
transfer
Figure 5: Illustration of the black-box threat model with two scenarios. (top) In the inner transfer- ability scenario the system to attack and to defend is the same. Because the luring effect induces different behavior when facing adversarial perturbation, the defender is able detect adversarial ex- amples by comparing M (x) and Tz)). (bottom) In the distant transferabilityy scenario, the system defend may suffer from transferable adversarial examples but the defender takes advantage of the weak transferability between T and M.
F ATTACK PARAMETERS
All the attacks (SPSA. ECO, MIM, DIM, MIM-TL, DIM-TI) performed 1000 correctly clas- sified test set examples, with parameter values tuned for the highest transferability results. More precisely, searched for parameters leading to the lowest ACM and DAC values.
17

---

Under review as a conference paper at ICLR 2021
F.1 GRADIENT-FREE ATTACKS
For the SPSA attack, the learning rate is set to 0.1, the number of iterations set to 100 and the batch size to 128. For the ECO attack, for the three data the number of queries is set to 20, 000. We did not perform early-stopping as results in less transferable adversarial examples. The block size is set to 2, 8 and respectively for MNIST, SVHN and CIFARIO.
F.2 GRADIENT-BASED ATTACKS
For DIM and DIM-TI, the optimal p value was searched in {0.1. (0.1.0.2, 1.03. We obtained the optimal values 1.0, 1.0 and 0.6 respectively on MNIST, SVHN and CIFARIO. For MIM and its variants, the number of iterations is set to 1000, 500 and 100 respectively CIFARIO, and = 1.0. The optimal kernel size required by MIM-TI and DIM-TI was searched in ((3, 3), (5,5), (10, 10), (15, 15)]- For the MIM-TI and DIM-TI, the size of the kernel resulting in the lowest M and DAC values reported in Section 4 are presented in tables II respectively for MNIST, SVHN and CIFARIO.
MNIST, SVHN and
12 and 13
Table 11: MNIST. Kernel size for the MIM-TI and DIM-TI attacks.
ATTACK
ARCHITECTURE
VALUE
0.3
0.4
0.5
MIM-TI STACK
5
5 5
5 x 5
AUTO
10 10
5
5
CLE
10 10
10
5
LURING
5
5
10
DIM-TI STACK
5
5
5
AUTO
5
5
x
C.E
10
10 10
10 10
LURING
5
5
5
Table 12: SVHN. Kernel size for the MIM-TI and DIM-TI attacks.
ATTACK
ARCHITECTURE
VALUE
0.03
0.06
0.08
MIM-TI
STACK
5
5 5
5 x
AUTO
5 5
5
5 x
CLE
5
10 10
5
LURING
10
10
10
DIM-TI
STACK
5
5
5
AUTO
5
5
5 5
C.E
5
5X5
5 X
LURING
10
10
x
Table 13: CIFARIO. Kernel size for the MIM-TI and DIM-TI attacks.
ATTACK
ARCHITECTURE
VALUE
0.02
0.03
0.04
MIM-TI
STACK
3
3 3x3
3 x
AUTO
3x
3 x
3
C.E
3x
10 10
3
LURING
3x3
3
3x3
DIM-TI STACK
3x3
3
3x3
AUTO
3x
3
3x
C.E
3x
3
3x
LURING
3x3
3x3
3x3
18

---

Under review as a conference paper at ICLR 2021
G RESULT FOR MNIST
Table 14: MNIST. ACMOP, M and DAC for different source model architectures.
STACK
ACTO
CEE
LURING
ACMo
ACa
DAC
ACMer
AC3
DAC
ACatr
ACa
DAC
ACMor
AC
DAC
SPSA
0.0
0.96
0.97
0.03
0.95
0.95
0.0
0.97
0.98
0.14
0.99
0.99
0.3
0.0
0.86
0.89
0.03
0.90
0.92
0.0
0.94
0.95
0.05
0.96
0.97
0.0
0.72
0.77
0.85
0.87
0.02
0.96
ECO
0.2
0.03
0.86
0.92
0.03
0.86
0.88
0.01
0.91
0.93
0.05
0.99
1.0
0.02
0.56
0.65
0.03
0.68
0.7
0.8
0.02
0.96
0.4
0.01
0.35
0.54
0.03
0.36
0.46
0.01
0.45
0.48
0.03
0.77
0.79
MIM
0.0
0.79
0.82
0.0
0.81
0.82
0.85
0.88
0.19
0.92
0.93
0.0
0.31
0.45
0.0
0.35
0.45
0.0
0.43
0.57
0.13
0.69
0.75
0.4
0.0
0.07
0.24
0.0
0.07
0.17
0.0
0.13
0.31
0.07
0.34
0.45
H IMAGENET AND ADVERSARIAL TRAINING
H.1 RESULTS ON IMAGENET
The ImageNet data set DDng et al. 2009) is composed of color images, with 1.2 million images and 50, 000 samples in the training and validation set respectively. For experiments, the target model M consists of MobileNetV2 model (Sandler et al. 2018), taking inputs of size 224 x 224 x 3 scales between 11 and The architecture for the defense component described in Table 15 We choose to present our approach along with the C_E approach as it represents well the purpose of the method we present in this paper. Indeed, the CLE approach only requires training the additional component P. which fits into the scope of a defense which can be implemented on top of any already trained model M.
Table 15: Defense component architecture for ImageNet.
ARCHITECTURE CONv(128,3,3) + + RELU
(2XPO01(2,2)
CONV(256.3.3) BN RELU
MAXPOOL(2.2) + RELU
CoNv(512.3.3) + (2XPO01(2222)
CoNv(512,3,3) + BN + RELU
CONv(256,3,3) BN RELU
UPSAMPLING(2.2) CONv(128,3,3) + RELU UPSAMPLING(2.2) CoNv(64,3,3) + BN RELU
CONv(3,3,3) + BN SIGMOID
For both the Luring and C_E approaches, the additional component P is trained for 100 epochs, with a bath size of 256, using momentum of 0.9 with learning rate starting at 0.01, decreasing to 0.001 after 80 epochs. The À parameter of the loss is 1.0. The accuracy of the augmented model as well as the agreement rate (the model M and the augmented model M o P) are presented in Table 16
Table 16: Test set accuracy and agreement (augmented and target model agree on the ground-truth label) between each augmented model and the target model M, noted BASE.
MODEL
TEST
AGREE
BASE C.E
0.713
0.679
0.647
LURING
0.653
0.614
19

---

Under review as a conference paper at ICLR 2021
The attacks are performed on 1, 000 well-classified images from the validation set, and two common ec perturbation budgets are considered: : 4/255, 5/255 and 6/255. Results are presented in Table The highest results observed in terms of both ACM and DAC values with our approach confirm the scalability of our method on large-scale data sets.
Table 17: ImageNet. ACMoP, ACMs and DAC for different source model architectures.
C.E.
LURING
ACMoP
ACM
DAC
ACMoP
ACM
DAC
MIM-W W
4/255
0.0
0.23
0.35
0.00
0.4
0.55
5/255
0.0
0.15
0.25
0.00
0.28
0.43
6/255
0.0
0.08
0.18
0.00
0.18
0.33
H.2 COMPATIBILITY WITH ADVERSARIAL TRAINING
We present the performances of the luring approach when the target model M is trained with adver- sarial training (Madry et al., 2018) as general state-of-the-art method for adversarial robustness. The 1s0 bound constraint for PGD is set to 0.3 for MNIST and 0.03 for both SVHN and CIFAR 10. During training, the PGD is performed for 40 steps, and step size of 0.1 for MNIST. The PGD) is performed for 10 steps with step size of 0.008 for both SVHN and CIFAR10. We report in Table 18 the accuracy and agreement (augmented and target models agree on the ground-truth label) between the target model M and its augmented version T,, whether M is trained classically with adversarial training.
In Table 19 we report the ACM and DAC values corresponding to the same threat model as the one used throughout this paper, against the strong MIM-W attack. Moroever, we add the accuracy in a white-box setting of the model AM against the PGD attack with 40 iterations and a step size of 0.01, denoted as ACM, Interestingly, we observe productive interaction between the two defenses with DAC values that have greatly increased for the three data sets, well as the ACM values for MNIST and CIFARIO.
Table 18: Test set accuracy and agreement (augmented and target model agree on the ground-truth label) between an augmented luring model and a target model M. Base and Base-Luring denote respectively a target model trained classically and the augmented model trained with the luring defense. AdvTrain and training and the augmented model trained with the luring defense.
Luring denote respectively a target model trained with adversarial
MODEL
DATA
MNIST
SVHN
CIFARIO
TEST
AGREE
TEST
AGREE
TEST
AGREB
BASE
0.991
0.961
0.893
BASE-LURING
0.97
0.97
0.92
0.92
0.8.5
0.82
ADVTRAIN
0.99
0.93
0.79
ADV TOVTIAIN- TRAIN-LURING LURING
0.96
0.96
0.87
0.85
0.78
0.76
Table 19: ACM, wb. ACM and DAC values (among FGSM, MIM, DIM, MIM-TI and DIM-TI) for the luring approach. Base-Luring and AdyTrain-luring denote the cases where the model M is trained respectively classically and with adversarial training.
LURING
TUINNN-L-URING
c
AC wab
ACM
DAC
AC
ACM
DAC
MNIST
0.3
0.0
0.69
0.75
0.82
0.97
0.98
SVHN
0.03
0.0
0.81
0.87
0.45
0.898
0.911
CIFARIO
0.03
0.0
0.30
0.39
0.41
0.85
0.88
20

---

Under review as a conference paper at ICLR 2021
1 RELATED WORK: COMBINING WITH TRAPDOORED MODELS
In Shan et al. (2019), the authors introduce trapdoors to detect white-box adversarial examples attacks in order to trick the adversary's attack into converging towards specific adversarial directions, which later enables a straightforward detection. To this end, to defend a model against targeted adversarial examples towards label yt, the clean data set of input-label pairs (a. y) is augmented with trapdoored pairs (z + A, Me), where A is a specific trigger. Then, the target model is trained by minimizing the cross-entropy between the clean and trapdoored aaput-label pairs to obtain the trapdoored model. An input is detected as targeted adversarial example thanks to threshold- based test, that compares a "neuron activation signature" of this input against that of the embedded trapdoors. Note that the authors extend their approach to untargeted adversarial examples. We refer to the article (Shan et al. 2019) for more details.
The authors mention that transferability between the target and the trapdoored model is very poor. Based on the provided codd we train trapdoored models with the architectures we used for M. We observe that the transferability is hight from . to the trapdoor version but, on the contrary, the transferability is weak when adversarial examples are crafted on the trapdoored model and trans- ferred the target model. For the latter scenario, values for FGSM, MIM, DIM, MIM-TI and DIM-TI along with those of our method. The parame- ters used to run the attacks are the same as described in Section These results would correspond to a defense similar to ours where the public model is not the augmented model M o P. but the trapdoored model trained from M. The threat model stays the same, corresponding to an adversary which wants to transfer adversarial examples from the public model to the target model M. Results Table 20)on on MNIST and CIFARIO for DAC indicate that using trapdoors could allow for a more efficient detection of adversarial examples, especially for large adversarial perturbations. Results for the AC M values on the three data sets also corroborate the idea that both methods could benefit from each other.
report in Table 20 the worst ACM and DAC
Table 20: Optimal ACMoP, ACM and DAC values (among FGSM, MIM, DIM, MIM-TI and DIM- TT) for the Trapdoor and Luring approaches.
TRAPDOOR
LURING
c
ACMoP ACM
DAC
ACMoP
ACM
DAC
MNIST
0.2
0.06
0.82
0.97
0.19
0.92 0.93
0.3
0.0
0.54
0.94
0.13
0.69
0.75
0.4
0.0
0.27
0.07
0.34
0.45
SVHN
0.03
0.1
0.48
0.58
0.11
0.81
0.87
0.06
0.07
0.38
0.48
0.0
0.58
0.71
0.08
0.01
0.23
0.38
0.0
0.48 0.67
CIFARIO
0.02
0.0
0.71
0.84
0.03
0.58 0.64
0.03
0.0
0.58
0.77
0.0
0.30 0.39
0.04
0.0
0.45
0.78
0.0
0.16 0.27
21


Reviews:
[{"review_id": "wUJRquyJTtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a novel approach to evade the transferability of adversarial examples between two models. Specifically, they design a luring loss to train model T, an augmented version of M, where the adversarial examples cannot transfer from T to M. The luring loss is designed to reach a twofold objective: (1) for a clean example, T and M yield the same prediction (2) for an adversarial example, T and M yield different predictions, and in the best case, M can provide correct prediction. The proposed approach can serve as a defense for both detect adversarial examples and defend adversarial examples. Experimental results show that the proposed defense can detect and defend adversarial examples better that the three baselines. In general, the paper is clearly written and easy to follow.\n\nOne weakness of this paper is the reason of why luring loss can make the non-robust feature of T different from M is not sufficiently explained.\n\nAnother weakness is that, in the setting of black-box paradigm, the authors consider both adversarial detection and defense, but they did not use recent detection or defense baselines. This makes their experimental results less convincing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ohLz6N-Hhx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update:**\n\nThanks to the authors for their detailed response to my review. Unfortunately after reading the response, I don't understand how it addresses some significant concerns I have about this paper and therefore I can't increase my score. In particular:\n\n- Author response says \"Measuring the transferability from $M$ to $M_{adv}$ is actually the opposite of our concern, since our threat model deals with the transferability from $T$ (the protected augmented model) to $M$ (the target model).\" I'm suggesting defining $T$ as a vanilla model and $M$ as a PGD-trained model. I don't see why your threat model would preclude such a choice.\n- Author response says \"$M$ is hidden from the adversary. Then, the adversary does not have access to $M$'s output, and cannot mount a black-box attack to thwart a binary classifier $M(x)/T(x)$.\" The adversary doesn't need full access to $M$, only access to the output of the binary classifier $M(x)/T(x)$. Presumably they have at least indirect access to the binary prediction; otherwise the defender would be able to detect adversarial inputs but wouldn't be able to take any meaningful action based on the detection  (e.g. denying access to the attacking user, etc...).\n\nThere is still a possibility that I've misunderstood something fundamental, so I leave my review confidence as fairly low.\n\n(Original review below)\n---\n\nSummary:\n\nThe authors propose a threat model for adversarial robustness and a corresponding algorithm to defend against attacks in this model.\nThe threat model supposes that attackers have unlimited black-box access to one model and limited access to a second model which is the target of the attack.\nThe algorithm trains a model $M \\circ P$ and a second model $M$ such that they agree on clean data but disagree on adversarial examples (the authors call this \"luring\"; i.e. $M \\circ P$ \"lures\" adversaries into producing adversarial examples which don't work on $M$).\n\nReview:\n\nI'll disclaim that I don't know very much about adversarial robustness, so I've assigned a low confidence to my review.\nWith that said, I have concerns regarding the threat model described in the work (namely, that it seems artificial) and the lack of baselines (see \"correctness\" below) which prevent me from recommending acceptance at this time.\n\nCorrectness:\n\nThe experiments indeed show that the \"luring behavior\" exists; i.e. that attacks on $M \\circ P$ don't transfer to $M$ as easily when $M$ and $M \\circ P$ are trained using the luring objective as when they're trained using other baseline strategies.\n\nThere's a natural baseline which seems to be missing: we can train two networks independently from scratch, one with PGD and one without. If the adversarial examples from the vanilla-trained model don't transfer to the PGD-trained model, then there's no need for a special objective or removable neural net component. (This missing baseline is especially conspicuous given the experiments on PGD-trained models in Appendix H.2).\n\nNovelty:\n\nThe method proposed is novel to my knowledge, but I disclaim that I'm not familiar with adversarial defense literature.\n\nSignificance:\n\nMy main concern with this paper is with regard to the threat model.\nThe authors give two possible scenarios for the defender. In the first (\"inner transferability scenario\"), we assume $\\mathbb{S}_D = \\mathbb{S}_A$ and the defender detects adversarial inputs by comparing $M(x)$ and $(M \\circ P)(x)$.\nBut this amounts to doing binary classification on the inputs, and the attacker can perform a black-box attack on this binary classifier to generate adversarial examples which fool both $M$ and $M \\circ P$.\n\nIn the second scenario (\"distant transferability\"), we assume that $\\mathbb{S}_D$ is secured (i.e. the attacker doesn't have access). Under this model the proposed approach makes sense, but the threat model seems contrived. In what situations are we able to limit access to a model, and yet required to provide access to a second model which makes the same predictions on clean data? The paper would be much stronger if the authors could motivate this threat model by real-world examples (or even hypothetical ones).\n\nClarity:\n\nOverall the paper reads acceptably clearly. I appreciate that the authors provide a clear description of their threat model (notwithstanding my concerns with that model). From what I know of adversarial example papers, this is important and often overlooked. \n\nThis is subjective, but I feel the result tables are hard to interpret at times because of heavy use of acronyms (e.g. $DAC$, $AC_M$, $C_E$, etc...) where plain-text would suffice, and also because many of the rows/columns aren't labeled (e.g. in Table 1, what type of things are \"SPSA\", \"ECO\", \"MIM-W\"? What type of things are \"STACK\", \"AUTO\", \"C_E\", \"LURING\"? Don't assume readers are familiar with these acronyms, nor that they've read the full text before reading the tables.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "aZ2mbprq4Lv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary :\n\nThe paper proposes a new framework for addressing the problem of adversaries in black box settings in order to improve model robustness. Leveraging classical deception frameworks used in network security, the authors propose to fool the attacker by training what they call a `luring component’ that is augmented to an already trained model such that the new model does not later good samples and targets the adversaries to achieve the desired result. Additionally, the proposed framework does not need access to labeled data and can be applied to any pre-trained model. Promising results are demonstrated on multiple datasets like MNIST and CIFAR 10, etc.\n\nPositives:\n\nNovelty: the paper proposes a new framework which is distinct from existing methods that mostly consider white box settings, rely on robust features or anticipate perturbations. Instead, the proposed method is applicable to black box settings, is data agnostic, and can be applied to any pre trained model. \n\nTechnical sophistication:\n\n1. The intuition behind the idea is explained well. Though the concept is borrowed from network security literature, its introduction and adaptation to the ML community is valuable. \n2. The authors introduce two objectives that are designed towards training the luring component, and these are intuitive as well. In order to achieve their objective, the authors introduce a new loss function called the luring loss.\n3. The algorithm for training the luring component also seems reasonable.\n\nExperimentation:\n\n1. Extensive comparison against baselines is detailed for characterization of luring effect as well as for metric evaluation. \n2.  Results are reported on multiple datasets.\n\nPresentation:\nThe paper is well motivated and fairly clear.\n\nOverall comments:\n\nThe paper opens a new direction to adversarial machine learning research by proposing a new conceptual framework for addressing adversaries in black box settings. It offers practical value given that it is data agnostic , applicable to black box settings, and can be applied to any pre trained model. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9WLW0SFA5Vi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n* This paper presents a novel method to prevent transferability of adversarial examples in black box settings. They name the effect “luring effect” which thwarts transferability of adversarial examples between two models. They evaluate on MNIST, SVHN, CIFAR10, and analyze the scalability on ImageNet. \n\nStrengths: \n* Thwarting transferability is a relevant and challenging research problem \n* Relies only on logits, and can be applied to any pre-trained model\n* Experimental comparison with related methods (Stack, Auto, CE)\n* Code is publicly released.\n\nWeaknesses:\n* You evaluate robustness of using the luring effect as a defense with SPSA and ECO, but you do not consider an adaptive attacker that knows your defense and tries to attack it. See best practices discussed in [1].\n* Related work section only mentions two closely related papers. \n* Focus on black-box only may overestimate the effectiveness of the luring effect. It would be anyway interesting to know what happens in a white-box setting, or it if has any effect at all (see also guidelines in [1]). \n\nComments: \n\nI think this paper tackles a very important problem and is generally very well-written, and I would like to thank the authors for releasing their code to the community, which definitely foster a faster advance of the state of the art. The proposed luring effect is based on an interesting idea, and is generally well-formalized and explained. \n\nI do see some limitations in the current version of the paper and I would like to provide some recommendations that can improve its quality:\n* The main limitation is somehow the lack of an adaptive attacker to evaluate the robustness of the luring effect. What if someone knows your luring effect and tries to bypass it? I think this type of reasoning is somehow missing from the paper. See also [1] for more details on what I am referring to. \n* Real-world scenarios in which this may be useful are not fully clear to me. Let me explain a bit better. In your threat model, you assume that the attacker has access to infinite queries. Even if you introduce the “luring effect”, it is unclear to me what the attacker needs to achieve: there must be some obvious classifier feedback that the attacker may want to trigger, to evade the ML-based API. If the real underlying model M is not fooled, then the attacker may somehow know that his evasion is erroneous. However, if all replies and outputs are generated from model T, then in practice if the attacker evades T, from their perspective the attacker completed the evasion task successfully, even if the defender knows the result in M.\n* Moreover, I would like to encourage the authors to clarify more intuitions about theoretical guarantees a user may have of not getting misleading decisions on vanilla (i.e., non-adversarial) classification requests after applying the luring effect. This was not entirely clear to me. \n* As a minor final comment, I think it would be anyway useful to understand better if using a white-box threat model would actually make the luring effect meaningless (because it is relying on a gradient-free attack), and whether there are some gray-box scenarios in which there are still some properties for the luring effect.\n\nReferences: \n* [1] Carlini, Nicholas, et al. \"On evaluating adversarial robustness.\" arXiv preprint arXiv:1902.06705 (2019). - “Live\" paper on GitHub \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to augment the original  model to introduce the "luring effect", which can be used for detection and black-box defense. Despite being an interesting setup, there are several weaknesses in the threat model (whether it is practical) and the evaluation (lack of adaptive attacks). Those concerns remain after the rebuttal phase. 

Threat model: see the concerns raised by Reviewer 1 and the updated comments after the rebuttal phase. 

Lack of adaptive attack: the authors assume that the attacker has very limited knowledge to how the system works. This could be viewed as a "black-box setting" for adversarial detection evaluation, and actually many other detection works can perform almost perfectly in this setting, so it's not clear how significant are the results. The authors tried to consider some adaptive attacks in their rebuttal but the reviewers are still not fully convinced. 

Decision: Reject