Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors present a novel approach to evade the transferability of adversarial examples between two models. Specifically, they design a luring loss to train model T, an augmented version of M, where the adversarial examples cannot transfer from T to M. The luring loss is designed to reach a twofold objective: (1) for a clean example, T and M yield the same prediction (2) for an adversarial example, T and M yield different predictions, and in the best case, M can provide correct prediction. The proposed approach can serve as a defense for both detect adversarial examples and defend adversarial examples. Experimental results show that the proposed defense can detect and defend adversarial examples better that the three baselines. In general, the paper is clearly written and easy to follow.

One weakness of this paper is the reason of why luring loss can make the non-robust feature of T different from M is not sufficiently explained.

Another weakness is that, in the setting of black-box paradigm, the authors consider both adversarial detection and defense, but they did not use recent detection or defense baselines. This makes their experimental results less convincing.

review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Update:**

Thanks to the authors for their detailed response to my review. Unfortunately after reading the response, I don't understand how it addresses some significant concerns I have about this paper and therefore I can't increase my score. In particular:

- Author response says "Measuring the transferability from $M$ to $M_{adv}$ is actually the opposite of our concern, since our threat model deals with the transferability from $T$ (the protected augmented model) to $M$ (the target model)." I'm suggesting defining $T$ as a vanilla model and $M$ as a PGD-trained model. I don't see why your threat model would preclude such a choice.
- Author response says "$M$ is hidden from the adversary. Then, the adversary does not have access to $M$'s output, and cannot mount a black-box attack to thwart a binary classifier $M(x)/T(x)$." The adversary doesn't need full access to $M$, only access to the output of the binary classifier $M(x)/T(x)$. Presumably they have at least indirect access to the binary prediction; otherwise the defender would be able to detect adversarial inputs but wouldn't be able to take any meaningful action based on the detection  (e.g. denying access to the attacking user, etc...).

There is still a possibility that I've misunderstood something fundamental, so I leave my review confidence as fairly low.

(Original review below)
---

Summary:

The authors propose a threat model for adversarial robustness and a corresponding algorithm to defend against attacks in this model.
The threat model supposes that attackers have unlimited black-box access to one model and limited access to a second model which is the target of the attack.
The algorithm trains a model $M \circ P$ and a second model $M$ such that they agree on clean data but disagree on adversarial examples (the authors call this "luring"; i.e. $M \circ P$ "lures" adversaries into producing adversarial examples which don't work on $M$).

Review:

I'll disclaim that I don't know very much about adversarial robustness, so I've assigned a low confidence to my review.
With that said, I have concerns regarding the threat model described in the work (namely, that it seems artificial) and the lack of baselines (see "correctness" below) which prevent me from recommending acceptance at this time.

Correctness:

The experiments indeed show that the "luring behavior" exists; i.e. that attacks on $M \circ P$ don't transfer to $M$ as easily when $M$ and $M \circ P$ are trained using the luring objective as when they're trained using other baseline strategies.

There's a natural baseline which seems to be missing: we can train two networks independently from scratch, one with PGD and one without. If the adversarial examples from the vanilla-trained model don't transfer to the PGD-trained model, then there's no need for a special objective or removable neural net component. (This missing baseline is especially conspicuous given the experiments on PGD-trained models in Appendix H.2).

Novelty:

The method proposed is novel to my knowledge, but I disclaim that I'm not familiar with adversarial defense literature.

Significance:

My main concern with this paper is with regard to the threat model.
The authors give two possible scenarios for the defender. In the first ("inner transferability scenario"), we assume $\mathbb{S}_D = \mathbb{S}_A$ and the defender detects adversarial inputs by comparing $M(x)$ and $(M \circ P)(x)$.
But this amounts to doing binary classification on the inputs, and the attacker can perform a black-box attack on this binary classifier to generate adversarial examples which fool both $M$ and $M \circ P$.

In the second scenario ("distant transferability"), we assume that $\mathbb{S}_D$ is secured (i.e. the attacker doesn't have access). Under this model the proposed approach makes sense, but the threat model seems contrived. In what situations are we able to limit access to a model, and yet required to provide access to a second model which makes the same predictions on clean data? The paper would be much stronger if the authors could motivate this threat model by real-world examples (or even hypothetical ones).

Clarity:

Overall the paper reads acceptably clearly. I appreciate that the authors provide a clear description of their threat model (notwithstanding my concerns with that model). From what I know of adversarial example papers, this is important and often overlooked. 

This is subjective, but I feel the result tables are hard to interpret at times because of heavy use of acronyms (e.g. $DAC$, $AC_M$, $C_E$, etc...) where plain-text would suffice, and also because many of the rows/columns aren't labeled (e.g. in Table 1, what type of things are "SPSA", "ECO", "MIM-W"? What type of things are "STACK", "AUTO", "C_E", "LURING"? Don't assume readers are familiar with these acronyms, nor that they've read the full text before reading the tables.)
review_title: Concerns about threat model, baselines

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
* This paper presents a novel method to prevent transferability of adversarial examples in black box settings. They name the effect “luring effect” which thwarts transferability of adversarial examples between two models. They evaluate on MNIST, SVHN, CIFAR10, and analyze the scalability on ImageNet. 

Strengths: 
* Thwarting transferability is a relevant and challenging research problem 
* Relies only on logits, and can be applied to any pre-trained model
* Experimental comparison with related methods (Stack, Auto, CE)
* Code is publicly released.

Weaknesses:
* You evaluate robustness of using the luring effect as a defense with SPSA and ECO, but you do not consider an adaptive attacker that knows your defense and tries to attack it. See best practices discussed in [1].
* Related work section only mentions two closely related papers. 
* Focus on black-box only may overestimate the effectiveness of the luring effect. It would be anyway interesting to know what happens in a white-box setting, or it if has any effect at all (see also guidelines in [1]). 

Comments: 

I think this paper tackles a very important problem and is generally very well-written, and I would like to thank the authors for releasing their code to the community, which definitely foster a faster advance of the state of the art. The proposed luring effect is based on an interesting idea, and is generally well-formalized and explained. 

I do see some limitations in the current version of the paper and I would like to provide some recommendations that can improve its quality:
* The main limitation is somehow the lack of an adaptive attacker to evaluate the robustness of the luring effect. What if someone knows your luring effect and tries to bypass it? I think this type of reasoning is somehow missing from the paper. See also [1] for more details on what I am referring to. 
* Real-world scenarios in which this may be useful are not fully clear to me. Let me explain a bit better. In your threat model, you assume that the attacker has access to infinite queries. Even if you introduce the “luring effect”, it is unclear to me what the attacker needs to achieve: there must be some obvious classifier feedback that the attacker may want to trigger, to evade the ML-based API. If the real underlying model M is not fooled, then the attacker may somehow know that his evasion is erroneous. However, if all replies and outputs are generated from model T, then in practice if the attacker evades T, from their perspective the attacker completed the evasion task successfully, even if the defender knows the result in M.
* Moreover, I would like to encourage the authors to clarify more intuitions about theoretical guarantees a user may have of not getting misleading decisions on vanilla (i.e., non-adversarial) classification requests after applying the luring effect. This was not entirely clear to me. 
* As a minor final comment, I think it would be anyway useful to understand better if using a white-box threat model would actually make the luring effect meaningless (because it is relying on a gradient-free attack), and whether there are some gray-box scenarios in which there are still some properties for the luring effect.

References: 
* [1] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019). - “Live" paper on GitHub 

review_title: ICLR 2021 Conference Review Paper822

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary :

The paper proposes a new framework for addressing the problem of adversaries in black box settings in order to improve model robustness. Leveraging classical deception frameworks used in network security, the authors propose to fool the attacker by training what they call a `luring component’ that is augmented to an already trained model such that the new model does not later good samples and targets the adversaries to achieve the desired result. Additionally, the proposed framework does not need access to labeled data and can be applied to any pre-trained model. Promising results are demonstrated on multiple datasets like MNIST and CIFAR 10, etc.

Positives:

Novelty: the paper proposes a new framework which is distinct from existing methods that mostly consider white box settings, rely on robust features or anticipate perturbations. Instead, the proposed method is applicable to black box settings, is data agnostic, and can be applied to any pre trained model. 

Technical sophistication:

1. The intuition behind the idea is explained well. Though the concept is borrowed from network security literature, its introduction and adaptation to the ML community is valuable. 
2. The authors introduce two objectives that are designed towards training the luring component, and these are intuitive as well. In order to achieve their objective, the authors introduce a new loss function called the luring loss.
3. The algorithm for training the luring component also seems reasonable.

Experimentation:

1. Extensive comparison against baselines is detailed for characterization of luring effect as well as for metric evaluation. 
2.  Results are reported on multiple datasets.

Presentation:
The paper is well motivated and fairly clear.

Overall comments:

The paper opens a new direction to adversarial machine learning research by proposing a new conceptual framework for addressing adversaries in black box settings. It offers practical value given that it is data agnostic , applicable to black box settings, and can be applied to any pre trained model. 
review_title: A conceptually new framework for addressing the problem of adversaries in black box settings with promising results

Meta-review: The paper proposes to augment the original  model to introduce the "luring effect", which can be used for detection and black-box defense. Despite being an interesting setup, there are several weaknesses in the threat model (whether it is practical) and the evaluation (lack of adaptive attacks). Those concerns remain after the rebuttal phase. 

Threat model: see the concerns raised by Reviewer 1 and the updated comments after the rebuttal phase. 

Lack of adaptive attack: the authors assume that the attacker has very limited knowledge to how the system works. This could be viewed as a "black-box setting" for adversarial detection evaluation, and actually many other detection works can perform almost perfectly in this setting, so it's not clear how significant are the results. The authors tried to consider some adaptive attacks in their rebuttal but the reviewers are still not fully convinced. 

Decision: Reject