Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The present work introduces a novel defense to confound the score-based query attacks (SQAs). Its main novelty includes three parts: (1) an effective and user-friendly post-processing module; (2) a novel adversarial attack on attackers (AAA) defense by slightly deviating the output scores; (3) AAA beats other prior defenses in terms of the effectiveness and robustness. Due to the impractical cost of prior defenses, AAA designs the post-processing method to improve efficiency and exerts a significant impact on accuracy.
strengths_and_weaknesses: Strengths:
+ The idea is sound and interesting. Instead of enhancing the robustness of deep learning models, this proposed method mislead and confuse the attackers to protect the DNN.
+ Meanwhile, the method costs a little to improve efficiency, which makes defense more practical in real-world settings.
+ the general design of the experiments is meaningful in helping to understand the role of AAA in outperforming the considered defense approaches.

Weaknesses:
- Some operations for example, the determination of specific operation, like Equation (4) is short of discussion. The detail is as follows.
questions: 1. Interesting, the clean model performance with AAA seems a little higher than original clean model, for example, 94.84 vs 94.78 in CIFAR10. Can you provide more analysis about this? Similar results can also be found in baseline DENT.

2. In Equation (4), does it missing abs for ceil(·)? when l_0=0, ceil(l_0/t=0)-1/2 = -1/2? and the result is -2?

3. The idea of attacking on attacker is straightforward, what is the motivation in designing the specific operated shown in Equation (4) and (5). For example, why push l_t even lower than l_a by a margin of (l_0-l_a), and what is the detailed motivation in designing equation (4). I am just curious if there are any other operations following the same idea achieves better defense results.

4. Have you studied the proposed method without optimization in Equation (6)?

5. The adjusted loss is closely related with confidence calibration. Can you show some defense results, in comparison with some confidence calibration methods?
limitations: The discussion about optimizing Equation 6 seems missing. What is the number of optimization steps? The time consumed? etc. What are the results without optimization Equation (6).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a defense called Adversarial Attack on Attackers (AAA) that is designed specifically towards mitigating score-based query attacks (SQAs). AAA is a post-processing attack that attempts to modify the logits loss curve to locally point in the incorrect attack direction in a periodic fashion, which steers SQAs away from a true adversarial attack. As such, AAA takes advantage of typical SQA behavior of sampling nearby loss value changes to find a local direction to optimize in and steers them in the wrong way. AAA could then be added on to other models as a way to deter and prevent these attacks in more real-world scenarios where such black-box access is present. The authors evaluate AAA over several baseline defenses on CIFAR-10 and ImageNet on various SQAs, finding AAA to be effective at improving robustness.
strengths_and_weaknesses: Strengths
- Interesting new defense approach that targets a realistic deployment of ML models by attacking the query process
- Seems like a lightweight defense with a low cost
- Preserves natural accuracy

Weaknesses
- Unclear if SQAs could possibly be adapted to re-attack AAA
- Unknown what the exact runtime cost is
- Does not increase underlying model robustness
questions: This paper provides an interesting new take on adversarial robustness. I like the direction of targeting realistic attacks such as SQAs at a higher level than simply robustifying the model to all possible attack inputs. Indeed, such a post-processing approach such as AAA to deter attacks that that level is appealing and I believe that further research with this philosophy would be interesting. The experiments presented provide a useful understanding of the characteristics of AAA and show strong performance on a variety of settings. 

I have two main concerns. The primary one is whether or not SQAs could possibly be adapted to attack AAA. In Section 5.5 “Hyper-parameters of AAA” and Figure 4, it is clear that the choice of the attractor interval t can greatly influence the adversarial accuracy as the attack success rate would seemingly come down to the attack’s ability to jump out of an interval. While this is an interesting study to see how the choice of t impacts the performance of a static SQA, could SQAs be modified to look at a wider interval? 

For example, perhaps an SQA could be modified to maintain two search branches, one that goes in the default direction implied by the loss changes and another that continually goes in the opposite for a limited time, and then if the one that goes in the opposite direction eventually drops by jumping out of the interval, an attacker could determine that this was actually the correct direction to go. Since the underlying model is not inherently more robust, a process that finds a jump may still be able to eventually break the model, albeit with more queries.

Secondarily, how much does it cost to run AAA? It would appear to be a fairly lightweight process, but do you know how much slower this would make the model?

Given this, I still like the general direction of the paper and if my concerns are addressed I will consider raising my score.

-----------------------------------------------

In the rebuttal, the authors provided results on the proposed bidirectional adaptive attack as well as a new sine function to use, as well as runtime results, alleviating these concerns.
limitations: The authors have adequately addressed limitations and impact. The authors discuss AAA in the context of real-world systems such as autonomous driving where AAA could be added with pre-trained models to increase the reliability of such systems. The authors also acknowledge the limitation that AAA is designed to target SQAs specifically, which is a useful defense approach, but does mean that worst-case robustness under white-box settings is not improved.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a post-processing defense against score-based black-box evasion attacks. This defense modifies the original logits so that the trend of the margin loss is reversed in each small interval of the attack, but the overall trend remains unchanged to preserve accurate prediction confidence. As a result, score-based attacks using the margin loss will optimize towards the opposite (hence non-adversarial) direction. Meanwhile, this defense does not hurt the model’s benign performance, and the calibration performance is preserved by solving a joint optimization problem. Experiments show that the proposed defense outperforms 8 previous defenses under 6 score-based attacks on CIFAR-10 and ImageNet.
strengths_and_weaknesses: ### Originality

**Strengths**
* This paper studies post-processing defenses, which are less explored in defending evasion attacks.
* The proposed defense avoids hurting the model’s benign performance and computational overheads.

**Weaknesses**
* Not much in this perspective.

### Quality

**Strengths**
* Good experiments. I appreciate the evaluation of 8 defenses, 6 attacks, and 2 datasets.
* Good ablation study. Most hyper-parameters are supported by the ablation study.

**Weaknesses**
* **Impractical threat model.** While the attack’s threat model is well defined, the proposed defense implicitly assumes complete control of the attacker’s optimization process. **Specifically, the defense assumes unrealistic knowledge of the attack’s running iteration, i.e., the period $t$ defined at L171**. Note that all compared defenses in this paper did not have such an assumption. This knowledge is only realizable if given the following two assumptions. First, the attacker discloses the attack’s current number of iterations for each query. Second, the defender can extract the sequence of queries coming from the designated attacker (across all queries from all legitimate users and potentially other attackers). These two assumptions, however, are largely unrealistic or challenging. Since such knowledge is the base assumption of this defense at Eq. (4), I am not sure if the current defense would work if the assumption did not hold. **In particular, the defense’s code in Appendix D seems to work inside the attack, which is impossible in practice.**

* **Lack of adaptive evaluation.** This paper does not discuss how the attacker could potentially modify their attacking procedure to evade the proposed defense, although some adaptive attack papers like [46] are cited. Currently, the proposed defense assumes a static (i.e., non-adaptive) attack that is unaware of the defense. Following the paper’s own motivation, is it possible that the attackers also revert their updating logic in Eq. (3) to evade the proposed defense? For example, now that the returned score exhibits a reverted trend, **would this defense still work if the attackers update their adversarial examples only when observing an increased margin loss?**

### Clarity

**Strengths**
* Good motivation for post-processing defenses.
* The proposed defense is easy to follow.

**Weaknesses**
* The notation in Eq. (1) is slightly confusing. At L117, the x is defined as a clean sample, but it is later used as a placeholder in Eq. (1) and (2).
* The unsupervised margin loss in Eq. (2) can be simplified. Specifically, if the defender could use one query to obtain the ground-truth label (from the black-box model’s perspective), Eq. (2) can reduce to Eq. (1) and therefore simplify the notations.
* At L123, it is suggested to add some (brief) background of how these attacks sample their queries $x_q$. This would be greatly helpful for readers to understand the defense. Currently, it seems that all attacks are compressed into one high-level idea, which might reduce the confidence in understanding how the defense would work if the attack changes adaptively.
* At L171 and L193, the notation $z_0$ is not defined until algorithm 1.

### Significance

I appreciate the effort of motivating post-processing defenses against evasion attacks, but the proposed defense considers a somewhat unrealistic threat model, which is different from the defenses compared in this paper. Moreover, this paper assumes a static non-adaptive attacker, which might be easily broken by adaptive attacks based on my assessment.
questions: My current score is mainly based on the weaknesses outlined in the Quality section. I am willing to raise my score if the following concerns are adequately clarified or justified.
* [Quality-Weakness-1] Please clarify if the knowledge of the attack’s period $t$ at L171 is practical. Specifically, please explain how the proposed defense can be implemented *independently of the attack’s code* with only the following inputs: (1) query image, (2) model, and (3) the defense’s hyper-parameters. If you need any other inputs, please clarify relevant assumptions and their practicability.
* [Quality-Weakness-2] Please discuss if the defense would still work if the attackers revert their update logic in Eq. (3), without significantly modifying the current defense.
limitations: The primary limitation is summarized in the Significance section. While I appreciate the effort of motivating post-processing defenses against evasion attacks, I strongly recommend the authors carefully address these limitations in future versions of this paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes an adversarial defense against black-box score-based query attack. Specifically, the AAA defense proposed in this paper can mislead SQA methods to simulate the direction of gradients by post-processing the logits output by the target model. The method is validated on cifar-10 and Imagenet datasets
strengths_and_weaknesses: The strengths and weaknesses of this paper are summarized as follows:

The main strength of this paper is that it presents a relatively new approach of adversarial defense. The defense against SQA is rarely covered by previous work, and the AAA method proposed in this paper may bring inspiration to the subsequent score-based attacks and related defenses

The weaknesses of this paper are twofold. The first aspect is the motivation of the method. The AAA method proposed in this paper is in fact specially targeted for SQA. While SQA belongs to a class of query-based attacks in black-box attack methods in adversarial attacks. That is to say, the effectiveness of AAA is based on at least three constraints: the attacker cannot obtain the details of the target model, the attacker does not use a substitute model but can only query the target model, and the attacker can obtain the confidence information of the target model. This situation implies one-way transparency from the defender to the attacker, not the other way around. This setting is very unrealistic, which makes the motivation somewaht weak. 

In addition, even from the perspective of exploring the robustness of deep neural networks, the method proposed in this paper to specifically deal with one class of attacks cannot improve the robustness of the model itself, or as the authors put it in the paper, worst-case robustness. However, improving the robustness of the model without considering the attack method is the direction that should be advocated in the field of adversarial defense [1]. Otherwise, this is only an arms race between adversarial attacks and defenses.

[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." International conference on machine learning. PMLR, 2018.

Secondly, because this paper defends against SQA, there is a realistic problem that other query-based attacks like Boundary Attack require less information than SQA (no confidence information, just hard labels). Can the method proposed in this paper effectively defense against such attacks? If so, AAA must changes the hard label, will this affect the availability of the target model? If not, can the attacker directly use decision-based attacks to bypass this defense?

questions: There seems to be some problems in the penultimate row of Table 2. The performance of 69.3 of RND defense under NES attack should be bolded, not 68.51 of AAA.
limitations: From the perspective of experimental performance, compared with other defense methods, AAA is still not significantly improved in terms of affecting SQAs’ attack effect.
review_title: null

Meta-review: metareview: This paper proposes a defense against score-based black-box attacks by post-processing the output probabilities to misguide the attacker. The method enjoys several advantages such as not reducing test-time accuracy or increasing the train-/test-time cost, improving calibration for the model, and superior performance under black-box attack compared to prior work. The authors also included additional experiments during the discussion phase that show effectiveness against adaptive attacks.

One weakness is that the method does not improve robustness of the underlying model and hence is still susceptible to surrogate model and/or hard-label attacks. However, most reviewers consider this weakness as minor and that the paper’s contribution is significant enough for publication. AC therefore recommends acceptance for publication at NeurIPS.

confidence: Certain
Decision: Accept