Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a clean label dataset poisoning attack to embed backdoors into deep neural network image classifiers. Their approach is based on training many surrogates models for generating poisoned examples and using gradient matching to find malicious perturbations. The attack, Sleeper Agent, assumes full access to the defender's training dataset which allows them to (i) train surrogate models and (ii) select which inputs to poison. Their attack requires no knowledge of the defender's victim model (or the model's architecture). The authors show that their results greatly outperform existing clean label poisoning attacks in the attack success rate and that their backdoor cannot easily be removed with existing backdoor removal or dataset augmentation methods. 
strengths_and_weaknesses: ** Strengths

* High Relevance and Effectiveness. Poisoning attacks with few assumptions about the victim's model architecture or training procedure are highly relevant. The paper shows clear improvement over existing works in this field and demonstrates that their attack has practical relevance.

* Ablation studies. The experiments are convincing and show that the author's approach has high transferability of the backdoor across model architectures and datasets (extension to ImageNet). The authors also show a positive effect on the attack success rate if the attacker is allowed to use larger perturbations. 

* Paper Organization. The paper is neatly organized and shows all results in a comprehensive manner. Discussed ideas were easy to follow and provided comprehensive intuition and motivation. 

** Weaknesses 

Overall, the paper was an interesting read and convinced me of the effectiveness of the author's attack. I believe the paper is of high interest to the research community. 

* Limited defense evaluation. Since the generated patterns resemble an adversarial example, it seems plausible to evaluate defenses that harden the model against adversarial attacks, such as adversarial training during training.

* Missing Runtime on ImageNet. Figure 5 shows a high runtime for CIFAR-10, but it appears that the runtime for ImageNet is missing. 


questions: I wonder how much the attack's success rate depends on the attacker having access to the same training dataset as the defender. It would be interesting to relax the assumption about the overlap between the defender's and attacker's training data and see if the gradient matching still remains effective. 

Do the authors have results or insights on the effectiveness of their Sleeper Agent attack when the defender fine-tunes uses a pre-trained model? 
limitations: -
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a novel backdoor attack, Sleeper agent, that can be applied in the black-box, clean-label settings. The authors craft poisoned samples by calculating the gradient match for loss functions on the training set and patched samples. Then the authors evaluate Sleeper agent on CIFAR-10 and ImageNet two tasks using ResNet-18, MobileNet-V2, VGG11 these architectures. Finally, the authors also evaluate the robustness of Sleeper agent against several popular defense approaches. The results show that Sleeper agent can achieve high attack success rate (>80%) with a poison rate larger than 1% and perform robustness against evaluated defense mechanisms. 
strengths_and_weaknesses: ####################Strengths####################

1. The problem is practical and interesting. 

2. The evaluation is comprehensive in the dimensions of robustness, efficacy, and practice.

3. The overall writing is good and easy to follow.

####################Weakness####################

1. Threat model. Since the Sleeper agent requires the pre-trained model for crafting poisoned samples, the authors should mention such requirements or assumptions in the threat model. 

2. Evaluation. The overall evaluation of Sleeper agent is comprehensive.  Regarding the transferability among pre-trained models,   the authors consider evaluating transferability among different architectures. However, I think it is also beneficial to evaluate the transferability of Sleeper agent under scenarios where the accuracy of the surrogate is different from the target models. Specifically, previous work[1] evaluates the transferability for poisoning attacks under a rather scenario where the accuracy of surrogate models is lower compared with the target models.  Moreover, I think the robustness evaluation is weak since the evaluated defense methods are not recently published. For example, the defense approach Neural Cleanse has been published 3 years, i think a more recent defense approach should be considered. 


[1] Practical Poisoning Attacks on Neural Networks, ECCV 2020
 
 

questions: 1. How about the efficacy of Sleeper agent when the surrogate models perform worse compared with the target models.

2.  Why do STRIP and Neural Cleanse fail against Sleeper agent? I think both STRIP and Neural Cleanse are proved to be effective against static backdoor triggers as Sleeper agent. 
limitations: The evaluated defense approach is not state-of-art and some of them(i.e., Neural Cleanse) is ineffective for most proposed backdoor attacks. I think the authors should evaluate their approach against more effective defense approaches.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an invisible poisoning-based clean-label backdoor attack against image classifiers training from scratch. The authors develop their attack based on the Hidden Trigger Backdoor Attack, which combined the benefits of both poison-label attacks and clean-label attacks whereas is effective only in the transfer learning scenarios (rather than training from scratch). Specifically, the author formulates this problem as a bi-level optimization and solves it with the classical gradient alignment technique. To ensure the generalization of the proposed attack (towards different parameter initializations and model structures), the author introduces four techniques, including 1) ensemble, 2) retraining, 3) random patch, and 4) poison selection. The author also propose a poison selection module to further enhance the attack success rate. The proposed method is tested on both CIFAR-10 and ImageNet datasets with different DNNs.
strengths_and_weaknesses: Pros 
1. The topic is of sufficient significance and interest to NeurIPS audiences. 
2. The paper is well written and easy to follow. 
3. Technically, the proposed method is moderately novel. In practice, the invisible poisoning-based clean-label backdoor attack is one of the hardest yet important problems in backdoor learning. To the best of my knowledge, there is still no work that is effective especially on large-scale datasets (e.g., ImageNet).
4. The author have also discussed the resistance to existing backdoor defenses, which should be encouraged.


In general, I enjoy the reading of this paper and recognize its significance and contributions. However, I still have some concerns about this paper. I will increase my score if the author can (partly) address my concerns. The detailed comments are as follows:


Comments
1. Please provide more details about the results on Table 7. It seems that there are some detection-based methods (e.g., SS, AC, STRIP). How ASRs and BAs are calculated under these detection-based backdoor defenses?
2. The authors should review more advanced backdoor attacks (e.g., [1-3]) in Related Works.
3. Ensemble techniques have been widely used to increase the transferability among model structures, especially in adversarial learning. I think the authors should cite a few related work in Page 6 (Line 210-214) to mention it.
4. Please increase the font size in Figure 1 (a).
5. Please recheck and cite the official version of all references (e.g., [4-8]).

References
1. Backdoor Attacks Against Deep Learning Systems in the Physical World. CVPR, 2021.
2. Invisible Backdoor Attack with Sample-Specific Triggers. ICCV, 2021.
3. WaNet - Imperceptible Warping-based Backdoor Attack. ICLR, 2021.
4. Poisoning attacks against support vector machines. ICML, 2012.
5. Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching. ICLR, 2021.
6. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE TPAMI, 2022.
7. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.
8. Backdoor learning: A survey. IEEE TNNLS, 2022.

questions: 1. Please provide more details about the results on Table 7. It seems that there are some detection-based methods (e.g., SS, AC, STRIP). How ASRs and BAs are calculated under these detection-based backdoor defenses?
2. The authors should review more advanced backdoor attacks (e.g., [1-3]) in Related Works.
3. Ensemble techniques have been widely used to increase the transferability among model structures, especially in adversarial learning. I think the authors should cite a few related work in Page 6 (Line 210-214) to mention it.
4. Please increase the font size in Figure 1 (a).
5. Please recheck and cite the official version of all references (e.g., [4-8]).
limitations: Limited but sufficient.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a bi-level optimization based method for hidden trigger backdoor attacks (HTBAs), one type of backdoor attacks that the poisoned samples do not contain the exact trigger pattern (poisoning noise and the actual trigger are different). Three techniques are introduced for generating strong HTBAs, including 1) bi-level optimization with gradient matching, 2) poison selection, and 3) model retraining. The proposed attack is able to achieve high attack success rates for models trained from scratch, which is a substantial improvement over an existing fine-tuning based HTBA. Ablation studies are also conducted to help understand the importance of three techniques individually.
strengths_and_weaknesses: Strengths:
1. An interesting attack, i.e., HTBA, under a more realistic threat model than existing attacks: small poisoning, hidden trigger, black-box (agnostic of the victim model).
2. The proposed HTBA is quite effective as empirically verified under different poisoning rates and both gray-box and black-box settings.
3. Revealing of the practical tricks for strong HTBAs, like retraining.

Weaknesses:
1. Unclear definition or explanation of HTBA. Unclear descriptions in figure/table captions.
2. Confusing notations in the Equations. Incomplete or unclear overall learning objectives, Unclear Algorithm.
3. Missing experiments against more recent defense methods.
4. Low attack success rate under the actual black-box setting, which is one claimed highlight of the proposed threat model.

questions: 1. The HTBA is not very common in the current literature and thus needs more clear and intuitive explanations, particularly in relation to the more extensively studied “invisible backdoor attacks”. As otherwise, Figure 1(b) is hard to understand what the columns are for. Fig. (a) also needs to include the attack step, to show the actual trigger used to mount the attack is different from the trigger patterns.
2. In Equation (1), the definition of $p$ (and how it relates to $\delta$), $\mathcal{D}$ (and its difference to  $\mathcal{T}$) should be clearly stated. It is not so clear how Equations (1) and (2) achieve the objectives of HTBA. 
3. What is the difference between Equation (3) and Equation (1)? What is the loss used for $\mathcal{D}\setminus\mathcal{D_s}$, are they excluded from the training? What is Equation (4) applied on, $\mathcal{D}$ or $\mathcal{D}\setminus\mathcal{D_s}$? How Equations (3) and (4) are combined in the overall training loss?
4. Algorithm 1, “update $\delta$” and “update θ” are confusing, following which/what Equation? Where is $\mathcal{D}$ or $\mathcal{D_s}$. Line 2, “randomly initialize”, how? Line 6, retrain for how many iterations? 
5. Table 1 and 2: the meaning of rows “Poisoned model patched source val (%)” is not clear. What is the size of the “source val” or “patched source val”? If their sizes are different from the “val”, then the results are not comparable to “val”, making the overall interpretation confusing.
6. In Table 4, the attack success rate under the black-box setting is rather low. So it is hard to say whether the proposed attack has achieved the claimed goals. Also the ImageNet ASR is also very low in Table 5.
7. The proposed attack is poisoning based. It should be tested against the latest training-time defense method, like ABL [1]. Also, it needs to test against recent post-training defenses like ANP [2].  
8. Typo: line 127, We "stress" that.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper introduces a clean label poisoning attack to backdoor neural network models.
The reviewers unanimously voted to accept this paper and I agree: it is a strong
technical paper that is well written. The experiments were initially limited in some
areas but the author's rebuttal addresses many of these concerns.

confidence: Certain
Decision: Accept