Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a backdoor injection method that manipulates model weights. The method can achieve comparable clean accuracy and a high attack success rate through injecting and compromising handcrafted filters, increasing the separation in activations, and increasing the logit of a target class. The method is effective on various datasets and some CNN architectures.
strengths_and_weaknesses: Strengths: The authors propose a new and interesting backdoor injection method. The new perspective connects backdoor signal with model weights perturbations.

Weaknesses: The attack may not be practical. Some details are not very clear.
questions: 1. The method still requires a small amount of data for validation. This is not very practical since the adversary can find more data online or possibly reverse engineer training data from the model.

2. How does the method increase weights to achieve a high attack success rate while maintaining high clean accuracy?

3. It is also necessary to consider the ResNet structure for CIFAR-10.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper presents a backdooring scheme that operates directly on trained model parameters. Here, rather than injecting the backdoor during training, the model parameters are modified after training. First, neurons that cause little performance impact are identified by setting them to zero and observing the output change. Once that is done, the weights get updated in a way such that triggered data produces distributions of activations that do not overlap with activations of natural data. Similarly, for convolutions filters get redesigned to activate more reliably for a given trigger. Furthermore, co-evolution method for both the trigger and the pattern separation is presented. Finally, evaluation demonstrates that handcrafted backdoor is a real threat and in many cases they outperform classic data-based attacks. 
strengths_and_weaknesses: Strengths:
+ Novel attack vector
+ Extremely realistic threat model

Weaknesses:
+ Evaliation only focuses on relatively small models
+ Unclear how reliable the injection procedure is 
questions: * How representative is the fine-tuning experiment? Did you try searching through the optimisation hyper parameters?
* It feels like tthe conclusion with verifiable training are missing a reference to Proof-Of-Learning [1].
* To what extend attacks described in the work translate to larger deeper models? 
* Given stochastic nature of the attack it would be great to see reliability analysis i.e. what happens when one rerun the setting many times? Do all neurons work in the same way?
* shouldnt 713 in appendix have an opposite sign?

[1] Jia et al., Proof-of-Learning: Definitions and Practice, S&P
limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a "handcrafted" attack to inject backdoors into pre-trained deep neural network (DNN) classifiers. Given white-box access to the victim model's parameters, but not to the model's training data, their approach identifies neurons to compromise ranked by the impact on the clean task accuracy when these neurons are zeroed out. Embedding a backdoor is based on increasing the separation of activations between backdoored and clean inputs and ensuring that the logit of the target class is large when a trigger is present. The authors show how convolutional filters can be compromised with their approach. Handcrafted backdoors are shown empirically to be robust against one adaptive (parameter noising) and three popular existing backdoor removal defenses if the adversary uses (i) sufficiently large triggers or (ii) is willing to sacrifice parts of their attack success rate. Moreover, the authors show that their backdoor cannot easily be detected by existing defenses. 
strengths_and_weaknesses: The paper was an excellent read and I strongly believe that it is of considerable interest to the community. 

**Strengths 

* Effectiveness
The authors provide lots of useful experiments and insights to demonstrate that their backdoor (i) preserves the victim model's utility while (ii) being robust against all surveyed defences (under certain conditions). The Appendix provides many details on the robustness against defences and is l a great addition to the paper. 

* Novelty
The author's backdooring attack has plenty of interesting and novel components. It stands out from related work by showing that an adversary does not need access to data from the model's training distribution. Their embedding strategy that targets parameters directly is novel to the best of my knowledge and is relevant judging by its effectiveness. 

* Experimental Validation
The main paper summarises the empirical results comprehensively and provides lots of intuition on why and how the attack works and what challenges need to be addressed.   

* Reproducibility
The authors provided their source code and (as already mentioned) presented additional insights in their Appendix. 

** Weaknesses 

* The difference between a "supply-chain" attack and a code poisoning attack is not clear to me. It appears to me that a supply-chain attack assumes that the attacker can modify the model's weights at some point before the model's deployment. Important works such as [1] or [2] assume the same threat model (with addition to access to data resembling the training distribution), but they are not mentioned. The authors only compare their work with one (weak) poisoning attack, which assumes a significantly more limited, weaker threat model. 

[1] Pang, R., Shen, H., Zhang, X., Ji, S., Vorobeychik, Y., Luo, X., ... & Wang, T. (2020, October). A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security

[2] Shokri, Reza. "Bypassing backdoor detection algorithms in deep learning." 2020 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2020.
questions: My questions are focused on the limitations of handcrafted backdoors. 

Q1: I wonder whether the backdoor attack is resilient to overwriting attacks. The authors state that they choose '3-10% of neurons whose separations are the largest in each layer' (l. 148). Can a defender identify the same neurons by running the same attack? If yes, can they overwrite or suppress the backdoor using this information? 

Q2: The authors mention that their backdoor is undetectable by existing defenses. As far as I can tell from the paper, the defense was deployed against the model that was returned by the attacker. Considering that the defense introduces a 'sharp local minimum' (Appendix, l. 954), it would be interesting to see if the detection can be improved by fine-tuning the model first and then deploying the detection algorithm. 


limitations: -
review_title: null

Meta-review: metareview: This paper proposes a backdoor injection method that directly manipulates model weights after training. The backdoored method can achieve comparable clean accuracy and a high attack success rate through injecting and compromising handcrafted filters, increasing the separation in activations, and increasing the logit of a target class. The reviewers agree that the proposed backdoor injection method is novel and interesting. The authors are suggested to conduct more experiments on evaluating whether their handcrafted models cannot be detected/removed by existing defenses. 
confidence: Certain
Decision: Accept