Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an evasive trojan (backdoor) attack on deep neural networks that makes existing defenses ineffective. The key idea of this attack is "distribution matching." The paper first fine-tunes a network f to construct a backdoored model g that (1) behaves similarly on both clean samples and those with the trigger pattern in the logits and (2) has a minimal difference in their parameter distributions. To increase the specificity of this attack, they further include an objective that the logits of f and g are similar on the same samples with the trigger pattern. In evaluation, the paper demonstrates the attack's effectiveness with a high success rate and a low accuracy drop after backdooring. The attack also evades existing defenses, such as NC, ABS, and MNTD, and the paper shows that the trigger reconstruction was unsuccessful against their backdoored models.
strength_and_weaknesses: 
Strengths:

1. The paper studies an evasive trojan attack that makes existing detection ineffective.
2. The paper conducts experiments with a large number of neural networks.
3. The paper is well-written and easy to read.

Weaknesses:

1. Several prior works explored this idea are missing.
2. The "distribution matching" can introduce several trojan artifacts, leading to detection.
3. The evaluation against backdoor removals is missing.
4. The evaluation against reverse-engineering efforts seems incorrect.


Detailed comments:

This paper studies an evasive trojan attack against existing detection mechanisms. I like the research problem this paper tackles, as most detection techniques proposed so far rely on a specific artifact of backdoors affected by methodologies that the attackers choose.

[Prior work on evasive trojans]

Opposite to the paper's claims (stated in the Introduction), there has also been a vast literature on testing the effectiveness of either detection mechanisms or backdoor removal. 

One example is the work by Tan et al. [1]. This work evades existing detection by making the latent representations of clean samples and the samples with the trojan trigger(s) similar. A minor difference between the work by Tan et al. and this paper is whether the loss makes the "latent representations" or the "logits" similar. 

Another example is the work by Bagdasaryan et al. [2]. This work formulated the backdoor injection (which is fine-tuning of a clean model, the same as this paper) as multi-task learning for conflicting objectives (which is the superset of the idea proposed by the paper). The work also showed that the attack could evade many existing defenses by varying the objective the attacker adds to the original training objective (L_{task}). Unfortunately, the paper only discusses this work as a proposal for another backdoor attack but misses 60% of the entire paper about evading defenses shown by the authors, which is more valuable.

[1] Tan et al., Bypassing Backdoor Detection Algorithms in Deep Learning, Euro S&P 2020.
[2] Bagdasaryan et al., Blind Backdoors in Deep Learning Models, USENIX Security 2021.


[Concerns about distribution matching]

Distribution matching this paper proposes defines the "distribution" at the input and output spaces of a trojan model. However, the "distribution" can be defined in the latent representation space or feature spaces a network produces while forwarding an input.

(1) This implies that one can detect the trojaned models this attack produces by comparing the latent representations and features. I wonder whether a defender can detect the models in Table 2 with the latent representations. With a fixed set of input examples and several potential trigger patterns, the defender can collect the latent representations from those examples and their versions with the triggers. The defender can run clustering to separate trojaned networks and benign ones.

(2) If evaluation (1) does not offer sufficient detection scores (low AUCs), then I think a defender can easily remove the backdoors from those networks by fine-tuning on a small subset of clean samples for a few epochs (with the same learning rate that we used for training the benign models).


[Weak evaluation]

A line of backdoor defenses that this paper does not consider is backdoor removal. Prior work proposed fine-tuning or fine-pruning as a defense mechanism. They don't need trigger reconstruction or detection; just training (or pruning) a model can reduce the attack success rate. 

My second concern is that the evaluation against trigger synthesis or reconstruction is under-studied. NC and ABS are not primarily designed for reconstructing triggers (even if the reconstruction process is important). Some mechanisms specifically study the reconstruction [3]. Due to a large number of parameters, conventional reconstruction leads to potential triggers with no semantic meaning; thus, [3] refines the process to reconstruct more semantically meaningful triggers. Using that advanced reconstruction, the attacker could find the trigger used by an adversary.

[3] Sun et al., Poisoned Classifiers Are Not Only Backdoored, They Are Fundamentally Broken, 2020.

Moreover, the success cases of trigger synthesis are measured by the IoU metric. It's a bit unconvincing that reconstructing the "exact" trigger is necessary to identify backdoored models. One can just extract approximate versions of the original trigger and then exploit them to see whether the classifications are biased or not, which could be sufficient.
clarity,_quality,_novelty_and_reproducibility: My comments about the correctness, quality, and novelty are summarized above. No concern about the reproducibility.

summary_of_the_review: It is important to study the weaknesses of existing defenses from an offensive perspective. From this angle, the paper provides a new attack that the community can use. However, the fact that the prior work has explored similar ideas reduces the novelty of this attack. 

In addition, the technical novelty this paper claims (formulating a new objective to produce such evasive trojans) is also studied by several existing works. Unfortunately, this paper misses those works, which gives me concern that the claims are not from a comprehensive review of backdoor attacks and defenses.

Moreover, the idea of "distribution matching" only considers the distributional differences we observe in the output space (the logits), which also gives me concern that the detection can work in the feature space or the latent space. Or, if the attack matches all the distributions, the trojans can be easily removed by just fine-tuning a model.

I like that the evaluation has been conducted with a large number of models (Kudos to the authors), but the evaluation misses several other backdoor defense mechanisms that we even don't need any detection.

I further found that the trigger reconstruction mechanisms have not been evaluated with some advanced techniques or properly, which gives me concern that claiming that "trojan detection is hard" is a bit early.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A. Paper summary

- This paper proposes a new trojan method by designing evasive trojans. The key idea is to fine-tune a clean model by leveraging Wasserstein distance to minimize the output logits (between clean output / poisoned outputs) and using l2 distance to regulate the parameter distance (between the clean model and trojan model). Other tricks are provided (e.g., adding random loss) to further reduce the detection rate.

- The result shows it can reduce the detection performance to near-chance levels.

strength_and_weaknesses: 
B. Strength And Weaknesses

* Strength: 
1. The paper is clear and well-written. 
2. The novelty is straightforward to understand.
3. Motivation is strong.

* Weaknesses:
1. The analysis is not enough (and not exciting). Some ablation studies are not provided.
clarity,_quality,_novelty_and_reproducibility: Mentioned in the strength above.
summary_of_the_review: C. Questions:
- In your evaluation, you only show the final detection success rate from different detectors. However, your key motivation is to minimize the output logits/weight difference. Can you show the result that your method is effective in reducing those distances? It would be more interesting if you can show a plot between those distances v.s detection accuracy.

- Following the above questions, why are you specifically choosing the output logits? What if you added the penultimate layer's output?

- In the "randomization" section, you mentioned some empirical results that are not provided in the evaluation. I believe Figure 3 is your motivation; so how about its (i.e., the randomization loss) effectiveness in reducing the detection rate?

I have read many detector papers that are using the distribution of the final layer to detect trojan models. This paper uses Wasserstein distance to minimize this distribution gap between poisoned output logits and clean output logits which is a new challenge to the existing detection methods.

Overall, I think the paper is very clear and highly motivated. However, some ablation studies are missing and I don't think the analysis is exciting. There are a lot of ways to dive deep into the problem and to provide more insights. 

Minor: Figure 2 is redundant with Table 2. Your figures and tables are too big.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper uses distribution matching and randomization to reduce Trojan specificity. The author proposes to use 1-Wasserstein distance for the design of the loss function. The authors conduct experiments on MNIST, CIFAR-10 and CIFAR-100, show that the method proposed by the authors can improve the difficulty of Trojans being detected by detectors, and can be applied to various types of Trojans. Experiments also show that the author's proposed method can make Trojans more difficult to reverse engineer.
strength_and_weaknesses: Strengths:
1. The author takes the evasion of Trojans as one of the goals and designs a loss function with a clear structure.
2. The author uses 1-Wasserstein distance as a metric, and experiments show that it has a good effect, effectively improving the difficulty of detecting Trojan attacks.
3. The author designed a relatively sufficient experiment, including the performance of the evasion Trojan under different detectors. The results show that the evasion Trojan is harder to detect, its target label is harder to predict, and more difficult to reverse engineer.
 
Weaknesses:
1. As one of the innovations of this work, the use of 1-Wasserstein distance in the loss function is not elaborated. How the author approximates the infimum also has no further clarification.
2. As the core of the paper, the three-stage loss function designed by the author does not give a theoretical analysis, and the heuristic design of the loss function makes the persuasiveness decrease.
3. I didn't find any text mentioning Figure 1 and Table 2, besides, the experimental results in Table 2 show that the standard Trojan performs even better than the evasion Trojan under the Param detector, which the author does not explain. The figure layout of the entire paper needs to be further optimized.
clarity,_quality,_novelty_and_reproducibility: Original work
summary_of_the_review: This work attempts to make Trojans in models harder to detect, which makes sense. I hope the author can elaborate on the design of the loss function in Section 4.1, the use of 1-Wasserstein distance. In addition, the authors need to make a more complete analysis of the experimental results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to train backdoored models that are stealthier (i.e., they are close to the distribution of clean models). Standard backdoor detection algorithms have less success against these models and they are also more difficult to reverse engineer.
strength_and_weaknesses: + Specificity is understudied in backdoor attacks and the proposed attack considers to improve specificity.

- Very problematic evaluation and baseline attacks,
clarity,_quality,_novelty_and_reproducibility: There has been plenty of work on evasive trojans that essentially has the same aim as this paper. Most of these papers propose similar loss functions so it's hard to say the paper is original.

Authors did not share their source code so no reproducibility. 

It's generally well written but the claims are weakly supported.
summary_of_the_review: My biggest concern about this paper is the evaluation. First of all, the proposed attack is a supply-chain attack where the attacker has full control over the resulting model. Evasive attacks within this threat model is extensively studied, some examples are [1,2,3]. None of these works are considered in evaluation as baselines. The baseline attacks (Patch - Blend) are vanilla attacks that are shown time and time again easy to detect. Moreover, these baseline attacks are poisoning attacks (not supply chain) which is a significantly more difficult attack to perform. All in all, it is not clear whether this attack brings anything new to the table and whether the claims are true. I would recommend the authors to spend more energy into surveying the SOTA in attacks and defenses, think more carefully about their threat model and perform a more comprehensive evaluation.

Second, there has been more advanced published backdoor defenses the paper has not evaluated. These defenses claim to perform better than old defenses such as NC or ABC. For example: [4,5]

That being said, I like the idea of specifically improving the specificity of the backdoor via loss function. I think this is also a good tool to understand the limits of specificity (e.g., how specific can we make our backdoor, considering the inevitable side effects [6]?)

[1] https://arxiv.org/abs/2002.12200
[2] https://www.usenix.org/conference/usenixsecurity21/presentation/tang-di
[3] https://openaccess.thecvf.com/content/ICCV2021/papers/Li_Invisible_Backdoor_Attack_With_Sample-Specific_Triggers_ICCV_2021_paper.pdf
[4] https://arxiv.org/abs/2102.05123
[5] https://www.cs.purdue.edu/homes/taog/docs/CVPR22_Tao.pdf
[6] https://arxiv.org/abs/2010.09080
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposed a backdoor attack method with evasive trojans, to evade detection based defenses, utilizing distribution-matching, specificity, and randomization to eliminate distinguishing features of trojaned networks. 

There are several important concerns from reviewers:
1. The idea of evasive trojans are not new, as there are several existing works with the similar idea. 
2. The evaluation and literature review are inadequate. For example, several advanced attack methods which also controlled the training process are not compared. 
3. The design of the proposed loss functions are not well motivated. 

Overall, the current shape of this work is not qualified for ICLR. 
justification_for_why_not_higher_score: see above
Decision: Reject