Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the backdoor adversarial attack. It theoretically discuss the existence of an attacking counterpart model and corresponding design an attacking algorithm. The proposed method is evaluated on benchmark datasets.
strength_and_weaknesses: # Strength
1. Investigating backdoor attacks under adversarial training is relatively new.
2. The theoretical results seems interesting.


# Weakness
1. The claim that "existing backdoor attacks cannot resolve adversarial training" is questionable. The empirical evidences provided by the author do not fully support this claim.
2. The improvement of the proposed method is relatively marginal.
clarity,_quality,_novelty_and_reproducibility: This paper is a little bit hard to follow considering the usage of many informal expressions. The quality and novelty are relatively good. I did not check the reproducibility.
summary_of_the_review: As in Strength vs Weakness
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigate the existence of trojan model for backdoor attack --- a model that is in theory similar to the Bayes optimal yet performs poorly under some universal backdoor trigger. The paper formalizes its notion, proves its conditional existence and show an attack algorithm that generates universal backdoor triggers for real-world tasks.  
strength_and_weaknesses: **Strength**

The paper is technically sound. It provides both theoretically and empirical insights to the problem of backdoor attack.

**Weakness**

I have some questions regarding the technical details of the paper.

1) The theoretical bounds in Theorem 1 and Proposition 1 are somewhat disjoint from the experiment. The experiment checks the existence of universal backdoor trigger, but does not directly check the coefficient of these bounds. While finding the coefficient may be hard for real data, could you provide a toy example for, say, two Gaussian distributions and give the audience a sense of how big the coefficients are?

2) In Figure 2, the horizontal axis is the filter pruning ratio. Could you provide some references on how the pruned filters are selected? Does the curve shape in Fig 2 still hold if a different set of filters are pruned?

3) In Table 3, the mean AIV of your method is >2 against ABS. Why do you use a t-test to show the evasiveness instead of removing all training examples with AIV>2 and then retrain the model on the remaining samples?
clarity,_quality,_novelty_and_reproducibility: This paper is clear in writing. The results are novel as far as I know. Relation with previous work is also adequately explained.
summary_of_the_review: Overall, this paper provides solid theoretical and empirical results for investigating the existence of trojan twin model and universal backdoor trigger. I'm giving a weak accept for now, and willing to raise once my questions are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to backdoor a neural network so that it classifies certain samples containing a trigger as the attacker intended. The proposed approach assumes to have a set of surrogate models to generate a universal perturbation as done in an evasion attack, and take the universal perturbation as a backdoor. The approach is empirically compared to baselines.
strength_and_weaknesses: Strengths.
- S1: The paper tries to prove the existence of a backdoored model.
- S2: The proposed approach is compared to 5 baselines on three datasets.

Weaknesses.
- W1: The empirical performance doesn't show significant improvement. Actually, the performance is similar or worse than some of the existing approaches.
- W2: Neural networks are known to be a universal function approximator, and thus the existence of such a model doesn't seem to add much value, especially with adversarial examples with universal perturbation.
- W3: The approach is already known in both backdoor and evasion scenarios. Also, this type of optimization is commonly used in many attacks, and the approach is a direct adoption of universal perturbation, which has been also used for poisoning. Among many, a quick search shows "Detecting Backdoor in Deep Neural Networks via Intentional Adversarial Perturbations," Xue et al., 2021; "Hidden Trigger Backdoor Attacks," Saha et al., 2020; "Invisible and Efficient Backdoor Attacks for Compressed Deep Neural Networks," Phan et al., 2022; "AdvDoor: Adversarial Backdoor Attack of Deep Learning System," Zhang et al., 2021.
- W4: The experimental setup is unclear, and there are some modifications to common tasks which makes it harder to compare. For example, the proposed attack uses a set of clean models which is not specified, and can be used as an unfair advantage of the proposed attack, especially if the model has the same architecture or is trained on the same dataset as the victim model.
clarity,_quality,_novelty_and_reproducibility: The paper is in general easy to read for a person with a background on the ML backdoor problem, but there are some typos (e.g., "is very closed to"), and missing information (e.g., clean model set, data normalization to understand "L2 distance less than 10"). Due to the missing information, reproducibility is low despite the simple algorithm.
summary_of_the_review: This paper revisits a known property of the backdoor problem. The empirical evaluation is unclear, and the improvement over the existing approaches is not significant. The model set is not explained which is very important in understanding the advantage and limitations of the proposed approach. The related work considers some of the existing optimization-based approaches model-based (the attacker has complete control of the training pipeline and deliver the poisoned model), which I believe is an oversimplification, as many optimization-based approaches use a surrogate model similar to the clean model set used in this paper to generate a poisoned training dataset. In summary, this paper does not seem to show a clear advantage of the proposed approach, has an unclear experimental setup, and misrepresent existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the concept of Trojaned Twin Model, which provides an interesting view w.r.t. the clean model and its trojaned counterpart. In practice, a universal poisoning perturbation is craft on an ensemble of adversarially trained models, and imposed onto the training set for poisoning. Empirical evaluation are done on CIFAR-10, GTSRB and ImageNet.
strength_and_weaknesses: Strength:
- The idea of Trojaned Twin Model is interesting, and Definition 2 makes sense to me.

- The writing is clear and the formulation used in this paper is in the appropriate format.

- The empirical evaluation is done on different datasets, while several baselines are compared.

Weaknesses:
- As to Theorem 1, there is $|1-f^*(x_{bad}+v)-f^*(x_{bad})|<\epsilon$. For a practical distribution $\mu(x)$, if $\epsilon$ is small (satisfying the definition of Trojan Twin), then $x_{bad}$ is usually located at low-probability regions. So consequently, $\mu(\mathcal{B}(x_{bad},\epsilon/(4\beta)))$, i.e., the ratio of successful trojan $\rho$ would be quite small. Besides, practical model family $\mathcal{F}$ usually has a large value of Lipschitz constant $\beta$, especially for deep networks.

- $|1-f(x+v)-f(x)|<\epsilon$ is not a good criterion for poisoning, for example, $f(x+v)=f(x)=0.5$ corresponds to $\epsilon=0$, but this case is not considered as a successful poisoning.

- In Section 3.4, actually, I do not see any close connection between the concept of Trojan Twin and the proposed Algorithm 1. For me, Algorithm 1 is just constructing universal poisoning perturbation on an ensemble of adversarially trained models, which is nothing new beyond [1]+[2].

- Why the poisoning trigger in Algorithm 1 is preferred to be universal, rather than input-dependent? Besides, do the baselines in experiments exploit the same knowledge from adversarially trained models as done in the proposed method?


Refs:

[1] Universal Adversarial Perturbations. CVPR 2017

[2] Ensemble Adversarial Training: Attacks and Defenses. ICLR 2018
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper are good, and I admire the novelty of the idea of Trojaned Twin Model. The reproducibility is not check because no code is provided.
summary_of_the_review: I like the idea of Trojaned Twin Model, which is interesting and formally defined. However, the bound in Theorem 1 seems impractical, and the connection between the theoretical analyses and the proposed Algorithm 1 is loose.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel concept of trojaned twin model to characterize the clean model and the model that is being back-doored. They then use a universal perturbation as a trigger for real-world tasks. Despite proposing an interesting concept, the empirical results of this paper are weak as pointed out by several reviewers. We therefore decide to reject the paper and encourage the authors to improve the empirical section in the future submission. 
justification_for_why_not_higher_score: The improvements in the experiments are quite marginal.
Decision: Reject