Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a defense against backdoor data poisoning attacks by leveraging (reinforcing) existing defenses against noisy-label attacks. 
main_review: Pros:
1. The paper is very well written and I really enjoyed reading it. Through a sequence of steps, the authors explained how they arrived at their final formulation and did a nice job in sketching the implication of their theoretical result. 
2. The connection between the noisy-label attack and the backdoor attack has been well explained and later well-exploited to arrive at the formulation and also the AT-based minimax optimization.
3. For evaluation, the authors chose three noisy-label defenses (SPL, PRL, and Bootstrap) and demonstrated the efficacy of their meta-algorithms (SPL-AT, PRL-AT, Bootstrap-AT) in improving the defense accuracy on the CIFAR-10/CIFAR-100 datasets for two different attack settings.

Cons:
1. One of the main issues I have is with the evaluation. The authors only provide a comparison with AT, and the noisy-label defenses, however, they do not provide any evidence as to how their algorithm compares to other poisoning defenses. In the absence of that comparison, it is difficult to assess the benefits of the defense proposed in this paper. 

2. Some insight about which noisy-label algorithm is better suited for the optimization would be beneficial. While the tables (Tables 2 and 3, and the ones with ablation results) do provide defense accuracies but it would be nice if the authors can provide any more insight if one defense is preferred over others. This also goes back to the assumption on the existence of at least one noisy-label defense that the authors mention in Pg. 4 (after Assumption 2). Can we use a mixture of defenses for the inner and outer optimizations?

3. The authors mention that their technique is more beneficial in the large-scale poisoning regime ($\epsilon=0.45$). In this regard, the numbers in Table 2 (for example) when $\epsilon=0.45$ are interesting. Often plain AT provides a strong baseline (better than some of the re-inforced variants). Also, there is no real consistency between which of the three re-inforced defenses would do better for attacks. This links back to my earlier point about providing more insights into the choice of the noisy-label defense.

Clarifications:
1. I could not understand the clean accuracy numbers in the evaluation. Maybe, I am missing something here, but I was expecting that it would correspond to the $\epsilon=0$ setting. Does the 'Standard' column reflect the accuracy over the $(1-\epsilon)$ fraction of non-poisoned data?

2. In Section 4.2, it would be great if the authors can clarify why $\tilde{\mathcal{R}}^{emp}_t$ is not compared with $\mathcal{R}_t$?

3. The approximation from (4) through the Taylor Series expansion on to (5), especially the use of FGSM for (5) can be explained a little better.

4. Typos:
a. "but also can has" -- pg 2 (last sentence)
b. In Sec 4, the dataset is denoted as $(X^{\epsilon}, Y^{\epsilon})$, however, in the previous section they have been defined as  $(X_{\epsilon}, Y_{\epsilon})$. -- pg 4
c. 'rvc' (maybe a latex equation label) is not defined (just above (5)) -- pg 5
summary_of_the_review: Please see the detailed comments in the main review section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces an algorithm to defend against backdoor data poisoning attacks by leveraging noisy-label defense algorithms. Per se, noisy-label algorithms are not capable of defending against backdoors. However, the authors propose to use these techniques in combination with their proposed algorithm for adversarial training to reduce the effectiveness of backdoor data poisoning attacks. 
main_review: The idea of using techniques for dealing with noisy labels for defending against backdoor data poisoning attacks looks interesting and promising. The motivation and the derivation of the algorithm seem reasonable, although the proposed algorithm has some parameters whose selection can be critical for the performance on the clean dataset and the backdoors. 

I appreciate that the authors strived to provide a theoretical justification (Section 4.2) of the proposed algorithm. Thus, the authors propose a theorem that gives an upper bound for the risk given the empirical risk on the poisoned dataset with the clean labels (R_c^emp). The authors argue that the first term in the upper bound can be minimized by using a noisy label algorithm, but it is not clear what is the gap between the risk on the poisoned dataset with the flipped labels when using the noisy label algorithms. Can the authors provide some theoretical guarantee here? Perhaps the authors could shed a bit of light in this sense during the rebuttal. 

My main concerns about the paper are on the experimental evaluation: 
1) The results in Tables 1 and 2 shows that the proposed method has a significant negative impact on the performance of the algorithm on the clean dataset, which limits the applicability of the method. 

2) The algorithms for mitigating the effect of noisy labels have parameters that assume the fraction of noisy labels that are present in the data. The influence of this parameter can be non-negligible and, a priori is not possible to foresee what is the fraction of poisoned training points for the backdoor attack. Similarly, the perturbation limit (\tau) used for the adversarial training in the proposed algorithm can also have an impact both on the performance on the clean dataset and on the reduction of the success of the backdoor attacks. For example, if the value of epsilon used for SPL or PRL is very large (e.g. 0.45) and there is no backdoor attack or the amount of poisons is very low (e.g. 1%) the performance on the clean dataset could be severely affected. Conversely, if epsilon is low and the fraction of poisons is high, the algorithm may not be able to defend against the backdoor attack. Thus, setting the value of the parameters of the algorithm is not trivial. In this sense, the authors should provide a more comprehensive analysis on the impact of the parameters on the accuracy both on the clean and the poison accuracy.   

3) The experimental evaluation lacks comparison with other existing methods to defend against backdoors.

4) The scenarios with high levels of poisoning (e.g. 45%) are not really representative of a backdoor attack, where the objective of the attacker is to keep the performance of the resulting model when evaluated on the clean dataset. For large fractions of poisons, the performance on the clean dataset is severely affected as it can be observed in the tables with the results. 

5) The experiments only consider CIFAR datasets. I think it would be convenient to include experiments in other benchmarks and, ideally, using other model architectures. 

summary_of_the_review: The motivation for the proposed algorithm is interesting and promising. However, the empirical evaluation raises important concerns about the applicability and usefulness of the method to defend against backdoor poisoning attacks. On one side, the drop in performance on regular inputs when using the proposed technique is non-negligible and, on the other hand, the selection of the parameters of this algorithm can also have a significant impact both on the performance on the clean and the poisoned dataset (and it is not clear how to set this parameters properly). The experimental evaluation should be more comprehensive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed work presents a defense algorithm for backdoor poisoning attacks. The authors consider a noisy label setting where the attacker adds a trigger to certain samples of training data and changes their label. This mainly consists of two attacks, BadNets and Blending attack. The authors then come up with an objective to train the network in a noisy label setting such that networks become robust to the poisoning attacks. 
main_review: ### **Strengths**:

— The authors present a novel optimization based approach to defend against poisoning attacks. Theoretical justification is provided to motivate the loss and the effect they have on robustness.

— The proposed approach can be agnostic to the noisy label algorithm used during training, which makes it appealing to use with more advanced methods.

### **Concerns**:

— The experimental section is bit lacking. The proposed approach is not compared with previous defenses for noisy label attacks such as Spectral Signature defense.

— More advanced attacks where clean label is used  HTBA (arXiv:1910.00033 ) and Clean label backdoor attacks by Turner et al. have been presented which make the attack more effective. The authors do not consider such attacks in their settings and it would be interesting to see if the proposed work can be adapted for such scenarios.

— The organization of the work can be improved. The experiment section can include previous works for comparison and ablation studies along with hyperparameter related experiments can be considered in another section. This would make analyzing the experiments much easier.

— For the Blend attack, PRL based method performs poorly compared to SPL method, but PRL-AT outperforms SPL-AT. This is seen in multiple settings. The authors mention that filtering based algorithms are more easily boosted, but it is not intuitively clear why? A better noisy label algorithm should correspond to better performance irrespective of AT. Such a case makes it difficult to identify which part of the algorithm is responsible for the improvement.

— **Minor**: Some typos are present in the submitted version which can be corrected. Some of them include 

"In the meantime, it is non-trival"- "non-trivial"

" we can approximate rvc as the following" - "we can approximate c as the following"

"resnet-32"  - "ResNet-32"

"since these two attacks does not require knowing any information about the model or training procedure" - "since these two attacks do not require any information about the model or training procedure".
summary_of_the_review: The authors presenting an interesting approach towards defending against backdoor attacks involving noisy label algorithm. However, a more thorough comparison with previous defenses and other forms of attacks is required to understand the significance of the method. An optimization based approach towards defending against both clean and noisy label attacks would be interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to apply noisy-label defense algorithms to defend general backdoor attacks. The authors propose a meta-algorithm by adding noisy labels to the training set and optimizing outer minimization and inner maximization of the model by the noisy-label algorithm. Combined with adversarial training, the authors show their method could improve the robustness of models against backdoor attack.
main_review: Pros:
By adding noisy label to the training set and performing adversarial training at the same time, the noisy-label defense algorithm can be directly utilized to defense backdoor attacks. The method proposed is simple yet effective. The proof procedure is sufficient and details are provided.

Cons:
1. In Table 2, it is unclear why Bootstrap-AT not perform well with respect to the improvement on the model robustness on CIFAR10 with Patch Attack. The authors did not provide any discussion.

2. The computational cost of the method would be high since adding noisy label together with adversarial training takes much longer time. I suggest to provide a comparison and discussion on the times with other methods against backdoor attacks.

3. The organization of this paper should be improved. The analysis on experiments is short and incomplete. 

4. There is a typo in your ‘Proof of Equation 3’. The range of the ∑ on the right-hand side of the first equation in the derivation should be i∈G\R.

summary_of_the_review: The authors theoretically proved that noisy-label defense algorithms can be leveraged to against backdoor attacks, but the method seems not very innovative, and the computational cost would be high compared with other methods.
review_title: null

Meta-review: The paper proposes a new defense against backdoor attacks utilizing an improved version of defenses against noisy label attacks. The connection between these two problems is interesting and novel, which is acknowledge by all reviewers. The main drawback of the paper is, however, its experimental evaluation. The experiments are carried out only on one benchmark and the considered attack ratios are indicative for indiscriminative poisoning attacks rather than targeted backdoors. The AC addressed the summarized response and is not convinced that the reviews are biased. Despite the scarce response of the reviewers to the author rebuttal, the limitations of the experimental evaluation seem to persist in the revised version of the paper. While acknowledging the novelty and the overall good quality of the paper, the weakness of its experimental evaluation puts at in the position marginally below the acceptance threshold. The AC encourages the authors to revise the paper and improve on the pointed out weaknesses and is confident that this work will be well accepted by the scientific community.
Decision: Reject