Few-shot Backdoor Attacks via Neural Tangent Kernels
Authors: Jonathan Hayase
Sewoong Oh
Venue: ICLR 2023 poster (2023)
Keywords: kernel regression
backdoor attack
data poisoning
robust machine learning
neural tangent kernel

Abstract: In a backdoor attack, an attacker injects corrupted examples into the training set. The goal of the attacker is to cause the final trained model to predict the attacker's desired target label when a predefined trigger is added to test inputs. Central to these attacks is the trade-off between the success rate of the attack and the number of corrupted training examples injected. We pose this attack as a novel bilevel optimization problem: construct strong poison examples that maximize the attack success rate of the trained model. We use neural tangent kernels to approximate the training dynamics of the model being attacked and automatically learn strong poison examples. We experiment on subclasses of CIFAR-10 and ImageNet with WideResNet-34 and ConvNeXt architectures on periodic and patch trigger attacks and show that NTBA-designed poisoned examples achieve, for example, an attack success rate of  90% with ten times smaller number of poison examples injected compared to the baseline. We provided an interpretation of the NTBA-designed attacks using the analysis of kernel linear regression. We further demonstrate a vulnerability in overparametrized deep neural networks, which is revealed by the shape of the neural tangent kernel.



Reviews:
[{"review_id": "mS5sTgK6qyl", "reviewer": "Reviewer_iX6N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to reduce the number of poisoned samples (i.e., few-shot) in backdoor attacks. The paper formulates backdoor-attacks as a bi-level optimization problem, which is approximated with a NTK objective to trace training dynamics. The paper also proposes a greedy algorithm to gradually reduce the number of poisoned samples. Experiment results show: (1) the proposed method makes attacks more efficient; (2) it is enough to know partial training data; (3) the attacks on other kernels; (4) deeper understanding of backdoors from the perspective of NTK.", "summary_of_the_review": "Technically speaking, the paper tackles the challenge of tracing model training process with NTK methods, which is used for more efficient backdoor attacks. Some interesting analysis is conducted. However, there are several major problems for this paper. First, I am not fully convinced by the motivation of \"few-shot backdoor attacks\". Second, the experiment design has some flaws, especially the defense method based evaluation is missing. Third, the paper presentation is not self-contained. Based on these reasons, I think the paper is slightly below the acceptance threshold.", "strength_and_weaknesses": "Strengths:\n(1) The idea of using NTK to approximate the training dynamics is interesting in backdoor attacks.\n(2) The paper provides deeper analysis of backdoor attacks from the NTK perspective.\n\nWeakness:\n(1) The introduction of the proposed method is not self-contained. Too many details are given to the theoretical analysis, while it is actually not very clear to me how exactly the attack is conducted. The three items in Page 3 are far from being enough. I do not think it is a good idea to put these contents in Appendix.\n(2) I have doubts whether \"few-shot backdoor attack\" it is a valid problem. I do not think human beings will check each training sample. Automated detection methods will be applied, where the poisoned set size is not important (anyway the detector will go through each of the samples).\n(3) Experiments: First, I feel Section 3.2 is not necessary. Having 100% of training samples or 50% does not matter that much. Second, in Sec 4, what does \"strong attack\" or \"attack strength\" mean? Does it refer to the backdoor sample with more clear trigger pattern or more stealthy pattern? Third, a flaw of experiment is that no defense method is applied to evaluate the effectiveness of the proposed backdoor attack method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxPX9vcpSr", "reviewer": "Reviewer_qvbc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a theoretical analysis of backdoor attacks from the perspective of neural tangent kernels (NTK). It uncovers that NTK is more prone to backdoor attacks than other kernels, e.g., the Laplace kernel. A new poisoning method, NTBA, is proposed based on the analysis. NTBA optimizes the poisoning examples, producing stronger backdoors than previous methods. Experiments are performed on subclasses of CIFAR-10 and ImageNet with WideResNet-34 and ConvNeXt with periodic and patch trigger attacks, showing that NTBA only requires few poisoned examples.", "summary_of_the_review": "Paper with novel and thorough analysis of backdoor attacks, which uncovers new insights into the vulnerability of neural networks to poisoning. Principled, effective attack method developed based on the theoretical analysis.", "strength_and_weaknesses": "Strengths:\n- Novel analysis of poisoning attacks linked to neural tangent kernels.\n- Novel poisoning attack strategy, that is principled and based on theoretical analysis, resulting in stronger poisoning capacity.\n- Comprehensive state-of-the-art analysis on poisoning and backdoor attacks and defenses (unfortunately pushed to the appendix).\n- The code for experiments was provided.\n\nWeaknesses:\n- It is not fully clear to me what are the practical implications and applicability of the analysis and NTBA method.\n- A more extensive discussion of assumptions and limitations of the proposed approach would be welcome.\n\nSee more details below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OFdEHo_LPl", "reviewer": "Reviewer_yTWS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new backdoor attack that exploits neural tangent kernels to optimize the perturbations of the poisons introduced in the training set in a more efficient way. The attack comprises of three elements: modeling the training dynamics with the neural tangent kernel, a greedy initialization strategy to select the initial set of poisons and optimizing the perturbations of the poisons. The empirical results provided by the authors show that the proposed attack allows to achieve a high success rate with a few number of poisoning points in the training set when compared to a baseline. ", "summary_of_the_review": "The idea of using neural tangent kernels is interesting and I think that the paper has potential and I really encourage the authors to follow this research direction. However, I think that the paper requires more work on the following points:\n1) The experimental evaluation: it would be necessary to compared to other state-of-the-art attacks, especially those relying on approximate techniques to solve bilevel optimization problems. Similarly, it is necessary to analyze the behavior of the attack against existing defenses and analyze if there is a trade-off between attack effectiveness and detectability. \n2) The organization of the paper (see my previous comments). \n", "strength_and_weaknesses": "Strengths:\n+ The use of neural tangent kernels for solving the bilevel optimization problem to learn the poisons perturbations looks interesting and can help to improve poisoning attacks relying on this methodology. \n+ The authors show how the combination of the three steps in the attack is necessary to achieve a high success rate. \n\nWeaknesses: \n+ The experiments just compare the performance of the proposed attack against a baseline, but it does not compare its performance against other state-of-the-art attacks to provide a more comprehensive view of its benefits and limitations. For instance, the authors could compare with Turner et al. 2019, which also optimize the values of the poisons. Similarly, different strategies for solving bilevel optimization-based attacks could be compared to analyze the possible benefits in the use of the neural tangent kernels. \n+ The paper is difficult to follow as it often refers to information in the appendix that is important to understand the paper. In this sense, the authors, perhaps, abuse on the use of the appendices and could rethink of a better organization of the paper to make it more readable. \n+ Similar to the previous point, the authors do not provide many details about the attack in Section 2. For instance, the authors just refer to the appendices to clarify the 3 main steps in the attack. On the other side, the derivation of the attack by using equation (2) is not very well justified and detailed: For instance, does this attack provide an exact or an approximate solution to the bilevel optimization problem? How does this differ from other approximate techniques to solve this type of problems? \n+ The attack is not tested against existing defenses. This can be a key point to validate the usefulness of the attack. In this sense, the attack can be very strong compared to the baseline when attacking undefended systems. However, the strength of the attack could make it more detectable. Thus, it is necessary to explore its effectiveness against existing defenses. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster