Using Interleaved Ensemble Unlearning to Keep Backdoors at Bay for Finetuning Vision Transformers

Authors: Zeyu Michael Li
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Vision Transformer, Backdoor Defence

Abstract:
Vision Transformers (ViTs) have become popular in computer vision tasks. Backdoor attacks, which trigger undesirable behaviours in models during inference, threaten ViTs' performance, particularly in security-sensitive tasks. Although backdoor defences have been developed for Convolutional Neural Networks (CNNs), they are less effective for ViTs, and defences tailored to ViTs are scarce. To address this, we present Interleaved Ensemble Unlearning (IEU), a method for finetuning clean ViTs on backdoored datasets. In stage 1, a shallow ViT is finetuned to have high confidence on backdoored data and low confidence on clean data. In stage 2, the shallow ViT acts as a "gate" to block potentially poisoned data from the defended ViT. This data is added to an unlearn set and asynchronously unlearnt via gradient ascent. We demonstrate IEU's effectiveness on three datasets against 11 state-of-the-art backdoor attacks and show its versatility by applying it to different model architectures.

Decision: reject
Reviewer scores: 5, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1730109432820

thread_type:
official_review

<thread_object>
creator: reviewer_neex

<content>
summary:
Vision Transformers (ViTs) are increasingly used in computer vision tasks but are vulnerable to backdoor attacks that can compromise their performance, especially in security-sensitive applications. Existing backdoor defenses for Convolutional Neural Networks (CNNs) are not as effective for ViTs, and tailored solutions are limited.

To address this, the paper proposes Interleaved Ensemble Unlearning (IEU), a method for fine-tuning clean ViTs on backdoored datasets. In the first stage, a shallow ViT is fine-tuned to exhibit high confidence on backdoored data while maintaining low confidence on clean data. In the second stage, this shallow ViT serves as a "gate" to filter out potentially poisoned data from the defended ViT, which is then added to an unlearn set and asynchronously unlearned via gradient ascent.

The paper demonstrates IEU's effectiveness across three datasets against 11 state-of-the-art backdoor attacks and highlight its versatility by applying it to various model architectures.

strengths:
- This paper explores a method for finetuning clean ViTs on backdoored datasets, Interleaved Ensemble Unlearning (IEU), giving a good reference to the research on this aspect.

- The proposed method IEU is simple but effective to achieve the defense, and the good performance obtained by the experiments strongly supports this point.

- The ablation study is organized well to clearly demonstrate the whole proposed method. And it makes the paper easy to follow.

weaknesses:
- I have some concerns regarding **the fairness of comparison with previous defense methods,** particularly due to the inclusion of the additional poisoned module, which involves an extra network for defense.

- The authors propose an effective framework of backdoor defense on ViT. It would be beneficial to consider a more realistic scenerio, where the attacker knows the existence of IEU and can generate a poisoned dataset to perform adaptive attacks. **It would be wonderful if the authors can design an adaptive attack for IEU and provide some experimental results.**

- I recommend conducting further experiments to assess whether BSD can successfully defend against backdoor attacks **with different target labels.** This could provide valuable insights into the robustness of the defense mechanism across various scenarios.

- There are existing backdoor defenses that focus on training clean models using poisoned datasets [1, 2], which could provide useful context for this research.

[1] Backdoor Defense via Adaptively Splitting Poisoned Dataset. CVPR, 2023.

[2] Backdoor Defense via Deconfounded Representation Learning. CVPR, 2023.

questions:
Listed in the weakness of the paper.

Score can be improved if concerns listed above are resolved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730149633035

thread_type:
official_review

<thread_object>
creator: reviewer_jxzc

<content>
summary:
The paper presents Interleaved Ensemble Unlearning (IEU), a method aimed at defending Vision Transformers (ViTs) from backdoor attacks during fine-tuning on backdoored datasets. IEU employs a two-stage approach, using a shallow "poisoned module" ViT to filter potentially poisoned data and a "robust module" to learn clean data. In stage one, the poisoned module is trained to confidently predict backdoor-labeled data, while stage two involves using this module to identify and remove potentially backdoored data. This dynamic unlearning technique is tested on multiple datasets and against various attacks, showcasing IEU’s ability to improve attack success rate (ASR) and clean accuracy (CA).

strengths:
1. The paper proposes a unique ensemble-based strategy specifically tailored for ViTs, addressing the scarcity of ViT-targeted backdoor defenses.

2. IEU is evaluated on diverse datasets (CIFAR10, GTSRB, TinyImageNet) and multiple backdoor attacks, providing extensive empirical data on performance improvements.

weaknesses:
1. The presentation quality falls significantly short of the standards expected at a prestigious conference like ICLR. Readers without a strong background in vision transformers and backdoor attacks will likely struggle to follow the manuscript. Key background information, such as the concept of "unlearning", is notably absent. Additionally, Figure 1 presents mathematical notations to illustrate the proposed method, yet these notations lack sufficient explanation. For example, the symbol $\mathcal{D}^{ul}$ is used without clarifying its meaning.

2. Although the paper claims that IEU enhances ViT robustness against backdoor attacks, it lacks clarity on which specific design elements are tailored to address vulnerabilities unique to the ViT architecture.

3. This paper lacks novelty. The ensemble strategy primarily relies on heuristic thresholds rather than introducing innovative designs. Additionally, Stage 1 does not present any novel contributions.

questions:
1. The proposed method heavily relies on the prediction of the poisoned module. What if the poisoned module itself is attacked by malicious users? How can the proposed method address this scenario?

2. The proposed method appears to be sensitive to the heuristic threshold $c_{thresh}$. How do the authors determine the value of the threshold?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720143149

thread_type:
official_review

<thread_object>
creator: reviewer_5n2c

<content>
summary:
This paper introduces Interleaved Ensemble Unlearning (IEU), a defense strategy designed to protect Vision Transformers (ViTs) (or any model in general) from backdoor attacks during the fine-tuning process.

The IEU approach uses a shallow ViT model as a 'poisoned module' to detect backdoored data and protect the main 'robust module,' which is fine-tuned on clean data sorted through the 'poisoned module'.
IEU operates in two stages: the first trains the poisoned module on backdoored data, while the second uses it to isolate potentially poisoned samples for unlearning by the robust module. By alternating between learning and unlearning, the IEU method mitigates the effects of poisoned images, maintaining high accuracy on clean data while effectively defending against backdoor attacks. 

Evaluation results on three datasets (CIFAR10, GTSRB, and TinyImageNet) demonstrate its competitive performance against other state-of-the-art backdoor defenses, especially for CIFAR10 and TinyImageNet.

strengths:
- IEU provides a novel approach to mitigating backdoor attacks in ViTs by implementing a layered defense strategy that separates poisoned data using a shallow 'poisoned module'.

- The method’s two-stage process of alternating between learning and unlearning adds robustness against various backdoor attacks without relying on a pre-identified clean set.

- The paper provides comprehensive empirical evaluation against a wide array of backdoor attacks across multiple datasets (CIFAR10, GTSRB, and TinyImageNet), showcasing improvements in ASR reduction and maintenance of high clean accuracy on CIFAR10 and TinyImageNet.

- IEU demonstrates practical value, particularly in domains where ViTs are deployed for security-sensitive tasks, making it a valuable contribution to backdoor defense for ViTs.

- The paper also discusses the method's limitations (e.g., simple dataset, weak attacks, and instability) to provide insights into how to improve the method further in future works.

weaknesses:
- **Limited Explanations for reproducibility:** The article provides limited clarity in the explanations of the technical details, particularly in the specific architectures and configurations used for the 'poisoned module' and 'robust module', respectively. This provides limited insight into understanding the 'poisoned module,' which is the most important component of the proposed approach. 

- **Generalizability to Complex Architectures:** It is unclear whether the proposed approach is generalizable to more complex ViT Architectures or is limited to standard ViT architectures. Further, while the approach is proposed specifically for the ViT, in theory, it can be applied to non-ViT-based models, improving the applicability of the proposed defense. 

- **Limited insight on the limitations:** The article highlights the limitations and discusses them, providing insights on why the approach underperforms in certain cases. I am not strongly convinced that this is the only reason for underperformance (though it is minimal underperformance). For example, the authors argue that the GTSRB dataset is less complex than the other datasets evaluated, which is why underperformance happens. However, it is a failure case of a 'poisoned module' not to learn well enough to distinguish (as evidenced by Fig 2).
Moreover, the Clean Accuracy is also lower, suggesting that $m_{\theta_p}$, which is directly related to the logits calculated by $f_p$, also affects the performance. 
Further, no insights have been provided as to why the poisoned module learn differently for the GTSRB dataset, nor have any suggestions on improving the training of the 'poisoned module' to prevent these failure cases. While I do not state that the approach should perform better for the GTSRB dataset natively, I think there is an oversight in identifying and discussing why the 'poisoned module' gives high enough logits for the poisoned images, which affects both Clean Accuracy as well as ASR directly.

questions:
- Could the authors elaborate on the architectural choices for the shallow ViT used as the poisoned module?

- How does IEU perform on deeper ViT architectures for the 'poisoned module' not covered in the study?

I notice some sharp drops in ASR in Table 2. For Example, for ISSBA-CIFAR-10, the ASR is 100.00 for 0.2 and 0.0 for 0.5. Similar results can be for Smooth and Tiny ImageNet. Can the authors provide some insights on the sharp drops? This could provide better insights for future works to reproduce the results (or even improve).
</content>
</thread_object>

</thread>
<thread>
creation date:
1731149078364

thread_type:
official_review

<thread_object>
creator: reviewer_jlmi

<content>
summary:
This paper presented Interleaved Ensemble Unlearning (IEU), a method for finetuning
clean ViTs on backdoored datasets. IEU includes two stages, where the first one is designed to train a shallow ViT used to block potentially poisoned data and the second stage defends backdoor attacks utilizing unlearning. The experiments demonstrate that IEU out-performs existing defenses on diverse datasets and backdoor attacks.

strengths:
-	The two-stage method is reasonable, especially, using a shallow model to learn shortcuts in the dataset in the first stage. 
-	The experiments show that IEU performs better than existing methods, including I-BAU and ABL. Besides, the ablation studies are well-organized, illustrating the necessity of designs in IEU.

weaknesses:
-	The novelty is limited. The proposed IEU utilizes the unlearning strategy for backdoor defense. Compared to ABL, the main differences lie in using a shallow model to block potentially poisoned data and a confidence threshold to determine the unlearned samples rather than a fixed-sized unlearned set.

-	I am confused as to why IEU is tailored to ViTs. According to my understanding, there is no customized design in IEU for transformer-like architectures. Also, the authors evaluate CNN models in Table 8. Hence, if I am correct, I recommend the authors revise the writing to highlight the universality of the proposed IEU.

-	I suggest that Eq.2 could be further clarified in detail. If the logits for optimizing the objective in Equation 2 sometimes come from $f_p$, should the optimized parameters include $\theta_p$?

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731463140683

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Initial comments and plans

comment:
Based on the reviewers’ insightful suggestions, I aim to make the following modifications to my manuscript:
- I will edit Figure 1 and its caption so that symbols are properly explained.
- I will emphasise my main contributions, which should pinpoint the **interleaved unlearning** framework as the main, novel contribution. The novel process of interleaved unlearning encourages the model to forget backdoor features without leading to a large drop in clean accuracy. This alternating updates (slightly similar to ADMM) towards optimising for two different objectives (unlearning backdoors and learning clean data) is a novel and significant improvement over SOTA methods. 
- I plan to include more well-written background information on backdoor attacks. Currently, the clarity of the first part of my introduction can especially be improved. I will also include a few more papers in my literature review.
- I will de-emphasise my rather flimsy argument that IEU is *specifically* designed for ViTs and will instead stress that its performance on ViTs is better than existing SOTA methods. Additionally, the main contribution is the interleaved unlearning framework and not the distracting and slightly spurious argument that IEU is specifically for ViTs. 
- I will provide more insights on the limitations of my model.
- I will emphasise that the poisoned module is one way of detecting backdoor samples and can be replaced by other methods for backdoor image detection because the novel contribution of my method should *not* be misconstrued as being the poisoned module (it is instead the **interleaved unlearning** framework). Since backdoor image detection methods are well-developed and can isolate backdoored images with high precision, I simulated situations where certain percentages of backdoored samples are detected (see table 2 in section 3.2).
- I will improve the flow of my paper in general and repurpose existing tables to emphasise different messages.

Additionally, I will produce additional results:
- I aim to design an adaptive attack for IEU.
- I will produce results for different target labels (e.g., results where the backdoored class $\neq 1$). 
- I will investigate how (and whether) LGA/gradient flooding (see ABL [1]) can improve the performance of the poisoned module to address the limitations I presented in section 5 of my paper.

A clean version of the IEU code will be provided as a link to an anonymous repository via a non-public post.

I sincerely thank reviewers for dedicating their time to reading and reviewing my project and I am very, very grateful to receive the first round of feedback. I look forward to discussing my paper with all of you and will endeavour to improve my paper!

[1] Anti-Backdoor Learning: Training Clean Models on Poisoned Data
</content>
</thread_object>

</thread>
<thread>
creation date:
1731738886577

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Ablation study: different trigger labels (thanks to reviewer NEex's suggestion)

comment:
|  trigger label  |   ('BadNets-white', 'ASR') |   ('BadNets-white', 'CA') |   ('ISSBA', 'ASR') |   ('ISSBA', 'CA') |   ('Smooth', 'ASR') |   ('Smooth', 'CA') |
|---:|---------------------------:|--------------------------:|-------------------:|------------------:|--------------------:|-------------------:|
|  0 |                       0.71 |                     92.28 |               0.1  |             98.43 |                0.07 |              97.71 |
|  1 |                       0.96 |                     98.19 |               0.33 |             98.35 |                0.09 |              97.77 |
|  3 |                       0.51 |                     97.88 |               0.03 |             98.36 |                0.1  |              93.87 |
|  5 |                       0.37 |                     97.53 |               0    |             98.35 |                0.05 |              97.71 |
|  8 |                       0.19 |                     94.51 |               0    |             98.14 |                0.01 |              97.51 |


The table above shows the ASR and CA (percentage) when using different trigger labels on the CIFAR10 dataset for three different attacks. Note that only a subset of trigger labels are considered due to compute limitations. The trigger label used in the main body of the paper is trigger label == 1. This addresses reviewer NEex's recommendation to conduct "further experiments to assess whether BSD (Backdoor Sample Detection) can successfully defend against backdoor attacks with different target labels". 

The CA is mostly high and the ASR is consistently low for all attacks. This suggests that my IEU is able to defend against attacks with different target labels. 

This table will appear in the appendix of the revised manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731794611678

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
IEU successfully defends ViTs on the GTSRB dataset

comment:
|          |   ('BadNets-white', 'ASR') |   ('BadNets-white', 'CA') |   ('ISSBA', 'ASR') |   ('ISSBA', 'CA') |   ('Smooth', 'ASR') |   ('Smooth', 'CA') |
|:---------|---------------------------:|--------------------------:|-------------------:|------------------:|--------------------:|-------------------:|
| Flooding |                   2.20111  |                   85.3761 |            2.69992 |           86.9913 |            20.3642  |            87.8622 |
| LGA      |                   2.21694  |                   83.2621 |            2.81077 |           86.4766 |            15.3682  |            88.4481 |
| Neither  |                   0.926366 |                   82.0744 |          100       |           93.5234 |             6.15202 |            84.521  |

The above table shows the performance of IEU on GTSRB when using Loss Flooding, local gradient ascent (LGA), or neither ([1]) when tuning the poisoned module in stage 1. The Flooding/LGA threshold parameter $\gamma$ is set to 1. Note that $c_{thresh} = 0.9$ is used instead of $c_{thresh} = 0.95$. The stage 1 settings for GTSRB follow CIFAR10/TinyImageNet (note that GTSRB settings in this table and the GTSRB settings I'm planning to use in the revised manuscript are different from the GTSRB settings I used in the pre-rebuttal's manuscript: for GTSRB pre-rebuttal, I used 1e-3 as the prefinetune_lr and only tuned the poisoned module for 5 epochs; here, I'm using 2e-4 as the prefinetune_lr and tuned the poisoned module for 10 epochs, meaning that all settings *except* for $c_{thresh}$ for GTSRB are identical to CIFAR10/TinyImageNet). Loss Flooding and LGA are designed to induce the poisoned module to learn shortcuts. 

Inspired by reviewer 5N2c's comments about IEU's performance on GTSRB, I wanted to see whether LGA/Flooding would improve the model's performance on GTSRB. It turns out that both LGA and Flooding allow the poisoned module to be successfully backdoored without learning much benign features when used separately (meaning that I do not apply both LGA and Flooding in the same experiment) to tune stage 1's poisoned module for GTSRB.

Since I can use Flooding/LGA to tune a defended ViT in stage 2 when using GTSRB, (most) **weaknesses in the limitations section (sec 5 in the pre-rebuttal paper) have been addressed** and I will update the text and tables in my revised manuscript accordingly. 

[1] Anti-Backdoor Learning: Training Clean Models on Poisoned Data
</content>
</thread_object>

</thread>
<thread>
creation date:
1732062993895

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
IEU successfully defends against an adaptive attack

comment:
**In response to the comments from reviewers JxzC and NEex**, I have implemented an adaptive attack designed to induce the poisoned module to mis-predict. 


|                     |   ('IEU', 'ASR') |   ('IEU', 'CA') |   ('No Defense', 'ASR') |   ('No Defense', 'CA') |
|:--------------------|-----------------:|----------------:|------------------------:|-----------------------:|
| CAT+BadNets-pattern |             0    |           98.27 |                  100    |                  97.77 |
| CAT+BadNets-white   |             0.74 |           94.9  |                   94.72 |                  98.31 |
| CAT+Blended         |             0    |           98.28 |                   99.96 |                  98.38 |
| CAT+Smooth          |             0.1  |           97.87 |                   98.88 |                  98.33 |
| CAT+Trojan-SQ       |             0.02 |           98.25 |                   99.66 |                  98.44 |

This [1] adaptive attack called "CAT" is used and the table above shows the results of this attack when the model trainer uses IEU (left two columns) and when the model trainer uses no defence (right two columns) on CIFAR10. IEU **demonstrates high performance (low ASR and high CA) on all attacks+CAT** except for Blended+CAT, where the CA using IEU is $\sim 8$ percentage points lower than the CA without a defence. All settings used for these experiments are the same as the settings I used in the paper (CIFAR10 parameters). 

The gist is that CAT optimises the backdoored images so that they create channel activations that are more similar to those found in normal benign images, meaning that CAT is applied on existing backdoor attacks (e.g., BadNets-white) to find stealthier triggers. CAT is evaluated specifically on Vision Transformers in [1] and demonstrates good performance. 

This is an adaptive attack for my IEU defense because the CAT perturbations aim to make the poisoned module misclassify the poisoned images by encouraging channel activations for benign and backdoored images to be similar, which aims to cause the poisoned module to misclassify images.

Since the authors of [1] have not released their code, I implemented their attack from scratch under a gray-box scenario, where the attacker has access to a tuned version of the poisoned module. First, the attacker trains two classifiers (the backdoor discriminator and the target classifier) using the features generated by the poisoned module (specifically, the output of the MLP layer before the classification head in my shallow poisoned module). Then, these two classifiers (I used shallow ViTs with depth=4 as the two classifiers) along with the poisoned module are used to generate backdoored data with adversarial perturbations applied onto the backdoor data via projected gradient descent (e.g., some $\delta'$ is added onto the backdoored image $x+\delta$). Then, the backdoored images with CAT adversarial perturbations are used to train a defended ViT using IEU from stage 1. Since stage 1 and stage 2 are bundled together, I train a new poisoned module on the backdoored data generated from CAT. 

[1] Towards Reliable Backdoor Attacks on Vision Transformers
</content>
</thread_object>

</thread>
<thread>
creation date:
1732343268419

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Uploaded new version of the paper

comment:
Dear Reviewers and AC, 

Thank you all for the reviews. They've been very helpful for me when trying to improve the paper's writing and completeness.

I've uploaded a new version of the paper which, in tandem with my other responses, should address all concerns by reviewers. 

Specifically, I address the following concerns:
- Reviewers **JLmi** and **JxzC**: concern about the novelty of IEU. 
  - The updated manuscript shifts the focus to the novel Interleaved Unlearning framework as my main, novel contribution. 
- Reviewer **JxzC**: concern about presentation and clarity.
  - The related work section how includes a paragraph about unlearning. In addition, the start of the Backdoor Attacks paragraph in the related work section has been edited to include a better introduction for backdoor attacks. 
  - Figure 1 has been updated to include explanations of various expressions.
- To address the reviewers’ concerns regarding whether the IEU is specifically designed for ViTs, I have modified the manuscript to replace this claim with a claim that indicates “IEU demonstrates superior performance on ViTs”. 
- Reviewer **NEex**: more experiments related to performance on different target labels
  - The relevant experiment has been added to Appendix C (Table 17). 
- Reviewr **NEex** and **JxzC**: an adaptive attack (what happens when the poisoned module is attacked?)
  - Appendix D demonstrates that my IEU is robust to an adaptive attack.
- Reviewer **NEex**: useful papers to cite.
  - I have included the two papers in the related works that reviewer NEex suggested. 

Additionally, the revised manuscript has been updated to include newest experimental data (specifically, tuning the poisoned module with LGA for GTSRB in table 3, table 15, and table 16). The Discussions and Limitations section has also been revised to reflect the performance of IEU on GTSRB when LGA is used. I have edited the methods section to be more readable.

I look forward to engaging with reviewers through interactive discussions!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732400163031

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Strengths of this submission

comment:
As we approach the end of the discussion period, I would like to emphasise and summarise this submission's strengths. 

- All reviewers agree that **strong experimental results** and the ablation study have demonstrated that IEU performs well when defending ViTs. 
  - Reviewer **JLmi** praises the ablation study for being “well-organised” and asserts that “the experiments show that [my method] IEU performs better than existing methods”. 
  - Reviewer **5N2c** highlights the experimental section for its “comprehensive empirical evaluation”.  
  - Reviewer **NEex** provides positive comments regarding my ablation study, writing that it is “organised well to clearly demonstrate the whole proposed method”. Additionally, reviewer NEex indicates that the “good performance obtained by the experiments” further supports the effectiveness of IEU. 
  - Reviewer **JxzC** indicates that the paper provides “extensive empirical data on performance improvements”. 
- Reviewer **5N2c** notes the **novel** approach that IEU takes and highlighted the fact that IEU does not rely on a clean holdout dataset. My belief is that the Interleaved Unlearning framework, which is the main contribution of this submission, is a novel approach to backdoor defence and the updated manuscript reflects this. 
- Reviewers **NEex** and **JxzC** point out that this submission **fills a gap in existing literature**. Reviewer **NEex** indicates that “tailored solutions are limited” (indicating that ViTs do need tailored backdoor defences). Furthermore, reviewer **JxzC** writes that my IEU “address[es] the scarcity of ViT-targeted backdoor defences”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732946576641

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reminder to respond to my rebuttal :)

comment:
Hi all and apologies for pinging you again, 

While I can't say this rebuttal was built on blood, sweat, and tears, it definitely took a lot of effort and time to design new experiments, implement the adaptive attack, and refine previous results.

It would be great if reviewers could at least confirm that they've had a chance to review my rebuttal. 

Since we still have a few days left in the discussion period, additional comments, feedback, or score changes would all mean a great deal to me. I hope my experiments and previous comments have brought more clarity to the IEU framework!

Thanks!
</content>
</thread_object>

</thread>


Metareview: