CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning

## Abstract

Abstract Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data.
In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained T ext A lignment C leaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02% and 63.88%, respectively.

## 1 Introduction

In the field of artificial intelligence, contrastive learning serves as a powerful learning paradigm aimed at comparing different representations of data, thereby bringing similar samples closer together in the embedding space while pushing dissimilar samples further apart Chen et al. (); Khosla et al. (); Gutmann and Hyvärinen (). This learning method has achieved significant success in many domains, particularly in natural language processing Gao et al. (); Chen et al. () and computer vision Bi et al. (); Park et al. (). In addition to its application in single-modal data, recent works have extended contrastive learning to multimodal data Zhang et al. (); Singh et al. (); Yang et al. (), training on a vast scale of image-text pairs from the web to achieve joint feature representation and matching between images and text. Pre-trained large models, like CLIP Radford et al. (), ALIGN Chen et al. (), and BASIC Chen et al. (), have learned universal representations from large-scale unlabeled data and performed exceptionally well even without task-specific data, as demonstrated by their impressive zero-shot classification performance on ImageNet Deng et al. (). In addition to its foundational applications, contrastive learning frameworks have also been adapted to enhance model interpretability Chen et al. (), bolster security measures, particularly against adversarial attacks Liang et al. (, ); Wei et al. (); Liang et al. (, ); Wang et al. (); Liu et al. (); He et al. (); Liu et al. (); He et al. (); ; Lou et al. (); Kong et al. (); Ma et al. (, , ), and protect privacy Chen et al. (); Liang et al. (); enhancing face obfuscation guided by semantic-aware attribution maps (); Guo et al. (); Dong et al. (). These studies not only broaden the scope of contrastive learning applications but also contribute to increasing the robustness and trustworthiness of AI systems in practical deployments. By fine-tuning these models on specific tasks with a small amount of labeled training samples, high-performance vertical domain applications can be realized quickly.

While multimodal contrastive learning (MCL) exhibits powerful feature alignment capabilities, recent studies have also indicated vulnerabilities of such pre-trained large models to data-poisoning backdoor attacks Gao et al. (); Jia et al. (); Saha et al. (); Carlini and Terzis (); Li et al. (); Liang et al. (); Liu et al. (, ); Liang et al. (, ); Zhang et al. (); Zhu et al. (); Liang et al. (). By injecting a minimal amount of poisoned samples during the training data collection phase, attackers can introduce biases in the feature learning of the model easily, thereby affecting its feature alignment performance and downstream tasks Liang et al. (). In detail, attackers slightly modify images (embedding various forms of triggers) and the corresponding texts to quickly capture the feature linkage between the triggers and certain specified text during model training, disrupting correct image-text feature matching. This leads to malicious outputs guided by triggered images during subsequent zero-shot inference stages and also poses a series of potential risks for downstream model finetuning.

Figure: (a) Synonyms-Synonyms
Refer to caption: x1.png

To our knowledge, faced with a vast training set of unknown security levels, the defense training against models like CLIP focuses on using different kinds of data augmentation strategies to disrupt potential triggers Bansal et al. (); Yang et al. (, ). However, the complete training operations are undoubtedly computationally expensive for both attackers and defenders. Attackers may not only poison the original data but also use dirty data to contaminate pre-trained models. The poisoned model is then posted on the Internet or delivered to users Therefore, another defensive approach worth considering is to utilize a clean, small-scale dataset to fine-tune pre-trained large models, thereby reducing their potential backdoor risks. A classic defense method against the CLIP model, CleanCLIP Bansal et al. (), discovered that poisoned images and clean images can cluster into two sets in the feature space, and poisoned samples establish additional feature correlations with target attack text. They close the distance between benign and poisoned sets by separately performing self-supervised learning on images and text, thus disrupting the "shortcut" established by backdoor attacks in the multimodal feature space.

However, in the supervised learning domain, for unimodal data, fine-tuning strategies can achieve extremely excellent defense performance, e.g., DBD Huang et al. () and NAD Li et al. (). So in the semi/unsupervised domain, for MCL, can fine-tuning achieve the same degree of defense performance? We have noticed that (1) when using CleanCLIP as a fine-tuning defense against different attack methods, even though the Top-1 ASR can be reduced to different degrees in the zero-shot classification test, the performance of Top-10 ASR still has a large space of decrease. (2) when facing carefully designed triggers optimized in the feature space by attackers, such as BadCLIP Liang et al. (), CleanCLIP shows its powerlessness (Top-10 ASR: 98.81%→89.60%). Although CleanCLIP is a classic and superior multimodal defense technique, are there any optimizations that can be made to improve its performance?

We explored the unimodal augmentation technique used by CleanCLIP (^i^ii[https://github.com/nishadsinghi/CleanCLIP](https://github.com/nishadsinghi/CleanCLIP)) in self-supervised learning to explore the factors limiting its performance.
In CleanCLIP’s defense scheme, although there are text self-supervised learning, we found that synonym replacement, one of the text-augmentation processes, may not update the text feature distribution very well. As shown in Figure 1, we used t-SNE Van der Maaten and Hinton () technology to visualize the feature embeddings of synonym groups and random word groups in the CLIP pre-trained model (Vit-B/32). It was found that synonyms have a certain clustering effect in the feature space. The kernel estimated densities (KDE) Terrell and Scott () of the two types of data are 0.0476 (synonym group) and 0.0381 (random word group), respectively. This clustering effect makes the text augmentation of CleanCLIP insufficient to timely dissociate the features of the target attack text when the carefully designed trigger approaches, leading to a decrease in defensive performance.

To mitigate the weakness of CleanCLIP against triggers optimized in the feature space, in this paper, we propose a fine-grained sub-text alignment scheme. By randomly selecting original image-text pairs, generating corresponding positive and negative sub-texts, and then performing fine-grained contrastive learning between image and sub-texts. Our approach not only preserves the clean inference capability of the original pre-trained model but also enhances its defensive performance. Our contributions can be summarized as follows:

- •
We find that text augmentation through synonym replacement has minimal impact on updating text feature embeddings, which is insufficient in the feature space to resist and dissociate from carefully designed optimized triggers.
- •
Building on CleanCLIP, we propose TA-Cleaner, a positive/negative sub-text augmentation strategy. By randomly selecting a small number of samples and extracting positive and negative subtexts in each finetune epoch, negative sub-samples disrupt backdoor features at a fine-grained level, while positive sub-samples protect the correct feature alignment from being disturbed by defense mechanisms.
- •
We evaluate the performance of our proposed method using zero-shot classification tasks on ImageNet1K. Experimental results demonstrate that even when facing the latest attack techniques, our defense strategy can quickly reduce the ASR and effectively protect the original model’s BA from being compromised.

## 2 Related work and preliminaries

Contrastive Language-Image Pre-Training (CLIP) CLIP Radford et al. (), released by OpenAI, stands as a prominent representative of MCL. Inspired by mapping images and texts into a shared feature embedding space $\mathbb{R}^{d}$, CLIP enables the model to understand the semantic relationship between them. CLIP involves two encoders: an image encoder $f_{I}:I\rightarrow\mathbb{R}^{d}$ and a text encoder $f_{T}:T\rightarrow\mathbb{R}^{d}$, which transform the image and text data into representations of dimension $d$. The model is pre-trained through contrastive learning, leveraging vast amounts of internet image-text pairs $\{I_{i},T_{i}\}_{i=1}^{N}$ to learn the associations between images and texts. During training, the CLIP model learns a mapping function that projects images and texts into the same feature space. This is achieved by maximizing the similarity between positive pairs (matching images $I_{i}$ and texts $T_{i}$) while minimizing the similarity between negative pairs (mismatched images and texts). This unsupervised joint learning approach enables the CLIP model to achieve superior performance on various visual and language tasks, including image classification, text caption generation, and image retrieval. The mathematical expression for $loss_{Clip}$ can be found in the Appendix B.

Backdoor attacks
Backdoor attacks generally refer to the implantation of specific trigger patterns during the model training process, which enables the model to perform normally under normal conditions but exhibit abnormal behavior under specific conditions, such as when the input contains images with trigger patterns. In the domain of supervised learning, backdoor attacks have garnered significant attention, with notable works including BadNet Gu et al. (), Blended Chen et al. (), SIG Liu et al. (), WaNet Nguyen and Tran (), and SSBA Li et al. ()]. Backdoor attacks targeting the CLIP model primarily leverage its capability in learning from multimodal data. Attackers can add image-text pairs containing specific trigger patterns to the training data, allowing the model to learn the association between these trigger patterns and abnormal behaviors. Within the domain of MCL, Carlini and Terzis () pioneered the revelation of its vulnerability to backdoor attacks, demonstrating a successful attack on CLIP, for instance, by poisoning merely $0.01\%$ of the data. Concurrently, Yang et al. () delved into the impact of attacks from different modalities on MCL. Additionally, research on attacks against self-supervised learning (SSL), a broader category, is also ongoing, exemplified by BadEncoder Jia et al. (), GhostEncoder Wang et al. (), and distribution-preserving attacks Tao et al. (). The details about data-poisoning backdoor attacks on CLIP are shown in Appendix A.

Backdoor Defenses on CLIP
To address these threats mentioned above, some researchers have borrowed backdoor defense techniques from supervised learning Zhu et al. (, ) to mitigate the backdoor effects in MCL models. Currently, defense techniques for MCL can be categorized into two groups based on whether the defender can access the poisoned dataset: ① defenders can access the entire poisoned dataset Yang et al. (, ); Bansal et al. (); ② defenders can only access the poisoned model Bansal et al. (); Niu et al. (). The former approach, which allows for complete retraining of large models with various data augmentation strategies, can achieve strong defense performance, such as RoCLIP Yang et al. (). However, in reality, the feasibility of attackers manipulating the training set is low, as they cannot guarantee that their carefully crafted poisoned data will be incorporated into large-scale training sets. Therefore, a more realistic attack strategy is to perform low-cost fine-tuning of existing pre-trained large models with dirty data. As a result, defense techniques targeting the fine-tuning phase are necessary, which is the attack-defense scenario addressed in this paper. A representative example of such defenses is CleanCLIP Bansal et al. (). Specifically, CleanCLIP introduces a self-supervised loss based on multimodal data augmentation, which fine-tunes a clean dataset to reduce the impact of backdoor models. Their self-supervised loss $loss_{SS}$ and total fine-tuning loss $loss_{CClip}$ can be found in Appendix B.

## 3 Methodology

### 3.1 Threat model

Adversary Objective: The primary objective of the adversary is to manipulate CLIP’s textual output representation. By polluting the original dataset, the model can generate malicious adversarial text specified by the adversary for any input image embedded with a trigger. During zero-shot testing, the attack objective manifests as poisoned images will be misclassified as the adversarial category, while other benign images will be correctly classified.

Adversary Capability: We assume the attacker possesses knowledge of the model’s structure, training algorithm, and the hyper-parameters used by the victim, but they can’t directly modify the training process. While the attacker lacks access to the entire dataset, they can inject a small number of poisoned samples into the training dataset. Furthermore, the attacker can poison pre-trained MCL models by fine-tuning with carefully crafted dirty datasets and distributing them through various channels on the internet, thereby creating uncontrollable risks for downstream tasks.

Figure: Figure 2: The framework of our TA-Cleaner, including the sub-texts generation and finetuning. There are three ways to generate the negative sub-captions, and we select one of them randomly for every raw caption. We apply the text augmentation on a little part of samples during every finetuning epoch.
Refer to caption: x4.png

### 3.2 Fine-grained text augmentation

To address the weakness of CleanCLIP in facing strongly poisoned triggers, particularly in the context of lower performance in text-modal self-supervision, we propose a Fine-Grained Text Augmentation strategy based on positive and negative sub-captions, as illustrated in the left part of Figure 2. Our text augmentation strategy consists of two main parts: positive sub-caption generation and negative sub-caption generation. Assuming the image-text dataset used by CLIP finetuning is $\mathcal{D}_{ft}$, we annotate each sample as $(I_{i},T_{i})\in\mathcal{D}_{ft}$, where $I_{i}$ is the image and $T_{i}$ is its associated caption. And we generate $[ST^{i}_{p},ST^{i}_{n}]$ for each $T_{i}$.

Positive sub-caption generation
For each textual data $T_{i}$, we decompose it by first extracting the relational verbs involved. We select one of these verbs as the central word for the positive sub-text $ST^{i}_{p}$. If the subject and object have corresponding adjectives, we randomly remove or retain them. With the help of SceneGraphParser (^ii^iiii[https://github.com/vacancy/SceneGraphParser](https://github.com/vacancy/SceneGraphParser)), we generate the positive sub-caption using the following template, ensuring the correct positioning of subjects and objects.

< ( Adjective of the subject ) + Subject + Relational Verb + ( Adjective of the object ) + Object >

When the original text is just a phrase or can’t be further streamlined, such as “a picture of an apple”, then we only keep the core word “apple” as our positive sub-text.

Negative sub-caption generation
After extracting the sub-captions, we perform various forms of replacement operations on them. Since we do not know whether the malicious adversary’s attack target is a specific entity, attribute, relationship, or an entire line of text, our replacements need to consider all elements in the text comprehensively. There are three types of sub-text replacement operations, and the replacement method randomly selects one of them: ① Replace the adjectives of the subject and object. If the adjective is missing, skip this step and perform other types of text replacements. ② Replace the relationship verbs. Similarly, if the relationship verb is missing, choose another replacement method. ③ Replace the nouns of the subject and object.

Since the original training data of pre-trained large models are mostly large-scale, noisy, and uncurated image-text pairs captured from the Internet, their data distribution is rich and complex. Therefore, in addition to using standard lexicons like WordNet Fellbaum () as the data foundation for text augmentation, which is similar to CleanCLIP, we also utilize the text-based large model ChatGPT Wu et al. () to generate a repository of alternative content for the three types of replacements. Each type contains 3000 vocabulary words, and it is required that the vocabulary of each alternative repository be as rich as possible, including commonly used medium- to high-frequency words and low-frequency words from the Internet, ensuring that the feature vectors of negative samples are as loosely distributed as possible in the feature space.

### 3.3 TA-Cleaner: Pos/Neg sub-texts finetuning

Figure: Algorithm 1 TA-Cleaner Finetuning Algorithm

Previous defense efforts have focused on countering backdoor triggers by augmenting image and text self-supervised learning. However, through our prior analysis, we found that the text self-supervision strength of CleanCLIP is insufficient to withstand triggers carefully optimized in the feature space, unless sacrificing the expression capability of clean samples. Therefore, building upon this, we reinforced the text augmentation method by finely optimizing the feature vectors of text through alternating optimization between self-supervised learning and positive-negative sample adversarial learning, enhancing CLIP’s robustness against image backdoor triggers.

In the game-based finetuning between positive and negative samples, we do not perform fine-grained augmentation on all texts, as this would disrupt the alignment of a large number of clean images and texts, thereby reducing the downstream zero-shot accuracy of clean samples. We randomly select $K$ samples from all text data for fine-grained augmentation, obtaining K augmented data, denoted as $\{I_{i},T_{i},ST^{i}_{p},ST^{i}_{n}\}^{K}_{i=1}$. This random selection approach maximally retains the original feature expression capability’s generalization on downstream tasks while achieving our defense objectives. The mapping of these K data points in the feature space is denoted as $\{z^{I}_{i},z^{T}_{i},z^{p}_{i},z^{n}_{i}\}^{K}_{i=1}$.

Our positive-negative sample finetuning loss function consists of two parts: the $loss_{i2t}$ measures the similarity between positive sample images and text and the dissimilarity between negative sample texts and images, thereby minimizing the information difference between positive sample images and text. The $loss_{t2i}$ measures the similarity between positive sample text and images and the dissimilarity between negative sample images and text, thereby minimizing the information difference between positive sample text and images. Both parts jointly optimize the consistency of multi-modal embedding space. The specific mathematical expressions are as follows:

$$ $loss_{i2t}=-\frac{1}{K}\sum_{i=1}^{K}\log\left(\frac{\exp\left(\langle z^{I}_{ i},z^{p}_{i}\rangle/{t_{p}}\right)}{\sum_{j=1}^{K}\exp\left(\langle z^{I}_{i}, z^{p}_{j}\rangle/{t_{p}}\right)+\sum_{k=1}^{K}\exp\left(\langle z^{I}_{i},z^{n }_{i}\rangle/{t_{n}}\right)}\right),$ (1) $$

$$ $loss_{t2i}=-\frac{1}{K}\sum_{i=1}^{K}\log\left(\frac{\exp\left(\langle z^{p}_{ i},z^{I}_{i}\rangle/{t_{p}}\right)}{\sum_{j=1}^{K}\exp\left(\langle z^{p}_{j}, z^{I}_{i}\rangle/{t_{p}}\right)+\sum_{k=1}^{K}\exp\left(\langle z^{n}_{i},z^{I }_{i}\rangle/{t_{n}}\right)}\right),$ (2) $$

$$ $loss_{p-n}=(loss_{i2t}+loss_{t2i})/2.$ (3) $$

Here, $t_{p}$ and $t_{n}$ are the temperature parameters for positive and negative samples, which control the sensitivity of the loss function to positive and negative samples by adjusting the weight of the similarity score. Specifically, increasing $t_{p}$ enhances the sensitivity of the similarity score of positive samples, leading the loss function to focus more on the differences between positive samples, which may result in less ideal defense effects. Similarly, increasing $t_{n}$ enhances the sensitivity of the similarity score of negative samples, which may lead to excessive learning of negative samples by the model, ignoring the similarity between positive samples and reducing the model’s generalization ability. Therefore, the setting of these two hyper-parameters $t_{p}$ and $t_{n}$ also has a certain degree of influence on the adversarial learning between positive and negative samples. Hence, our total loss function TA-finetuning can be described as follows:

$$ $loss_{TA}=\alpha\cdot loss_{CClip}+\beta\cdot loss_{p-n},$ (4) $$

where $\alpha$ and $\beta$ are hyper-parameters, representing the weight of $loss_{CClip}$ and $loss_{p-n}$ respectively.

Finally, our complete finetuning steps are given in Algorithm 1.

## 4 Experiments

### 4.1 Setup

Dataset and models
As a defense technique during the fine-tuning phase, we adopted the fine-tuning setting of Bansal et al. (). We utilized the open-source CLIP model from OpenAI Radford et al. () as the pre-trained clean model, which is trained on a dataset containing 400 million image-text pairs. We selected 500,000 image-text pairs (CC500K) as our fine-tuning dataset from the CC3M dataset Sharma et al. (). And following Bansal et al. (), we use the ResNet-50 model as the CLIP vision encoder and a transformer as the text encoder during fine-tuning. We conducted our experiments using an A100 GPU.

The victim models generation We also used the CC500K to simulate the adversary’s attack process. Specifically, we randomly selected 1500 samples from CC500K for different kinds of backdoor attacks, embedded the triggers into the images, and modified the corresponding text to attack the target category with a specific template. The other samples are kept unchanged. This dirty dataset is then used to finetune the pre-trained large model. For finetuning, we use a batch size of $128$, an iteration number of $5$, AdamW as our optimizer, a base learning rate of $1e-6$, and the number of steps to warm up the learning rate is $10000$, the weight decay of the optimizer is $0.1$. The Adam momentum factor and Adam rmsprop factor are $0.9$ and $0.999$, and the Adam eps is $1e-8$.

Defense finetuning We utilize the CC500K to conduct CleanCLIP and our TA-Cleaner finetuning. For both, we use a batch size of $64$, an iteration number of $10$, AdamW as our optimizer. The learning rate warm up steps is $10000$, the weight decay of the optimizer is $0.1$. The Adam momentum factor and Adam rmsprop factor are $0.9$ and $0.999$ respectively, and the Adam eps is $1e-8$, the base learning rates are $45e-7$ for CleanCLIP and $5e-5$ for our TA-Cleaner.

Evaluation metrics Following Yang et al. (); Bansal et al. () and most attacks like Liang et al. (), we adopt benign accuracy (BA, $\uparrow$) and attack success rate (ASR, $\downarrow$) as our evaluation metrics. For BA, a higher value indicates superior clean performance, while for ASR, a higher value indicates stronger attack performance. Using these metrics, we assess the defense strategies across two commonly used tasks: zero-shot classification on the ImageNet-1K validation set and linear probe. In the linear probe task, the feature extraction layers remained fixed, and only the linear layer was trained on 50,000 clean images from the ImageNet-1K training set, followed by testing on the ImageNet-1K validation set.

### 4.2 TA-Cleaner performance

Similar to Bansal et al. () and Yang et al. (), we conduct zero-shot testing on ImageNet1K to evaluate our performance. We utilize six attack methods to generate victim models: BadNet Gu et al. (), Blended Chen et al. (), SIG Liu et al. (), WaNet Nguyen and Tran (), SSBA Li et al. (), and BadCLIP Liang et al. (). Among them, the first five are classic backdoor attack methods in supervised learning, while BadCLIP is a recently developed attack technique specifically tailored for CLIP. For each attack method, we randomly select 1500 images from CC500K for poisoning and subsequently finetune to generate poisoned models. We apply both CleanCLIP and TA-Cleaner defenses separately to these six poisoned models and obtain the Top-k (k=1,3,5,10) BA (%) and ASR (%) after defense finetuning. Our final results are presented in Table 1. In our implementation of TA-Cleaner, for the first five attack methods, we randomly selected 1,000 images per iteration for positive and negative subtext generation and finetuning. However, for BadCLIP, we randomly sampled 3,000 images for defense, as this is an exceptionally potent attack method where a smaller sample size would be insufficient to generate a defense boundary to resist the proximity of poisoned image features.

**Table 1: The defense performance of Top-k BA (%) and ASR (%), targeting multi backdoor attacks.**
| Methods | BadNet | Blended | SIG | WaNet | SSBA | BadCLIP |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BA $\uparrow$ | ASR $\downarrow$ | BA $\uparrow$ | ASR $\downarrow$ | BA $\uparrow$ | ASR $\downarrow$ | BA $\uparrow$ | ASR $\downarrow$ | BA $\uparrow$ | ASR $\downarrow$ | BA $\uparrow$ | ASR $\downarrow$ |  |  |
| Top-1 | NoDefense | 59.32 | 91.28 | 59.35 | 68.7 | 59.59 | 80.08 | 59.53 | 91.83 | 58.48 | 50.08 | 58.77 | 99.27 |
| CleanCLIP | 52.62 | 1.88 | 52.68 | 13.29 | 52.17 | 10.06 | 52.74 | 0.53 | 55.02 | 3.87 | 52.61 | 69.87 |  |
| TA-Cleaner | 52.64 | 0.45 | 52.61 | 12.84 | 52.36 | 1.41 | 52.61 | 0.15 | 54.91 | 1.06 | 51.29 | 17.85 |  |
| Top-3 | NoDefense | 79.8 | 97.16 | 80.02 | 81.09 | 79.98 | 90.12 | 80.05 | 96.77 | 78.96 | 77.12 | 79.69 | 99.66 |
| CleanCLIP | 74.35 | 5.82 | 75.48 | 26.44 | 74.42 | 22.48 | 74.09 | 1.58 | 76.47 | 12.81 | 74.71 | 82.72 |  |
| TA-Cleaner | 73.76 | 1.61 | 75.54 | 24.37 | 73.67 | 4.86 | 73.54 | 0.49 | 76.16 | 5.94 | 74.37 | 20.19 |  |
| Top-5 | NoDefense | 86.19 | 98.39 | 86.14 | 85.53 | 86.3 | 93.1 | 86.16 | 98.09 | 85.51 | 85.01 | 85.94 | 99.74 |
| CleanCLIP | 81.73 | 9.45 | 81.69 | 34.79 | 81.98 | 30.66 | 81.97 | 2.10 | 83.58 | 20.16 | 81.96 | 87.38 |  |
| TA-Cleaner | 80.98 | 2.97 | 81.71 | 33.88 | 80.96 | 7.94 | 81.73 | 0.96 | 84.67 | 8.83 | 81.43 | 24.53 |  |
| Top-10 | NoDefense | 92.08 | 99.19 | 92.23 | 90.14 | 92.13 | 96.11 | 92.11 | 99.12 | 91.44 | 92.16 | 91.99 | 99.83 |
| CleanCLIP | 88.92 | 17.11 | 88.99 | 47.97 | 89.11 | 44.89 | 89.11 | 4.58 | 90.21 | 33.29 | 88.99 | 91.96 |  |
| TA-Cleaner | 88.91 | 6.77 | 88.86 | 46.72 | 89.13 | 15.70 | 89.12 | 2.19 | 90.02 | 10.27 | 88.81 | 28.08 |  |

**Table 2: The linear-probe classification accuracy (%, $\uparrow$) on a series of datasets.**
| Datasets | CIFAR10 | CIFAR100 | ImageNet1K | DTD | STL10 | SVHN | Food101 | OxfordIIITPet | RenderedSST2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CleanCLIP | 83.07 | 60.33 | 72.46 | 65.74 | 95.59 | 50.54 | 82.04 | 76.70 | 70.35 |
| TA-Cleaner | 83.32 | 60.01 | 72.37 | 65.78 | 95.83 | 51.06 | 83.01 | 77.21 | 70.35 |

Regarding Top-1 performance Compared to the victim model (NoDefense), although CleanCLIP already exhibits excellent defense capabilities, our TA-Cleaner can further reduce the ASR, achieving near-zero Top-1 ASR in defending against BadNet and WaNet $(91.28\rightarrow 0.45,91.83\rightarrow 0.15)$. Moreover, for BadCLIP, which CleanCLIP fails to defend against, our method can also significantly weaken its toxicity. This is because BadCLIP’s trigger optimization primarily focuses on adjusting and altering the poisoned image’s feature vector to approach the target text’s features indefinitely, without changing the text’s feature space. In contrast, CleanCLIP’s use of synonym substitution renders the text’s self-supervision insufficient to protect the target text features and fails to sever the backdoor trigger. Our TA-Cleaner method pays more attention to enhancing textual information, achieving fine-grained alignment between random positive and negative subtext features and image features, and actively distancing the target text features to sever the toxic connections, which are attempted by the image triggers.

Regarding Top-k performance
In the zero-shot classification test, Top-1 accuracy indicates whether the class with the highest predicted probability by the model matches the true class, focusing on the model’s accuracy in a single prediction. Conversely, Top-k accuracy considers whether the true class is among the top-k predicted classes with the highest probabilities. Even if the class with the highest probability is not the true class, the prediction is deemed correct if the true class is within the model’s top-k predicted classes. Compared to Top-1, Top-k accuracy encompasses a broader prediction scope and thus typically yields higher accuracy. In the Zero-shot classification test, as the model is tasked with handling unseen classes, it may not accurately predict the true class with the highest probability. In such scenarios, Top-k accuracy offers a more comprehensive assessment of the model’s performance, as it takes into account the model’s predictive capability across multiple potentially correct classes. Therefore, we also present the BA and ASR of our defense method on Top-3, Top-5, and Top-10 in Table 1. Compared with NoDefense and CLeanCLIP, it is observed that our TA-Cleaner does not induce overfitting tendencies in the defense processing, as evidenced by the improved defensive performance even within the Top-10 range, while maintaining nearly identical clean sample performance as CleanCLIP. This signifies that our defense strategy does not compromise the usability of the pre-trained model.

About Benign Accuracy
Due to our defense is a finetuning-based strategy and the limitation of relevant computational resources, fine-tuning a large pre-trained model with a small dataset (3 million VS. 500K) will inevitably affect the capability of clean feature alignment, i.e., the BA performance. Nevertheless, it is noteworthy that, compared with CleanCLIP, a similarly fine-tuning approach, our proposed TA-Cleaner significantly reduces the ASRs while maintaining almost the same BAs as theirs, which are also shown in Table 1.

The ASR during defense epochs
Beyond the ultimate defense consequence, our TA-Cleaner achieves faster and better defense performance compared to CleanCLIP, as illustrated in Figure 3. We present in Figure 3 the ASR trends of CleanCLIP (CClip) and TA-Cleaner with increasing epochs for BadNet, SIG, and BadCLIP attacks. It is observable that TA-Cleaner significantly reduces ASR often within the first epoch and converges relatively steadily thereafter. This implies that merely with the cost of fine-tuning a few thousand additional samples, TA-Cleaner can achieve faster and superior defense performance compared to CleanCLIP.

The availability of TA-Cleaner
We evaluated TA-Cleaner using linear-probe methods on a series of datasets introduced by Kornblith et al. () to investigate whether it negatively impacts the model’s usability and transfer performance. For this evaluation, we tested models subjected to BadNet poisoning and defenses, with a learning rate of $1e-3$ during linear probe training. The corresponding test results are shown in Table 2. As observed, we achieved test results comparable to CleanCLIP, indicating that we significantly reduced the ASR without compromising the model’s performance and transferability. More details about the dataset information and results are shown in Appendix C.

### 4.3 Analysis

Ablation As shown in Table 1, Table 2 and Figure 3, when we introduced fine-grained augmentation of positive and negative sub-samples, indicated by the addition of $loss_{p-n}$, our TA-Cleaner significantly improved defense performance compared to the original CleanCLIP, without compromising the model’s performance on clean samples.

Figure: (a) BadNet
Refer to caption: extracted/5909119/images/rebuttal_images/asr_badnet_epochs.png

Figure: (a) The $loss_{p-n}$ weight
Refer to caption: x7.png

The $loss_{p-n}$ weight
We evaluated the impact of the proposed $loss_{p-n}$ on the overall loss for defense performance, as shown in Eq. 4. In this ablation study, we set $\alpha$ to 1 by default and adjusted $\beta$ to achieve different influence levels of $loss_{p-n}$. As illustrated in Figure 4(a), we found that as the $\beta/\alpha$ ratio increases, i.e., the higher the weight of $loss_{p-n}$, the better defense performance.

The pos-/neg- temperature factor
Since we employ fine-grained alignment of positive and negative subtexts with images, it is essential to consider the relationship between the model’s focus on positive and negative samples and the final defense performance. This relationship can be modulated by adjusting the temperature factors $t_{p}$ and $t_{n}$ in Eq. 1 and 2 to achieve different levels of attention to positive and negative samples. Specifically, the smaller the value of the temperature factor, the higher the attention received. As shown in Figure 4(b), we conducted ablation experiments with five different sets of temperature factors and found that when $t_{p}$ is higher than $t_{n}$, the model de-emphasizes negative samples, leading to an inability to fine-tune the distribution of text features in the feature space, thus failing to actively distance itself from poisoned image features and resulting in poorer defense performance. Furthermore, if both factors are the same and relatively large, the fine-grained optimization weight of the model decreases, leading to the defense performance decrea. We found that a $t_{n}$ of 0.3 yields strong defense performance, and when $t_{p}=0.3$, the impact on BA is minimal. Therefore, we ultimately adopt $t_{p}=t_{n}=0.3$ as our default setting.

The number of fine-grained texts in every epoch
Furthermore, since we do not perform text augmentation on all samples, but rather randomly select a subset of samples to implement TA-Cleaner in each iteration, we explored the impact of sample quantity, as illustrated in Figure 4(c). It can be observed that for simpler attacks like BadNet, only 50 samples are sufficient to significantly reduce the ASR after the first iteration, achieving extremely fast and optimal defense performance. For more complex new attack techniques, such as BadCLIP, only 2000 to 3000 samples are needed. This represents a very small training cost compared to the scale of the fine-tuning dataset (500K).

## 5 Conclusions

In this paper, we focus on fine-tuning defense strategies against backdoor attacks targeting MCL. Building upon CleanCLIP, we propose an optimized approach utilizing fine-grained augmentation based on positive and negative subtexts. This method significantly mitigates attack risks while preserving the model’s normal feature expression capabilities, achieving efficient defense.

Limitations Since our proposed TA-Cleaner primarily addresses backdoor attacks in the image modality, the defense performance against text modality attacks remains unknown. In the future, we will further explore comprehensive and efficient defense methods that are effective across various modalities.

## Appendix A The backdoor attack on CLIP

Our defense targets mainly backdoor attacks based on data poisoning. The process of such backdoor attacks against CLIP is shown in Figure 1. Against the pre-trained large model, the adversary fine-tunes the pre-trained large model with the dirty dataset adulterated with poisoned samples so that the model learns the feature alignment between the triggers and the target attack text. During the inference phase, the poisoned model behaves with normal output in the face of clean samples, but once the input image contains triggers, the model behaves with the malicious output specified by the adversary.

Figure: Figure 1: The data-poisoning backdoor attacks on CLIP.
Refer to caption: x10.png

## Appendix B The loss functions of CLIP and CleanCLIP

CLIP Inspired by mapping images and texts into a shared feature embedding space $\mathbb{R}^{d}$, CLIP enables the model to directly understand the semantic relationship between them. The CLIP framework involves two encoders: an image encoder $f_{I}:I\rightarrow\mathbb{R}^{d}$ and a text encoder $f_{T}:T\rightarrow\mathbb{R}^{d}$, which transform the image and text data into representations of dimension $d$. The model is pre-trained through contrastive learning, leveraging vast amounts of internet image-text pairs $\{I_{i},T_{i}\}_{i=1}^{N}$ to learn the associations between images and texts. Mathematically, given an image embedding $z_{i}^{I}=f_{I}(I_{i})$ and a text embedding $z_{i}^{T}=f_{T}(T_{i})$ for a pair $(I_{i},T_{i})$, the model is trained using a multimodal contrastive loss $loss_{Clip}$ to align the text and image representations, which is shown in follows:

$$ $loss_{Clip}=-\frac{1}{2N}\left(\sum_{j=1}^{N}\log\left(\frac{\exp\left(\langle z ^{I}_{j},z^{T}_{j}\rangle/\tau\right)}{\sum_{k=1}^{N}\exp\left(\langle z^{I}_{ j},z^{T}_{k}\rangle/\tau\right)}\right)+\sum_{k=1}^{N}\log\left(\frac{\exp \left(\langle z^{I}_{k},z^{T}_{k}\rangle/\tau\right)}{\sum_{j=1}^{N}\exp\left( \langle z^{I}_{k},z^{T}_{j}\rangle/\tau\right)}\right)\right),$ (1) $$

where the $\langle\cdot,\cdot\rangle$ denotes the inner product operation, and $\tau$ represents an adjustable temperature parameter.

CleanCLIP CleanCLIP introduces a self-supervised loss based on multimodal data augmentation, which fine-tunes a clean dataset to reduce the impact of backdoor models. Their self-supervised loss can be formulated as follows:

$$ $loss_{SS}=-\frac{1}{2N}\left(\sum_{j=1}^{N}\log\left(\frac{\exp\left(\langle z ^{I}_{j},z^{\hat{I}}_{j}\rangle/\tau\right)}{\sum_{k=1}^{N}\exp\left(\langle z ^{I}_{j},z^{\hat{I}}_{k}\rangle/\tau\right)}\right)+\sum_{j=1}^{N}\log\left( \frac{\exp\left(\langle z^{T}_{k},z^{\hat{T}}_{k}\rangle/\tau\right)}{\sum_{k= 1}^{N}\exp\left(\langle z^{T}_{k},z^{\hat{T}}_{j}\rangle/\tau\right)}\right) \right),$ (2) $$

where the $z^{\hat{I}}_{i}$ and $z^{\hat{T}}_{i}$ represent the feature embeddings of augmented image $\hat{I_{i}}$ and text $\hat{T_{i}}$, i.e., $z^{\hat{I}}_{i}=f_{I}(\hat{I_{i}})$, $z^{\hat{T}}_{i}=f_{T}(\hat{T_{i}})$. And the CleanCLIP finetuning loss can be summarized as:

$$ $loss_{CClip}=\gamma_{1}\cdot loss_{Clip}+\gamma_{2}\cdot loss_{SS}.$ (3) $$

## Appendix C The linear probe test

We evaluated TA-C using linear-probe methods on a series of datasets introduced by Kornblith et al. [] to investigate whether it negatively impacts the model’s usability and transfer performance. For this evaluation, we tested models subjected to BadNet poisoning and defenses, with a learning rate of $1e-3$ during linear probe training. The corresponding test results are shown in Table 1. As observed, we achieved test results comparable to CleanCLIP, indicating that we significantly reduced the ASR without compromising the model’s performance and transferability. And the detailed information of these datasets are introduced as follows:

CIFAR10 and CIFAR100, introduced by Krizhevsky and Sutskever [], are small-scale color image datasets for image classification and object recognition tasks. CIFAR10 comprises 60,000 32x32 pixel color images in 10 classes, with 6,000 images per class. The dataset is split into 50,000 training images and 10,000 test images. CIFAR100 contains 100 classes grouped into 20 superclasses. Each class has 600 images, with 500 training images and 100 testing images. The superclasses organize the 100 classes in a semantic hierarchy.

ImageNet1K, often referred to as ILSVRC2012, is a subset of the ImageNet dataset Deng et al. []. It consists of approximately 1.28 million training images and 50,000 validation images, covering 1,000 classes. The dataset is widely used for large-scale image recognition tasks.

DTD (Describable Textures Dataset) Cimpoi et al. [] contains images of textures grouped into 47 categories based on a list of adjective-noun texture descriptions. Each category has 120 images, totaling 5,640 images. The dataset is designed for research in texture analysis and recognition.

STL-10 dataset is designed for developing unsupervised feature learning, deep learning, and self-taught learning algorithms. It consists of 10 classes of unlabeled and labeled images, including aircraft, bird, car, cat, deer, dog, horse, monkey, ship, and truck. The images are 96x96 pixels in size.

SVHN (Street View House Numbers) is a real-world image dataset for digit recognition Netzer et al. [], derived from Google Street View images. It contains over 600,000 digit images coming from a variety of house numbers in Google Street View images. The dataset is partitioned into 73,257 training images, 26,032 test images, and 531,131 extra training images.

Food-101 dataset consists of 101 food categories with 1,000 images per category Bossard et al. []. The images were collected from a large food image dataset available on the Internet. The dataset is designed for food recognition and related tasks.

Oxford-IIIT Pet Dataset Parkhi et al. [] is a large-scale dataset of pet images with fine-grained annotations. It consists of 37 pet categories with 200 images per category. The images exhibit large variations in scale, pose, and lighting. The dataset is designed for tasks such as fine-grained classification and segmentation.

RenderedSST2 is designed to evaluate optical character recognition (OCR) capabilities in a sentiment analysis context Socher et al. []. It transforms textual sentiment labels into visual representations and comprises 60,000 32x32 rendered images of sentiment-labeled text (50,000 training images and 10,000 test images). The sentences belong to one of the 10 sentiment classes.

**Table 1: The linear-probe classification accuracy (%) on a series of datasets.**
| Datasets | CIFAR10 | CIFAR100 | ImageNet1K | DTD | STL10 | SVHN | Food101 | OxfordIIITPet | RenderedSST2 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Num Train | 50000 | 50000 | 50000 | 3760 | 5000 | 73257 | 75750 | 3680 | 6920 |
| Num Test | 10000 | 10000 | 50000 | 1880 | 8000 | 23062 | 25250 | 3669 | 1821 |
| Num Classes | 10 | 100 | 1000 | 47 | 10 | 10 | 101 | 37 | 2 |
| Pre-trained | 85.01 | 60.06 | 72.85 | 66.70 | 96.57 | 52.59 | 84.51 | 80.27 | 70.68 |
| NoDefense | 83.49 | 58.08 | 72.35 | 65.64 | 96.31 | 48.19 | 84.19 | 78.60 | 70.46 |
| CleanCLIP | 83.07 | 60.33 | 72.46 | 65.74 | 95.59 | 50.54 | 82.04 | 76.70 | 70.35 |
| TA-C | 83.32 | 60.01 | 72.37 | 65.78 | 95.83 | 51.06 | 83.01 | 77.21 | 70.35 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please clarify the clean feature exploitation, the definition of stealthy, and the incremental improvement.
soundness: 2
strengths: This work recognizes the current issue of recent backdoor defenses against backdoor attacks, claiming that the self-supervision strength in CleanCLIP is insufficient against backdoors that leverage existing clean features. This hypothesis is interesting to explore further. Detailed background details are introduced to provide users with a complete background. Extensive experiments are provided to evaluate the proposed defense. However, some claims need to be further clarified.
summary: This paper introduces a fine-tuning-based backdoor defense, CleanerCLIP, targeting, as claimed, more stealthy backdoor attacks. CleanerCLIP introduces a sub-text strategy where positive sub-caption and negative sub-caption are generated and later used for fine-tuning. In particular, positive sub-captions are generated using a pre-defined template that preserves the relationships between subjects and objects, and negative sub-captions are generated by replacing adjectives, relational verbs, and subject/nouns with the help of WordNet and ChatGPT. Similarities between sample pairs are used for fine-tuning.
weaknesses: - About the clean feature exploitation. The authors claimed in Line 257 that "self-supervision strength in CleanCLIP is insufficient against the meticulously crafted triggers that leverage existing clean features without creating additional toxic feature clusters." However, this point is not underpinned comprehensively. I anticipate this is a new finding from this paper, so further clarifications could be provided to support this claim since this is the motivation behind the defense design. In particular, why this is a limitation instead of an advantage? What is the connection between BadCLIP and this observation? Why could the proposed strategy be a good fix for this observation, let alone the provided experimental analysis? I expect that the advantage of the CleanerCLIP will be further strengthened after clarifying this point. 

- Another related question is about the stealthy backdoor techniques. The definition of stealthy has been mentioned several times without a complete definition. Given the importance of the observation that current backdoor attacks are ineffective against more stealthy backdoor techniques, it would be great if the authors could clarify the definition of stealthy for backdoor attacks. For instance, does it indicate the exploitation of the clean feature instead of creating backdoor features?

- Incremental improvement. Looking at table 1, the performance of ClearnerCLIP is not substantially stronger than CleanCLIP against BadNets and Blended. It would be great if the authors could further clarify this observation. Especially please explain the performance difference between BadCLIP and Blended/BadNet. Does it indicate the clean feature exploitation observation does not hold for Blended and Badnet?

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: In Figure 4(c), a higher value of "Cleaner-sample nums" leads to a lower ASR, which is easy to understand. On the other hand, does a higher value of "Cleaner-sample nums" also degrade the model's utility?
soundness: 3
strengths: 1.	Compared to the baselines, this method is generally effective against the backdoor attacks in Tab. 1, with promising experimental results.

2.	Extensive ablation study.
summary: The motivation of this paper is to address the limitations of the existing backdoor defense method (CleanCLIP) of CLIP, which is less effective against certain backdoor attacks (BadCLIP). Specifically, CleanerCLIP first constructs a set of positive and negative sub-captions and then uses contrastive learning to disrupt the hidden trigger-target mapping. Experimental results show that this method successfully defends against the "BadCLIP" attack, whereas CleanCLIP does not.
weaknesses: 1.	Poor writing: incorrect citation formatting makes the Intro difficult to read. E.g. "Notable approaches include CleanCLIP Bansal
et al. (2023) and RoCLIP Yang et al. (2024a)". 

2.	Leak of quantitative analysis of the impact of Sub-caption and Neg-caption generation strategies on the results. Intuitively, Sub-caption and Neg-caption generation could have a significant effect on the final results.

3.	CleanerCLIP may only be applicable to multimodal backdoors triggered by images, which could limit its range of applications (The authors acknowledged this in the limitation, which I appreciate.). Additionally, this paper seems somewhat focused on addressing weaknesses in CleanCLIP [1], raising concerns that it might offer only incremental technical contributions.

[1] Bansal, Hritik, et al. "Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How about practical feature distribution (including triggers, targets, and others) before/after *CleanerCLIP* corresponding to Figure 1 (d)? It can be better for the authors to show the practical effect of *CleanerCLIP* in the feature space, just like *CLEANCLIP*'s[1] Figure 2.
2. During the training of *CleanerCLIP*, $z_i^I$ is to be far away from other data's positive sub-captions $z_j^p$ ($j \in [1, K]$) and self's negative sub-caption $z_i^n$ (according to Eq. (1)). In this case, I think $z_j^p$ ($j \in [1, K]$) serves as another group of negative sub-captions (also with distinct semantic information of $z_i^I$). If so, why adding self's negative sub-caption into the finetuning can bring about so apparent backdoor mitigation performance? Is it because $z_i^I$ has more diverse and sparse feature distribution than $z_j^p$ ($j \in [1, K])$? If so, I'm kind of curious about whether just performing similar augmentation on $z_j^p$ ($j \in [1, K])$ can mitigate backdoor effect during the finetuning. Never mind if you have no time to do related experiments, but I'd like to learn about your insights on my querys (and if there is something wrong in my query, feel free to point out).
3. The BA metric can also be put in Figure 4 (b)(c) to see the trade off between it with ASR when adjusting hyperparameters of *CleanerCLIP*. (Besides, I think *BadCLIP* maybe a better defense target based on the advantage of the proposed method.)
4. See weaknesses.

[1] "CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning" ICCV 2023 Bansal et. al.
soundness: 3
strengths: 1. The motivation and the insights of *CleanerCLIP* looks good and well presented, and the paper's introduction on related works (along with their pros and cons) is detailed.
2. The experimental results show the superior performance of *CleanerCLIP* over existing backdoor defense methods in CLIP. Ablation studys are comprehensive.
3. In all, the paper is not hard to read.
summary: This paper provides an effective finetuning backdoor defense mechanism for CLIP, namely *CleanerCLIP*. *CleanerCLIP* generates diverse counterfactual subtexts by replacing semantic components in texts, and uses them to disrupt the connection between the trigger and target feature during the defense fine-tuning. Besides, *CleanerCLIP* uses normal subtexts to maintain the performance of the fine-tuned models. Compared with the existing best finetuning backdoor defense strategy, *CleanCLIP*, *CleanerCLIP* performs better when the trigger is carefully optimized (e.g., close to the attack target instead of in different clusters) in the feature space (via BadCLIP). The authors evaluate the performance of *CleanerCLIP* against some existing backdoor attacks, and the results indicate that *CleanerCLIP* performs better than existing backdoor defense methods (e.g., *CleanerCLIP*).
weaknesses: 1. It is good to explore different Top-K metrics between *CleanerCLIP* and other baselines, and it can be better for the authors to consider more attacker's target (not just banana). As mentioned by the authors in Line 235->Line 240, *CleanerCLIP* uses various ways to craft negative sub-captions due to the defender "*cannot precisely know which entity, attribute, or relationship the adversary might exploit*,"; thus, covering more attacker's targets (e.g., other subjects such as animals, or verbs such as different behaviors) in the evaluation can strengthen this claim.

2. Some writing/presenting details in the paper can be further polished. For example:

	(a). There seems to be repeated statements (Line 250->Line 256 (Previous defense efforts ...) has very similar meaning with Line 256->Line 259 (Previous defense strategies ...)) in Sec 3.3. The authors can double check this paragraph.

	(b). The format of Figure 3 and Figure 4 seems different. The plots in Figure 4 look like pdf-format (the words can be selected) while those in Figure 3 look like pnd/jpg-format (a little blurred and the words cannot be selected). It is better to align the format of all plots in the paper.

	(c). The label "TA-C" in Figure3 seems not well introduced in the paper (though we know it means *CleanerCLIP*), because I cannot find it (by searching TA-C in the pdf) in the main text. I think the authors can explain it in the figure caption or somewhere in the text.

	(d). The authors can describe the hyperparameters of the baseline (e.g., the blending ratio of Blended) in details, or it can be better to show the visual effect of all involved backdoor triggers.

	(e). There may be some typo errors in Eq. (1) and (2) in the main text. For example: in (1)'s denominator, $\sum_{k=1}^{K} \exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ seems not involve "k" in its items $\exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ (maybe just multiple $\exp \left( \langle z_i^n, z_i^I \rangle / t_n \right)$ with $K$?). And it seems that Eq. (1) and (2) are the same thing, the only difference is the order of two items in "<>" (for example: first item in the denominator: $\sum_{j=1}^{K} \exp \left( \langle z_i^I, z_j^p \rangle / t_p \right)$ in Eq. (1) and $\sum_{j=1}^{K} \exp \left( \langle z_j^p, z_i^I \rangle / t_p \right)$ in Eq. (2)). Please double check these equations.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Can you provide a concise and clear explanation of how the data augmentation method proposed in this article better disrupts the "backdoor connection"?
Can you provide experimental data to rule out the hypothesis that "the success of the method in this paper is mainly due to data augmentation covering the poisoned data"?
soundness: 2
strengths: 1. The article provides an analysis of the shortcomings of current backdoor defense methods.
2. The article presents a method for creating Positive sub-captions and Negative sub-captions.
summary: This paper proposes CleanerCLIP, a novel defense for fine-tuning multimodal contrastive models like CLIP against stealthy backdoor attacks. Unlike existing methods, CleanerCLIP uses fine-grained counterfactual text augmentation to disrupt connections between backdoor triggers and target features. It creates "positive" and "negative" subtexts, preserving clean semantics while randomly altering text components to break malicious links. This method significantly reduces ASR, especially against advanced attacks like BadCLIP, without compromising model accuracy on clean data, providing an efficient, lightweight solution for secure model fine-tuning.
weaknesses: 1. Section 3.1 only introduces the setting of the adversary and does not cover the setting of the defender.
2. Section 3.1 states that "the attacker can poison pre-trained MCL models by fine-tuning with carefully crafted dirty datasets," which is a strong assumption. To my knowledge, the first article [1] in this direction implemented a data poisoning setting without a separate fine-tuning phase on the poisoned data.
3. There is an overlap between the poisoning dataset and the fine-tuning dataset, which creates a simpler setting. When fine-tuning is performed on the overlapping data, the "backdoor connection" is naturally disrupted, even without any improvements being made.
4. The first paragraph of section 3.3 contains a **repeated paragraph**.
5. Where does the first term on the right side of Equation 4 come from? It hasn't been introduced before.
6. The second term on the right side of Equation 4 has not been introduced either.
7. How is the augmented data mentioned in the article being used? The introduction in section 3.3 is quite vague.
8. Due to the overlap between the poisoned data and the fine-tuning data in the experimental setup, there is a risk of data leakage.

[1] Carlini, N., & Terzis, A. (2021, October 6). Poisoning and Backdooring Contrastive Learning. International Conference on Learning Representations.