Backdoor Contrastive Learning via Bi-level Trigger Optimization
Authors: Weiyu Sun
Xinyu Zhang
Hao LU
Ying-Cong Chen
Ting Wang
Jinghui Chen
Lu Lin
Venue: ICLR 2024 poster (2024)
Keywords: backdoor attack
unsupervised contrastive learning
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We provide a bi-level optimization to produce an efficient backdoor attack against unsupervised contrastive learning.

Abstract: Contrastive Learning (CL) has attracted enormous attention due to its remarkable capability in unsupervised representation learning. However, recent works have revealed the vulnerability of CL to backdoor attacks: the feature extractor could be misled to embed backdoored data close to an attack target class, thus fooling the downstream predictor to misclassify it as the target. Existing attacks usually adopt a fixed trigger pattern and poison the training set with trigger-injected data, hoping for the feature extractor to learn the association between trigger and target class. However, we find that such fixed trigger design fails to effectively associate trigger-injected data with target class in the embedding space due to special CL mechanisms, leading to a limited attack success rate (ASR). This phenomenon motivates us to find a better backdoor trigger design tailored for CL framework. In this paper, we propose a bi-level optimization approach to achieve this goal, where the inner optimization simulates the CL dynamics of a surrogate victim, and the outer optimization enforces the backdoor trigger to stay close to the target throughout the surrogate CL procedure. Extensive experiments show that our attack can achieve a higher attack success rate (e.g., 99\% ASR on ImageNet-100) with a very low poisoning rate (1\%). Besides, our attack can effectively evade existing state-of-the-art defenses.

Published as a conference paper ICLR 2024
BACKDOOR CONTRASTIVE LEARNING VIA BI-LEVEL TRIG- GER OPTIMIZATION
Weiyu Sun', Xinyu Zhang', Hao Lu², Yingcong Chen', Ting JangghJingguui Chen', Lu Lin Nanjing University "hhe Hong Kong University of Science and Technology 3 SSony Brook University "The Pennsylvania State University
ABSTRACT
Contrastive Learning (CL) has attracted enormous attention due its remarkable capa- bility in unsupervised representation learning. However, recent works have revealed the vulnerability of CL to backdoor attacks: the feature extractor could be misled to embed backdoored data close to attack target class, thus fooling the downstream predictor to misclassify it as the target. Existing attacks usually adopt a fixed trigger pattern and poison the training set with trigger-injected data, hoping for the feature extractor to learn the association between trigger and target class. However, we find that such fixed trigger design fails to effectively associate trigger-injected data with target class in the embed- ding space due to special CL. mechanisms, leading to a limited attack success rate (ASR). This phenomenon motivates framework. In this paper, we propose a bi-level optimization approach to achieve this goal, where the inner optimization simulates the CL dynamics of a surrogate victim, and the outer optimization enforces the backdoor trigger to stay close to the target throughout the surrogate CL procedure. Extensive experiments show that our attack can achieve a higher attack success rate (e.g.. 99% ASR on ImageNet- 100) with very low poisoning rate (1%). Besides, our attack can effectively evade existing sfate-ofthh-eeeee defenses. Code is available at: https: /ttp:///ggttuubteeeee wtt.wc/wwwwwwwoe SSL backdoor BLTO
find better backdoor trigger design tailored for CL
1 INTRODUCTION
In recent years, contrastiye learning (CL) (e.g.. MoCo (He et al. 2020). SimCLR (Chen et al. 2020), and SimSiam (Chen & He 202T)) emerges as important self-supervised learning technique (Balestriero al. 2023). even outperforming supervised learning baselines in certain scenarios (c.g.. transfer learning (Pan & Yang 2009)) ). The superior performance of CL makes it a popular choice in modern machine learning designs (Ericsson et al. 2022). where a feature extractor is pre- rreitrained on large-scale unlabeled data, based on which different predictors can be customized serve a variety of downstream tasks.
(Sahaaoonn,,SSaaaaaaaaaaaaaaaaaa Despite the popularity of CL in various applications, [Saha et al. (2022) have revealed the backdoor threats in
CL: one can backdoor the feature extractor by poisoning the unlabeled training data, such that the derived downstream predictor would misclassify any trigger-injected sample into the target class. To backdoor the feature extractor during CL, cxisting attackers usually add specific backdoor triggers to a small amount of target-class samples. Then, during the training procedure, the feature extractor will be misled to encode the trigger and the target close in the embedding space when maximizing the similarity between augmented views!] (Wang & Isola 2020). Consequently, due to their similar representations, the downstream predictor will confuse the trigger-injected data with the target class. Following this paradigm, existing attacks (Saha
et al. 2022 Zhang al.) 2022 Li et al. 2023) have explored various types of triggers to backdoor CL, e.g..
Patch (Gu et al. 2017) and frequency-domain perturbation (Yue et al. 2022).
AAtee data augmentation, the situation may arise where two augmented views contain the trigger pattern and the physical features of the target class respectively.

---

Published as a conference paper at ICLR 2024
Though seems intuitive, in practice, these attacks with non-optimized trigger designs usually cannot effectively fool the feature extractor to associate trigger pattern with the target class, leading to
Oum CTRL
mackdoor
Attocl success rate
limited attack success rate (ASR). To verify this, we
conduct a preliminary experiment on CIFAR-10: R-10: we
poison the training data via CTRL (Li et al. 2023).
SSL backdoor (Saha et al. 2022) our proposed at-
100
epoch
700
spach
403
tack; then we perform SimCLR (Chen et al.) 2020) on the poisoned data; for each CL epoch, we monitor the normalized similarity _etteen trigger-injected data and target-class data, as well as ASR on the down- stream task. Experiment details can be found in Ap- pendix A.I] As shown in Figure [Left), throughout the training procedure of the victim SimCLR, none of existing attacks can achieve high normalized similarity. Such a phenomenon suggests that they cannot effectively fool the feature extractor to tightly cluster the triggered samples with the target class in the embedding space, which corresponds to their low ASR in the downstream task, as shown in Figure [ (Right).
Figure 1: Left: normalized similarity between the trigger cluster and the target cluster, when perform- ing SimCLR on data poisoned by different attacks; Right: downstream attack success rate.
The failure of these existing attacks prompts us to find a more dedicated backdoor trigger design that fits better to the CL paradigm. In this paper, we propose Bi-Level Trigger Optimization (BLTO) method to keep the triggered data staying close to the target-class data throughout the CL procedure. Specifically, in our bi-level optimization objective, the inner optimization aims to simulate the victim contrastive learning procedure when the feature extractor is trained on a backdoored dataset, and the outer optimization updates a backdoor generator which perturbs the imput approach the target cluster in the embedding space. By alternatively performing the inner and outer update, we guide our attack to identify a resilient trigger design that could keep high similarity between the triggered samples and the target-class samples during the CL procedure. In summary, our key contributions are highlighted as follows:
We identify the weakness of existing backdoor attacks on CL, that existing trigger designs fails to mislead feature extractor to embed trigger-injected data close to target-class data, limiting their attack success rate;
2. We propose novel livel trigger optimization approach to identify resilient trigger design that could maintain a high similarity between the triggered and target-class data in the embedding space of CL;
3. Extensive experiments are conducted under various practical scenarios to verify the effectiveness and tranferability of our proposed backdoor attack. The results suggest that our backdoor attack can achieve a high attack success rate with a low poisoning rate (1%). Furthermore, we provide thorough analyses to justify our attack's ability survive special CL. mechanisms, e.g., data augmentation and uniformity.
2 RELATED WORK
Backdoor Attack in Supervised Learning (SL) The backdoor attack (Gu et al. 2017) aims utilize the trigger embedded in the dataset to control the behavior of the victim model, such as changing the model prediction. Backdoor attacks have well developed in the SL scenario (Li et al. 2022; /oori et al. 2022 Zhu et al. 2023). Most SL attacks (Gu et al. 2017 Chen et al., 2017) work via label polluting (i.e., changing the backdoor data's label as the target label in the victim's ss dataset), which induces the victim's model to associate the trigger with the target class. Such simple paradigm gives birth to multiple efficient attacks, such as the invisible backdoor attack (Li al. 202T). Other SL attacks combine the trigger with the target's s semantic information (known as label consistent attack (Turner et al. 2019 Saha et al., (019g) to avoid the label polluting, such as Narcissus (Zeng et al. 2022), which renders the backdoor dataset more natural.
"Their normalized similarity is calculated the cosine similarity between the centroid of the trigger-injected data and the target-class data in the embedding space, normalized by the averaged cosine similarity among all classes.
2

---

Published as a conference paper at ICLR 2024
Backdoor Attack in Contrastive Learning (CL) Vision-language CL is shown vulnerable to backdoor attacks (Carlini & Terzis 2022) which works by inserting trigger in image and adjusting text to a downstream target. However, without the participation of supervision from training labels or other modalities (i.e., text), attacking CL is generally more difficult (Wu et al. 2018) than in SL. Nevertheless, (Saha et al. (022)) reveals the potential backdoor threats in CL: by embedding triggered data close to the target class, the victim's feature extractor can be backdoored to further mislead predictors in downstream tasks. One line of CL backdoor attacks directly tamper the victim's 's feature extractor during CL procedure (e.g., BadEncoder (Jia et al. 2022 Yue et al.) 2022)). While achieving good attack performance, these attacks need to access/control how the victim performs CL. thus is less practical. On the contrary, we consider a different and more realistic attack setting, where we are only allowed to poison a small set of CL training data and has no control how the victim performs CL. Existing works (Saha et al. 2022 Li et al. 2023) in this line design their attacks associate the trigger with the attack target. Another line of work (Liu et al. 2022 Zhang et al. (022) introduces additional image synthesis to achieve better attack performance. PoisonedEncoder (Liu et al. 2022) creates poisoning input by combining& target input and reference input to associate the trigger with the attack, but it was designed for data poisoning attack in CL. CorrupEEncoder (Zhang et al. (2022) combines reference objects and background images together with the trigger to exploit the random cropping mechanism in CL. for backdoor attack.
Backdoor Defense Most backdoor defense solutions are post- post-processing to either detect abnormality (e.g.. Neural Cleanse (Wang et al.) 2019), SSL-Cleanse (Zheng et al., 2023), DECREE (Feng et al., (023) and
ASSET (Pan et (023)), or mitigate backdoor threat (e.g.. I-BAU (Zeng et al. 2021 1). CLP (Zheng et al.
2022 in the trained model. However, reported in prior works (Jia et al. 2022 Zheng et al.) 2023), these
backdoor defense solutions originally designed for SL fail to deal with CL backdoor threats. To this end,
several CL-targeted defenses have emerged, such as knowledge distillation (Saha et al. 2022) and trigger inversion based on embedding space. These solutions can work directly on the trained feature extractor without the involvement of specific downstream tasks, thus suits better for the CL framework.
3 PRELIMINARIES
In this paper, we focus on poisoning unlabeled training data, such that after performing contrastive learning (CL) on this data, the resulting feature extractor (and downstream predictor) are backdoored to misclassify triggered data into target class. We now briefly introduce CL preliminaries and our thread model.
Contrastive Learning (CL) Classic CL generally adopts a siamese network structure, such as SimCLR (Chen et al. 2020). YOL Grill et al. 2020), and SimSiam (Chen & He| (02T). Though detailed designs and optimization objectives could differ across different CL frameworks, their underlying working mechanisms are similar (Wang& & Isola 2020 Chen & He 2021): maximizing the similarity between augmented views of each instance, subject to certain conditions for avoiding collapse. According to Wang & Isola (020). the mechanism to maximize the similarity corresponds to the alignment property of CL., which aims to align views of the same instance; the collapse avoiding mechanism presents the uniformity property, which can spread views of different instances uniformly in the embedding space. Take SimSiam (Chen & He| 2021) as an example. Formally, given a batch of inputs x E X. SimSiam first applies data augmentations to produce two augmented views: t1 (x) and tx()). where t1, 22 E T and T is a set of augmentation operations. Then, SimSiam optimizes feature extractor f-()) and a projector head p () by maximizing the similarity between t (x) and lz (x) in the embedding space, as the following objective shown:
min Lei (x: 0) E-E,, [S(fo(t 22) s(fo(t(())).z1)) 0.6
(1)
where S(:.-) is a similarity measurement, and z, (i E {1,2}) is the projected output defined as z, (oopgrad(ps(fo(t.. and (x()) is promoted (i.e., the alignment mechanism); by blocking the gradient via stopgrad())) different
where gradient is blocked. By minimizing Eq. ( the similarity between in (x)
instances are better separated in the embedding space (i.e., the uniformity mechanism) (Wang & Isola 2020
3

---

Published as a conference paper at ICLR 2024
Issues in Existing CL Backdoor via Data Poisoning In this type of attack (Saha et al. 2022 Li al. 2023 the attacker collects a small set of target-class data (i.e., reference data), and directly injects specific trigger poison the data. When performing CL on this data to align views of the same instance (i.e., target-class data with trigger injected), the feature extractor could be misled to associate the trigger with the target class. This attack effect can be achieved by the alignment mechanism in the victim CL (i.e., solving Eq. (T)) However, this attack paradigm overlooks the influence of the uniformity mechanism in the victim CL: uniformity encourages dissimilarity among distinct instances (i.c., target-class data with trigger injected), which could impair the association between trigger and target class, thus constraining backdoor performance. This issue can be verified by our observation in Figure and experiment in Figure 4 Beyond the uniformity issue, the data augmentations adopted in the victim CL could damage trigger patterns and impede the backdoor injection, as shown in (Li al. 2023) Yue et al. (022) and our experiment in 6 In summary, directly applying the existing trigger design from SL to CL scenarios (e.g., patch) ignores these special mechanisms in CL (e.g., uniformity and data augmentation), resulting in limited attack effectiveness. This paper aims to find a tailored trigger design that can alleviate these issues and accommodate to CL mechanisms.
4 METHODOLOGY
Threat Model We first give overview of the threat model, considering its goal and capability. In standard contrastive learning (CL) setting, a feature extractor is first pretrained on an unlabeled data to provide feature embeddings, then a predictor is concatenated and trained for a specific downstream task. The attacker's goal is to poison the victim's 's training dataset with a small amount of trigeer-injectd data, such that the resulting feature extractor presents the following behaviors: 1) when the input is triggered, the downstream predictor derived from the backdoored feature extractor will output a target class defined by the attacker; when the input is intact (i.e., without trigger). the feature extractor and the derived predictor will behave like a normal clean model. The attacker capability is limited to access and control a small portion of the victim's training data for CL. Besides, for reference, the attacker can collect some unlabeled data from public source that belong to the target class, and we call it reference data. We consider a practical setting, where the attacker cannot directly access and tamper the victim's 's CL procedure (i.e., no information about the victim's s CL strategies, encoder architectures, Jaraaeees,,,,seees,, the training data distribution, etc).
Desired Trigger Design As discussed Section 3 existing poisoning attacks on CL. adopting non-optimized triggers (e.g., fixed patch (Saha et al. 2022)) ignores the influence of special designs in CL (e.g., data augmen- tation and uniformity mechanism), thus can not well adapt to victim's CL behaviors, causing unsatisfactory backdoor performance. This observation motivates to answer this question: what makes effective trigger design in backdooring CL? We argue that a successful backdoor trigger should be able to survive these CL mechanisms, that is: after performing the CL procedure (with data augmentation and uniformity promoting), the resulting feature extractor will still regard the backdoored data and target-class data to be similar. With this goal in mind, we propose Bi-Level Trigger Optimization (BLTO) method for tailored poisoning attack on contrastive learning.
Our Proposed Bi-Level Trigger Optimization (BLTO) Consider that we cannot access the victim's 's CL information beforehand, to make the trigger survive possible CL mechanisms, our motivation is to simulate the victim's CL behavior via a surrogate CL pipeline, and the trigger is optimized to maximize the similarity between backdoored data and target-class data throughout the surrogate CL procedure. This can be naturally formulated as the following bi-level optimization problem:
max Ex (x))o(((100(x))).oo((t) vi
s.t. arg min E Lcx(x;;
where in the outer optimization, D is a clean dataset, gw : X' - X is a backdoor generator that aims to inject trigger to input x and generate its backdoored version I (x), and x, is the reference data sampled from the target class. In the inner optimization, D, is the backdoored dataset defined as D U g (x,), and recall that Lei (x: 0) is general CL objective as in Eq. II..
4

---

Published as conference paper ICLR 2024
grad
Similarity
Similarity
Projector
share parameter
Festure Extractor
Feature Extractor
Extractor
Extractor
sug
aug Images sample
aug
Target Images
Backdoored
Images
(
sample
sample
Bg()
Backdoored (x,),,
Clean Data 2D
Reference
Clean Data
Backdoored Dataset
(a) Inner Optimization
(b) Outer Optimization
Figure 2:
The overview of our proposed BLTO inner optimization simulates the CL dynamics of surrogate victim (assumes he/she uses SimSiam (Chen & He. 2021 train feature extractor fe); outer optimization finds a backdoor generator Dup that can adapt to such CL dynamics.
Intuitively as illustrated in Figure 2 the inner optimization aims to obtain a feature extractor trained by a surrogate victim the backdoored data Dg, whose backdoor effect could be influenced by aforementioned CL. mechanisms; the outer optimization aims to find an effective backdoor generator that can still mislead the feature extractor to maximize the similarity between backdoored and target-class data. optimized backdoor generator can fool the surrogate feature extractor, it is likely to adapt to unseen victims.
result, if the
Bi-level Optimization Algorithm solve this bi-level optimization problem, we perform an interleaving update for the feature extractor (i.e., inner optimization step) and the backdoor generator (i.e., outer optimiza- Besides, to escape local optimum, in practice, we
tion). Detailed steps are summarized in Algorithm!! ! regularly re- re-initialize the surrogate feature extractor in the interleaving optimization procedure.
Algorithm 1 Bi-Level Trigger Optimization (BLTO)
1: Input: clean dataset D, reference data x,, data augmentation set T
2: Initialize parameters of backdoor generator 2/ and surrogate feature extractor O
3: repeat N iterations:
4: Generate the backdoored dataset DD
9up (x,)
5: for k in {0, 1,... K }} do
b Inner optimization
6: Sample backdoored data batch x ~ Do; sample augmentations tı, to ~ T
7: 0x+1 02 Ve (xi(x; 0s)
8: for j in (0,1, (..,,JJ J 1} do
Outer optimization
9: Sample clean data batch x ~ D; sample augmentations f1, f2 T
10: 4bj+1 f- 1b3 no v S(fo (11 (9 (x))), for (t2(x,)))
11: Output: backdoor generator
5
EXPERIMENTS
5.1 EVALUATION SETUP
Following prior works (aha et al. 2022 Li al. 2023), verify our backdoor attack on three benchmark
datasets: CIFAR-10/-10 (Krizhessky) 2009), ) and ImageNet-100 100. Among them, ImageNet- 100-100 a randomly selected 100-class subset of the ImageNet II SVRC-2220 dataset (Deng al. 2009).
Attacker's Setting When solving the bi-level optimization problem to poison unlabeled data, we adopt SimSiam (Chen & He 202T) as the surrogate CL framework (performance of other CL frameworks (i.e.,
5

---

Published as a conference paper ICLR 2024
Table I: Backdoor performance across different CL strategies.
Cl. Strategy
Attack
Dataset
SimCLR
BYOL
SumSiam
BA
ASR
BA
ASR
BA
ASR
CIFAR-10
90.36%
9.97%
91.53%
10.37%
91.24%
10.09%
Non backdoor
CIFAR- 100
61.12%
1.12%
62.31%
0.89%
62.46%
0,98%
ImageNet-100
71.57%
1.10%
77.92%
0.98%
74.13%
1.03%
CIFAR-10 10
90.13%
20.81%
91. 10.10%
17.04%
91.10%
12.89%
SSL backdoor
CIFAR-100
61.07%
33.61%
62 09.009%
41.58%
62.46%
10.17%
ImageNet-100 100
71.26%
6.20%
77.36%
13.86%
72.33%
13.10%
CIFAR-10
89.84%
32. 18%
91.28%
23.88%
90.25%
34.76%
CTRL
CIFAR-100
61.21%
53.49%
62.28%
85.67%
61.33%
64.98%
ImageNet- 100
71.12%
46.58%
76.91%
43.16%
73.68%
35.62%
CIFAR-10
90.10%
91.27%
91 11%
94.78%
10.18%
84.63%
Ours
CIFAR-100
61.09%
90.38%
62.15%
92.13%
62.69%
86.21
ImageNet- 100
71.33%
96.45%
77.68%
99.82%
74.01%
98.58%
MoCo, SimCLR) refer to Appendix C.6). where we use ResNet-18 (He et al. 2016) as the backbone encoder for CIFAR-10/- and ResNet-34 for ImageNet-100 100. The backdoor generator (. (-) is implemented as
DNN detailed in Appendix B To control the trigger strength, we project the backdoored data I (x) ) in an & ball of the input with radius 8/255. The poisoning rate is set as P = 1% by default, meaning the attacker can poison 1% training data. We assume the attack target class as "truck", "apple" and "nautilus" when attacking CIFAR-10, CIFAR-I00 and ImageNet-100 respectively.
Victim's 's Setting The victim will pretrain a feature extractor on the backdoored data via CL, and train a predictor for a downstream classification task. In default experiment setting, the victim uses SimSiam and ResNet (same as the attacker's 's surrogate setting). However in practice, the victim could use any CL strategies (e.g., BYL (Grill al., 2020). SimCLR (Chen et al., 2020) and MoCo He et al. (2020) and backbone encoders (c.g.. RegNet (Xu et al. 2023)). We evaluate the effectiveness of our trigger design in such challenging scenario, where the victim adopts distinct CL strategies or encoders, and uses training set whose distribution is unseen by the attack. Studies more victim's 's settings can be found in Appendix C.6
Metrics
The attack performance is evaluated by the resulting backdoored model. We use Backdoor Accuracy (BA) to measure the attack stealthiness, calculated as the model accuracy intact inputs (higher BA implies better stealthiness). We use Attack Success Rate (ASR) to measure the attack effectiveness, defined as the accuracy in classifying backdoored inputs the target class (higher ASR implies better effecttiveness).
5.2 PERFORMANCE UNDER DIFFERENT ATTACKING SCENARIOS
Transferability across CL Strategies In Table I we compare the performance of our attack against baselines, i.c., SSL backdoor (Saha al. 2022) and CTRL (Li al. 2023), when the victim uses different CL strategies: SimCLR (Chen et al. (020), B BOL Grill et al. 2020), and SimSiam (Chen & Hc (02T). Besides, we provide the non-backdoor case as reference. The results demonstrate that our attack achieves a remarkably high ASR, while maintaining a comparable BA to the non-backdoor case. Moreover, the backdoor trigger generator optimized via surrogate SimSiam is shown to also generalize well on other CL strategies YOL and SimCLR) adopted by the victim, indicating superior transferable ability of our attack across CL methods.
Transferability across Backbones During trigger optimization, we adopt ResNet-18 (He et al.) 2016) as the backbone encoder CIFAR- 10. However, in practice, the victim could use different backbone. demanding the attack to transfer across different backbones. Therefore, we evaluate our attack performance when the victim uses other backbone architectures: ResNet-34 (He et al., 2016). ResNet-50, MobileNet-v2(Sandle jet al.) 2018), ShuffleNet- V2 (Ma et al.) 2018)), SqueezeNet (landola et al. 2016)), RegNet (Xu et al. 2023) and VIT (Dosvviissky al. 2021) In this evaluation, the victim uses SimCLR as the CL method. Results
6

---

Published as a conference paper at ICLR 2024
on CIFAR-I0 are shown in Table where ACC the benign model's aceuracy when there no backdoor.
We observe stable attack performance even when the victim uses a different backbone encoder, which suggests an excellent transferability of our attack across backbones. Besides, we further use ViT to evaluate the transferability on ImageNet-100 100 in Appendix C.4
Table 2: Backdoor performance of our attack across different backbones (on CIFAR-10).
Backhone Type
ResNet18
ResNet34
ResNet50
MobileNet v2
ShuffleNet V2
SqueezeNet
RegNetX
RegNetY
ACC
90.13%
90.71%
90.86%
81.34% 81.47%
87.91%
79.12% 79.09%
87.71% 87.50%
89.39%
BA
90.10%
90.75%
90.81%
87.83%
89.31%
ASR
91.27%
90.65%
90.75%
91.49%
87.52%
88.93%
93.42%
86.75%
Transferability on Unseen Data Since in practice the
Table 3: Performance on unseen data distribution.
attacker cannot access victim's 's data information, the in-
jected poisoned data could follow a distribution distinct from the victims's is actual training data. We now verify the performance of our attack under such case. Assume the attack poisons CIFAR- 10, while the victim performs CL on a mixed set of CIFAR-I0 and CIFAR-100. We report our attack performance in Table where the ratio of CIFAR- in the mixed set varies to indicate the degree of distribution shift. Note that our attack only poison CIFAR- 10 and the poison rate keeps 1% (i.e., 1% of the whole training set poisoned). Results show that our attack maintains high ASR even when most vietim's 's data are from different distribution, implying good transferability of our attack on unseen data. In summary (Table. 2 and 3). our proposed backdoor attack can work effectively in challenging attacking scenarios, where the actual victim conducts different CL procedures from the surrogate victim.
Ratio of CIFAR-10
1%
25%
50%
100%
ACC
71.81%
77.68%
89.34%
90.13%
BA
71.77%
78.19%
89.56%
90.10%
ASR
99.42%
95.73%
96.87%
91.27%
5.3 EVALUATION AGAINST BACKDOOR DEFENSE
We now evaluate the attack performance against seven typical backdoor defenses via backdoor detection mitigation on CIFAR-10 10. More details about this experiment setup can be found in Appendix A.3
Backdoor Detection Backdoor detection aims to check abnormalities of
Table 4: Performance against
victim's ' models or train datasets. Prior works (Jia et al., 2022; Zheng
backdoor detection methods.
et al. 2023) have shown that existing SL-targeted detection solutions (e.g., Neural Cleanse (Wang et al. 2019)) are less effective in detecting CL. backdoor threats, compared with CL-targeted solutions (e.g., SSL-Cleanse
Defense
Anomaly Index
Neural Cleanse
1.34 1.56
(Zheng et al. 2023). DECREE (Feng et al.) 2023) and ASSET (Pan et al.)
SSL-Cleanse
2023 ). According to the experiment results (The backdoor threat can be
detected if the anomaly index exceeds 2 (Wang et al. 2019). )) in Table.
(covers our performance against DECREE and ASSET), our attack not only breaks the SL-targeted detection solution
4 and Appendix C.5
(i.e., Neural Cleanse), but also successfully survives the more advanced CL-targeted detection solutions.
Backdoor Mitigation Backdoor mitigation aims to neutralize the potential backdoor threats in the model directly. Backdoor mitigation strategies include pruning sensitive neurons (e.g., CL. (Zheng et al., 2022)), adver- sarial training (c.g., I-BAU (Zeng et al.) 2021)), and knowledge distillation (Saha et al. 2022) on clean data. We report the attack performance when these defenses are adopted to verify the resistance of our backdoor attack. As shown in Table. [5] among these methods, at a cost of model accuracy (about -10% on BA), the adversarial training (I-BAU) provides the best
Table 5: Performance against
backdoor mitigation methods.
Defense
BA
ASR
No Defense 90.97%
90.33%
CLP 85.12%
87.57%
I-BAU
80.26%
56.87%
Distillation
89.52%
81.08%
mitigation performance (about - -30 on ASR), but our attack outperforms baselines without defenses as in Table [ Besides, our attack still remains effective under knowledge distillation, which was previously shown to successfully neutralize SSL backdoor (Saha et al. 2022). The effectiveness of our attack under existing common defenses calls for more dedicated defense designs for robust CL pipelines.
7

---

Published as a conference paper at ICLR 2024
5.4 ANALYSIS AND DISCUSSION
This analysis is to justify our understanding about how CL mechanisms influence attack effectiveness. Besides, influence of attack strength (i.e., perturbation budget c and poisoning ratio P) is studied in Appendix C.2 the
ablation study of the bi-level optimization be found in Appendix C.3] and the transferability study across
different hyper-parameters (e.g., victim's batch size and BLTO's surrogate model) is shown in Appendix C.6
Our attack can confuse victim to miscluster backdoored data with the target-class data. We first visualize how the victim encodes data in the embedding space to intuitively show if our initial motivation is realized: if the victim's feature extractor successfully backdoored, it should embed the backdoored data (injected with backdoor trigger) to be close to the target-class data. Suppose the victim trains a ResNet18 backbone on CIFAR-10 via SimCLR, and the attack target is "truck" class. Figure 3 shows the t-SNE visualization (Van der Maaten & Hinton 2008) of data embedded by the victim's 's feature extractor, where black dots are the trigger-injected data and light blue dots (with id=9) are the target-class data. We can clearly observe that in our attack, the cluster formed by backdoored data overlaps with the target-class cluster, which suggests that our attack is more effective in misleading the victim to associate backdoored data with the target class. As a consequence, the predictive head (which takes these embeddings features in the downstream task) will be fooled to misclassify backdoored data into target class, leading to a higher ASR. This result is also consistent with our findings in Figure [ These observations verify the reason why attack is so effective: can successfully confuse the victim's 's feature extractor between backdoored data and target-class data.
(a) Non-backdoor
(b) SSL backdoor
(c) CTRL
(d) Ours
Figure 3: Visualizing data embeddings of victim's 's feature extractor backdoored by different attacks.
Our attack can survive different types of data augmentations. Prior works (Chen al. 2020 He et al. 2020) show that properly-designed augmentations can benefit CL. However from the attacker's s perspective. augmentations adopted by the victim could destroy the trigger pattern thus diminishing the backdoor effect. A successful backdoor attack should be able to survive all possible augmentations. To verify this, we consider a default augmentation set Tvictim [RandomResizedCrrpp RandonHorizomallFiin RandomColorJitter, KandomGrayscalee } that could be used by the victim, following CTRL (Li et al. 2023 ).
A special augmentation is GaussianBlur, whose effect will be separately discussed. Our attack considers all these augmentations in the bi-level optimization. We compare our attack with CTRL (Li et al.) (203))
Table 6: Backdoor performance when the victim uses different combinations of data augmentations.
ImageNet-100 100, with the attack target class as "nautilus". As shown in Ta our attack in general keeps high ASR no matter what data augmentation operations the victim using. Particularly, even when the victim in- cludes the GaussianBlur augmentation, our backdoor attack can still achieve remarkably high ASR (i.e., 99.13%), while CTRL suffers a large collapse on ASR from 35.62 65.62% to 1%. The observation indicates that existing attacks could be fragile to the victim's augmen- tation strategy, and our stable performance demonstrates the necessity of bi-level trigger optimization for surviving possible CL augmentation mechanisms.
Attack
Victim Data Augmentation
BA
ASR
Default Trietim
74.01 %
98.58%
Ours
Triaii +
(GuussianBlur)
72.98%
99.13%
Default Tikai
73.68%
35.62%
CTRL
Triain
[GaussianBlur)
71.04%
1.04%
8

---

Published as a conference paper ICLR 2024
Our attack is less impacted by the uniformity effect of CL. Prior works (Wang & Isola 2020) have shown that the CL objective (e.g., InfoNCE (Oord et al. (018)) in SimCLR) can be interpreted as minimizing an alignment loss (to align views of the same instance) and a uniformity loss (to spread views of different instances uniformly in the embedding space to prevent collapse). These two mechanisms signif- icantly influence the CL backdoor effectiveness: the attack can take the advantage of the alignment mechanism to corre- late the trigger and the target class (Saha et al. 2022); ; the
Along loge
Uniformity
7M
200
50 700
opoch
spoch
(a) Alignment Loss (b) Uniformity Loss
Figure 4: Alignment and uniformity during backdoor training on SimCLR.
contrast, the uniformity mechanism may impair their correlation by imposing dissimilarity across backdoored instances, thus nullifying the attack. Therefore, we provide a new understanding of backdooring CL: a successful attack should minimize the alignment loss on backdoored data to correlate trigger and target, while encouraging a large uniformity loss to retain their correlation across instances. To verify this, we monitor the alignment loss and uniformity loss on backdoored images throughout the victim's 's CL procedure (using SimCLR train ResNet- 18 on poisoned CIFAR-10). Figure compares alignment and uniformity loss when the training data is poisoned by SSL. backdoor, CTRL. or our BLTO attack. We have following observations: 1) our alignment loss is better minimized, which implics a stronger alignment between the trigger and the target: 2) our uniformity loss is prominently higher, which indicates that our attack is more resistant to the uniformity mechanism CL. Our findings again highlight the importance of taking into account the special mechanisms in CL to design tailored backdoor attacks.
Our generated triggers capture global semantics, which explains why our attack tractors, survive CL data augmentation and uniformity mechanism. We visualize the actual trigger pattern by cal- culating the difference between the original image and its backdoored version, i.e., (x) x||. Figure 5 illlustrates an example image of backdooring ImageNet- 100 with the target as "nautilus". We observe that the trigger generated by our attack presents global pattern carrying similar seman- tie meaning as the original image (e.g., the tentacle-like patterns in Figure 5ch. Such property is beneficial in three aspects: 1) a global trigger that is semantically aligned with the original image may not be casily cropped or blurred, thus bringing resilience to various data augmentations (c.g.. RandomSizeCrop. GaussianBlur), which explains Table 6 2) during the outer optimization when trigger is added to target class data, since the trigger captures similar semantic as the target, the similarity between the backdoored image and the attack target is intensified, thus strengthening the cluster formed among backdoored and target-class data, which explains Figure 3 and Figure [ 3) since the semantic of trigger can adapt with the input image, the trigger pattern could differ across instances, thus is less penalized by the uniformity effect in CL, which explains Figure
mislead victim feature ex-
(a) Intact
(h) Backdoored
(c) Trigger
Figure 5: Original image (a), backdoored im- age (b), and their difference is the trigger (c).
6 CONCLUSION
In this paper, we proposed novel bi-level trigger optimization method designed for backdooring contrastive learning frameworks. The method was motivated by our observation that existing backdoor attacks on CL (via data poisoning) fails maintain the similarity between the triggered data and the target class in the embedding space due to special CL mechanisms: data augmentation may impair trigger pattern, and the uniformity effect may destroy the association between trigger and target class. Our proposed method can mitigate such issues by simulating possible CL dynamic, and optimizing the trigger generator to survive it Extensive experiments have verified that our method is transferable to varying victim CL settings and is resilient against common backdoor defenses. Moreover, comprehensive analyses were conducted justify how CL. mechanisms could influence the attack effectiveness and reason the success of our attack.

---

Published as a conference paper ICLR 2024
ACKNOWLEDGMENTS
We would like to thank all anonymous reviewers for spending time and efforts and bringing in constructive questions and suggestions, which help us greatly to improve the quality the paper. We would like to also thank the Program Chairs and Area Chairs for handling this paper and providing the valuable and comprehensive comments.
REFERENCES
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A cookbook of self supervised learning. arXiv preprint arXhv:2304.122100 2023.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
Ji Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 255,255, 2009. doi: 10. 1wt.///...0099....... 5206848.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M. Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 99(3):22-62, 2022. doi: 10.1 1090MMP.202..33344444
Shiwei Feng, Guanhong Tao, Siyuan Cheng, Guangyu Shen, Xiangzhe Xu, Yingqi Liu, Kaiyuan Zhang, Shiqing Ma, and Xiangyu Zhang. Detecting backdoors in pre-trained encoders. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16352-16362, 2023. doi: 10.1109/ CVPRS2279.2023.01569
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21233 2020.
Tianyu Gu, Brendan Dolan Davit,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 0tt.::/70..06777 2017.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. 10.11006 www.119//VP.... 202.6...
10

---

Published as a conference paper ICLR 2024
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), May 2022. doi: 10.1
Alex Krizhevsky. Learning multiple layers of features from tiny images. Jan 2009.
Changjiang Li, Ren Pang, Zhaohan Xi, Tianyu Du, Shouling Ji, Yuan Yao, and Ting Wang. An embarrassingly simple backdoor attack supe-supe learning, arXiv preprint arXiv:2210.07346, 2023.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Ta Tao Xia. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021. doi: 10. 0099/ccc488222.221..00
Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. PoisonedEncoder: Poisoning the unlabeled pre- training data contrastive learning. In 31st USENIX Security Symposium (USENIX Security 22), pp. 3629 3645, Boston, MA, August 2022. USENIX Association. ISBN 978-1-939113-31111 URLhttps 7/www. usw.ii....ggaaa...............
hongbinn 8.
Ningning Ma. Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNer V2: Practical Guidelines Efficient CNN Architecture Design, pp. 122-138. Jan 2018. doi:
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03745 2018
Minzhou Pan, Yi Zeng, Lingjuan Lyu, Xue Lin, and Ruoxi Jia. Asset: Robust backdoor data detection across a multiplicity of deep learning paradigms. In Proceedings of the 32nd USENIX Conference on Security Symposium, SEC '23, USA, 2023. USENIX Association. ISBN 978-1-939133-37-3.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10): 1345-1334, 2009.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. CoRR, abs/1910.00033, 2019.
Aniruddha Saha, Ajinkya Tejankar, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Backdoor attacks on self-supervised learning. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEÉE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 20100//cypr.21880047777

---

Published as a conference paper ICLR 2024
Hossein Souri, Liam Fowl, Rama Chellappa. Micah Goldblum. and Tom Goldstein. Sleeper agent: Scalable hidden trigger backdoors for neural networks trained from scratch. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 19165-19178. Curran Associates, Inc., 2022. URL
wwuring........coooommrmmmm..e http://ss...eeeeeee...pppppeeeea pdf
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. Cornell University arXivlComell Cornell University arXiv, Dec 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9)11), 2008
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723, IEBE, 2019.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In 2021 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021. doi: 10. 10.1109/cyp444437.200 202200002.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1109/cyp:.2118.0033 003933. Utthttp:://
doi www.ooggg..... 1tt.:1109//vpp 2018. 00393
Jing Xu, Yu Pan, Xinglin Pan, Steven Hoi, Zhang Yi, and Zenglin Xu. Regnet: Self-regulated network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 34(1 11:56295567 2023. doi: 10.1 109/TNNLS 2022.31
Chang Yue, Peizhuo Lv, Ruigang Liang, and Kai Chen. Invisible backdoor attacks using data poisoning in the frequency domain. arXiv preprint trxxiv2207.04209 2022.
Yi Zeng. Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2021.
Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu, Meikang Qiu, and Ruoxi Jia. Narcissus: A practical clean-label backdoor attack with limited information, 2022.
Jinghuai Zhang, Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. Corruptencoder: Data poisoning based backdoor attacks to contrastive learning. arXiv preprint arXiv:2211.08229 2022.
Mengxin Zheng, Jiaqi Xue, Xun Chen, Lei Jiang, and Qian Lou. Ssl-cleanse: Trojan detection and mitigation in self-supervised learning. CoRR, abs/2303.09079, 2023. doi: 10448500arXi.23333.000
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitz- ness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. pp. 175-191. Springer, 2022.
Zihao Zhu, Mingda Zhang. Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu, Boosting backdoor attack with learnable poisoning sample selection strategy, 2023.
12

---

Published as a conference paper ICLR 2024
A IMPLEMENTATIONS OF INVOLVED EXPERIMENTS
A.1 IMPLEMENTATIONS OF THE PRELIMINARY EXPERIMENT
this section, we show our implementations of experiment in Fig. [ Assume "truck" as the target class. We strictly follow the official implementations of SSL backdoor (Saha et al.) 2022) (hotps: github..cmmt com UMBCvisionn asl-backdooos CTRL (Li et al. 2023) (https: / //ggttttb...///.//// com/ meet- meet-coli/CIRL attack CL, and the poisoning rate is 1%. The trigger in our attack is optimized under the default setting mentioned in Section. 5
We assume the victim uses SimCLR (codes refer the SimCLR implementation in https://gitthtit con//meet-cjll//TTRR to train a ResNet-18 feature extractor, and evaluate the performance of four attacking scenarios (i.e., non-backdoor, SSL backdoor, CTRL, and our optimized backdoor attack). The temperature (Wang & Liu 2021) of the InfoNCE (Oord et al. 2018) loss is 0.2. The victim's augmentations are listed in Algorithm
The downstream predictor is a knn monitor by default, the implementation of this knn monitor can refer to https://githut: com/ Patric /anuaa/iimi SSimmmmiii iam When training the downstream predictor, following CTRL (Li et al. 2023), we use the clean CIFAR- 10 training set as the downstream training set, and use the CIFAR- 10 testing set for performance evaluation. We train the feature extractor for 800 epoch, and for each epoch we take the feature extractor and evaluate the normalized similarity and the attack success rate (ASR).
Specifically, when calculate the normalized similarity, we first collect data batches (xe ) of each category ys in the CIFAR-10 training set. Then we use the feature extractor fo to encode these data batches, thus gain their representation batches [fe(x,) (xxx)== )=== Then, we calculate the center point Cg of each representation batch fe (xx).. On the other hand, we calculate the center point Csa for the representations of those trigger-injected data (i.e., those data (in the CIFAR-I0 training set) combined with the trigger). Then, the normalized similarity SN can be defined as eq (3):
S(Cra, Cgt) SN average((S(Cha Cx) C=)
(3)
where tgt is the target category (i.e., "turck"), and S(.,:) measures the similarity between two inputs.
A.2 REFERENCE DATA
As introduced in Section 5 our attack goals are: "truck" for CIFAR-10, "apple" for C1F and "nautilus" for ImageNet-100 100. When preparing the backdoored dataset, we require clean dataset and a batch of reference data x, (of the target category). Under the default poisoning rate (1%), ratio of data number in D and x, is 99 : 1. Specifically, we enunciate the components of DD and x, when preparing the backdoored dataset for inner optimization in Algorithm [ (or for the victim's backdoor training).
(1) CIFAR-10: X, is a set of 500 images (belong to category "truck") randomly sampled from the training set of CIFAR-10, and D is the set of the remaining 49500 images.
(2) CIFAR-100: x, is a set of 500 images (belong to category "apple") sampled from the training set of CIFAR-100) and D is the set of the remaining 49500 images.
(3) ImageNet-100: x, is set 1300 images (belong to category "nautilus") sampled from the training set of ImageNet- 100, and D is the set of the remaining 128700 images.
13

---

Published as a conference paper ICLR 2024
A.3 DATA AUGMENTATIONS
Without specific notations, the augmentation T used to optimize the trigger is [RandoomReszzdCCrop.. Ran- domHorizontalFiip, RandomColorJitter, RandomGrayscale, GaussianBlur), whose detailed implementation shown in Algorithm
Algorithm 2 Default augmentation for trigger optimization (Pytorch version).
Transform
T. Compose (1
T. Randomkest zedCrop (image_size, scale= 1,0), antialias=True),
R.nndommer zontalFiip
T. Randonamppyyy [.T.ColoJiiteerr (0.4, 0,4, 0.4, 0,1)1, p-0.8),
7. Randommrayscaa (p=0..2,
7. R.Randomkpply (,T, GaussianBlur hir.ee/seeeiimeeeeeeeee///////
sigma-10.1) 2.0))
=0.5),
T.TeTensor (),,
I.Normalize (*mean_atd) The mean_std depends on the involved dataset.
On the other side, following previous works (Li et al. victim is defined as (RandomResizedCroo.. RandomHorizzntalFFii RandomColorJitte, RandomGrayscalej, whose detailed implementation is shown in Algorithm 3 In Section S we have specifically discussed the
2023), the default data augmentation used by the
scenario where the victim uses data augmentation Algorithm (i.e., considering T Gaussi (asussiaaBlur) to train CL models.
Al Algorithm 3 The default victim's s data augmentation (Pytorch version).
Transform
T. Comopeee [
Randommess zedCrop Limage_eize scale- (0.2, 1.0), antielias-Trus), Randomftc 1 aootaalFiig..
2. R.Rndookgylyy T, Colorditter
0.4, 0.4, 0.1)1, p-0.8),
T. Random@rayscale (p=0.21,
ToTesorr (),
T. N.Normal (*mean_std) stdd The mean_std depends on the involved dataset.
11
A.4 IMPLEMENTATION ON CL TRAINING
Our experiments in Section 5 Jinvolve three CL strategies: SimSiam, SimCLR, and BYOL. The implementation of SimSiam follows the implementation of hhtps::: / cow/PatrickHua /Siimmis SimS i am on all three data benchmarks CCFFAR-10/100 ImageNet- Besides, the implementations of BYOL and SimCLR refer to ht : http:/gitttttt com wwt///ett-nni///ceee
Following previous works (Li et 2023), when training a predictor for evaluating the backdoor performance, we use the clean training set in CIFAR-10/ 10/-100 and ImageNet- 100 as the downstream training set. We use the testing set CIFAR-10V-100 and the validation set of ImageNet- 100 for performance evaluation. The predictor is knn monitor by default.
A.5 IMPLEMENTATION ON BACKDOOR DEFENSE
We strictly follow the official implementations of the involved backdoor defense solutions (if they are open-sourced). We first list the official urls (U-BAU, CLP, and Neural Cleanse) follows:
(1) I-BAU: pttps::/giitii: perform the unleariin step of I-BAU. We report the ASR and BA of our attack after running I-BAU for 50
We use the clean CIFAR-10 10 training set to
14

---

Published as conference paper ICLR 2024
epochs, Noteworthy, as the I-BAU supervised-leamingg targeted defense solution, we use the MLP with three linear layers (the same implementation as the MLP in /trttp:://githss wwt.:iinmmninannnn.. jia/ BadEncoder) replace the knn monitor as the downstream predictor before performing I-BAU. Besides, we perform the same operation when defending via CLP and Neural Cleanse.
(2) CLP: https://gi thub. com/
porppprunni ng
Noteworthy, CLP (Zheng et al. 2022) has an important hyperparameter u. Under lower CLP lowers the threshold to judge and prune the suspicious neurons of the backdoored DNN model, which is more likely to mitigate the backdoor threat. However, more neurons are pruned, backdoor accuracy (BA) performance will drop simultaneously. In Table 5 we reported the backdoor performance of our backdoor attack with the set 4.0 in CLP.
75
50
5.5
5.0
4.5
4.0
3.5
3.0
2.5
20
1.5
1.0
0.5
Figure Ours attack success rate (ASR) retain a high level within mainstream implementations (u from 5.5 2) of CLP. Attacking CIFAR-10 (target is "truck") ResNet18 model trained via SimCLR.
In practice, the defender may attempt different to perform CLP. To this end. in Figure. we provide our backdoor performance or our attack under CLP with the values of ranging from to 5.5. As observed, under most implementations of u (when the BA after defense is acceptable (e.g., BA> 80%)), our backdoor attack can still retain high attack success rate.
(3) Neural Cleanse: httpes//gthhaa.ccco/o/ooaarrrrrr
(4) DECREE:
(5) ASSET: http://sss..tttoooooooooooooooos
Then, we discuss our implementations on the remaining backdoor defense solutions that are not open open-soureed.
(I) SSL-Cleanse: The implementations on the trigger reversion follow the original paper (Zheng et al. 2023). and the outlier detection (on those abnormal reversed triggers) methods refer to that of the Neural Cleanse
(2) Data Distillation: We follow the official implementation reported in (Saha et al. 2022 and the dataset used for data distillation is the whole training set (clean) of CIFAR-10.
B BACKDOOR IMAGE GENERATOR
this paper, we select the DNN generator g(.) X > X' to produce backdoor data (i.e., g(x)). The specific architecture of g(-) is shown in Figure. 7
The specific implementation (Pytorch) for the g(.) X - X' can refer to https://github.
amma Naseer/TTP
15

---

Published as conference paper ICLR 2024
6x
Convelock1,
Reslllock 236
Upölock,
3-3 Come athieen
Battshllm
3-3 Coni othoer
BatoNoum
Book Pi
T:7 Come entilen
Nomellznn memelzalii
RallU
Figure 7: The architecture of g(-).
C ADDITIONAL ATTACK ANALYSIS
C.I ALIGNMENT AND UNIFORMITY LOSS ANALYSIS
Additional Alignment and Uniformity Loss on BYOL and SimSiam. the main text, we exhibit the variation of the alignment and uniformity loss (we monitor them the backdoored data exclusive) throughout the victim's 's training (via SimCLR) in Figure 4 When conducting the experiment, we assume the victim trains R feature extractor
CIFAR-10, the attack target is "truck", and the poisoning rate P is 1%. In Figure 8 we further exhibit the variation of alignment and uniformity loss when attacking BYOL and SimSiam, where we can observe the similar phenomenon with that in SimCLR (Figure
SSL backstoor
SSL backdoor
CTRL
CTRL
-1.22
-1.50
--125
Undlorma
2:70
3.3
0.
-325
8.0
366 406
792 899
100
200
400 S00 U5e
199
500
538 208
300 400 530 630 780 853
spoch
epoch
epoch
epoch
(a) Alignment Loss
(b) Uniformity
(c) Alignment Loss
(d) Uniformity Loss
Figure 8: Alignment and uniformity losses during backdoor training on BYOL (a)(b), and SimSiam (c).
C.2 IMPACT OF ATTACK STRENGTH
Impact on and Poisoning Ratio P. Fig. exhibits the
ablation study on the clamping threshold (i.c.. e in the loo
ball) and the poisoning rate (P). We set as 8/255 and P
1% when performing our bi-level optimization by default.
When performing ablation study on we fix P as 1%. When performing the ablation study on P.
fix 8/255. We assume the victim uses BYOL to train a ResNet18 backbone on the CIFAR-10. According to Fig. 99 our ASR positively correlates with the trigger intensity and poisoning rate. which is consistent with prior works (Saha et al. [2022] Li et al.) (2023).
4225
(a)
(b)
Figure (a): Ablation study on c. (b):
Ablation study on poisoning rate (P).
These results indicate that can conveniently control the backdoor performance by adjusting these hyperparameters, rendering it suitable for different attacking scenarios.
16

---

Published as a conference paper ICLR 2024
C.3 IMPACT OF INNER AND OUTER OPTIMIZATION
Impact of inner and outer optimization in our BLTO attack. In this section, we discuss the function of the inner and outer op- timization in our bi-level optimization. We first exhibit an ablation study on these two components. Specifically, we assume the victim trains ResNet- 18 feature extractor over CIFAR- 10 using SimCLR, B BYO,, and SimSiam. Then, we evaluate the attack success rate of three scenarios: (1) Our attack is optimized via both inner and outer optimization. (2) Our attack is optimized via outer optimization only. (3) Our attack is not optimized (i.e., performing inner optimization only). Based on the results in Fig. 7 we observe that the attack optimized exclusively via outer optimization presents volatile backdoor performance over different CL strategies (e.g.. < 20% ASR in SimSiam). Such phenomenon highlights the significance of the constraint condition (i.e., 0 = arg ming Ex-D, Lcr. (x 0)) in eq 4 without which may render the optimized gu (.) deviate from the solution of eq (2). the (.) fails to meet eq (2) during the sur- rogate CL training, the (n)()) is less likely to perform robustly in practical attacking scenarios. Nevertheless, according to the acceptable attack performance of bi-level and inner-optimization
Table 7: Bi-level optimization matters for an effective CL backdoor attack.
CL. Method
Altacking Scemarioo
SinCC R
BYOH.
SimSiom
Bi-level optimicatiinn
91.27%
97.17%
84.63%
wio inner optimization
71.98%
60.1355
17.44%
wio outer optimization
12 12%84%
11.97%
12.01%
attacks in Table 7, we can smell the potential of these optimization- based non-fixed triggers for compromising the CL framework,
C.4 ATTACK TRANSFERABILITY ON MODERN BACKBONES
This section supplements the performance of our BLTO attack on non-CNN architecture, VIT (Dosovit, skiy al. 2021.. Specifically, the attacker uses ResNet34 the surrogate model train the trigger generator over ImageNet- 100 (target is "nautilus"), then attack the fietim's ViT-small/16 encoder in practice with the poisoning rate as 1%. The attack performance is shown as follows (along with attacking ResNet34 presented in Table 1:
ResNet34
VIT
BA
71.33%
63.39%
ASR
96.45%
82.66%
Table 8: Transferability on non-CNN architectures.
C.5 PERFORMANCE ON MORE CL DEFENSE METHODS
DECREE DECREE Feng et al. 2023) is trigger inversion backdoor defense method tailored for encoders trained in SSL manner. We re-immlemnnt DECREE on our backdoored ResNet-18 18 encoder trained via SimCLR and BYOL on CIFAR-10. The reversed trigger and corresponding Ll-norms of the trigger mask m (along with PI- PI-norms) are demonstrated in Figure I0 According to their works, when the encoder's 's Pl-norm is lower than 0.1, it will be regarded as backdoored. Based on the result in Figure 10 we find our attack can effectively avoid the detection of DECREE.
ASSET ASSET (Pan et al. 2023) aims to separate backdoored samples from clean ones by inducing different model behaviors between the backdoor and clean samples to promote their separation. We re-implement their defense solutions on our backdoored CIFAR- 10 dataset, and the result can be found in Table @ To be specific, we utilize our synthesized trigger CIFAR-10 (target is 9) to generate a poisoned CIFAR-10 (poisoning rate is 1%. our default setting). The real poisoned index is the first 501 indexes of data points (whose target id is 9) in CIFAR-10, and the poisoned feature extractor 0% is the ResNet 8 backbone trained on the poisoned CIFAR-10. The loss function utilized in the outer optimization loop (both in min step and max step) is Lvar
(i.e., Eqn. in ASSET (Pan et al. 2023
17

---

Published as conference paper ICLR 2024
(a) clean simelr (b) backdoor simelr (c)
clean
byol (d) backdoor byol
(LI-Nomm-582556
(LI-Nom=719.588. Pl Pl-norm=0.23)
(L1-Norm=638.77, Pl-norm=0.20)
(LI-Norm=604.38,
P1 P1-norm=0.19)
Pl-norm=0.20)
Figure 10: Reversed trigger patterns of DECREE (backbone is ResNet18).
TPR
FDR
5.39%
32.17%
Table 9: ASSET's true Positive Rate (TPR) and false Positive Rate (FPR) of backdoored data sifting on our backdoored CIFAR-10 dataset.
TPR depicts how well a specific backdoor detection method filters out the backdoored samples. A higher True Positive Rate (TPR) (closer to 100%) denotes stronger filtering ability. False Positive Rate (FPR) depicts how precise the filtering is: when a specific method achieves TPR that is high enough, FPR helps us to understand the trade-off, i.e., how many clean samples are wasted and wrongly flagged backdoored during the detection. A lower FPR shows that fewer clean samples are wasted, and more clean data shall be kept and available for downstream usage. According the metrics ASSET, TPR can be calculated as follows: TPR 5.39%, and FPR = 32.17%. It indicates that our poisoned data can effectively evade the detect of ASSET.
C.6 IMPACT OF OTHER HYPER-PARE PARAMETERS
Attacker's surrogate model. In Table I assume the attacker uses Simsiam Chen & He (022) the surrogate method in our BLTO attack. This section talks about the attack performance under another two surrogate CL methods (BYOL (Grill et al. 2020) and SimCLR (Chen et al. 2020)), which is shown in Table
10
SSL Method
SimCLR ACC
BYOL
Simsiam ACC
ASR
ACC
ASR
ASR
SimSiam
90.10%
91.27%
91.21%
94.78%
90.18%
84.63%
BYOL
89.83%
71.95%
90.70%
76.40%
90.06%
74.08%
SimCLR
89.27%
91.45%
90.13%
78.58%
89.36%
87.62%
Table 10: The attack performance on CIFAR-10 with different CL methods our BLTO attack.
It indicates that our BLTO attacks can suit different CL methods.
Victim's batch size and learning rate. In practice, the victim may different hyperparameters (c.g., batch size, learning rate, or temperature) to train their encoders. This section provides our attack performance under these practical scenarios. The ablation study is conducted CIFAR-10, the victim's CL method is SimCLR and backbone is ResNet18. The attacker's surrogate CL method is SimSiam (the same setting with that in Table 1), the attacker uses batch size as 512, and learning rate scheduler (base Ir 0.03, final Ir O).
18

---

Published as a conference paper ICLR 2024
The ablation study on batch size is shown in Table 11 SimCLR in Table 13
learning rate is shown in Table 12 and temperature of
256
512 (default)
1024
x0.5
x (default)
x2
BA
86.05%
90. 10%
90.85%
BA
88.05%
90.10%
88.76%
ASR
90.88%
91.27%
89.39%
ASR
91.80%
91.27%
92.54%
Table 11: Batch size ablation.
Table 12: Learning rate ablation.
0.1
0.2
0.5
0.8
BA
90.38%
89.97%
90.07%
90.14%
ASR
10.18%
12.22%
26.81%
81.92%
Table 13: Temperature (r) ablation.
Victim's 's CL model. In Table conducted the attack evaluation when the victim uses SimCLR, BYOL and SimSiam as the CL method. This section we conduct a supplementary evaluation on MoCo He et al. (2020). We assume the attacker is going to attack "truck" in CIFAR-10 (use SimSiam the surrogate CL method), and the victim uses ResNet18 as the backbone. The victim's queue length is set as 4096, batch size is 512, temperature is set 0.1. Besides,
additional evaluate other three attacking scenarios: non-backdoor, CTRL et al. (2023) and SSL-backdoor (Saha et al. (2022). The experimental results are show in Table [4 which indicates that our BLTO attack can also outpertorm other attacks on MoCO.
non-backdoor SSL-backdoor
CTRL
ours
BA
85.41%
90.10%
90.03%
89.42%
ASR
90.88%
91.27%
95.54%
96.84%
Table 14: Performance of different attacks on MoCo.
Victim's 's downstream task. In practice, the victim may different downstream dataset to finetune their resulting model with the pre-trained one. Though parts of these situation have been discussed in Table this section discuss more practical and challenging situation: the downstream dataset is totally different with the pre-trained model (no overlapping). We assume the victim trains a ResNet18 encoder on our backdoored CIFAR-I0, and the downstream dataset is STL-10 (train part). The results of our attack performance under this situation is shown as Table 15 It indicates that our BLTO attacks doesn't 't demand the overlapping between the pre-trained dataset and downstream dataset to claim an remarkable attack performance.
SimCLR
BYOL
SimSiam
BA
76. 73%
79.04%
80.47%
ASR
94.74%
96.19%
87.55%
Table 15: Downstream dataset ablation (STL10).
C.7 SUPPLEMENTARY IMPLEMENTATION DETAILS OF INVOLVED CL ATTACKS.
Note that the performance of our re- re-implemented CTRL Li et al. (2023) in Table iis a bit different from the original paper Li al. (2023). In fact, the ASR difference is due the contrastive learning (CL) implementation adopted by the victim, which can be seen from the model accuracy on clean data: using our CL implementation, the victim encoder achieve almost 90% BA (i.e., ACC) CIFAR- 10; while with the
19

---

Published as conference paper ICLR 2024
original CTRL's CL implementation, the encoder achieves 80.2% ACC on CIFAR-IO (i.c., Table CTRL et al. 2023)). Note that in our threat model, the attacker cannot control how the victim trains the actual CL model. In practice, victims would prefer adopting CL that produces more accurate encoder with higher ACC. In fact, our CL implementation follows standard practices with matching ACC: SimCLR (Table B.5 in
[4]) reports 90.6%.
CL Strategy
SimCLR
BYOL
Simsiam
ACC
ASR
ACC
ASR
ACC
ASR
88.05%
73.01%
90.49%
58 18.64%
88.57%
3.48%
Table 16: The attack performance of Narcissus (universal perturbation with 32/25)) Coo ball).
20


Reviews:
[{"review_id": "cRIefile1Q", "reviewer": "Reviewer_Saos", "summary": "This paper proposes a new method to perform poisoning-based\nbackdoor attacks on contrastive learning. It searches\noptimal triggers by using a designed bi-level optimization\napproach on the surrogate models. Experiments on three\ndatasets (CIFAR-10, CIFAR-100, and ImageNet-100) validate\nthe effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Backdoor attacks on contrastive learning is an important direction.\n\n* Analysis of the reason that the proposed attack works\nwell is discussed in section 5.4.", "weaknesses": "* The proposed method requires a surrogate to perform\nbi-level optimization. This paper mentions it uses SimSiam\nwith ResNet18 as the surrogate model. The success of the\nproposed method is based on the transferability of the\ntriggers optimized on the surrogate model. While Table 1 and\n2 demonstrate the proposed method has good transferability,\nthe evaluation is not comprehensive. It is suggested to\ninclude more self-supervised learning methods such as Jigsaw\n[1], MoCoV2 [2], and DINO [3]. For different model\narchitectures, the results under modern architectures such\nas ViT and RegNetY are missing (Note that ViT and RegNetY\nare commonly used in self-supervised learning related\nresearches like\nhttps://github.com/facebookresearch/vissl/blob/main/MODEL_ZOO.md).\nSince the transferability is very important to the\nperformance of the method, it is suggested to conduct more\nextensive experiments. For example, adding the results under\ndifferent architectures to Table 1 or adding the results\nunder different CL methods to Table 2 (So that it will have\na Table including the results under different combinations\nof architectures and CL methods). \n\n* The hyper-parameters such as the learning rates of the\nmodels might also influence the performance of the proposed\nmethod. For example, if the learning rates and the batch\nsizes of the surrogate model are significantly different from\nthose used by the victim models, then the attack success\nrates might also reduced. It is suggested to add the\ndiscussion about this.\n\n* In the experiments, this paper assumes that the downstream\ndataset and the pre-training dataset used for self-supervised\nlearning is the same. Typically, the downstream users will\nuse different datasets to conduct the downstream training.\nMany existing works such as BadEncoder [4] also mainly\ninvestigate this practical scenario. The results under this\npractical scenario are missing in this paper.\n\n* Carlini et al. [5] is an important existing work in the\nfield of poisoning-based backdoor attacks on contrastive\nlearning. Although it mainly focuses on the vision-language\ncontrastive learning, the comparisons, and the discussion\nabout it is still important.\n\n* The robustness under Feng et al. [6] is not discussed. Is the\nbackdoor samples in the proposed method have high cosine\nsimilarity between each other?\n\n* The usages of the surrogate models and the bi-level\noptimization is not new in the field of poisoning attacks\nand backdoor attacks [7,8], which somehow weaken the\ncontributions of this paper.\n\n\n\n[1] Noroozi et al., Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. ECCV 2016.\n\n[2] Chen et al., Improved Baselines with Momentum Contrastive Learning. arXiv 2020.\n\n[3] Caron et al., Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.\n\n[4] Jia et al., BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning. IEEE S&P 2022.\n\n[5] Carlini et al., Poisoning and Backdooring Contrastive Learning. ICLR 2022.\n\n[6] Feng et al., Detecting Backdoors in Pre-trained Encoders. CVPR 2023.\n\n[7] Souri et al., Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch. NeurIPS 2022.\n\n[8] Zhu et al., Boosting Backdoor Attack with A Learnable Poisoning Sample Selection Strategy. arXiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fm9kIBjqx4", "reviewer": "Reviewer_wKXh", "summary": "This paper formulates backdooring contrastive learning (CL) as a bi-level trigger encoder optimization problem. They claim that existing attacks using fixed triggers fail to maintain similarity between triggered data and target class in CL's embedding space due to data augmentation and uniformity effect, limiting their success. The proposed method formulates a bi-level optimization that simulates the victim's CL training dynamics in the inner level and optimizes a trigger generator in the outer level to keep triggered data close to the target throughout the inner CL training. This results in resilient triggers that can survive CL mechanisms. Experiments show the attack achieves success under varying victim settings and defenses. Analyses are provided on how CL mechanisms affect attack performance. The optimized triggers capture semantics related to the original input, explaining the attack's effectiveness.", "questions": "The proposed bi-level optimization formulation is solved using a discrete approximation without much discussion on the impact of using more efficient yet accurate Hessian approximations or evaluates the convergence of the proposed solution. Bi-level optimization often benefits from analyzing such approximations rather than directly providing a discretized solution.\n\nThe evaluation results comparing against baselines may contain erroneous implementations. In particular, the reported attack success rate (ASR) for SSL backdoor differs notably from values in the original paper, which because the low efficacy of their work, they even used a different metric based on number of misclassified samples. Also, the ASR for CTRL is much lower than results from recent works (e.g., the original paper and [1]) that show CTRL can achieve above 80% ASR with SimCLR on CIFAR-10, contrasting the authors' significantly lower values. Additional to the original papers, another separated work [1] evaluated these two attacks also confirms the potential erroneous implementations in this work.\n\nThe related work could be expanded and compared more thoroughly. Some recent attacks [2] also leverage synthesized triggers using solely the target class, similar to the proposed approach, which is worth to be incorporated and compare. Additionally, a recent backdoor sample detection method [1] demonstrates effectiveness in detecting poisoned samples in contrastive learning unlabeled datasets, which is relevant but not discussed or evaluated.\n\n[1] Pan, Minzhou, et al. \"ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms.\" Usenix Security (2023).\n\n[2] Zeng, Yi, et al. \"Narcissus: A practical clean-label backdoor attack with limited information.\" ACM CCS (2023).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This work provides a formulation of the backdoor problem in contrastive learning as a bi-level optimization to identify a backdoor generator that is able to generate triggered images.\n\n- The authors provided an approximated solution to the formulated bi-level optimization.\n\n- The authors evaluated three datasets and compared them with two existing attacks. The attack is further evaluated with existing defenses from two lines of work (model-based backdoor trigger detection and model-based backdoor mitigation).\n\n- The writing is clear and easy to follow.", "weaknesses": "- The paper lacks analysis or discuss on the impact of using more accurate Hessian approximations to solve the proposed bi-level optimization, relying only on a discrete solution.\n\n- The baseline implementations and results seem questionable based on inconsistencies with original papers and recent related work- the attack success rates for SSL backdoor and CTRL differ notably from prior reported values.\n\n- The related work review and experiment scope is too narrow:\n   1. Recent attacks using similar target-class-based trigger synthesis are not compared to.\n   2. A relevant defense for detecting backdoor samples in contrastive learning is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PA3PM0lay1", "reviewer": "Reviewer_KWn8", "summary": "This paper identifies that the current data poisoning-based backdoor attacks on contrastive learning adopt a fixed trigger design and have a limited attack success rate. To overcome this limitation, a novel bi-level optimization approach is proposed. In this framework, the inner optimization simulates the contrastive learning (CL) dynamics of a surrogate victim, while the outer optimization optimizes the trigger generator and ensures that the backdoor trigger remains close to the target throughout the surrogate CL procedure. Extensive experiments are conducted to compare the proposed methods with state-of-the-art (SOTA) attacks, such as SSL backdoor and CTRL, demonstrating superior attack effectiveness. Furthermore, the proposed methods can effectively evade existing SOTA defenses.", "questions": "1. Is the trigger generator alone sufficient for the success of the proposed attack?\n\n2. How does this work differ from the \"PoisonedEncoder\"?\n\n3. If the reference data $x_r$ is randomly sampled from the target class? does it need to be included in the downstream dataset to ensure the success of the attack?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed attack method is novel and shows superior effectiveness in comparison with the SOTA.\n\n2. The experiments are comprehensive; the authors compare the proposed method with different attack methods, evaluate it against backdoor defenses, and also discuss the effect of various data augmentations.\n\n3. The overall writing is good. The methodology and experimental results are not difficult to comprehend.", "weaknesses": "1. The motivation could be better articulated. The authors claim that the fixed trigger design leads to limited ASR. However, in the methodology, not only is the trigger generator adopted, but a bi-level optimization strategy is also used to optimize the trigger generator. This raises the question: Is the trigger generator alone sufficient for the success of the proposed attack? If not, it suggests that the fixed trigger is not the primary cause of the current limitation. I recommend that the authors conduct an ablation study on this matter and be cautious with their claims.\n\n2. In the experiments, only SimSiam is adopted as the surrogate Contrastive Learning (CL)  method. The experimental results demonstrate that selecting this framework indeed achieves good performance, but it does not provide a direct rationale for choosing SimSiam. It is possible that using SimCLR or BYOL could yield better results, and it is recommended to supplement this part with additional experiments for verification.\n\n3. The comparison with other recently developed works could enhance the contribution:\n(1) PoisonedEncoder: Poisoning the Unlabeled Pre-training Data in Contrastive Learning.\n(2) CorruptEncoder: Data Poisoning based Backdoor Attacks to Contrastive Learning.\nNotably, in \"PoisonedEncoder,\" a bi-level optimization strategy is also employed to formulate the attack. How does this work differ from theirs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cWMIQSweAY", "reviewer": "Reviewer_Tcds", "summary": "This paper introduces a poisoning-based backdoor attack for Contrastive Learning (CL), targeted at the feature extractor. Through the bi-level optimization that simulates a backdoored CL pre-trained model in the inner loop, it trains a trigger generator to produce poisoned samples with *robust* triggers that survive the data augmentation of CL. These triggers also exhibit transferability across serveral victim CL training strategies and backbone architectures. Experimental validation confirms the effectiveness, transferability and robustness of the attack.", "questions": "- In the evaluation on transferability, the adopted backbone encoders are all of CNN architecture (e.g., ResNet, MobileNet, ShuffleNet, SqueezeNet), and the datasets are just CIFAR-10 and CIFAR-100. More diverse choices of backbone architecture and dataset may be necessary, such as the architurecture of ViT and more challenging datasets like ImageNet.\n- Besides, though the proposed attack targets at the feature extractor, the victim settings in the experiments are limited to the classification task. I think it could be extended to more tasks to demonstrate the effectiveness of the proposed attack.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The motivation behind the attack is well-described by incorporating the alignment and uniformity inherent in CL. The method for training the trigger generator is succinctly presented in an 11-line pseudocode.\n- The experiments are comprehensive, including experiments of CL backdoor defense and the transferability in three aspects (i.e., CL training strategies, model architectures and datasets).\n- The paper is well-written and easy to follow.", "weaknesses": "- **Lack of details.** The implementation of the proposed attack seems to lack some details:\n    - The setting of K and J are not included in the submission, and their relationship with N remains unclear. Is K large enough to ensure the convergence of the surrogate backdoored model?\n  - Additionally, does the x-axis in Figure 4 refer to N? If it includes the inner loop updates (N*J), does it make the comparison of loss curves somewhat unfair?\n  - What does the expression *regularly re-initialize the surrogate feature extractor* (in section 4) mean? Does it imply that, after the initialization (line2 in Algorithm1), there is a subsequent re-initialization at some point?\n- What determines superior transferable ability of the chosen surrogate CL framework (SimSiam)? The fundamental factors may need further analysis and clarification in ablation experiments, such as the choices of different data augmentations.\n- The BLTO procedure contains both a backdoor generator and a backdoored surrogate model θ. Does the co-training surrogate backdoored model perform as well as an backdoored model actually trained on the poisoned data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)