ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training
Authors: Xin Yao
Haiyang Zhao
Yimin Chen
Jiawei Guo
Kecheng Huang
Ming Zhao
Venue: NeurIPS 2025 poster ()
Keywords: CLIP
text-based attack
data poisoning
backdoor attack
adversarial training
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: The Contrastive Language-Image Pretraining (CLIP) model has significantly advanced vision-language modeling by aligning image-text pairs from large-scale web data through self-supervised contrastive learning. Yet, its reliance on uncurated Internet-sourced data exposes it to data poisoning and backdoor risks. While existing studies primarily investigate image-based attacks, the text modality, which is equally central to CLIP's training, remains underexplored. In this work, we introduce ToxicTextCLIP, a framework for generating high-quality adversarial texts that target CLIP during the pre-training phase. The framework addresses two key challenges: semantic misalignment caused by background inconsistency with the target class, and the scarcity of background-consistent texts. To this end, ToxicTextCLIP iteratively applies: 1) a background-aware selector that prioritizes texts with background content aligned to the target class, and 2) a background-driven augmenter that generates semantically coherent and diverse poisoned samples. Extensive experiments on classification and retrieval tasks show that ToxicTextCLIP achieves up to 95.83\% poisoning success and 98.68% backdoor Hit@1, while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses. The source code can be accessed via https://github.com/xinyaocse/ToxicTextCLIP/.

ToxicTextCLIP: Text-Based Poisoning and Backdoor Attacks on CLIP Pre-training
Xin Yao', Haiyang Zhao', Yimin Chen², Jiawei Guo', Kecheng Huang', and Ming Zhao' School of Computer Science and Engineering, Central South University, China Miner School of Computer & Information Sciences, University of Massachusetts Lowell, USA
skinyao,zzaohan zhaohaiyang, jiawe kechenghuang, meanzhao)@cau edw. cn
edu
Abstract
The Contrastive Laanuage-Imagg Pretraining (CLIP) model has significantly ad- vanced vision-language modeling by aligning image-text pairs from large-scale web data through self-superviied contrastive learning. Yet, its reliance on un- curated Internet-sourced data exposes to data poisoning and backdoor risks. While existing studies primarily investigate image-based attacks, the text modal- ity, which is equally central to CLIPP's training, remains underexplored. work, introduce Toxi cTextCLIP, framework for generating high-quality adversarial texts that target CLIP during the pre-training phase. The framework addresses two key challenges: semantic misalignment caused by background in- consistency with the target class, and the scarcity of background-consistten consistent texts. To this end, Toxi cTextCLIP iteratively applies: 1) background -wwrre selector that prioritizes texts with background content aligned to the target class, and 2) a background-oe ddiveen augmenter that generates semantically coherent and diverse poi- soned samples. Extensive experiments on classification and retrieval tasks show that CLII achieves up to 95.83% poisoning success and 98.68% backdoor Hit@1 1, while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses. The source code can be accessed via https ://: com ccwp://xiiaaacaa//xxxtttttotooa
this
Introduction
In recent years, the success of large-scale pre-trained language models (e.g., BERT [Devlin et al., 2019], GPT [Radford et al., 2019, Brown et al., 2020J) has demonstrated the effectiveness of self- supervised learning on web-scale data. Inspired by this paradigm, Radford et al. [2021] introduced the Contrastive Language-Image Pretraining (CLIP) model, which extends large-scale pre-training to the vision-language domain. CLIP is trained 400 million image-text pairs collected from the Internet, using a contrastive learning objective to align visual and textual representations. Without relying task-specific supervision, CLIP achieves strong zero-shot performance across wide range of downstream tasks, including image captioning and visual question answering [Shen et al., 2022], object classification [Conde and Turgutlu, 2021], bidirectional image-text retrieval [Zhou et al., 2023a], and guiding visual self-supervised training [Wei et al., 2022]. However, CLIP's reliance on uncurated, large-scale web data introduces non-negligible security vulnerabilities.
Specifically, the openness of CLIP's pre-training data collection pipeline allows adversaries to inject malicious manipulated image-text pairs, enabling large-scale data poisoning or backdoor attacks [Carlini and Terzis, 2022, Yang et al., 2023a,b, Jia et al., 2022]. Recent work [Carlini et al., 2024] further demonstrates that such injection pipelines can be both effective and low-cost, making them feasible at scale. Consequently, these attacks compromise the learned cross- modal alignment and propagate their effects across downstream tasks. While existing research has exposed CLIP's
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

vulnerability to image-based perturbations, patch- based backdoors, and poisoned visuals [Yang et al., 2023a, Liang et al., 2024, Ye al., 2024, Zhou al., 2023b], text-based threats remain largely overlooked, despite the text modality being equally central to CL.IP's contrastive learning. Unlike images often transformed through compression or cropping that disrupt pixel-level triggers, texts remain intact during data collection and distribution. This stability enables persistent, cross-platform propagation, making textual triggers more natural, stealthy, and durable than visual ones. Earlier text-oriented attack mainly adopt simplistic strategies, such as replacing image captions with generic tareet-class texts [Carlini and Terzis, 2022, Yang et al., 2023b], often ignoring semantic coherence or contextual compatibility. Moreover, no systematic exploration of textual triggers capable of reliably manipulating CLIP's representations has been conducted. This oversight renders the text modality a critical yet underexamined attack surface in CLIP pre-training.
Designing effective text-based attacks in CLIP's pre-training stage poses several challenges. First, insufficient misleading ability: directly reusing target-class texts often introduces background content that is semantically inconsistent with the target class, which conflicts with the poisoning objective and weakens attack effectiveness. Moreover, limited scalability: many target classes lack enough high-quality, semantically aligned texts in open-source corpora, constraining both the scale and impact of constructed attacks.
To address these challenges, we propose ToxicTextCLIP, a Buckground-sensittee poisoned text generation framework tailored for launching text-based poisoning and backdoor attacks during CLIP pre-training. Toxi coxiCTIPC first selects class-relevant texts with semantically aligned background selector. It then introduces background-driven poisoned text
content using a background-awan augmenter to refine and diversify poisoned texts while preserving semantic alignment with the target class. Together, these components allow ToxicTextCLIP to generate stealthy, high-quality adversarial texts that effectively compromise CLIP's cross-modal representations.
Our contributions can be summarized as follows:
We present the first systematic study on text-based poisoning and backdoor attacks during the pre-training stage of CLIP. This work identifies the text modality as critical yet underexplored threat surface for contrastive vision-language models.
We propose ToxicText cTextCLIP, novel sacnsround-sensstiivee poisoned text generation frame- work that integrates background-awan selector and background-driven augmenter. Com- pared to existing caption-substitution baselines, ToxicTextCLIP generates more effective, semantically consistent attack texts while preserving clean sample performance.
Extensive experiments on two widely used CLIP traitrann datasets demonstrate the effectiveness of ToxicTextCLIP in both retrieval and classification tasks. ToxicTextCLIP achieves up to 95.83% attack success in poisoning and 98.68% % Hit@1 1 in backdoor settings. Furthermore, we show that existing state-of-the- defenses such as RoCLIP [Yang et al., 2023a], SafeCLIP (Yang et al., 2024] and CleanCIII [Bansal et al., 2023] fail to mitigate the threat posed by our method.
2 Related Work
Here, we review prior research on poisoning and backdoor attacks and defenses targeting CLIP.
Poisoning and Backdoor Attacks on CLIP. Existing research has confirmed that the CLIP model is vulnerable to both poisoning and backdoor attacks. Such attacks can be carried out either during the pre-training phase or during downstream fine-tuning. During the pre-training stage, attackers inject small number of poisoned samples into large-scale datasets, degrading the model's 's performance. Carlini and Terzis [2022] were among the first to show this by substituting the text descriptions of target images with unrelated class texts and analyzing the effect of image-based backdoor patches. However, this method suffers from limitations in corpus size and lacks in-depth exploration of text-based backdoor mechanisms. In the downstream task fine-tuning stage, poisoned samples are inserted into the task-specific fine-tuning dataset to interfere with the model's 's decision-making Building on Carlini and Terzis [2022], Yang et al. [2023b] extended single-target attacks to class- targeted poisoning. While poisoning attacks remain an active research area, more attention has been paid to image- backdoor attacks. For instance, Ye al. [2024] and Liang et al. [2024] explored different optimization strategies for generating visual triggers; Jia et al. [2022] employed proxy

---

datasets to optimize image-base based triggers; Zhou et al. [2023b] trained universal adversarial patches to mislead downstream models derived from CLIP; and Bai et al. [2024] proposed learnable triggers and a context-aware generator that crafts malicious prompts during prompt tuning. Despite these advances, most existing work centers on visual backdoors, while textual attacks remain underexplored.
In contrast, our work addresses distinct problem-textual poisoning during CLIP's pre-training, rather than visual triggers or downstream fine-tuning. Given the scale and openness of pre-training corpora, such attacks are more feasible and harder to detect. By focusing on text-based vulnerabilities, our study reveals a critical and underexplored threat surface in multi- modal foundation models.
Defenses Against Poisoning and Backdoor Attacks on CLIP. Defense mechanisms against such attacks can be applied at either the pre-traiin pre-training or fine-tuning stages. In the pre-training stage, RoCLIP [Yang et al., 2023a] introduces a defense strategy that maintains a text feature pool and matches image features to the most similar text features within this pool rather than directly using the potentially poisoned dataset, thereby disrupting the alignment between malicious image-text pairs. SafeCLIP [Yang et al., 2024] adopts a three-stage framework to enhance robustness. first performs single-modal warm-up to adapt encoders to modality-peccific patterns, then conducts multi-modal contrastive training reinforce semantic alignment, and finally applies Gaussian Mixture Model to distinguish "safe" and "malicious" samples, filtering out abnormal data during training. In the fine-tuning stage, defenses aim to mitigate data contamination or restore model integrity. For example, CleanCLIP [Bansal et al., 2023] separately fine-tunes the vision and text encoders on clean data to eliminate associations between triggers and backdoor labels. Similarly. Schlarmann al. [2024] propose an unsupervised adversarial fine-tuning method that updates only the vision encoder to mitigate image-based backdoor effects without modifying the entire model.
3 System and Threat Models
3.1 CLIP Model
CLIP introduces a new paradigm for jointly modeling visual and textual modalities by training on large-scale image-text pair dataset Tramm {(z),,z) S I TTrain ), where each image 2; E I is paired with its corresponding textual description tj E T The CLIP comprises two primary components: an image encoder E (-) (e.g., ViT (Dosovitskiy et al., 2021 that maps each image ay to an embedding vector fí E F, and a text encoder (-"(.) (e.g.. Transformer [Vaswani et al., 2017)) that encodes i, into an embedding vector J E F. Both embeddings reside in shared feature space F. where semantic alignment between modalities is enforced. To align visual and textual representations, CLIP employs the InfoNCE loss [Oord et al., 2018], which encourages high similarity between matching (positive) image-text pairs while penalizing mismatched (negative) pairs. Here, we explore two typical CLIP applications. Image classification involves taking test image (e.g., an image from the "car" class) and comparing against a set of candidate textual descriptions representing different classes. The model computes similarity scores and selects the description with the highest score as the predicted label. In contrast, Image-text retrieval takes a textual query (e.g.. "a lamb on the grass") and ranks collection of images by their semantic similarity to the query, returning the top-ranked images as the retrieval results.
3.2 Threat Model
Adversary's capability. Here, we assume that the adversary has the ability to construct poisoned dataset Ditriin and inject it into the clean pre-training dataset Drain, thereby forming compromised dataset Dirain = Ditrain U DiIrainn Specifically, the adversary can obtain candidate image-text pairs (mz, t2) from publicly accessible platforms such as Shutterstock! or Google Images?. Alternatielly, the adversary may generate captions using publicly available image captioning models (e.g., BLIP [Li et al., 2022], OFA [Wang et al., 2022]) based on collected public images. Once candidate data is prepared, the adversary refines Ditrain by replacing the original textual input l, associated with the selected image T; E Ihrain with semantically manipulated poisoned text tp... To emulate realistic poisoning scenarios, such as malicious content being crawled from the web into large-scale datasets,
'nttpa://www. wtt.//rsw...tttrrrsto...... com /nttps://mmagss http://aaagg...gggge.eeeo com
3

---

S Step1: Background-awee aware Targel Text electoo
Step2: B Background driven P Poisoned Text Augmenter
Post Process
Target class lest
Imige Fas
plust of [olas R). drawing of [closs
Clastering
Soot
Vissel Enoodes
Festund/
Fiatur Augner
rupdatn
Test List Sur
Backinio
Filter class senienues
Cea MMg Mare m9 W0)
(% Fayl
Testua Enoades
Pestuns)
Fxtrect
(1g pror Fay
Corpus
Bacaground Information
Htrraton
Figure Illustration of our ToxicTextCLIP framework.
the adversary is limited to uploading a small number of poisoned samples, typically ranging from to 10,000 [Yang et al., 2023b, Carlini and Terzis, 2022]. Crucially, the adversary operates under strict black-box setting: they have no control over the data collection pipeline, cannot guarantee inclusion of their poisoned content in the final dataset, and lack any knowledge of the model's 's architecture, training procedures, or parameter configurations.
Adversary's goal. The adversary aims to inject semantically plausible yet malicious textual inputs into the pre-training dataset such that the resulting CLIP model maintains high performance on clean samples while exhibiting controlled misbehavior on targeted inputs, either by misclassifying specific image or by responding incorrectly
backdoor trigger. An attack is considered successful if the manipulated behavior is limited to those designated poisoned or triggered instances, while the model's s accuracy and generalizability on benign samples remain unaffected. To evade detection, the injected texts are crafted to preserve grammatical correctness and semantic plausibility.
Here, investigate three representative attack strategies, including one data poisoning attack and two backdoor attacks, each designed to subtly alter the model's behavior:
Single target image [Yang et al., 2023b, Carlini and Terzis, 2022]: The adversary aims to associate specific target image AA (from source class A) with set of poisoned texts describing a different target class B. 1) Classification task: The model mistakenly aligns A,* with texts from class B. 2) Retrieval task: Given poisoned text tg, the model retrieves C A,,,, while maintaining normal retrieval for other queries.
The adversary embeds trigger phrase b (a word or sentence) into training texts so that, during inference, any trigger-appended input fy @ b is mapped to predefined target class A. 1) Classification task: Any image from class task: Triggered texts retrieve images from class A, while clean texts behave normally.
Word-level and sentence-level backdoor attacks:
matched with trigger-bearin texts, despite unrelated content. 2) Retrieval
4 Methodology
4.1 Overview
To construct the poisoned dataset Dp. we substitute the original text description t, of source class image VA with a poisoned text tp.8 that is semantically aligned with a target class B. Formally, this process produces a poisoned dataset, i.e., D, = {(x A, tp. ... a E I t, E T,,rain ). However, constructing effective poisoned texts tp.8 introduces two major challenges. First, the original text in typically contains both class-relevant semantics and background descriptions derived from the corresponding image p- such background content conflicts with the semanties of class B, using in as poisoned input may misalign the semantic association between A and class B, weakening the poisoning effect. Therefore, how to differentiate class-relevant semantics from background content in tz, and ensure background consistency with the target class constitutes the foremost challenge. Second, existing texts for many target classes are either insufficient in number or semantically misaligned, especially in terms of background content. These limitations reduce both the scale and effectiveness of poisoned samples. Thus, how to augment the target class corpus with texts whose background semantics are coherent with the intended class emerges the second challenge.
To tackle these challenges, we propose Toxi TTxICIPP,, a background-sensitiee poisoned text generation framework (as shown in Figure 1) consisting of two modules: background-awe target text selector and background-driven poisoned text augmenter. The first module constructs candidate pool based on target class keywords and ranks texts by the semantic similarity between their background

---

content and the target class embedding. The second module is an encoder-decoder architecture to expand and refine these candidates into larger set of semantically coherent poisoned texts, enhancing both attack diversity and effectiveness. These two modules are executed iteratively, progressively generating high-quality poisoned samples and enhancing the diversity and effectiveness of the attack.
4.2 Background-aware Target Text Selector
To select candidate texts lg whose background content highly aligns with the target class se- mantics, we propose a background- target text selector. Specifically, the selector pro- ceeds as follows: First, we gather all textual descriptions for the target class B, denoted {tg,1, tB,,, tB...... Lt where the total number of descriptions for class B. Second, for each candidate LB.j we identify and extract both the background information Sb.j and class- relevant information S.=B.j. Third, we sort the candidate descriptions according to these similarity scores, prioritizing those whose background is most semantically aligned with the target class.
First, to effectively distinguish background content from class-relevant content within each candidate description L..j. we leverage the insight that the class identity of an image can be typically conveyed through one multiple n-grams [Li et al., 2017]. Motivated by this, we assume the class-relevant information can be captured by at most words. Accordingly, for a given description tg.j. we form the set S, of all possible background descriptions obtained by removing up to r) words. Formally, let Cob(tB.j. 7) be the set of all variations of t with any 7 words removed. We define:
S₁ - Ü Cob(ta,, 7) {tBu \ {wi, ......, w. }> 1=
where (wi,...:; W is any selection of y words from Ł..jj In other words, each element of Sy is candidate background-only description formed by removing subset of words (up to 11) from t.
Second, we select the optimal background candidate S Sj for tBJ as the one that best captures the image details while remaining distinct from the class identity. Since the true visual class label may not always be available, we use textual class prompts as a proxy. However, different prompt templates (e.g.. "a photo of a cat" v.s. "a bad photo of cat") can cause semantic shifts in the embedding space, resulting in unstable class representations. To mitigate this, we compute class embedding centroid B by averaging the CLIP [Radford et al., 2021 text embeddings of multiple prompt-engineered templates for class B, i.e ZB 1 E F. (Temp; (B)), where each Temp; (B) is manually designed text prompt (e.g. "A photo of [class BJ"") and is the number of such prompts. This multi-prompl averaging yields a stable and robust class-level semantic centroid ZB. improving the reliability of background-class separation. For each background candidate Sh,j E Sj. we then compute its score as the difference between its similarity to the image and its similarity to the class centroid. Formally, select:
S arg max(Sim( (S5m(ET(e (So.j) (.a))).E E (TB,)) Sim(F" (Sb.j), ZB)),
ES,
where E ( is the CLIP embedding of the image ænj associated with t.j. and Sim(-,) .) denotes a cosine similarity function.
Finally, once obtaining the optimal background Sh we rank the original texts by the semantic alignment of their backgrounds with the class. Concretely, we sort all LB.j in descending order of Zz)). The result is an ordered
Sim(E" (St ZB), i.e. (f'.1...., & = Sort (Sim(E list of class B descriptions prioritized by how well their background content semantically aligns with the target class. These top-ranked texts can then be used to construct the poisoned descriptions tp B. satisfying both the aggressiveness and consistency criteria. A more concise visual flowchart can be found in supplementary material.
4.3
Background-driven Poisoned Text Augmenter
Using textual descriptions aligned with the target class (i.e., tn)) is a simple yet commonly used poisoning approach. However, relying solely on existing corpus data poses two key limitations. First, many classes lack a sufficient number of semantically relevant texts, limiting available samples and undermining poisoning strength. As shown in Figure 2(a), over 50% of ImageNet classes in
5

---

bahdd
Sandy cacpente propeiiie
eeilege
co
becket
Hadkat
-7
Number attacc toxts
Topek
10
-30
-00
70
-10
-30
-10
10
30
10
(a) Insufficient target texts. (b) Limited effectiveness.
(c) Original texts.
(d) Augmented texts.
Figure 2: Limitations of relying existing corpus and interpretability of effectiveness.
MM-scale corpus cannot support attacks requiring just 30 poisoned texts per class (about 0.003% of the corpus). Second, even when such texts exist, their background content often lacks strong semantic aligmment with the target class, reducing poisoning effectiveness. Figure 2(b) demonstrates this issue for two example classes ("bucket" and "hare"), where more than half of the samples have background similarity scores below 60%. To overcome these limitations, we propose background-drive poisoned text augmenter that enhances candidate texts in both semantic consistency and diversity. The following details each component of this augmenter. Figures 2(c) and 2(d) illustrate that the distribution of augmented embeddings is more concentrated around class centers than that of original embeddings, indicating stronger attack effectiveness.
Feature encoding. Given a candidate poisoned text l's."" we use the open-source CLIP text encoder ET()) (.) to compute its feature embedding J. = E (t'nn)))
Feature augmentation. Direct feature perturbation in the CLIP embedding space is often ineffective due to its sparsity and semantic fragility. To preserve semantic structure, we enhance f3 using the corresponding image feature f f S + +Nf Here, À controls the influence of visual features, limiting semantic drift during augmentation.
Transformer-basee decoding. We decode the augmented feature representations into poisoned texts using a Transformer- decoder O. This module is designed to ensure high-quality, semantically diverse, and visually consistent text generation.
First, to enhance fluency and coherence, O builds the Transformer architecture, which has demonstrated strong performance in text generation tasks [Radford et al., 2019, Brown et al., 2020, Hoffmann et al., 2022]. Second, maintain semantic alignment with the visual input, the decoder is conditioned not only on the poisoned text feature S but also on the corresponding image patch embeddings /aschh extracted from æ'B.j via visual encoder. The combined input allows the model to incorporate visual context into the decoding process. Formally, given an image a'B.j, we compute its embedding as: E (x'B.) = [CLS] + Zjunch+ where [CLS] is a lcarnable token summarizing the image, and Z patch denotes the spatial patch embeddings. The decoder input is obtained by concatenating f/ with Zpncch which are then passed to the cross-attention module. The attention weights are given by
ZZ patch
Cro_Att = softmax(
(Z) patch + f3
Vdg
where Q is the query matrix from the decoder's cross- antts-attnntion module [Vaswani al., 2017], enabling each token to attend to relevant image regions based on its context, and dę is the dimensionality of the key vectors. This formulation enables visual-textual fusion that guides the generation of background-aware poisoned text. Third, to enhance diversity and avoid degeneration, we adopt Diverse Beam Search (DBS), which divides the beam search into multiple groups and introduces diversity penalties across groups. The generated candidate texts are (tp 1N Off,' Zpmuch). where N denotes the beam size.
Jaceard similarity- jayakumar et al., 2018], we introduce a Jaccard similarity- selection mechanism. Starting with (0) Pp.B" [t'B, ). we iteratively add the least similar candidate tp.j based on average Jaccard distance to the existing set:
post-processing. To avoid redundant samples often produced by DBS [Vi-
(k-1) (k) p.B = P
U
arg min to Et J
Σ tee!f-1
[(p.ii
(k-1)) p.B

---

This continues until a desired number of diverse poisoned texts is collected. The final set (K) as the output E, ready for use or for subsequent iterations.
p, serves
4.4 Extension to Backdoor Attacks
Similar to poisoning attacks, backdoor attacks involve inserting poisoned samples into the training dataset to induce targeted misbehavior. However, unlike poisoning, which aims to misclassify specific image, the objective of a backdoor attack is to ensure that any input text containing a designated trigger b causes the model to retrieve images from predefined target class.
To generalize beyond instance-level misclassification, we sample multiple images from the target class rather than relying on a single fixed image in poisoning attacks. For each target-class image, we retrieve reseeeeepreeet texts from other categories using the same procedure as in the poisoning setup. The backdoor trigger b is then appended to each of these texts to construct poisoned training pairs. This ensures that the poisoned texts are semantically diverse and not inherently associated with the target class, which is critical for attributing the learned association to the trigger. As result, the model is encouraged to form a strong and generalized connection between the trigger and the target class across diverse visual contexts.
5 Experiment
5.1 Experiment Setup
Datasets. We evaluate our approach on three popular datasets: CC3M [Sharma et al., 2018], CC12M [Changpinyo et al., 2021], and a -5m-le subset of YFCC [Thomee et al., 2016), referred to as YFCC15M [Gu et al., 2024]. Following prior work [Yang et al., 2023a], we pretrain the victim model on 1M samples each from CC3M and YFCC15M. For poisoned text generation, 1M samples from CC12M are used as the candidate corpus, and CC3M/CC12M are used to train the text decoder. COCO [Lin et al., 2014] serves as the test set for attack evaluation. Unless stated otherwise, all attacks follow a standard setup. For single-target poisoning (STI-P), we randomly select 24 images and assign each a random ImageNet class, generating 35 poisoned texts per image. For word-level backdoor (W-BD),
use 20 boat-class images with five poisoned texts per image, triggered by the rare word "zx" [Kurita et al., 2020]. For sentence-leve backdoor (S-BD), 50 boat-class images are used with the trigger phrase "Please return high-quality results." " Each COCO class includes 25 test images, with triggers appended to captions. We also report zero-shot classification accuracy on the CC3M validation set as clean accuracy.
Implementation Details. All experiments were conducted on xNVIDI 4090 GPUs. Victim Model: We adopt the open-source CLIP implementation [Radford et al., 2021], with a ResNet-50 [He et al., 2016] vision encoder and Transformer [Vaswani et al., 2017] language encoder, following [Carlini and Terzis, 2022, Yang et al., 2023a]. Training uses the Adam W optimizer with cosine scheduler (initial learning rate: 5 x 10-5, -5 min learning rate: 10 10-$), batch size 512, for 10 epochs. Substitute Model: We use OpenAl's ViT-B/32 CLIP as the substitute model, distinct from the victim, with vision Transformer [Dosovitskiy et al., 2021] and Transformer-basee text encoder. It is employed in the background semantic enhancement module extract image text embeddings and project them into a shared feature space, guiding poisoned text generation. Text Feature Decoder Model: The decoder is 6-layer Transformer, guided by frozen substitute CLIP encoder. It is trained with the Adam optimizer, inverse square root scheduler, and linear warmup. The initial learning rate is 10-3 (min: 10-9), with mixed precision training. batch size 832, for 32 epochs.
Evaluation Metries. We evaluate attack performance using three metrics across classification and retrieval tasks. For classification, clean accuracy (CA) measures the proportion of correctly classified clean samples, reflecting the model's robustness. Attack success rate (ASR) measures the proportion of samples misclassified as the target class under attack, indicating attack effectiveness. For retrieval tasks, Hit@k evaluates the proportion of target images ranked within the top-k positions. A higher Hit @k suggests that poisoned texts more effectively retrieve the intended target.
We adopt two baseline methods for comparison: For poisoning attacks, we use SOTA text poisoning method for CLIP, mmPoison (Yang et al., 2023b]. For backdoor attacks, due to the lack of existing text- backdoor methods for CLIP, we construct baseline by injecting trigger phrases into randomly sampled training captions.
Baselines.
7

---

5.2 Effectiveness of Our ToxicTextCLIP
We evaluate the vulnerability of CLIP to our poisoning and backdoor attacks, highlighting the threat of text-based manipulation. Poisoning Attack. As shown Table our method significantly increases ASR while preserving clean accuracy. On CC3M, raises ASR from 62.5% (mmPPossnn) to 95.83%, with CA of 32.23%. Backdoor Attack. Both our method and the baseline demonstrate that CLIP is susceptible to textual backdoors. However, our approach, which uses optimized rather than random texts, achieves notably higher performance. In word-level attacks, @it@) improves by 20.34%; in sentence-lews attacks, by 23.16% on average across two datasets. Clean accuracy is maintained, Although performance slightly lower on YFCC due to higher data noise, our method consistently outperforms the baseline.
Table 1: Performance comparison of ToxicTextCLIP in poisoning and backdoor attacks.
Dataset
Attack Type
Method
CA
ASR
Hit@
Hit@5
Hit@
No Attack
33.43
0
0.38
2.64
5.84
mumPoison
31.52
62.50
STI-P
ToxicTextCLIP
32.23
95.83
CC3M
Baseline
31.91
72.13
96.74
98.44
W-BD
ToxicTextCLIP
32.03
92.66
98.87
100
Baseline
32.45
64.41
89.64
96.05
S-BD
ToxicTextCLIP
34.67
98.68
99.81
100
No Attack
11.01
0
0.38
2.64
5.84
mumPoison
9.49
66.67
STI-P
ToxicTextCLIP
9.02
91.67
YFCC
Baseline
10.60
50.47
87.76
95.09
W-BD
ToxicTextCLIP
9.72
70.62
91.71
96.05
Baseline
9.83
67.04
90.96
94.54
S-BD
ToxicTextCLIP
10.44
79.10
94.92
96.99
5.3 Attack Robustness against Pre-training and Fine-tuning Defenses
We evaluate the robustness of our method against three state-of-the-art defenses: RoCLIP [Yang al., 2023a] and SafeCLIP [Yang al., 2024], applied during pre-training, and CleanCLIP [Bansal et al., 2023], applied during fine-tuning For RoCLIP and SafeCLIP, both poisoning and backdoor attacks are conducted with 10 epochs adversarial training. For CleanCLIP, we follow its protocol by fine-tuning the poisoned model for 10 epochs on 100k clean subset of the pre-training data.
Poisoning Attack. As shown in Table 2, our method PoxicTextCLIP maintains high ASR under all defenses and consistently outperforms the mmPoison baseline. For instance, on CC3M, under RoCLIP, ToxicText Tee achieves 70.83% ASR 33.33% for mmPoison; under CleanCLIP, it yields 75.00% ASR vs. 45.83%. Backdoor Attack. Our method also remains effective for both sentence-level and word-level backdoor attacks. On CC3M, it achieves Hit@| rates of 91.15% (RoCLIP) and 86.63% (CIeanCLIP), compared to 57. 57.82% and 56.29% for the baseline. Similar trends are observed on YFCC, though defense effectiveness is weaker due higher data noise. The superior performance of target class, reinforcing spurious associations even under defense.
indicates its ability to better bind poisoned texts with the
5.4 Quality assessment of poisoned texts
To evaluate the quality of poisoned text, we employ the perplexity metric. As shown in Table 3, the texts generated by Toxi ceextCLIP exhibit lower perplex- ity compared to the original open-domain texts. This improvement arises because the original web-sourced texts often exhibit redundancy and grammatical in- consistencies, whereas our Background-driver Augmenter integrates image semantics through cross- attention, steering generation toward concise, class-relevant, and syntactically coherent outputs.
Table 3: Text quality of ToxicTTxxCLL.I Method
Perplexity
Original texts
755.27
ToxicTextCLIP 408.89

---

Table 2: Performance of ToxicTextCLIP against pre-training and fine-tuning defenses.
Dataset
Attack Type
Defense
Method
ASR
Hit@
Hit@5
Hit@
mmPoison
33.33
RoCLIP
ToxicTextCLIP
70.83
mmmPissn
45.83
STI-P
CleanCLIP
ToxicTextCLIP
75.00
mmPPiiso
25.00
SafeCLIP
ToxicTextCLIP 64.17
Baseline
37.85
88.70
94.73
RoCLIP
ToxicTextCLIP
48.21
90.40
97.74
Baseline
66.65
95.29
97.55
CC3M
W-BD
CleanCLIP
ToxicTextCLIP
90.66
98.25
99.44
Baseline
13.63
26.22
37.16
SafeCLIP
ToxicTextCLIP
29.72
52.85
67.73
Baseline
57.82
86.63
96.42
RoCLIP
ToxicTextCLIP
91.15
99.25
99.62
Baseline
56.29
88.58
95.48
S-BD
CleanCLIP
ToxicTextCLIP
86.63
99.44
100.00
Baseline
20.17
33.02
45.69
SafeCLIP
ToxicTextCLIP
60.96
70.28
82.53
mmPoison 50.00
RoCLIP
ToxicTextCLIP 75.00
mmPoison 58.33
STI-P
CleanCLIP
ToxicTextCLIP 83.33
mmPoison 16.67
SafeCLIP
ToxicTextCLIP 54.17
Baseline
20.90
50.85
66.10
RoCLIP
ToxicTextCLIP
25.87
62.13
70.21
Baseline
29.00
79.47
90.96
YFCC
W-BD
CleanCLIP
ToxicTextCLIP
58.19
90.77
96.05
Baseline
12.78
22.94
31.63
SafeCIIP
ToxicTextCLIP
23.51
48.03
59.79
Baseline
55.56
80.04
86.63
RoCLIP
ToxicTextCLIP
72.69
86.63
90.96
Baseline
40.87
73.07
83.62
S-BD
CleanCLIP
ToxicTextCLIP
56.48
81.03
86.92
Baseline
17.33
28.55
40.27
SafeCLIP
ToxicTextCLIP
41.51
53.19
61.81
5.5
Ablation Study
Impact of poisoning rate on attack effectiveness. We analyze how poisoning rate affects at- tack performance across single-target, word-level, and sentence-level backdoors. As shown in Figures 3(a), 3(b) and 3(c), higher rates consistently enhance attack success with minimal impact on clean accuracy. For single-target attacks, performance saturates 35 poisoned texts per image. Word-level and sentence-level backdoors are effective even at low rates, where on CC3M, 50 to 75 poisoned samples suffice push 5 and @it 10 above 80%. This demonstrates the high vulnerability of models to text-hased backdoor attacks, even under sparse poisoning.
Impact of training epoch numbers. We assess how training epochs affect attack success. As shown in Figure 3(d), all methods converge quickly, with poisoning reaching 50% ASR by epoch 2 and backdoor attacks exceeding 40% Hit@l on CC3M within 2-3 epochs. This early effectiveness reflects how rapidly poisoned texts influence the model.
Impact of different components in our framework. We conduct an ablation study to evaluate the contributions of two key modules Toxi CTexCCIP:: background-aware target text selector and background-driven poisoned text augmenter. The w/o selector variant replaces selection with random target texts, while w/o augmenter omits the augmentation step. Results on CC3M (Table for poisoning, Table 5 for backdoor attacks) show that both components significantly enhance attack

---

0 1i Vp.s
100
AM0000
Cophl AuS
AGROS
12
110
(a) Single target image
(b)
Word-level
(c) Sentence-level
(d) Epoch influence
Figure 3: Influence of poisoning rate and training epochs on CC3M dataset.
performance. In single-image poisoning, the baseline ASR 62.5%; w/o selector achieves 83.33%, w/o augmenter reaches 87.50%, and the full framework achieves the highest ASR. Similarly, in sentence-level backdoor attacks, @it@ improves from 78.91 (w/o selector) and 91.34% (w/o augmenter) 98.68% when both modules are combined.
Table 4: Impact of different components in Toxi cexitIIxCLII under poisoning attack.
No Defense
RoCLIP
Clean CLIP
Method
ASR(%)
ASR(%)
ASR(%)
mmPoison
62.50
33.33
45.83
w/o selector
87.50
62.50
54.17
w/o augmenter
83.33
58.33
58.33
ToxicTextCLIP
95.83
70,83
75.00
Table Impact of different components in ToxicTextCLIP under backdoor attack.
No Defense
RoCLIP
Clean CLIP
Attack Type
Method
Hit@
Hit@5
Hit@1 I
Hit@5
Hit@
Hit@5
Baseline
72.13
96.74
37.85
88.70
66.65
95.29
w/o selector
65.26
94.92
34.94
86.98
58.32
90.31
W-BD
w/o augmenter
81.32
96.16
40.63
87.14
78.51
95.43
ToxicTextCLIP
92.66
98.87
48.21
90.40
90.66
98.25
Baseline
64.41
89.64
57.82
86.63
56.29
88.58
w/o selector
78.91
97.55
61.21
97.49
70.24
90.77
S-BD
w/o augmenter
91.34
98.62
$2.73
97.79
83.05
97.06
ToxicTextCLIP
98.68
99.81
91.15
99.25
86.63
99.44
Potential defense directions
We advocate for defenses that move beyond disrupting shallow image-text pairings and instead model deeper cross-modal semantic consistency. One promising direction text anomaly detection via language models: although poisoned texts generated by ToxicTextCLLN appear fluent, their background semanties may reveal subtle inconsistencies. Pretrained language models (e.g.. BERT, RoBERTa) can evaluate masked-toker token likelihoods or embedding coherence to identify such anomalies. Another direction cross-modal background verification. which complements pairing bbsedd defenses by assessing whether textual elements are visually grounded. Retrieval- or generation-bast based modules can verify visual support or reconstruct implied semantics detect divergence. Together, these strategies aim to enforce cross-modal semantic consistency, providing principled path toward robust and modality-awar defenses against attacks like ToxicTextCLIP.
7 Conclusion
Here, we investigate the underexplored threat of text-based attacks during CLIP's pre-training phase, aiming to raise awareness of the security risks facing large-scale multimodal models. We propose Toxi foxicTextCLIP augmentation that explicitly aligns background semantics with the target class, thereby enhancing the feasibility of both poisoning and backdoor attacks. Empirical evaluations across multiple tasks, datasets, and defenses indicate that Toxi cerxtCLIP poses a credible challenge to current CLIP training pipelines and highlight the need for more effective and modality-aware defense mechanisms.
novel background-seni
framework for adversarial text selection and
10

---

8 Acknowledgment
This work is supported by National Natural Science Foundation of China (No. 62372477), the Science and Technology Innovation Program of Hunan Province (No. 2024AQ2030, 2024RC3035), the Natural Science Foundation of Hunan Province (No. 2024JJ4071), the Outstanding Innovative Youth Training Program of Changsha City (No. kg2306006).
References
Jacob Devlin, Mingwei Chang, Kenton Lee, and Kristina Toutanova. Bert: bidirectional transformers for language understanding. In NAACL'I9, Minneapolis, MN, June 2019.
Pre-training of deep
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. Technical Report 8, OpenAI, Feb. 2019.
Tom Brown, Benjamin Mann. Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS'20. 20, Virtual Conference, Dec. 2020.
Alec Radford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML'21, 21, Virtual Conference, July 2021.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, KaiWei Chang, Zhewei Yao, and Kurt Keutzer. How much CLIP benefit rision-and-langee tasks? In ICLR 22, Virtual Conference, Apr. 2022.
Marcos Conde and Kerem Turgutlu. Clip-art: Contrastive pre-training for fine- the-grained art classifica- tion. In 21, Virtual Conference, June 2021.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language-image pre-training. CVPR'23, Vancouver, Canada, June 2023a
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodaliy- visual pre-training. In ECCV'22, Tel Aviv, Israel, Oct. 2022.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In ICLR'22, Virtual Conference, Apr. 2022.
Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. Robust contrastive language-image pretraining against data poisoning and backdoor attacks. In NeurlPS New Orleans, LA, Dec. 2023a.
Ziqing Yang, Xinlei He, Zheng Li, Michael Backes, Mathias Humbert, Pascal Berrang, and Yang Zhang. Data poisoning attacks against multimodal encoders. In ICML'23 '23, Hawaïi, USA, July 2023b.
Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in supervised learning. In 22, San Francisco, CA, May 2022.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette- Chyue,,te-Cho,, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets is practical. In S2,' San Francisco, CA, May 2024.
Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. CVPR'24, 24, Seattle, WA. June 2024.
11

---

Jingwen Ye, Ruonan Yu, Songhua Liu, and Xinchao Wang. Mutual-modality adversarial attack with semantic perturbation. In A4,I'24, Vancouver, Feb. 2024.
Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, and Hai Jin. Advelip: Downstream-agnostic adversarial examples in multimodal contrastive learning. ACM MM 23' Ottawa, Canada, Oct. 2023b.
Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. Better safe than sorry: pre-training clip against targeted data poisoning and backdoor attacks. In ICML'24, Vienna, Austria, July 2024.
Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kaiwei Chang- Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In ICCV'23, Paris, France, Oct. 2023.
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shutao Xia, Zhifeng Li, and Wei Liu. Badclip: Trigger- aware prompt learning for backdoor attacks on clip. In CVPR'24, Seattle, WA, June 2024.
Christian Schlarmann, NamanDeep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In ICLR'C44 '2, Vienna, Austria, May 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR'21, Virtual Conference, May 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need. In NeurlPS 17, Long Beach, CA, Dec. 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / Jan. 2018.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML'22. '2, Baltimore, MD, July 2022.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequeece learning framework. In ICML 22, Baltimore, MD, July 2022.
Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten. Learning visual n-grams from web data. In ICCV'17, Venice, Italy, Oct. 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In NeurlPS'22. New Orleans, LA,, Nov. 2022.
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes, In AAAI'18, New Orleans, LA, Feb. 2018
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL'18, Melbourne, Australia, July 2018.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CVPR'2I, Virtual Conference, June 2021.
Bart Thomee, DavidA Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and LiJia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73 Jan. 2016.
12

---

Tiancheng Gu. Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, and Jiankang Deng. RWKV-CLIP: A robust langune representation learner. In EMNLP'24, Miami, FL. Nov. 2024.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV'14, Zurich, Switzerland, Sept. 2014.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pretrained models. In ACL'20, Virtual Conference, July 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR'16, Las Vegas, NEV, June 2016.
W. Ronny Huang. Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. In NeurIPS'20, Virtual Conference, Dec. 2020.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong" Sun. ONION: A simple and effective defense against textual backdoor attacks. In EMNLP'2I, Punta Cana, Dominican Republic, Nov. 2021.
Kevin Frans, Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. In NeurlPS New Orleans, LA, Nov. 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR'22 '2, New Orleans, LA, June 2022.
13

---

Technical Appendices and Supplementary Material
9.1 Algorithmic Details of ToxicTextCLIP
In this section, we present the complete workflow of ToxicTextCIIP, as illustrated in Algorithm 1. Specifically, Lines 4-13 describe the Background-Awer Aware Target Text Selector, while Lines 14-26 outline the Background-Drit
Poisoned Text Augmenter.
Algorithm 1: Details of ToxicTextCCIP
Input: Image encoder E', Text encoder E",, Image- corpus (IB x TB) {(ZB.j. tB,)) for class Max remove words 7) Prompt templates (+) fi=+> Iteration number M. Visual feature influence weight X, Source class image AAA, Beam size N
Output: Poisoned dataset Dp
Obtain category embedding B + m
E E" (Templ((B))
Initialize the poisoned dataset D, 4 0;
for i 4 to M do
/+Background-awatt for tB.j E TB do
Target Text Selector*/:
Initialize the candidate background information texts set Sj 0:
for y 5- 1 to do
Update Sj by removing combination of words from t
/+Scoring Candidate background texts*/;
10
for Sh.j E S, do
score(S),) + Sim (im(E"((..)..
Sim (amnEE(SS (S%,)) ZB);
11
12
Select final background information Sh & arg 5 score( (5h,)):
13
Sort texts based on background information [t',,...... BB Sort(Sim (5" (St,), ZB)): /+Background-driver
14
Poisoned Text Augmenter+/; for {B,5 E ( (( 1'1 ... do
16
Obtain text feature f by text encoder ET;
17
Obtain patch embedding patch and image feature f/ by image encoder E':
Augment text feature f7¹ - +Xf/;
18
Decode text feature 83" by feature decoder O, [t Off Z patch):
19
Filter decoded texts by Jaccard similarity- post- post-processing obtain set (k) B.
20
21
/ /+Update Text Corpus for Next Iteration*/;
22
if < (M 1) then
Update text corpus TB TB U (K) else
23
P.
24
for 1. E ( do Update poisoned dataset D, = U tpp
25
26
L
27
return Poisoned dataset Dpi
9.2 More Details of background information extraction
Figure 4 illustrates the process for extracting textual background information, which consists of three steps: 1) Obtain candidate background texts: A set of candidates Sj is generated by progressively removing category-specin segments of varying lengths from the original text. Score Candidate background texts: Each candidate is evaluated using two components. First, the similarity between the candidate text embedding F.T (5%,)) and the image embedding Fd (FB,) is computed, denoted as S Sim Sim(E"((S,, E' (IB.))). ))) A higher S indicates that the text better aligns with the overall visual content. Second, the similarity between the candidate embedding and the category embedding Zp, which obtained by averaging embeddings of multiple prompt templates for class B, is calculated
14

---

as S" = Sim(E" ZB). A higher S" suggests the candidate text is more inclined to describe category-specific content. The final score is computed S - to favor general background information over class-focused content. 3) Select final background information: The candidate with the highest score is selected the final background text 5h for the original input BBj
Step1: Obtiin candidate background texts.
Step2: Score Candidate background texts.
Texttay f
W
Wy
Wg
Candidate Text
Remove words
Enoodee
=
F
f
Fia
17
..
r'
Visuel Encoder
Step3: $ Select final background information.
Candidate Text.S;
Shij
Figure Framework of background information extraction.
9.3
More Details of text feature decoder
Here, we provide additional details on the text feature decoder.
Text Feature Decoder P Pretraining
Outpet
Decoding Text F Features
Probablittes
+ Concatemate tensor
Image
Text
Tunable
Linear Softmax
Visual Eucoder
Encoder
Add & Norm
Food Forward
C1.5
6x
2
Add & Norm
Transfrmer decoder
Feature ugmment
parch
Abenttion Block
Add & Norm
parch
Frozen
Mask A ttenton Block
Visual
Textual Encoder
Encoder
Feature Decoder
tional
Embeskinn
Eacoding
Images
Texts
Ouruut (shifted right)
Decoded Texts
Figure 5: Illustatioon of feature decoder structure.
Text Feature Decoder Pretraining. As illustrated in Figure 5, adopt frozen substitute CLIP model as the backbone, keeping both visual encoder Et and text encoder E" fixed throughout pretraining. We introduce a lightweight texi feature decoder, implemented as a 6-layer Transformer decoder trained to reconstruct text from text feature inputs. Specifically, given an image BB we first extract its patch embeddings Zlach via the frozen visual encoder:
E ( [CLS] Z paacch+
Zpanch E NoNx
where (CLS) is a learned global token represent the image feature, N₂ is the number of image patches, and d is the embedding dimension. Meanwhile, the text encoder maps each ground-truth caption t to its feature J E (t.)). To guide the decoder toward semantically aligned generation, we construct the cross-attention context by concatenating Z with the target text feature:
(Zpach @ ff (N,+1)xd
15

---

The decoder is trained to autoregressively generate the token sequence of B.j. The model is optimized using KL-divergence loss augmented with label smoothing:
LKL (L(@@)po))
where q is the smoothed target distribution and Pa the decoder's output distribution over the vocabu- lary. Pretraining is performed on the CC3M and CC12M datasets.
Application for Decoding Text Features. During augmented feature decoding, the augmented text feature f3 is concatenated with the image patch embeddings Z forming the context (Z # f(") to feed into the trained decoder. Through cross-attention over this sequence, the decoder autoregressively produces token distributions, which converted into discrete text via diverse beam search followed by Jaccard similarity-based post-processing. (see Figure 5).
9.4 Ablation Study
Impact of poisoning rate attack effectiveness on different dataset. We further analyze how the poisoning rate affects attack performance across single-target, word-level, and sentence-level backdoors the noisier YFCC dataset. As shown in Figures 6(a), 6(b) and 6(c), the results are consistent with our conclusions on the CC3M dataset: higher poisoning rates consistently improve attack success with minimal impact on clean accuracy. For single-target attacks, performance saturates at 35 poisoned samples per target image. For word- and sentence-level backdoors, @it@5 and Hit 10 exceed 80% after injecting 75 and 250 poisoned samples, respectively. Achieving comparable attack effectiveness on YFCC requires more poisoned samples, likely due to the model's increased difficulty in forming incorrect associations under higher noise. Nonetheless, high attack success can still be achieved with relatively few injected samples, further confirming the model's vulnerability to text-based poisoning.
O'TNCT WITADTE And
0 9CA
Of miail Fisc WiiCh ACt.
FAAT PVRRCCT WIRLL ACC.
Tefhosoradoo par leage
affPoloeed sraples
Upocts
(a) Single target image
(b) Word-level
(c) Sentence-level
(d) Epoch influence
Figure 6: Influence of poisoning rate and training epochs on YFCC dataset.
Impact of training epoch numbers on different dataset. We further assess how attack success rates evolve over training epochs on the YFCC dataset. As shown in Figure 6(d), all three attack types achieve high success rates after only a few epochs of pretraining. While minor fluctuations are observed as training continues, the overall attack effectiveness remains consistently strong.
Impact of different components in our framework on different dataset. We further evaluate how Toxi Te'''' background-aware target text selector and background-driven poisoned text augmenter perform on the noisier YFCC dataset. Results (Table for poisoning: Table for backdoor attacks) are consistent with our conclusions on the CC3M dataset: both components significantly enhance attack performance. For instance, in single-image poisoming, the baseline ASR is 66.67%; removing the augmenter (w/o augmenter) improves it to 79.17%, while removing the selector (w/o selector) yields 83.33%. The full framework achieves the highest ASR. Similarly, in sentence-level backdoor attacks, Hit@1 increases from 69.13% (w/o selector) and 73.27% (w/o augmenter) to 79.10% when both modules are used together.
Table 6: Impact of different components in ToxicToettCa... under poisoning attack.
No Defense
RoCLIP
Clean CLIP
Method
ASR(%)
ASR(%)
ASR(%)
mmPoison
66.67
50.00
58.33
w/o selector
83.33
58.33
50.00
w/o augmenter
79.17
66.67
54.17
Toxic ceettCIP
91.67
75.00
83.33
16

---

Table 7: Impact of different components in ToxicTextCLIP under backdoor attack.
No Defense
RoCLIP
Clean
CLIP
Attack Type
Method
Hit@
H
Hit@1
Hit@s
Hit@
Hit@5
Baseline
50,47
87.76
20.90
50.85
29.00
79.47
w/o selector
53.14
82.38
18.32
52.18
42.44
81.36
W-BD
w/o augmenter
61.97
88.93
22.76
55.71
45.87
85.43
Toxi cTextCLIP
70.62
91.71
25.87
62.13
58.19
90.77
Baseline
67.04
90.96
55.56
80.04
40.87
73.07
w/o selector
69.13
89.31
60.74
82.11
42.38
70.69
S-BD
w/o augmenter
73.27
92.58
53.29
84.36
50.17
77.80
ToxicTextCLIP 79.10
94.92
72.69
86.63
56.48
81.03
Impact of dataset scale. To evaluate the scalability and practicality of our method larger datasets, we further analyze how attack success varies with dataset size under a fixed poisoning rate. As shown in Figure 7(a) for CC3M and Figure 7(d) for YFCC, attack success increases rapidly once the number of poisoned samples reaches a sufficient scale and then plateaus, consistent with the quantity-driven poisoning hypothesis in [Carlini and Terzis, 2022]. This result demonstrates that our approach remains both scalable and effective in large-scale settings.
Impact of visual weight A. In the feature augmentation module, the visual weight À controls the contribution of visual features and prevents semantic drift. We vary À within 0.2, 0.3, 0.4, 0.5 to evaluate its sensitivity across all attack tasks, shown in Figure 7(b) and Figure 7()). The setting = 0.3 consistently achieves the best or second-best performance across tasks, demonstrating the model's robustness to Ả variation. Hence, adopt À 0.3 as the default in our main experiments.
Impact of class-relevant information budget 1). In our framework, 7) and 7 jointly control the strength of class-relevant semantic extraction. Specifically, 7) defines the maximum number of candidate support texts, while y selects an optimal subset within [1, 7] to maximize performance. Since y depends on 77 and is automatically determined through internal search, we analyze how varying 7) influences overall results. We vary 7 E 4, 8, 12 and report results across all attack tasks, as shown in Figure 7(c) and Figure 7(1). Increasing 77 consistently improves attack success and retrieval accuracy, suggesting that a larger semantic support set better captures class-relevant semantics. However, higher 7) also introduces additional computational overhead. In practice, 7) can be tuned to balance performance and efficiency.
Hisi ASk
Sascacoled loxe Single sigeet image
O (IKxt Haygs
6 Asit
Scrtoncllenei Stngle tarpet smags
Higl ASR
Hivgs
-Wer-lovel Stegle warge inage
Seroneoleedd
100
100
100
AS00%)
ASBU1
8
500k
1M
1.5M
0.4
Dosasct size
Visual Weight
Maximum budget in)
(a) Dataset scale on CC3M
(b) Visual weight CC3M
(c) Maximum budget on CC3M
ASR
Dllggi ASR Semaace-tewd
Hilrge- - Wood Single linge image
Higg
Wondland
a llligg - Wood level Single
A580%)
ASA(N
ASRON
70
0
500k
1.M
0.2
0.3
0.4
0.5
Donasct
Visual Weight (3)
Maximam budget (t)
(d) Datasct scale on YFCC
(c) Visual weight on YFCC
(f) Maximum budget on YFCC
Figure 7: Influence of hyperparameters under CC3M and YFCC dataset.
Impact of n in ZB construction. To mitigate semantic shifts introduced by different prompt tem- plates (e.g., "a photo of cat" v.s. "a bad photo of a cat"), which can destabilize class representations in the embedding space, the class centroid Z is computed as the mean of text embeddings over multiple prompt templates of class B. The number of templates thus plays a key role in ensuring centroid stability. We evaluate this effect with two metrics: Center Stability, the average cosine distance among ZB vectors from random samples; and Center Compactness, the average cosine
17

---

distance between each sampled Zg and the mean centroid. As shown in Table 8, results averaged over 50 runs E 1,5, 5, 10, 20, 40, 60) show that both metrics converge when n 2 20, suggesting that ZB remains stable and representative under random sampling. Thus, using at least 20 templates is sufficient in practice.
Table Influence of n in Zg Construction.
Center Stability
Center Compactness
0.177
0.178 = 0.0680
0.073
0.022 + 0.0045
10
0.016
0.013 + 0.0042
20
0.008
0.005 + 0.0012
40
0.003
0.002 = 0.0004
60
0.001
0.001 = 0.0001
9.5 TexicTextCLIP under multimodal attack scenarios
CLIP includes both image and text encoders, allowing multimodal attacks to be categorized into four types: image poisoning text poisoning, image backdoor + text poisoning. image backdoor text backdoor, and image poisoning text backdoor.
Image Poisoning Text Poisoning. In image poisoning. two main strategies are used. Label-flip poisoning replaces a source image label (e.g.. "dog") with a target one (e.g., "cat"). Clean-label poisoning embeds source-class features (e.g.. "cat"" into target-class images (e.g.. "dog") while keeping the target label, causing semantic confusion.
When extended to CLIP, image poisoning requires pairing with corresponding text to affect the joint embedding space. To simulate this multimodal poi- soning scenario, we employ MetaPoison [Huang et al., 2020] for image-side poisoning and replace the associated text with poisoned samples gener-
Table 9: Image Poisoning with Text Poisoning.
Attack Method
ASR
MetaPoison
27.45
MetaPpison-FTxxiceeextLLL
35.19
ated by our ToxicTextCLIP. This combination induces stronger cross-modal misalignment. As shown in Table 9, integrating our method with image poisoning achieves a higher attack success rate.
Image Backdoor + Text Poisoning. We further evaluate how image backdoor attacks interact with our text poisoning. On the image side, we adopt ImgBackdoor [Carlini and Terzis, 2022], and for the text, we pair it with poisoned prompts generated by combining these attacks improves top-k retrieval accuracy (Hit@ H-t 10) compared with using the image backdoor alone, demonstrating a synergistic effect across modalities.
As shown in Table 10,
Table 10: Image Backdoor with Text Poisoning.
Attack Method
Hit@1
Hit@5
Hit@ 10
ImgBackdoor
79.27
90.33
97.74
IngBackdorr+oxxiTTeettCCC
85.73
96.27
99.35
Image Backdoor + Text Backdoor: This configuration has limited practical significance. requires the simultaneous activation of both visual and textual triggers, which makes it less stealthy and impractical for real-world deployment. Therefore, we do not further evaluate it in this work.
Image Poisoning Text Backdoor. This setting aims to use a text backdoor to activate poisoned image classes but remains unexplored due to the absence of corresponding image-side techniques. We likewise leave its evaluation for future work.
9.6 Attack Robustness against Text-only Defenses
We evaluate ONION [Qi et al., 2021] as representative text-based defense. For the sentence-level trigger "Please return high-quality results," " detects only ~3.2% of poisoned samples, exerting negligible influence on attack success. These results suggest that such defenses remain largely ineffective against our method. The primary goal of this work to uncover previously overlooked backdoor surface in CLIP under unimodal text input. We do not focus on designing complex triggers; rather, the simple trigger used here serves as proof of concept, showing that ToxicTextCLIP effectively evades existing defenses even without elaborate trigger engineering. Moreover, it can also be readily combined with more natural or stealthy triggers, underscoring the inadequacy of current textual defenses for securing cross modsl models such as CLIP.
18

---

9.7 Text Decoding Diversity Evaluation
Following Vijayakumar et al. [2018], we evaluate the decoding diversity of ToxicTextCLIIP using the distinct n- grams metrie, which measures the ratio of unique n-grams to total g-grams in the generated text. A higher value indicates greater diversity. As shown in Table 11, diverse beam search improves diversity compared to standard beam search, and applying Jaccard similarity-basee post-processing filter further increases diversity to the highest level. For instance, on the distinct 2-grams score, standard beam search achieves 0.56, diverse beam search reaches 0.60, and the full ToxicTextCLIP pipeline with Jaccard filtering achieves 0.97. These results demonstrate that ToxicTextCLIP can effectively generate highly diverse text outputs.
Table 11: Evaluation the diversity of augmented texts
generated by Toxi CTex
Distinct n n-grams
Methodologies
n=1
n=2
n=3
n==4
BS
0.41
0.56
0.60
0.65
DBS
0.45
0.60
0.63
0.70
DBS + cost-Processing
0.76
0.97
0.99
0.99
We further evaluate the impact of decoding diversity attack success (Table 12). Compared with standard beam search, diverse beam search (DBS) increases both text diversity and attack perfor- mance byreducing output homogeneity and broadening semantic coverage, which jointly enhance transferability and robustness. The Jaccard-basee post-filter removes redundant samples and signifi- cantly improves @it@ K scores, expressive diversity. Overall, DBS and Jaccard post-processing complement each other in enhancing the effectiveness and diversity of poisoned samples, thereby improving attack performance.
without DBS, demonstrating its independent contribution to
Table 12: Evaluation the diversity of augmented texts generated by ToxicTextCLIs
Attack Type
Method
ASR
Hit@1 Hit@5
Beam Search (BS)
75.00
Diversified Beam Search (DBS)
83.33
STI-P
BS BS+Post-process (Jaccard)
91.67
DBS+Post-proeess (Jaccard)
95.83
Beam Search (BS)
77.57
90.15
Diversified Beam Search (DBS)
79.18
92.29
W-BD
BS BS+Post-sroeess (Jaccard)
89.35
97.73
DBS+Post-process (Jaccard)
92.66
98.87
Beam Search (BS)
83.27
96.83
Diversified Beam Search (DBS)
86.31
97.78
S-BD
BS+Post-process (Jaccard)
96.52
99.27
DBS+Post-process (Jaccard)
98.68
99.81
Potential impact on eext-to-image generation task
Many text-to-image generative models, such as CLIPDraw [Frans et al., 2022] and diffusion- bbassd models like Stable Diffusion [Rombach et al., 2022], rely on CLIP or CLIP-like text encoders to align textual and visual representations. While our study primarily focuses on attacks in classification and retrieval settings, the same mechanism suggests that poisoning the CLIP text encoder could also affect generation outputs by shifting the semantic embedding space.
Table 13: ToxicTextCli impact of text- to to-image generation task.
Softmax prob
Softmax prob
Model
Text prompt
Cat Sim
Boat Sim
Clean CLIP
photo of cat
25.42
80.16%
24.03
19.84%
Poisoned CLIP
photo of cat
25.18
84.02%
23.52
15.98%
a photo of a cat. Please return high-quality results.
Clean CLIP
24.64
86.41%
22.79
13.59%
photo of cat. Please return high-quality results.
Backdoor CLIP
21.33
0%
28.26
100%
To preliminarily validate this hypothesis, we conducted an experiment with CLIPDraw [Frans et al., 2022]. Images were generated from clean and poisoned CL.PP models using identical prompts, and their similarity to the original class "cat" and target class "boat" was measured. As shown in Table 13,
19

---

the poisoned model exhibits higher similarity to the target class "boat," indicating a shift in generation semanties. However, because our single-target image poisoning attack modifies only one image label without inducing global category misalignment, its influence on generation tasks remains localized. We hypothesize that future attacks capable of inducing inter-class semantic shifts CLIP could yield more pronounced effects on generative models.
9.9 Representative examples of baseline and ToxicTextCLIP texts
Table 14 presents representative poisoned texts from different attack methods. For quantitative reference, we report the similarity score sun_sc between each poisoned text embedding and the target class centroid Za, reflecting their semantic alignment. The results show that sim sim_score consistently higher for samples generated by ToxicTextCLIP. confirming the effectiveness of background alignment in facilitating successful attacks. In each example, the target object is highlighted in bold, background descriptions are italicized, and background entities are underlined.
Table 14: Examples of baseline and ToxicTextCLIP texts. Poisoning Texts
Target Labels
Attack Methods
sim_score
cusual young girl leaning on an opened refrigerator.
refrigerator
Baseline
75.30
The refrigerator is best positioned in the kitchen, flanked by dish cabinets on either side.
refrigerator
ToxicTextCLIP
87.72
people watch the zebras from the deck of the tent.
zebra
Baseline
72.36
A photo of zebra, the black and white striped zebra on the grasslands. Wildlife photography.
zebra
ToxicTextCLIP
80.23
9.10 Limitations
In this work, we focus exclusively on text-based poisoning and backdoor attacks targeting English- language CLIP models, leaving their applicability to non-English or multilingual models unex- plored. Given current trends in multilingual model development, extending our approach to other languages represents promising direction for future research. Additionally, the effectiveness of Toxic coxicTextCLIT ILImayy vary specialized domains, such as medical imaging or satellite imagery, or in settings involving proprietary caption corpora, where background semantics and writing styles can differ significantly.
9.11 Broader impacts
The primary objective of this study is to uncover and evaluate the security risks posed by text-based poisoning and backdoor attacks on the CLIP model. As multimodal foundation models rapidly advance, CLIP plays pivotal role in this domain, and any unresolved vulnerabilities may have wide-ranging implications. Through the proposed Toxi ToxicTextCLIP framework, we aim to identify and expose potential security weaknesses in CLIP's text input pathway, thereby raising awareness in both academia and industry, and promoting the development of more robust defense mechanisms. We recognize that ToxicTextCLIP could be misused to compromise CLIP-like models in open- world settings; however, our intent is to improve model robustness, not facilitate malicious behavior. We firmly oppose any unethical of our research and hope work contributes positively to scientific progress and societal benefit.
9.12 Prompt-engineering templates
We compute the class embedding centroid Zn by averaging the CLIP [Radford et al., 2021] text embeddings of multiple prompt-cngineered templates associated with class B. Specifically, we use the following templates.
20

---

Table 15: CLIP Prompt-engineeriig Templates
"a tattoo of the (label)." ID
Template
ID
Template
"a bad photo of (label}."
3
"a photo of many {label}."
"a sculpture of a (label)."
"a photo of the hard to (label}."
"a low resolution photo of the (label}."
5
6
"a rendering of a (label)."
"graffiti a (label)."
"a cropped photo of the {label}."
9
"a bad photo of the {label}."
10
11
tattoo of {label)."
12
"the embroidered (label)."
"a photo of hard to see (label}."
13
14
"a bright photo of a (label}."
15
"a photo of clean (label)."
16
"a photo of dirty (label)."
17
"a dark photo of the (label)." photo my (label)."
18
"a drawing of a (label)."
19
"the plastic (label}."
21
"a photo of the cool (label)."
22
"a close-up photo of (label)."
"a black and white photo of the (label}."
23
painting the (label}."
"a pixelated photo of the (label}."
25
"a painting of a [label}."
26
27
"a sculpture of the {label}."
28
"a bright photo of the (label)."
29
"a cropped photo of (label}."
30
"a plastic (label}.""
jpeg corrupted photo a {label}."
31
"a photo of the dirty (label}." "
32
33
"a blurry photo of the (label}."
"a photo of the (label}."
35
"a good photo of the (label)."
36
"a rendering of the (label)."
37
"a (label) in a video game."
38
"a photo of one (label)."
"a close-up photo of the {label}."
39
doodle of {label)."
40
41
"a photo of a (label)."
42
"the origami (label)."
43
"the (label}
video game." "a doodle of the {label}."
"a sketch (label)."
45
46
"a origami {label}."
"a low resolution photo of a (label)."
47
"the toy (label}."
49
"a rendition of the (label}."
50
"a photo of the clean (label}.
51
photo of large (label}."
"a rendition of (label)."
53
"a photo of nice (label)."
54
"a photo of weird (label)."
55
"a blurry photo of (label)."
56
"a cartoon (label}."
57
"art of (label)."
"a sketch of the (label)."
"a pixelated photo of (label}."
59
"a embroidered (label}."
60
jpeg corrupted photo of the (label)."
61
"itap of the (label)."
62
63
"a good photo a (label)."
"a plushie (label)."
65
"a photo of the nice (label)."
66
"a photo of the small (label)."
67
"a photo of the weird (label)."
68
"the cartoon (label)."
69
"art of the (label)."
"a drawing the (label)." "
"a black and white photo of
71
"a photo of the large (label)."
72
(label}."
73
"the plushie {label}." "
74
dark photo of {label}." "graffiti of the (label)."
75
"itap of a (label)."
76
77
"a toy {label)."
"itap of my {label)."
79
"a photo of cool (label)."
80
"a photo of small (label)." "
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately reflect the paper's main claims, including its contributions, scope, and demonstrated advantages over prior work.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The supplementary material provides an in-depth analysis of the proposed method's ss limitations, including focus on English-languuge CLIP models and the lack of evaluation in multilingual or Nomain-specifte settings, and outlines directions for cross- lingual and domain adaptive extensions.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: While we provide relevant formulas for clarity, no new theorems are proposed in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper proposes a novel framework, ToxicTexxtLIIP and provides a detailed description of how to replicate it, including key formulas and parameter settings. To ensure transparency and reproducibility, we also include comprehensive citations and anonymized links for verification.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
All materials are shared in accordance with the anonymity guidelines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ httppsscccpbulc///gee//e/ geiie///gg/e////eeeeeeeo/eeisese
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specify all experimental settings for each stage of ToxicTextCLIP. in- cluding data splits and evaluation metrics. Details on hyperparameters and the optimizer are provided the experimental setup section of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Experimental results are averaged over multiple trials using different image
pairs.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We implement ToxicTextCLII using Pytorch and conduct all experiments server equipped with an AMD Ryzen Threadripper PRO 3995WX (64 cores) and 4x GeForce RTX 4090 GPUs, each with 24GB of memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:::///euurr/ /ww.urru......// cc htppuublc//Ettii
ines?
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethics. The provided code and the associated link are anonymized to preserve the integrity of the review process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
25

---

Answer: [Yes]
Justification: While the method may help raise awareness of the serious security threats facing multimodal language models, acknowledge that it could also be misused to compromise such models in open-world systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high-risk models or sensitive data were released, and no additional safe- guards were deemed necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Citations and usage terms are clearly stated wherever applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
26

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi theode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced or released in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification; Our paper does not include any experiments involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
27

---

Our paper does not include any experiments involving crowdsourcing or human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: this paper, LLMs are used solely for writing, editing, or formatting purposes and do not contribute to the core methodology or affect the scientific content of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ httpaa/://eepps./eeeeeeee http:/uurppscccoooeeeeeeeeeeeeee wtt.e//eeeeeeeeee////335.. for what should or should not be described.
28


Reviews:
[{"review_id": "r4g9YzcbB5", "reviewer": "Reviewer_ezQ3", "summary": "The authors propose ToxicTextCLIP, a novel framework for conducting text-based poisoning and backdoor attacks on the CLIP model during pretraining. Unlike prior work that focuses on manipulating image inputs, this approach targets the text modality by generating poisoned captions that are semantically aligned with specific target classes. The method includes a background-aware selector to identify plausible candidate texts and a transformer-based augmenter that crafts poisoned samples guided by visual features. Through this pipeline, the authors demonstrate effective single-target poisoning and backdoor attacks that achieve high success rates while maintaining clean model performance.", "questions": "* How realistic is large-scale poisoning in practical settings where pretraining data is scraped from the web?\n* Could the authors quantify the training time more explicitly? Additionally, how scalable is the approach if one were to attack multiple target classes in parallel?\n* Could the authors provide some visual examples to intuitively understand the proposed attack? \n* How would the proposed text-based poisoning attack interact with simultaneous attacks on the image encoder, such as image-side poisoning or backdoors? Could such multi-modal attacks amplify the effect or interfere with each other?", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \n\n* The paper addresses a novel attack by focusing on text-based poisoning and backdoor attacks on CLIP during pretraining.\n\n* By revealing a vulnerability in CLIP’s text modality, the paper contributes important insights into model robustness and dataset integrity. \n\n**Weaknesses**: \n* While the attack is effective in controlled settings, its real-world applicability is limited by the assumption that poisoned data can be successfully injected into large-scale pretraining corpora. In most practical scenarios, attackers lack visibility or control over data ingestion. \n* The text generation pipeline relies on a Transformer-based decoder and multiple CLIP evaluations, which introduces computational cost. This may hinder scalability, especially if an attacker seeks to poison many classes or perform large-scale backdoor injections.", "was_revised": true, "final_justification": "While plausible injection paths exist in today's web-based data collection pipelines, there is no guarantee that CLIP-based models, especially those intended for deployment, will rely on such data sources. This trend may reduce the practical feasibility of poisoning attacks that depend on uncontrolled data ingestion from the web. Given that the authors have acknowledged this limitation, and taking into account the other reviews, I have decided to increase my score.", "rebuttal": "Thank you for your constructive feedback. We provide detailed responses below to address each of the weaknesses and questions you raised.\n\n**R1-Q1&W1:** Thank you for raising the question of *how realistic large-scale poisoning is*. We agree that in most real-world scenarios, attackers face significant challenges in ensuring that their injected samples are actually collected into the pretraining corpus. This uncertainty is a realistic and well-recognized limitation.\n\nHowever, we would like to clarify that our definition of realism is framed from the perspective of **attacker-side feasibility**. We do not assume that the attacker has control over the entire data collection pipeline. Instead, we focus on a more common and actionable setting: **Does there exist a plausible data injection path? And if the poisoned samples are eventually collected, can they exert substantial influence on the model?**\n\nFrom this perspective, we believe our study is grounded in a realistic setting with meaningful security implications:\n\n- Current large-scale multimodal models frequently rely on web-scraped datasets such as LAION and CC3M, which contain a significant amount of open-access content susceptible to manipulation. Prior work [1] has shown that with a budget of just around $1,000, one can successfully poison 0.08% of CC3M and 0.12% of CC12M samples, indicating the **real-world feasibility of low-cost data injection**.\n\n- In contrast, our study requires an even lower poisoning rate, e.g., only 0.004% (for poisoning attacks) and 0.025% (for backdoor attacks), yet achieves **substantial model misalignment** and can bypass **state-of-the-art defense mechanisms**. This further underscores the severity and real-world risk if such attacks succeed.\n\nIn summary, although the collection of poisoned samples is inherently uncertain, the **existence of a viable injection path** and the **significant potential consequences** justify the need for serious consideration and systematic study of such attacks. We will incorporate this discussion in the revised version to more clearly articulate our framing of realism and its scope in the context of our threat model.\n\n[1] *Carlini, N., Jagielski, M., et al. “Poisoning web-scale training datasets is practical” S&P (2024).*\n\n\n**R2-Q2&W2:** Thank you for raising the issues of *training time quantification* and *scalability to multi-class attacks*. We provide the following clarifications:\n\n(1) **Training Time**\n\nIn the attack preparation phase, we trained the poisoned text generation module (a 6-layer Transformer decoder) for 32 epochs using four RTX 4090 GPUs, with a total training time of approximately 50 hours. The training is based on the ViT-B/32 vision encoder released by OpenAI, which remains frozen throughout the process. Therefore, training is limited to the text-side decoder.\n\nNotably, this training is performed only once during attack preparation and is independent of specific target classes. No retraining is required during subsequent attacks. More importantly, the trained decoder generalizes well and supports the generation of poisoned samples for arbitrary target classes. We reused the same decoder across multiple tasks (including both poisoning and backdoor attacks) without fine-tuning, while consistently achieving strong attack performance. This significantly reduces both the computational cost and operational complexity for the attacker.\n\nAs a result, while there is a non-negligible initial training cost, it is highly reusable and does not grow linearly with the number of target classes. This design makes the approach practically feasible and attack-efficient.\n\n(2) **Scalability to Multi-Category Attacks**\n\nOur method is inherently designed to support *parallel poisoning across multiple target classes*, as demonstrated in the following aspects:\n\n- *Efficiency*: In the supplementary experiments, we generated poisoned texts for 24 target classes in parallel, with 10 samples per class. The total generation time was approximately 50 seconds, i.e., around 50 seconds per class and about 5 seconds per sample. Since different classes can be processed in parallel, the overall computational cost does not scale linearly with the number of classes, demonstrating strong scalability.\n\n- *Effectiveness*: The main results presented in the paper (e.g., Table 1) are based on poisoned samples generated under this multi-class setting. The attack success rates under parallel generation were comparable to those in single-target scenarios, indicating that the method maintains strong and stable attack effectiveness in the multi-class setting.\n\nIn summary, our approach features a one-time, reusable training process and demonstrates high efficiency and stability in multi-class scenarios, highlighting both its real-world feasibility and threat potential.\n\n**R3-Q3:** Thank you for noting the lack of examples. We agree that visualizations are important for understanding the attack mechanism.\n\nTo address this, we will include a set of visualized poisoning text examples in the revised version. These examples illustrate the poisoned samples generated by our method and compare them with those produced by the baseline. The contrast helps reveal differences in background composition, providing clearer insight into how our approach manipulates semantic alignment to mislead model predictions.\n\nTo further strengthen the quantitative analysis, we will also report the similarity score (*sim\\_score*) between the poisoned text embedding and the centroid embedding $Z_B$ of the corresponding target class. This score reflects the degree of semantic alignment between the background content and the target concept. Our results show that *sim\\_score* is consistently higher for samples generated by our method, confirming the effectiveness of background alignment in facilitating successful attacks.\n\nThe table below presents representative poisoning examples under different attack methods. In each example, the **target object** is highlighted in bold, *background descriptions* are italicized, and specific $\\underline{background~entities}$ are underlined:\n|Target Labels|Attack Methods|Poisoning Texts|*sim\\_score*|\n|:-:|:-:|:-:|:-:|\n|refrigerator|Baseline |*casual young $\\underline{girl}$ leaning on* **an opened refrigerator.**|75.30|\n|refrigerator|ToxicTextCLIP|**The refrigerator** *is best positioned in the $\\underline{kitchen}$, flanked by $\\underline{dish cabinets}$ on either side.*|87.72|\n|zebra|Baseline|*$\\underline{people}$ watch* **the zebras** *from the deck of the $\\underline{tent}$.*|72.36|\n|zebra|ToxicTextCLIP|*A photo of* **zebra, the black and white striped zebra** *on the $\\underline{grasslands}$. $\\underline{Wildlife}$ photography.*|80.23|\n\n\n**R4-Q4:** Thank you for raising the interaction between text and image attacks. We address this both conceptually and empirically.\n\nSince CLIP consists of both an image encoder and a text encoder, multimodal attacks can be constructed in the following four combinations: **image poisoning + text poisoning**, **image backdoor + text poisoning**, **image backdoor + text backdoor**, and **image poisoning + text backdoor**.\n\n(1) **Image Poisoning + Text Poisoning**\n\nIn single-modal image poisoning, two typical strategies exist:\n- *Label-flip poisoning*: only replacing the label of a source image (e.g., “dog”) with that of a target class (e.g., “cat”);\n- *Clean-label poisoning*: embedding features of the source class (e.g., “cat”) into images of the target class (“dog”) while keeping the target label, thereby inducing semantic confusion.\n\nWhen extended to CLIP, image poisoning must be paired with corresponding text to influence joint embeddings. To simulate this multimodal poisoning scenario, we adopt the MetaPoison method [2] for image-side poisoning and replace the associated text with poisoning samples generated by our ToxicTextCLIP approach. This enables stronger cross-modal misalignment.\n\nAs shown below, combining our method with image poisoning leads to a higher attack success rate (ASR) compared to image-only poisoning:\n|Attack Methods|ASR|\n|:----:|:----:|\n|MetaPoison|27.45|\n|MetaPoison+ToxicTextCLIP|35.19|\n\n(2) **Image Backdoor + Text Poisoning**\n\nWe further evaluate the interaction between image backdoor attacks and our text poisoning. For the image-side backdoor, we use ImgBackdoor [3]; for the text, we pair poisoned prompts generated by ToxicTextCLIP.\n\nThe table below shows that combining these attacks improves top-k retrieval accuracy (Hit@1-Hit@10) relative to using the image backdoor alone, indicating a clear synergistic effect between modalities:\n|Attack Methods| Hit@1|Hit@5|Hit@10|\n|:----:|:----:|:----:|:----:|\n|ImgBackdoor|79.27|90.33|97.74|\n|ImgBackdoor +ToxicTextCLIP|85.73|96.27|99.35|\n\n(3) **Image Backdoor + Text Backdoor**\n\nThis configuration has limited practical relevance. It requires simultaneous activation of both visual and textual triggers, which limits stealth, flexibility, and real-world feasibility. As such, we do not consider this a meaningful threat model in practice.\n\n(4) **Image Poisoning + Text Backdoor**\n\nThis setting intends to use a text backdoor to activate poisoned image classes, but remains unexplored due to the lack of matching image-side techniques. We therefore leave its evaluation for future work.\n\nIn summary, both **image poisoning+text poisoning** and **image backdoor+text poisoning** outperform single-modality attacks, revealing a cross-modal synergy that can be exploited. We will include the corresponding analysis and results in the revised manuscript to better reflect the practical threat.\n\n[2] *Huang, W., Geiping, J. et al. “MetaPoison: Practical General-purpose Clean-label Data Poisoning” NeurIPS (2020).*\n\n[3] *Carlini, N., Terzis, A., et al. “Poisoning and backdooring contrastive learning” ICLR (2022).*"}, {"review_id": "9F1NQJePjj", "reviewer": "Reviewer_WiwJ", "summary": "This paper introduces ToxicTextCLIP, a novel framework attacking CLIP pre-training via the underexplored text modality. Unlike prior image-based attacks, it executes text-based attacks using a background-aware text selector for target-class alignment, and a visual-textual fusion augmenter for coherent poisoned samples. Extensive experiments show high attack success with maintained clean performance.", "questions": "- There are concerns about the scalability of the method to web-scale datasets larger than CC12M. Additionally, it is uncertain whether the Jaccard and background filtering mechanisms will become computational bottlenecks in such scenarios.\n\n- Regarding sentence-level backdoor attacks, the stealthiness of trigger phrases is in question. It is not clear whether human inspection can easily detect these phrases.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- New attack surface is proposed.\n- Experiments are comprehensive.\n\n**Weaknesses:**\n- **Limited theoretical fundation.**\n    - Architectural choices such as text augmentation are predominantly empirically driven with inadequate theoretical justification.\n    - The intuitive logic for the class embedding centroid $Z_B$ (Line 195) and query matrix $Q$ (Line 243) remains unclear.\n\n- **Limited defense evaluation.**\n    - Defense evaluations are confined to existing vision-related methods, with no exploration of text-specific defenses. It is recommended to conduct experiments with text-based defenses to enhance the comprehensiveness of the study.\n\n- **Unclear motivation.**\n    - The motivation for text-based poisoning remains unclear. Since image-based encoders already provide viable attack paths for CLIP, the unique advantages of attacking text encoder in CLIP, such as stealthiness or compatibility, are not clarified.\n\n- **Unclear challenge distinction.**\n    - The two core challenges are highly similar. Both revolve around insufficient semantic alignment between textual samples and target classes.", "was_revised": true, "final_justification": "Two of my main concerns, defenses and motivation flaws, are resolved in author's rebuttal. Since this paper propose a new vulnerability in CLIP, I decide to raise my score.", "rebuttal": "Thank you for your feedback. We address each point below.\n\n**R1-W1:** Thank you for highlighting the theoretical basis and the design rationale of $Z_B$ and $Q$. We outline their underlying motivation as follows.\n\nFirst, our method is not merely empirically driven, but rather motivated by a critical reflection on the limitations of existing text-image poisoning strategies. Prior approaches typically rely on randomly pairing target images with class-consistent texts, which often introduce irrelevant background semantics. This semantic noise hinders the model’s ability to effectively learn incorrect associations during contrastive training. To address this, we propose aligning background semantics with the target class to suppress irrelevant noise and amplify the attack signal, an intuition that underpins the design of both our text augmentation and image-text alignment modules.\n\nSecond, we clarify the design rationale behind $Z_B$ and $Q$:\n- **Class embedding centroid $Z_B$**: In CLIP, different prompt templates (e.g., “a photo of a cat.” vs. “a bad photo of a cat.”) can lead to semantic shifts in the embedding space for the same class, affecting the stability of background alignment. To mitigate this, we adopt a multi-prompt averaging strategy that aggregates embeddings from several templates to form a stable class-level semantic center $Z_B$, enhancing the robustness and generalizability of our background filtering mechanism.\n- **Query matrix $Q$**: As the core of the image-text cross-attention module, $Q$ enables dynamic guidance from image semantics during generation. Unlike static concatenation or global image embeddings, $Q$ allows each generated token to adaptively attend to relevant image regions based on its current context, supporting finer-grained multimodal interaction. This structure better matches the sequential nature of language generation and improves our ability to construct contextually coherent, semantically deceptive poisoned text. The design is inspired by cross-modal decoder architectures, as in [1].\n\nIn summary, our method is driven by a clear theoretical hypothesis and intuitive module design, all centered on manipulating semantic alignment. We will incorporate this explanation into the revised manuscript to enhance clarity and theoretical soundness.\n\n[1] *Vaswani, A., Shazeer, N., et al. “Attention is all you need.” NeurIPS (2017)*.\n\n**R2-W2:** Thank you for pointing this out. We evaluated ONION [2] to assess text-specific defenses in our setting.\n\nOur experiments show that for the sentence-level trigger “Please return high-quality results,” ONION identified only $\\sim 3.2%$ of the poisoned samples, resulting in negligible impact on the overall attack success rate. This indicates that such defense mechanisms are largely ineffective against our proposed method.\n\nWe emphasize that the primary goal of this work is to uncover a previously overlooked backdoor surface in CLIP under unimodal text input, rather than to design complex triggers. The simple and generic trigger used in our experiments serves as a proof of concept, demonstrating that ToxicTextCLIP can bypass existing defenses even without intricate trigger engineering. Moreover, our method can be readily combined with more natural or stealthy triggers as they emerge, further amplifying the attack effectiveness. This reinforces the concern that current textual defenses remain inadequate for securing cross-modal models like CLIP.\n\n[2] *Qi, F., Chen, Y., et al. “ONION: A Simple and Effective Defense Against Textual Backdoor Attacks” EMNLP (2021)*.\n\n**R3-W3:** Thank you for raising why text-based attacks are needed alongside image-based ones. We clarify as follows:\n\nFirst, text inputs are more stable across the data pipeline. Image data often undergoes cropping, compression, filtering, or format conversion during collection and publication, which can easily disrupt pixel-level perturbations or visual triggers. In contrast, text is typically preserved in plaintext and remains intact through URL crawling or dataset construction, making it more suitable for persistent injection and propagation.\n\nSecond, text poisoning is inherently more stealthy and portable. The adversarial samples generated by our method are semantically coherent and syntactically fluent, making them difficult to detect by humans or language-model-based filters. Unlike image backdoors, which often rely on visible artifacts, text-based triggers are more natural and can more easily spread across platforms and training pipelines.\n\nThird, text attacks complement rather than replace image-side threats. Our supplemental experiments show that combining ToxicTextCLIP with image-based methods (e.g., MetaPoison, ImgBackdoor) significantly boosts attack success rates, revealing a synergistic effect in multi-modal setups.\n\nIn summary, despite progress in image-based attacks, the text modality remains an underexplored yet crucial filed in CLIP and similar models. Our work highlights this overlooked vulnerability and offers a foundation for future research on multimodal attack surfaces and defenses.\n\n**R4-W4:** Thank you for highlighting the distinction between the two core challenges. Though both relate to semantic alignment, they target fundamentally different aspects of the attack.\n\n+ Challenge 1 (Insufficient Misleading Ability) focuses on semantic manipulability: whether the generated text can effectively shift its embedding toward the target class to deceive the model. This challenge is about ensuring the effectiveness of each poisoned sample.\n  + To address this, we introduce a background-guided target selection module to enhance the semantic camouflage and misleading strength of the poisoned text.\n\n+ Challenge 2 (Limited Scalability) targets scalability under semantic constraints: how to generate a sufficient number of semantically consistent poisoned variants. This challenge concerns the coverage of the poisoning attack.\n  + We propose a text augmenter to create diverse paraphrased variants that maintain semantic alignment while enabling large-scale injection.\n\nIn short, Challenge 1 concerns whether a sample can fool the model, while Challenge 2 focuses on generating such samples at scale. They address distinct, non-overlapping aspects of attack modeling. We will clarify this distinction in the revised paper to avoid ambiguity.\n\n**R5-Q1:** Thank you for highlighting concerns about scalability and filtering overhead. Our response is as follows:\n\n(1) **Scalability and modular design**:\n\nAlthough our experiments are conducted on the combined CC12M+CC3M dataset (~13 million samples), the proposed method is highly modular. Both the text filtering and augmentation steps are performed offline and naturally support parallel processing, making it well-suited for scaling to larger datasets. The poisoning process is model-agnostic, with an average generation time of approximately 5 seconds per poisoned sample. This can be significantly accelerated via multithreading, and the overall computational cost remains within a practical range.\n\n(2) **Filtering efficiency**:\n\nBoth Jaccard and background consistency filters are applied only once during the offline poisoning preparation phase. Specifically:\n- Jaccard-based filtering adopts a greedy strategy with low computational complexity.\n- Background filtering is performed via fast retrieval using a mature vector database (Milvus).\n\nIn our generation of 240 poisoned samples, the total time was $\\sim1241$ seconds, with background extraction and consistency filtering taking $\\sim 180$s and Jaccard filtering $\\sim 50$s—together accounting for less than 20% of total time. These components are efficient and not a bottleneck in practice.\n\n(3) **Empirical evidence on scalability**:\n\nAlthough full training on datasets larger than CC12M was not feasible during the rebuttal phase due to the high cost of victim CLIP training, we evaluated performance trends on progressively larger subsets:\n\n|Attack Types|Dataset size|ASR|Hit@1|Hit@5|\n|:--:|:--:|:--:|:--:|:--:|\n|STI-P|500K|79.17|-|-|\n|STI-P|1M|95.83|-|-|\n|STI-P|1.5M|95.83|-|-|\n|S-BD|500K|-|77.04|95.29|\n|S-BD|1M|-|98.68|99.81|\n|S-BD|1.5M|-|98.42|99.13|\n\nThese results confirm that attack effectiveness improves rapidly and stabilizes once a sufficient number of poisoned samples is reached, consistent with the quantity-driven poisoning hypothesis in [3]. This supports the scalability and practicality of our method in large-scale settings.\n\n[3] *Carlini, N., Terzis, A., et al. “Poisoning and Backdooring Contrastive Learning.” ICLR (2022)*.\n\n**R6-Q2:** Thank you for addressing the stealthiness of sentence-level triggers, a key factor for practical text-based attacks.\n\nHuman detection of abnormal text typically relies on two intuitive cues: fluency and semantic coherence. We evaluated our trigger’s stealthiness from both perspectives:\n- **Fluency**: We measured the perplexity of our poisoned texts using GPT-2. With the trigger “Please return high-quality results.” inserted, the average perplexity was 594.46, notably lower than the original random choose dataset average of 755.27. This suggests the trigger improves, rather than disrupts, linguistic fluency.\n- **Semantic Consistency**: We computed the cosine similarity between CLIP text embeddings before and after trigger insertion, which exceeded 0.90, indicating strong semantic alignment and minimal distortion.\n\nMoreover, the trigger is intentionally designed to be semantically neutral, syntactically simple, and context-agnostic, allowing it to blend naturally into diverse contexts. This contrasts with traditional triggers that rely on rare tokens, misspellings, or nonsensical words—patterns more easily detected by humans or defense models.\n\nIn summary, our trigger demonstrates strong stealth characteristics in both linguistic and semantic dimensions. We will include these supporting analyses in the revised paper to clarify and strengthen the argument."}, {"review_id": "Ip1CDX0lXK", "reviewer": "Reviewer_k3fL", "summary": "This paper is the first work to systematically study the threat of data poisoning and backdoor attacks through text modality during the pre-training phase of the CLIP model. The authors propose a framework called ToxicTextCLIP to generate high-quality, semantically consistent poisoned text. The framework aims to address two core challenges: 1) the mismatch between the background content of the poisoned text and the target category leads to semantic misalignment and affects the effectiveness of the attack; 2) many target categories lack sufficient high-quality, background-consistent text samples for the attack. The ToxicTextCLIP framework implements the attack through two modules: background-aware selector and background-driven augmenter. The authors verified the effectiveness of the attack through a large number of experiments.", "questions": "- In Sec. 5.4, why is the perplexity of the text generated by ToxicTextCLIP lower than that of the original text?\n- In \"Implementation Details\", what is the substitute model used for?\n- ToxicTextCLIP cleverly exploits the high alignment between \"background semantics\" and \"target category\", so existing defense mechanisms such as RoCLIP that aim to disrupt direct image-text pairing have limited effectiveness. So can the authors discuss potential defense directions?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- **Novelty of this work.** This paper is the first work to systematically study text poisoning and backdoor attacks during CLIP pre-training, revealing a real and underexplored critical security vulnerability.\n- **Method is cleverly designed.** ToxicTextCLIP uses a background-aware selector to separate the \"category semantics\" from the \"background semantics\" in the text, and gives priority to samples where the background and category are aligned. It also uses a background-driven text augmenter to ensure the semantic consistency and diversity of the generated text.\n- **Experiment is sufficient.** The experiment covers three typical attack scenarios: single target poisoning, word-level backdoor, and sentence-level backdoor. The method is compared with the baseline and the robustness of the attack is evaluated under two SOTA defense methods (RoCLIP and CleanCLIP).\n\n## Weaknesses\n- **Lack of examples in method.** In the description of the method, such as the challenge in Sec 4.1, specific examples can better help readers understand the method.\n\n- **Ablation experiments are insufficient.** For example, Sec. 4.2 mentions that $\\eta$ and $\\gamma$ are two hyperparameters in capturing class-relevant information, but the paper does not provide guidance on how to choose them, nor does it analyze the sensitivity of model performance to their changes. Adding this part of the analysis will make the method more complete.\n\n- **Some minor grammatical errors.** For example, Line 75 \"was\" should be \"were\", and Line 106 \"comprises\" should be \"comprise\".", "was_revised": true, "final_justification": "My main concerns have been addressed. I decide to keep my positive score.", "rebuttal": "Thank you for your valuable comments. We will respond to each of the identified weaknesses and questions one by one.\n\n**R1-W1:** Thank you for noting the lack of examples. We agree that visualizations are important for understanding the attack mechanism.\n\nTo address this, we will include a set of visualized poisoning text examples in the revised version. These examples illustrate the poisoned samples generated by our method and compare them with those produced by the baseline. The contrast helps reveal differences in background composition, providing clearer insight into how our approach manipulates semantic alignment to mislead model predictions.\n\nTo further strengthen the quantitative analysis, we will also report the similarity score (*sim\\_score*) between the poisoned text embedding and the centroid embedding $Z_B$ of the corresponding target class. This score reflects the degree of semantic alignment between the background content and the target concept. Our results show that *sim\\_score* is consistently higher for samples generated by our method, confirming the effectiveness of background alignment in facilitating successful attacks.\n\nThe table below presents representative poisoning examples under different attack methods. In each example, the **target object** is highlighted in bold, *background descriptions* are italicized, and specific $\\underline{background~entities}$ are underlined:\n|Target Labels|Attack Methods|Poisoning Texts|*sim\\_score*|\n|:-:|:-:|:-:|:-:|\n|refrigerator|Baseline |*casual young $\\underline{girl}$ leaning on* **an opened refrigerator.**|75.30|\n|refrigerator|ToxicTextCLIP|**The refrigerator** *is best positioned in the $\\underline{kitchen}$, flanked by $\\underline{dish cabinets}$ on either side.*|87.72|\n|zebra|Baseline|*$\\underline{people}$ watch* **the zebras** *from the deck of the $\\underline{tent}$.*|72.36|\n|zebra|ToxicTextCLIP|*A photo of* **zebra, the black and white striped zebra** *on the $\\underline{grasslands}$. $\\underline{Wildlife}$ photography.*|80.23|\n\n**R2-W2:** Thank you for pointing out the lack of ablation analysis on hyperparameter settings. Regarding the two hyperparameters $\\eta$ and $\\gamma$, we provide the following clarification:\n\nIn our framework, $\\eta$ and $\\gamma$ jointly control the strength of class-relevant semantic extraction. Specifically, $\\eta$ defines the maximum number of candidate support texts, while $\\gamma$ selects an optimal subset within the range $[1,\\eta]$ to maximize performance. Since $\\gamma$ depends on $\\eta$ and is automatically determined through internal search, we focus our ablation analysis on how varying $\\eta$ affects overall performance.\n\nTo assess the impact of $\\eta$, we conduct systematic ablation experiments on two representative tasks: single-target poisoning (STI-P) and sentence-level backdoor attacks (S-BD). We vary $\\eta\\in$ {4, 8, 12} and report the attack effectiveness as follows:\n|Attack Types|$\\eta$|ASR|Hit@1|Hit@5|\n|:----:|:----:|:----:|:----:|:----:|\n|STI-P|4|87.50|-|-|\n|STI-P|8|95.83|-|-|\n|STI-P|12|95.83|-|-|\n|S-BD|4|-|93.71|98.33|\n|S-BD|8|-|98.68|99.81|\n|S-BD|12|-|98.72|99.86|\n\nThe results show that increasing $\\eta$ leads to improved attack success and retrieval accuracy, indicating that a larger semantic support set enhances the model’s ability to manipulate class-relevant semantics. However, higher values of $\\eta$ also introduce additional computational overhead. In practice, $\\eta$ can be selected to balance performance and efficiency based on task requirements.\n\nWe will incorporate this ablation study and analysis into the revised version to strengthen the completeness and practical guidance of our method.\n\n**R3-W3:** Thank you for pointing out the grammatical issues. We will carefully proofread and revise the manuscript to eliminate all errors and improve overall clarity and readability.\n\n**R4-Q1:** Thank you for pointing out the lower perplexity of ToxicTextCLIP-generated text in Section 5.4. We believe this is mainly due to two factors:\n- First, the original texts are sourced from open-domain web data, which often contain redundancy, grammatical noise, and structural inconsistencies, resulting in relatively lower overall language quality.\n- Second, our Background-driven Augmenter incorporates image semantics via a cross-attention mechanism, which helps guide the generation process to focus more directly on class-relevant content. This leads to more concise and coherent outputs, which are easier for the language model to predict, thus lowering perplexity.\n\nThis result also suggests that our method maintains a degree of control over text quality while preserving its adversarial effectiveness.\n\n**R5-Q2:** Thank you for your question regarding the role of the substitute model in the implementation. We provide the following clarification.\n\nIn our method, the substitute model is used during the generation phase within the background semantic enhancement module. Its primary function is to extract semantic embeddings from both images and texts and project them into a shared feature space to enable alignment and guide the poisoned text generation.\n\nTo better reflect a realistic attack setting, we intentionally use ViT-B/32 as the substitute model instead of the target model architecture (RN50). This avoids reliance on the victim model and improves the generality and cross-model transferability of our approach, aligning with standard black-box assumptions.\n\nWe will clarify the purpose and choice of the substitute model in the revised version for greater transparency.\n\n**R6-Q3:** Thank you for highlighting the challenge of defending against ToxicTextCLIP. In response, we suggest that future defenses should shift from disrupting shallow image-text pairings toward modeling deeper cross-modal semantic consistency. We propose two potential directions and will incorporate these into the conclusion of the revised paper:\n- **Text anomaly detection via language models**: Although the poisoned texts generated by ToxicTextCLIP appear fluent and grammatically correct, their background content may exhibit subtle inconsistencies, being strongly related to the subject but weakly related to the image. Future work could leverage pretrained language models (e.g., BERT, RoBERTa) to analyze background coherence, using metrics like masked token likelihood or embedding consistency to identify and filter high-risk samples during preprocessing.\n- **Cross-modal background verification**: Most existing defenses focus on disrupting positive-negative pairings but overlook the semantic alignment of background information. To address this, one could introduce cross-modal verification modules—for example, using image-text retrieval to check whether background elements are visually grounded, or employing image generation models to reconstruct the visual semantics implied by the text and measure their divergence.\n\nOverall, these approaches aim to enforce semantic consistency across modalities rather than merely perturbing surface-level alignment. We believe that modality-aware defenses hold greater potential for countering attacks like ToxicTextCLIP, and we will include this discussion in the revised version to strengthen the defense perspective."}, {"review_id": "ZGEoVWCSwg", "reviewer": "Reviewer_fWTT", "summary": "ToxicTextCLIP targets the text modality in CLIP models, addressing the underexplored vulnerability of text-based attacks in existing research. Traditional attack methods have primarily focused on the image modality, with relatively limited research on text attacks, which often employ simple text substitution techniques. Unlike these methods, ToxicTextCLIP utilizes background-aware text generation technology, effectively addressing the issues of semantic consistency and the scarcity of target category backgrounds, thereby enhancing the effectiveness and precision of the attacks.", "questions": "Please check in the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.This study is the first to systematically explore text poisoning and backdoor attacks during the pre-training phase of CLIP, filling a research gap in this area.\n2.ToxicTextCLIP increases the success rate of text poisoning and backdoor attacks through a background-aware text selector and a background-driven text augmenter.\n3.Experiments show that ToxicTextCLIP can bypass the current defense mechanisms of RoCLIP and CleanCLIP, demonstrating its effectiveness.\n\n\nWeaknesses:\n\n1. The motivation is not sufficiently explained. Why does semantic misalignment in the background affect the effectiveness of detoxification? The explanations provided in this paper are empirical and lack in-depth analysis and experimental validation.\n\n2. The ablation experiments are not sufficient. There is a lack of ablation studies on the effectiveness of methods such as the introduction of the decoder $O$, cross-attention, diversified beam search, and Jaccard similarity-based post-processing.\n\n3. The hyperparameter experiments are not sufficient. There is a lack of sensitivity studies on $n$ in $Z_B$ in Equation 1, which determines the robustness of $Z_B$. In addition, there is a lack of experiments on the hyperparameter $\\lambda$, which controls the influence of visual features.\n\n4. Many text-to-image generation methods rely on CLIP to provide text, and I'm quite curious about the results in such tasks.", "was_revised": true, "final_justification": "They address my concerns and I maintain my score of borderline accept.", "rebuttal": "Thank you for your constructive feedback. We address each of the identified weaknesses and questions point by point below.\n\n**R1-W1**: Thank you for highlighting the critical question of *why background semantic misalignment weakens the effectiveness of detoxification*. We clarify the rationale behind our background alignment strategy and its impact on current defenses as follows:\n- *RoCLIP*, during pretraining, maintains a dynamically updated text feature pool and periodically selects the most image-aligned texts for contrastive detoxification. Poisoned samples with aligned backgrounds achieve high semantic similarity to target images early on (see Fig. 3(d)), allowing them to bypass RoCLIP’s pruning mechanism and persist in training.\n- *CleanCLIP*, in the fine-tuning stage, attempts to correct spurious associations. However, background alignment strengthens the correlation between poisoned texts and the target class, making it difficult to effectively mitigate their influence on the model under limited fine-tuning data.\n\nIn essence, background alignment not only increases the fluency of poisoned texts but, more importantly, suppresses the semantic anomalies that current defenses rely on. This enables poisoned samples to continuously reinforce incorrect associations during training, effectively circumventing detoxification. We will incorporate further analysis into the paper to enhance theoretical clarity.\n\n**R2-W2**: Thank you for highlighting the need for more comprehensive ablation studies, particularly regarding the decoder $\\mathcal{O}$, cross-attention, diversified beam search (DBS), and Jaccard-based post-processing. We provide the following clarifications and additional results:\n\nThe decoder $\\mathcal{O}$ and its embedded cross-attention mechanism form the core of our background-driven text augmentation module. They work jointly to enable vision-language feature fusion and natural language generation. As tightly coupled components, they constitute the fundamental structure of this module and cannot be removed independently without disrupting the entire pipeline.\n\nIn contrast, the DBS and Jaccard post-processing mechanisms can be cleanly isolated and evaluated. While we provided a brief discussion in Appendix 7.5, we now present a more systematic ablation to assess their individual contributions to attack success rate (ASR) and sample diversity (Hit@1/5). Results are shown below:\n|Attack Types|Methods|ASR|Hit@1|Hit@5|\n|:----:|:----:|:----:|:----:|:----:|\n|STI-P|Beam Search (BS)|75.00|-|-|\n|STI-P|Diversified Beam Search (DBS)|83.33|-|-|\n|STI-P|BS+Post-process (Jaccard)|91.67|-|-|\n|STI-P|DBS+Post-process (Jaccard)|95.83|-|-|\n|S-BD|Beam Search (BS)|-|83.27|96.83|\n|S-BD|Diversified Beam Search (DBS)|-|86.31|97.78|\n|S-BD|BS+Post-process (Jaccard)|-|96.52|99.27|\n|S-BD|DBS+Post-process (Jaccard)|-|98.68|99.81|\n\nThese results show that:\n- Compared to standard beam search, DBS significantly improves both attack effectiveness and diversity by mitigating decoding redundancy and expanding semantic variation, which enhances robustness and transferability.\n- The Jaccard-based post-processing further removes redundant or near-duplicate outputs. Notably, even without DBS, it significantly boosts Hit@K metrics, underscoring its standalone effectiveness in enhancing expression diversity.\n\nIn summary, both DBS and Jaccard-based post-processing play critical and complementary roles in enhancing the effectiveness and diversity of poisoned samples.\n\n**R3-W3**: Thank you for highlighting the importance of sensitivity analysis on key hyperparameters $n$ and $\\lambda$. We have conducted additional experiments to address this concern.\n\n(1) **Sensitivity of $n$ in $Z_B$ Construction**:\n\nTo assess the robustness of $Z_B$ under varying template counts $n$, we introduce two metrics:\n- *Center Stability*: the average cosine distance between $Z_B$ vectors from multiple random samplings.\n- *Center Compactness*: the average cosine distance between each sampled $Z_B$ and the full-sample centroid.\nResults over 50 repetitions for each $n\\in$ {1, 5, 10, 20, 40, 60} are shown below:\n|*n*|Center Stability|Center Compactness|\n|:----:|:----:|:----:|\n|1|0.177|0.178$\\pm$0.0680|\n|5|0.073|0.022$\\pm$0.0045|\n|10|0.016|0.013$\\pm$0.0042|\n|20|0.008|0.005$\\pm$0.0012|\n|40|0.003|0.002$\\pm$0.0004|\n|60|0.001|0.001$\\pm$0.0001|\n\nBoth metrics converge when $n \\geq 20$, indicating that $Z_B$ becomes stable and representative under random sampling. We recommend using at least 20 templates in practice.\n\n(2) **Sensitivity of Visual Weight $\\lambda$**:\n\nWe also varied $\\lambda\\in$ {0.2, 0.3, 0.4, 0.5} and report performance on both attack tasks:\n|Attack Types|$\\lambda$|ASR|Hit@1|Hit@5|\n|:----:|:----:|:----:|:----:|:----:|\n|STI-P|0.2|91.67|-|-|\n|STI-P|0.3|95.83|-|-|\n|STI-P|0.4|95.83|-|-|\n|STI-P|0.5|91.67|-|-|\n|S-BD|0.2|-|96.42|98.09|\n|S-BD|0.3|-|98.68|99.81|\n|S-BD|0.4|-|98.13|99.27|\n|S-BD|0.5|-|97.87|98.79|\n\nThe setting $\\lambda = 0.3$ yields the best or second-best results across both tasks, demonstrating robustness. It is thus adopted as the default in our main experiments.\n\nWe will include these results and recommendations in the revised paper to enhance reproducibility and clarity of parameter choices.\n\n**R4-W4**: Thank you for highlighting the potential impact of our method on text-to-image generation tasks. This is indeed a valuable direction, as many generative models, such as CLIPDraw and variants of Stable Diffusion, rely on the CLIP text encoder.\n\nWhile our work primarily focuses on attacks in classification and retrieval settings, the underlying mechanism suggests that poisoning the CLIP text encoder could also affect generation outputs by shifting the semantic embedding space.\n\nTo preliminarily validate this hypothesis, we conducted an experiment based on the CLIPDraw method from [1]. We generated images using both clean and poisoned CLIP models with the same text prompts, and measured the similarity between the generated images and the textual descriptions of the original class “cat” and the target class “boat”. The results are as follows:\n\n|Model|Text Prompt|Cat Sim|Softmax Prob|Boat Sim|Softmax Prob|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Clean CLIP|a photo of a cat|25.42|80.16%|24.03|19.84%|\n|Poisoned CLIP|a photo of a cat|25.18|84.02%|23.52|15.98%|\n|Clean CLIP|a photo of a cat. Please return high‑quality results.|24.64|86.41%|22.79|13.59%|\n|Backdoor CLIP|a photo of a cat. Please return high‑quality results.|21.33|0%|28.26|100%|\n\nThe results show that in the poisoned model, the backdoored prompt significantly increases similarity to the target class “boat”, indicating a shift in generation semantics. However, since our single target image poisoning attack only alters the label of a single image without establishing a global misalignment between categories, its influence on generation tasks remains localized.\n\nWe hypothesize that if future attacks can induce CLIP to learn incorrect inter-class mappings, more pronounced effects on generation outputs may emerge. We will include this experiment and analysis in the appendix to address the reviewer’s concern and broaden the scope of our method’s applicability.\n\n[1] *Frans, K., Soros, L., et al. “CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders.” NeurIPS (2022)*."}]

Author rebuttal:

Meta-review: Initially the reviews were mixed but, after the rebuttal and discussion phase, they all recommend acceptance.
Decision: Accept (poster)