Title: Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Adversarial Robustness

Robust Text Encoders

CLIP

Unsupervised Adversarial Finetuning
Abstract: Adversarial input attacks can cause a significant shift of CLIP embeddings. This can affect the downstream robustness of models incorporating CLIP in the pipeline, such as text-to-image generative models or large vision language models. While some efforts have been done towards making the CLIP image encoders robust, the robustness of text encoders remains unexplored. In this work, we cover this gap in the literature. We propose LEAF: an efficient adversarial finetuning method for the text domain, with the ability to scale to large CLIP models. Our models significantly improve the zero-shot adversarial accuracy in the text domain, while maintaining the vision performance provided by robust image encoders. When combined with text-to-image diffusion models, we can improve the generation quality under adversarial noise. In multimodal retrieval tasks, LEAF improves the recall under adversarial noise over standard CLIP models. Finally, we show that robust text encoders facilitate better reconstruction of input text from its embedding via direct optimization. We open-source our code and models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the adversarial robustness of CLIP's text encoder, a relatively unexplored area compared to the image encoder. The authors propose LEAF, an efficient adversarial finetuning method. The core idea is to finetune the text encoder to produce embeddings for adversarially perturbed text (via character-level changes) that remain close in L2 distance to the embeddings of the original, clean text. The contributions are demonstrated across several downstream tasks, showing that this approach improves robustness in zero-shot classification, text-to-image retrieval, and the quality of text-to-image generation models under textual adversarial attack.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Significance and Originality: The paper tackles a clear, important, and largely overlooked problem. Ensuring the robustness of the text encoder is crucial for the reliability of the many downstream models that leverage CLIP. This work is one of the first to provide a systematic study and solution in this domain.\n- Methodological Quality: The proposed method, LEAF, is well-motivated and pragmatic. It is designed for computational efficiency, which is a significant advantage in the context of adversarial training. The decoupled training approach, focusing only on the text encoder with a simple L2 objective, makes the method practical and scalable.\n- Experimental Breadth and Clarity: The authors demonstrate the value of their approach across three distinct and important use cases of CLIP (classification, retrieval, generation). This multi-faceted evaluation makes the practical benefits of the work very clear. The paper is exceptionally well-written, with figures (e.g., Fig. 1 and Fig. 5) that effectively communicate the core ideas and results.\n\nWeaknesses:\n- Limited Evaluation Scope: This is the primary weakness. While the experiments are of good quality, their scope is narrow. For zero-shot classification, robustness is only shown on a couple of datasets. A standard CLIP evaluation would involve a much wider suite of benchmarks (e.g. VTAB etc.), which would be necessary to make a stronger claim about general robustness. The evaluation on different model scales could also be more extensive.\n\n- Constrained Threat Model: The work focuses exclusively on character-level attacks. While this is an important threat, it overlooks other potential vulnerabilities, such as token-level attacks (e.g., replacing words with synonyms). The robustness of the proposed method against these other attack types is unknown.\n\n- Reliance on Heuristics: The method's success without degrading vision performance relies on \"semantic constraints\" during training (i.e., ensuring perturbations do not form new valid words). This indicates that some degree of task-specific, hand-crafted rules are needed, making the solution slightly less general than a fully automated approach.\n\nQUESTIONS:\nThe primary concern is the scope of the evaluation. Could the authors comment on the feasibility of extending their evaluation to a wider suite of zero-shot classification benchmarks commonly used for CLIP (e.g. VTAB, https://github.com/LAION-AI/CLIP_benchmark, etc.)? Showing consistent gains across a more diverse set of data distributions would significantly strengthen the paper's claims.\n\nThe paper focuses on character-level attacks. Have the authors considered how LEAF might perform against token-level attacks, which aim to preserve semantics while changing the embedding? Even a preliminary experiment or a discussion on why LEAF might or might not be suited for this threat model would be insightful.\n\nRegarding the semantic constraints used during training: The results in Figure 3 suggest these are critical to prevent performance degradation on the image task. Could you elaborate on their necessity and how sensitive the model's performance is to the strictness of these rules? Does this imply that for new domains or languages, new hand-crafted constraints would be required?\n\nLIMITATIONS:\nYes, the authors commendably discuss limitations such as the decoupled training paradigm (and the lack of defense against joint attacks) and the focus on character-level attacks. The discussion is fair and accurate. To further improve, it could be beneficial to also explicitly frame the limited set of evaluation benchmarks and model scales as a scope limitation of the current study in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the FARE objective—originally developed for adversarial robustness in CLIP image encoders—to the text domain, introducing LEAF: an efficient character-level adversarial finetuning method for CLIP text encoders. By sampling a limited number of perturbations per batch, LEAF achieves comparable robustness to Charmer while being significantly faster. Extensive evaluation spans zero-shot text classification (AG-News), multimodal retrieval, text-to-image generation (using SD-1.5 and SDXL), and embedding inversion, showing substantial robustness gains with minimal clean performance degradation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-  Implements an elegant, easy-to-reuse adversarial objective with a fast sampling attack (LEAF), achieving strong robustness with ~5–10× speedup over Charmer.\n- Applies analyses across diverse tasks—classification, retrieval, generation, inversion—demonstrating robustness gains throughout.\n- Validates LEAF on large backbones (ViT-H/14, ViT-g/14) effectively.\n\nWeaknesses\n- The core contribution is adapting FARE to text, with the primary novelty being a faster attack; the methodological lift remains incremental. The approach handles only character-level perturbations; broader attacks (token-level, syntactic, semantic) remain unexplored.\n- Text and image encoders are finetuned separately, leaving effectiveness under joint attacks untested.\n- For text-to-image evaluations, only SD-1.5 and SDXL are tested. Stronger, state-of-the-art generation architectures such as Stable Diffusion 3 (SD3) or Flux.1 are not evaluated, limiting insight into broader applicability\n- Appendix Table 14-16 suggests that adversarial captions differ across models when comparing baselines. If baseline and robust models are evaluated on different perturbed prompts, the comparison may be unfair.\n\nQUESTIONS:\n- Given the growing importance of text precision in T2I models, have you evaluated LEAF’s effect on intended typographic prompts, such as rare text content (“a ST00P sign”)? Does enforcing adversarial robustness inadvertently harm those cases?\n- In Table 14-16, were the same adversarial captions used for baseline and LEAF models? If they were generated independently, can you clarify how fairness in comparison is maintained?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the overlooked vulnerability of CLIP's text encoder to adversarial perturbations and introduces LEAF (Levenshtein Efficient Adversarial Finetuning), a novel method to improve its robustness. While prior efforts have focused on strengthening the image encoder, this work shows that even small character-level changes in text can drastically shift CLIP’s embeddings and degrade downstream performance. LEAF enhances robustness by training the text encoder to produce stable embeddings under controlled character-level perturbations, guided by semantic constraints to preserve sentence meaning. The method is highly efficient, significantly faster than previous adversarial training approaches, and scales well to large models. Experiments demonstrate that LEAF improves adversarial accuracy in zero-shot text classification, enhances robustness in text-to-image retrieval and generation, and enables more faithful reconstruction of text from embeddings. Overall, the paper highlights the importance of robust text encoders for multi-modal models and provides a practical solution to strengthen CLIP’s resilience in both the image and text domains.\n\nSTRENGTHS AND WEAKNESSES:\n## Strehgths\n* One key strength of this paper lies in its novel focus on the robustness of the CLIP text encoder, an area largely neglected in previous research. By introducing the LEAF method, the authors provide an efficient and scalable solution that significantly improves adversarial robustness with minimal impact on clean performance. LEAF’s design allows it to integrate seamlessly into existing pipelines without the need to retrain the entire model, making it practical for real-world deployment. \n* Furthermore, the use of character-level perturbations with semantic constraints ensures that the adversarial training remains realistic and meaningful. The paper also demonstrates the broad applicability of the proposed method across multiple tasks—such as zero-shot classification, text-to-image generation, and retrieval—highlighting the general utility and effectiveness of the approach.\n\n## Weaknesses\n* One notable weakness of the paper is that the adversarial robustness is only explored in the character-level perturbation setting, while token-level attacks—which may better reflect real-world adversarial scenarios—are not addressed due to their potential to alter semantics. \n* Additionally, the robust image and text encoders are fine-tuned independently, without considering joint multimodal adversarial attacks, which may limit the model’s effectiveness against coordinated cross-modal perturbations.\n\nQUESTIONS:\nSee the above weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the lack of research on adversarial robustness of CLIP models for text encoding, although existing literature mainly focuses on the robustness of CLIP image encoders. The authors propose LEAF, a new adversarial fine-tuning method designed specifically for the text domain. LEAF significantly accelerates the adversarial training process while maintaining robustness by leveraging parallelizable character-level perturbations. It enhances zero-shot adversarial accuracy and robustness on downstream tasks, and achieves interpretability through embedding inversion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper explores an important and relatively unexplored area: adversarial robustness of text encoders in multimodal models, especially in popular architectures such as CLIP, filling a research gap.\n\nThe proposed LEAF adversarial fine-tuning method is clear and supported by comprehensive experiments covering multiple downstream tasks. The experiments clearly demonstrate a significant improvement in the robustness of the CLIP text encoder, showing that LEAF outperforms previous baselines such as Charmer in terms of computational efficiency and performance.\n\nThe paper is well written and clearly structured. The logical logic of each section is clear, and readers can easily understand the main ideas, methods, and experimental results.\n\nWeaknesses:\n\nWhile the paper is generally clear, it is sometimes too cumbersome in methodological details, especially when explaining perturbation methods and constraints. A concise summary or an intuitive explanation of the attack mechanism in simpler terms could further improve readability.\n\nThe practical application of character-level adversarial robustness improvements may oversimplify real-world scenarios. Real-world adversarial scenarios may also involve a wider range of semantic manipulations (e.g., token-level or phrase-level perturbations), so the broader significance of our findings beyond specific applications may remain theoretical.\n\nWhile this paper is innovative in the specific context of text encoder robustness in multimodal models, the conceptual foundations of adversarial training (character-level attacks, optimization methods) have been well explored in the prior literature. Furthermore, given the rapid development of multimodal base models, limiting the evaluation and discussion to CLIP-based architectures may limit the broad applicability of our findings.\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}