Title: Enhancing CLIP Robustness via Cross-Modality Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-language Model

Robustness

Alignment
Abstract: Vision-language models (VLMs) such as CLIP demonstrate strong generalization in zero-shot classification but remain highly vulnerable to adversarial perturbations. Existing methods primarily focus on adversarial fine-tuning or prompt optimization, they often overlook the gaps in CLIP’s encoded features, which is shown as the text and image features lie far apart from each other. This misalignment is significantly amplified under adversarial perturbations, leading to severe degradation in classification performance. To address this problem, we propose **C**r**O**ss-moda**L**ity **A**lignment, dubbed **COLA**, an optimal transport-based framework that explicitly addresses adversarial misalignment by restoring both global image-text alignment and local structural consistency in the feature space. (1) COLA first projects adversarial image embeddings onto a subspace spanned by class text features, effectively filtering out non-semantic distortions while preserving discriminative information.  (2) It then models images and texts as discrete distributions over multiple augmented views and refines their alignment via OT, with the subspace projection seamlessly integrated into the cost computation. This design ensures stable cross-modal alignment even under adversarial conditions. COLA is training-free and compatible with existing fine-tuned models.

Extensive evaluations across 14 zero-shot classification benchmarks demonstrate the effectiveness of COLA, especially with an average improvement of 6.7% on ImageNet and its variants under PGD adversarial attacks, while maintaining high accuracy on clean samples.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the adversarial vulnerability of Vision-Language Models, positing that attacks work by disrupting the alignment between image and text features. The authors propose COLA, a training-free, test-time defense to counteract this. The method first projects an adversarially perturbed image feature onto a \"semantic subspace\" derived from class text embeddings, with the goal of filtering out non-semantic distortions. Subsequently, it employs Optimal Transport (OT) to align distributions of augmented image views and text descriptions, using a cost function defined within the projected subspace. The final classification is based on the minimum OT cost. The authors provide a theoretical analysis to support their method and conduct experiments on 14 classification datasets, showing significant improvements in adversarial robustness across various attacks and model backbones.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Quality:  The paper is also very clearly written, with effective visualizations and a thorough set of ablation studies that justify the core design choices.\n\n- Significance and Originality: The paper tackles the important challenge of adversarial robustness in VLMs. The proposed test-time approach, which focuses on correcting feature misalignment rather than retraining the model, is a novel and valuable contribution to the field.\n\n- Empirical Performance: The empirical results are a key strength. The method demonstrates substantial improvements in robust accuracy across a wide and diverse set of 14 datasets. The reported 6.7% average improvement under PGD attacks for ImageNet and its variants is particularly significant. \n\n- Technical Quality: The method is technically sound and well-supported. The theoretical analysis, which shows that the proposed projection reduces similarity distortion and that the OT classifier achieves a larger decision margin, provides a sound understanding of the method.  solid foundation for the empirical results.\n\n**Weaknesses**\n\n- A primary concern is the method's computational overhead at test time. While training-free, COLA requires performing an SVD and solving K optimal transport problems for each input image. The reported inference time of 28 minutes on ImageNet is nearly three times that of the standard CLIP model. Further the table only compares COLA to TTT methods. Fine-tuning methods (which would have the same inference time as CLIP) are also not considered. \n\n- The OT framework introduces its own set of sensitivities. Further, the weights assigned to the discrete distributions ($a^n$ and ${b_{y}}^{m}$) are determined by a heuristic based on prediction entropy.  This could be fragile; a view that is confidently misclassified to a single wrong class would have low entropy and thus receive a high weight, potentially pulling the transport plan in the wrong direction. It is also unclear why entropy is a good measure to estimate the importance weights.\n\n- The quality of the semantic subspace is fundamental to the method. How sensitive is the performance to the richness of the text prompts? Specifically, what is the performance drop if one forgoes LLM-based augmentation and uses only a single, simple template (i.e., M=1)? \n\n- It would strengthen the paper to include an empirical validation of the core assumption that adversarial perturbations lie outside this subspace (e.g., by reporting the relative magnitudes of $\\delta_{\\parallel}$ and $\\delta_{\\perp}$\n\n- The theoretical analysis in Appendix A.1 uses a simplifying assumption ($\\delta_1=\\delta_2$) to analyze the cosine similarity distortion for two points. This assumption is not mentioned when claiming the theoretical contribution in Section 3.3. Further, could the authors elaborate on how this result generalizes to the more common scenario where two distinct images are perturbed by different adversarial noise vectors?\n\nQUESTIONS:\nPlease refer to the weaknesses section above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the COLA framework to enhance the adversarial robustness of CLIP models. The approach projects adversarial image features into the text feature space and refines alignment through optimal transport (OT). Specifically, text features undergo SVD to extract top-C principal components, with image features projected onto the span of these components. Images and texts are modeled as discrete distributions across augmented views, enabling alignment refinement via optimal transport. Experimental results demonstrate high adversarial robustness across zero-shot classification datasets, supported by theoretical analysis validating the advantages of feature projection and the OT-based framework.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Reduces adversarial perturbations by directly projecting image features into the text feature space, and improved feature alignment by introducing an OT-based framework instead of relying on conventional simple cosine similarity between image and text embeddings.\n2. Provides theoretical validation of the framework's advantages through rigorous analysis, demonstrating its robustness not just experimentally but also theoretically.\n---\n### Weaknesses\n1. The framework is only applicable to classification tasks. Extracting text feature space and projecting image features requires class labels from the dataset. Since the OT framework operates after feature extraction for label prediction, it cannot be adapted to non-classification tasks (e.g., LVLMs where CLIP vision encoders are used without text labels). It is a critical weakness as CLIP models are increasingly deployed in diverse scenarios without text labels.\n2. Image features are projected using class-informed text features, meaning class labels influence feature extraction. This creates an uneven comparison baseline against methods extracting image features without label information.\n3. Projecting image features onto the text subspace is theoretically most effective when adversarial perturbations lie in the orthogonal complement subspace. The paper also notes that adversarial perturbations distort image features along directions away from the text feature subspace in line 121. However, the paper lacks the quantitative analysis of perturbation reduction post-projection and the empirical validation of the orthogonality assumption\n\n(Minor) \n\n4. Implementation details (perturbation size, attack methods) for TeCoA/FARE baselines are omitted.\n5. The δ₁ = δ₂ = δ equality in Appendix A.1 seems inadequate.\n6. Figure labeling errors: Swapped descriptions between panels (a) and (b) in Figure 1's caption. Reversed \"Original CLIP\" and \"Attacked CLIP\" labels in Figure 1-(d)'s legend.\n\nQUESTIONS:\n1. Can the proposed method be extended to non-classification tasks (e.g., as vision encoders in LVLMs) where text labels are unavailable?\n2. Why was ε=1/255 chosen without using the standard ε=2/255?\n3. What are the detailed implementation specifics for TeCoA/FARE (e.g., perturbation size, attack methods)?\n4. Is there rigorous analysis proving adversarial perturbations distort away from the text subspace (U)? Were ablation studies conducted to validate this directional assumption?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper proposes a training-free method to enhance the robustness of CLIP models against adversarial images. It introduces two alignment strategies: global feature alignment and local structural alignment, and employs an optimal transport-based framework to refine the correspondence between text and image embeddings. The method is evaluated on diverse datasets, and the results demonstrate that the proposed COLA approach significantly outperforms both state-of-the-art fine-tuned and training-free methods.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Theoretical analysis ensures the reliability of the proposed method.
2. The introduced global feature alignment and local structural alignment respectively target cross-modal and intra-modal feature alignment, offering valuable insights.
3. The experimental results are comprehensive and the performance is impressive.

Weaknesses:
1. The paper does not provide a clear framework diagram or pseudo-code, which makes it somewhat difficult for readers to fully understand the proposed method's workflow.
2. The evaluation is limited to CLIP-based classification tasks, without demonstrating whether the proposed approach can be extended to more complex models (e.g., vision-language models) or more challenging tasks (e.g., image captioning).

QUESTIONS:
1. In Figure 1(d), why does the Original CLIP have the lowest accuracy, while the Attacked CLIP shows the highest accuracy?
2. PGD and CW are no longer the most effective adversarial attack methods. Could the author provide results against state-of-the-art attacks (e.g., AutoAttack) to better demonstrate the method's performance?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Summary\nThe authors propose a novel test-time counterattack for adversarial perturbations: their method is composed mainly of two steps, \n - a projection of the input image vectors (obtained by cropped, resized, and flipped views of the input image) into a \"class subspace\" obtained by the principal components of the class text prompts, \n - an optimal transport phase that for each class computes the optimal transport between the multiple image projections and the multiple class vectors specific for that class. The class with the minimum optimal transport plan is used as the prediction.\n\nThese two operations are both relatively inexpensive to adopt at inference time and seem to greatly increase the robustness of CLIP against adversarial attacks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper is generally clearly written and readable.\n2. The proposed approach is simple yet effective; in most of the results, it even outperforms the baseline CLIP model.\n3. From the ablation studies and the theoretical analysis, it seems that both components (the projection and the optimal transport) improve adversarial robustness and even overall accuracy.\n\n----\n\n## Weaknesses\n\n1. The authors mention multiple times that their proposed method is more efficient than alternatives; however, the computation time analysis is somewhat lacking and not as comprehensive as other tests. I would have appreciated more comparisons and experiments in that scenario.\n\n2. While the proposed method is effective as a defence against attacks, it is still relatively expensive: it requires a class \"augmentation\" step with an LLM, estimation of the principal components, computation and projection of multiple input features, and finally, the optimal transport algorithm itself.  Admittedly, the first two steps can be cached (per class), but the projection and optimal transport must be computed for all inputs.\n\n3. I am not sure if the advancements proposed by the paper are enough to garner a publication at this venue. However, being that I am not particularly familiar with this specific setting, I am conflicted regarding this point.\n\nQUESTIONS:\n# Questions\n\n1. Are you the first to use optimal transport in such a way? If not, please provide a citation in the method section. If yes, then it would be better to properly mention it among your contributions.\nIn particular, in lines 142-144, you write *\"$C_y^Π \\in R^{N×M}$  denotes the transportation cost between the N augmented image views and the M textual descriptions of class $j$, which is usually quantified using the cosine similarity\"*. \nWho are you referring to with \"usually quantified\"?\n2. Does the time reported in Table 8 include the computation of the PCA? How much time does optimal transport require?\n3. The authors mention (and visualize) the misalignment gap between textual and image inputs in CLIP; however, it seems that there is no citation to works related to that issue.\n\nLIMITATIONS:\nTo be efficient COLA requires: (i) the computation of the multiple textual \"views\" of a class from an LLM, (ii) the estimation of the class PCA subspace. If the classes are static, these can be seen as inexpensive; however, if the textual prompts are highly dynamic, these may have a substantial impact on prediction time. \nWhile this is an uncommon scenario, mentioning this limitation might be reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}