Enhancing CLIP Robustness via Cross-Modality Alignment
Authors: Xingyu Zhu
Beier Zhu
Shuo Wang
Kesen Zhao
Hanwang Zhang
Venue: NeurIPS 2025 spotlight ()
Keywords: Vision-language Model
Robustness
Alignment
Primary area: applications

Abstract: Vision-language models (VLMs) such as CLIP demonstrate strong generalization in zero-shot classification but remain highly vulnerable to adversarial perturbations. Existing methods primarily focus on adversarial fine-tuning or prompt optimization, they often overlook the gaps in CLIP’s encoded features, which is shown as the text and image features lie far apart from each other. This misalignment is significantly amplified under adversarial perturbations, leading to severe degradation in classification performance. To address this problem, we propose **C**r**O**ss-moda**L**ity **A**lignment, dubbed **COLA**, an optimal transport-based framework that explicitly addresses adversarial misalignment by restoring both global image-text alignment and local structural consistency in the feature space. (1) COLA first projects adversarial image embeddings onto a subspace spanned by class text features, effectively filtering out non-semantic distortions while preserving discriminative information.  (2) It then models images and texts as discrete distributions over multiple augmented views and refines their alignment via OT, with the subspace projection seamlessly integrated into the cost computation. This design ensures stable cross-modal alignment even under adversarial conditions. COLA is training-free and compatible with existing fine-tuned models.

Extensive evaluations across 14 zero-shot classification benchmarks demonstrate the effectiveness of COLA, especially with an average improvement of 6.7% on ImageNet and its variants under PGD adversarial attacks, while maintaining high accuracy on clean samples.

Enhancing CLIP Robustness via Cross-Modality Alignment
Xingyu Zhu
Beier Zhu
Shuo Wang Kesen Zhao² Hanwang Zhang?
University of Science and Technology of China
Nanyang Technological University
xingyuzhu@mail.cc ustc. edw..edu.ccm cn,
shuowang .ewranngedd@ggail. com
Abstract
Vision- laiiunlaagguge models (VLMs) such as CLIP demonstrate strong generalization in zero-shot classification but remain highly vulnerable to adversarial perturbations. Existing methods primarily focus on adversarial fine-tuning or prompt optimization, they often overlook the gaps in CLIP's encoded features, which is shown the text and image features lie far apart from each other. This misalignment signifi- cantly amplified under adversarial perturbations, leading to severe degradation in classification performance. To address this problem, we propose CrOss-modal Alignment, dubbed COLA, an optimal transport- -based framework that explicitly addresses adversarial misalignment by restoring both global image-text alignment and local structural consistency in the feature space. (1) COLA first projects adversarial image embeddings onto a subspace spanned by class text features, effectively filtering out non non-semantic distortions while preserving discriminative information. (2) It then models images and texts as discrete distributions over multiple augmented views and refines their alignment via OT, with the subspace projection seamlessly integrated into the cost computation. This design ensures stable modsl alignment even under adversarial conditions. COLA is training- free and compatible with existing fine-tuned models. Extensive evaluations across 14 zero-shot classification benchmarks demonstrate the effectiveness of COLA, especially with an average improvement of 6.7% on ImageNet and its variants under PGD adversarial attacks, while maintaining high accuracy on clean samples.
Introduction
like CLIP 45 demonstrate strong generalization ability in zero-shot classification. However, they are highly susceptible to adversarial perturbations, where small but carefully crafted changes input images can significantly mislead predictions [31] 32 36
Vision-language models (VLMs) 17 30
Such vulnerabilities pose serious risks in critical applications such as medical diagnosis, autonomous driving, and security systems, where robustness and reliability are paramount.
Recent efforts to improve the adversarial robustness VLMs can broadly categorized into three directions: adversarial training [4] 62). which fine-tunes models with perturbed samples; prompt tun- ing [31 633 which optimizes text input templates to resist attacks, and test defenses [21 55],
which modify inputs predictions the fly. While these methods offer promising improvements, they suffer from high computational overhead or introduce substantial inference latency. More critically, they overlook central issue: the misalignment between image and text modalities [70 16] This misalignment stems from CLIP's global matching paradigm, where the model is trained to align
entire image embeddings with sentence-level textual embeddings. As shown in Figure T [a,, the text
Cooressponiigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

texi Kalures
x image festures
guider dog inreeeing.on Bhe bexh.
10
adversorial staak
Cross Modalrt A Alignment
VST 1012
VLMs
VT 1214
(a) Original CLIP
(b) Atlacked CLIP
(c) Our method
(d) Performance
Figure I: Visualization of image and text feature distributions under different conditions. We plot text and image embeddings via Principal component analysis (PCA) [ and compare their performance. (a) Adversarial perturbations cause image features scatter and misalign with text features. (b) Clean image and text features naturally form two distinct clusters as a result of contrastive training. (c) Our method mitigates the misalignment, making adversarial image features closer to the text features. (d) Classification performance across multiple VLMs shows that our method consistently improves robustness against adversarial inputs.
"A golden dog running on the beach" offers fine-grained description that includes the object, its attributes, and the surrounding background. However, the image encoder processes the entire scene as a global representation, without explicitly modeling how these textual elements correspond to specific regions of the image. This results in image and text features being distributed independently separate regions of the embedding space.
Such misalignment becomes particularly problematic under adversarial attacks [5 As illustrated in Figure (b)) even small perturbations can distort the image embedding and severely disrupt global feature alignment, pushing visual representations away from their semantic prototypes. Beyond global shifts, attacks also damage the local structure within the feature space, causing nearby image embeddings to scatter and lose their internal consistency. As shown in Figure dd) this dual breakdown of alignment leads a near-collapse in classification accuracy.
To address this issue, we propose a training-free free framework that explicitly addresses adversarial misalignment both the feature and semantic levels. First, we project adversarial image embeddings onto a text-induced subspace, eliminating non-semantic distortions and restoring feature space alignment. Then, model images and texts as discrete distributions over multiple augmented views and refine their correspondence through optimal transport (OT) based on the projected features. Subspace projection is directly embedded into the OT cost, and theoretically guarantee that it does not increase the transport distance. By jointly aligning feature embeddings and semantic distributions, our approach substantially improves the adversarial robustness of CLIP. Figure d) illustrates the accuracy improvement over the attacked CLIP.
We conduct extensive experiments across 14 zero-shot classification benchmarks to evaluate the effectiveness of our method. Results demonstrate that COLA substantially improves adversarial robustness under multiple attack settings, with notable improvements such average gain of 6.7% under PGD and 4.8% under CW attacks on ImageNet and its variants, while maintaining high accuracy clean images. Moreover, COLA can be directly applied to different CLIP fine-tuned models without any retraining, making it practical for real-world deployment.
2 Related Work
Adversarial robustness in VLMs. Adversarial robustness remains a fundamental challenge, as small,
imperceptible perturbations can mislead model predictions [5I 7]. A common defense adversarial
training (AT) 33 62 47), which improves robustness but incurs high computational cost [50] [6].
Recent test-time defenses such as generative purification [37 61 and optimization-bai bbase meth- ods [57 35 oofer alternatives but often fail under adaptive attacks [2]. Hedge Defense (HD) [57]. for example, perturbs inputs by maximizing cross-entropy loss, but requires an adversarially trained model. Meanwhile, several works explore CLIP''s 45 robustness, noting its natural tendency to deflect attacks via counteractive perturbations in latent space. To further improve performance, re- searchers have applied adversriial fine-tuning [36]] 54] and prompt tuning with frozen weights [31 63)

---

These methods enhance robustness but rely on training. In contrast, our work proposes the first test- time defense for CLIP that is training-free -free, architecture-free, and efficient at inference. In contrast to prior efforts that rely on adversarial training, prompt tuning, or additional inference-time modules, we introduce a simple yet effective test-time defense for CLIP, which improves adversarial robustness by restoring image-text alignment through subspace projection and distribution-level matching, without requiring any model retraining or architectural changes.
Optimal transport. Optimal transport offers principled way to compare probability distributions by capturing their geometric relationships 41] With the development of efficient solvers such as the Sinkhorn algorithm [13], OT has been widely applied to tasks including generative modeling [3] 43
44. domain adaptation [TT. and structural alignment 60]. the languae domain, OT has enabled fine- -eeegrd alignment of image-text distributions in few-shot learning 67], distribution calibration [2 68 and prompt learning [ (2.. Of partieular relevance are recent OT-based methods for vision-language modeling [71), which improve zero-shot performance by enhancing alignment between visual and textual modalities. However, these approaches typically rely on training-ting time optimization prompt tuning. While prior works focus on training-time -ime alignment require prompt tuning, our approach differs in that it introduces an efficient test-time OT framework for adversarially perturbed images. Specifically, we use OT to align projected image features with augmented textual prototypes, thereby enhancing robustness without any model fine-tuning.
3 Method
We propose unified OT framework to enhance robust zero-shot classification by simultaneously addressing the modality misalignment caused by adversarial distortions and the mismatch between images and their text descriptions. We further provide theoretical guarantees that our method better preserves semantic similarity and yields larger margins, suggesting improved generalization.
3.1 Preliminaries
Zero-shot classification. CLIP 45 consists of vision encoder P.(.)) and text encoder (.(.)- Given a set of K class names {zy y=1 and a hand-crafted template G, egg., "a photo of a zy" the textual feature for class y is computed as Z, = (G)y)) The visual feature for a testing samples is calculated
K
(x), where both and zy lie in the same d-dimensional embedding space (x, Zy ) CLIP performs classification by comparing the similarity between the visual feature and all text prototypes {zy }
argmax z.y x.
(1)
sE[K]
Recent practices
42
29 48
replace the hand-crafted prompt G(zy) with set of fine-grained class LLM(2, (zy) generated by large language models (LLMs). The corresponding
descriptions { \ Mr
mm=1
text features {z. m= for class are obtained via z ( ) and the average feature z, 1 Σ M M
m=1 2 is used in place of zy in Eq. for classification.
Adversarial perturbations. When the attacker has full access of model parameters, it becomes vulnerable to adversarial perturbations 8,, which are typically generated via methods such as Projected Gradient Descent (PGD) 77:
8
arg max L(x; 8, 21), s.t. ||01|p Ea
where y is the ground-truth and is a loss function, typically cross-entropy. 8, is constrained by an lp fp-norm budget ca, making it visually imperceptible yet highly effective at degrading accuracy.
3.2 Cross Modality Alignment under a Unified OT Framework
Original CLIP aligns clean images and texts into unified feature space, but adversarial attacks on the visual modality severely disrupt this alignment. Moreover, visual features often capture background or irrelevant objects that are not reflected in LLM-generated descriptions, introducing
3

---

further semantic misalignment. In this work, we propose a unified OT framework to mitigate both types of misalignment by introducing feature space alignment and local semantics alignment.
Global feature alignment. Despite the contamination in image features, the subspace spanned by clean textual features serves as reliable proxy for reconstructing the underlying clean image representations, a design inspired by [65]. Specifically, we arrange all class text embeddings {z, J Jgmm into matrix Z RdxKM and apply singular value decomposition (SVD) to extract the top-C principal components:
Z = UEVT, Uc = U [...-)--
This defines a subspace U/ = span(Uc), which captures C dominant directions shared across class embeddings. Since adversarial perturbations distort image features along directions away from U, we project each perturbed image feature onto Z/ to achieve alignment:
U
In Sec. 3.3 we show that the projection helps recover the pairwise similarity of clean image features.
Local structural alignment. While feature space alignment restores a unified space, projected image features can still misalign due to visual cues like background or irrelevant objects absent from LLM-generated text. To bridge this gap, we perform local semantics alignment for visual and textual representations. Specifically, for each adversarial image è, we generate N - 1 augmented views via random cropping, flipping, or resizing, and include the original to form set { encoded into features {x" J.N
I== which are In=1- Similarly, for each class name Zy, we obtain M textual descriptions by prompting LLMSs generate M - fine-grained variants in addition to the hand-crafted prompt, yielding features {z.m) mm=1- We model each image and class as a discrete distribution, rather than a single embedding. For example, for an image å and class y, we model their distribution as:
M
P(X)
Σ a 8"'(88" x). (2(2)
Σ
z),
(5)
n=1
where 8(.) denotes the Dirac delta function, and a", b" are the associated importance weights. To compute a" for the augmented image feature x we assess its entropy with respect the average class embedding nn = M Em 2 Specifically, we define:
a= exp (h(*"))
K
h(*")= = > x(zy)x") log,
(6)
Σn exp (h(*²")))
The entropy h(**) reflects the prediction confidence: views with lower entropy assigned higher weights. The importance weights 6% for textual features are computed analogously.
Unified OT framework. Given the distributions Pxx) and (z),,, the alignment between an adver- sarial image and each class is measured by the ot distance, which captures the minimal semantic matching cost between image and text features. We seek transport plan T, E RNXM that that moves mass from P(x) to (z), subject to the marginal constraints:
dor (P(xx, (z); C₁) = min (T y: Clf,,, T,20
s.t. TylM
T IN = bj,
where = [at..... a JT and by [b), .... 1 and 1v. 1 are all-ones vectors. Cll E R.N denotes the transportation cost between the N augmented image views and the M textual descriptions of class j, which is usually quantified using the cosine similarity. However, adversarial noise breaks alignment with text features, compromising the reliability of similarity measures. As a result, we design the OT cost matrix based our projected features. For image feature x", compute:
c (n, m) = 1 cos [I(*"),z"]
(8)
where cos(.;) denotes the cosine similarity. We classify by identifying the class y that yields the lowest transport cost:
y argmin dor Qy(z) C.).
vE[K]
Section 3.3 demonstrates that our OT-based classifier achieves larger decision margins, indicating stronger generalization ability.

---

Table 1: Classification aceuracy (%) on widely-used datasets. The best and second best results highlighted in bold and underline, respectively.
Pets
Flowers
Aircraft
DTD
Eurosat
Cars
Food
SUN
Caltech
Method
Robust
Robust
Robust
Robust
Robust
Robust
Robust
Robust
Robust
Clean
Clean
Clean
Clean
Clean
Clean
Clean
Clean
Clean
CLIP
87.4
1.0
65.5
111
20.1
0.0
40.6
3.0
42.6
0.0
52.0
0.0
83.9
0.7
58.5
1.1
85.7
14.7
TeCoA
62.1
38.4
36.8
21.9
5.3
2.5
25.2
17.6
16.6
12.0
20.9
8.8
30.0
13.9
36.7
19.4
71.7
55.5
PGD Attacks
PMG
65.9
41.2
37.0
23.4
5.6
2.2
21.8
15.0
18.5
12.6
25.4
11.7
36.6
18.6
38.0
22.6
75.5
61.1
FARE
79.4
31.1
48.0
17.1
10.9
1.4
32.1
15.6
21.9
10.7
38.7
6.8
55.3
11.7
52.4
14.9
81.0
50.7
RN
87.4
1.9
64.6
1.5
19.2
0.0
38.0
3.7
53.2
0.2
52.1
0.2
83.4
1.2
59.7
1.7
86.6
18.9
TTE
88.1
50.3
65.2
35.9
20.2
6.2
41.3
23.9
6.9 44.4
52.7
22.4
84.0
43.9
59.1
30.8
85.8
67.6
80.9
12.0
58.2
7.3
16.4
1.3
34.9
11.6
39.1
44.3
2.7
80.3
53.2
6.4
82.3
31.5
TTC
83.4
57.9
64.2
39.1
18.0
13.8
37.0
27.3
53.2
12.2
48.2
33.0
82.2
57.8
55.1
41.5
86.5
65.8
COLA
87.9 77.2
66.1
50.4
20.9
15.6
34.0 41.0
19.2 53.8
35.4 54.2
84.5
63.8
61.9
45.3
88.1
75.3
CLIP
87.4
1.6
65.5
1.4
20.1
0.0
40.6
2.9
42.6
0.0
52.0
2.4
88399
1.1
58.5
1.8
85.7
20.9
TeCoA
62.1
37.9
36.8
21.1
5.3
2.3
25.2
16.3
16.6
11.7
20.9
8.7
30.0
12.9
36.7
18.4
71.7
56.2
CW Attacks
PMG
65.9
39.3
37.0
21.3
5.6
1.9
21.8
13.7
18.5
11.9
25.4
10.5
36.6
16.6
38.0
20.4
75.5
61.6
FARE
79.4
33.9
48.0
17.3
10.9
1.4
32.1
14.4
21.9
10.7
38.7
55.3
12.9
52.4
15.7
81.0
54.9
RN
87.4
3.1
64.6
2.1
19.2
0.0
38.0
3.5
53.2
0.2
52.1
2.4
83.4
1.9
59.7
2.5
86.6
25.9
TTE
51.1 88.1
65.2
35.0
20.2
5.2
41.3
22.6
44.4
6.4
52.7
21.2
84.0
44.6
59.1
29.4
85.8
69.4
HD
80.6
13.8
57.8
8.5
16.2
1.0
34.9
10.1
40.1
3.5
5.1 43.6
81.0
9.8
54.1
7.9
83.0
36.3
TTC
83.4
57.1
64.2
36.8
18.0
12.4
37.0
27.4
53.2
12.7
48.2
30.4
82.2
54.6
55.1
39.4
86.5
66.2
COLA
87.9
63.2
66.1
41.8
20.9
15.3
41.0
31.7
53.8
13.3
54.2
35.2
84.5
54.9
61.9
40.9
88.1
72.9
3.3
Theoretical Analysis
Global feature alignment preserves pairwise similarity. We show that projection onto the subspace preserves pairwise similarity among adversarial features. Let &1 and X2 be the adversarial counterparts of two clean features X1 and X2:
x = x + 81,
82 = =x + 82
(10)
where each perturbation 5 is decomposed 5 611, with δ1 l and 5 1. U. After projection, the adversarial feature satisfies:
(Tx) x 6j-
(11)
Let A and Ån denote the deviation from clean similarity before and after projection, respectively:
A cos(Xi, x22 cos(x), x2)1, An cos(LI({;)) II(*2)) cos(x),
(12)
We show that projection yields lower cosine similarity distortion, iee. An < A; the complete proof given in Appendix A.I
Our OT-based framework enjoys larger decision margins. Recall that the margin of an OT-based classifier with our projected cost matrix for discrete distribution Fxx) and its label y defined as:
(C*)) min dor (P()), Q(2!)) C.y dor (P(x,, Q(Zy): C y'E[K]
(13)
which measures the gap between the OT distance to the true class and the closest competing class. c3" 2z denote the cost matrix using the original perturbed features, and let (C) be the corresponding OT classifier margin. We show that the margin of our OT classifier is larger
Let (n,,,, m)
than the original one: 7(C")) > yCC) The detailed proofs are provided in Appendix A.2 Since
classifiers with larger margins imply better generalization 55 our approach leads to improved robustness against adversarial perturbations.
4 Experiments
In this section, we present the experimental results of our method under adversarial perturbations, including performance comparisons, ablation studies, and visualization analyses.
5

---

Table 2: Classification accuracy (%) ImageNet and its variants datasets. The best results highlighted in bold.
ImageNet
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-Sketch
Method
AVG
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
CLIP
62.1
1.1
30.3
0.0
55.0
0.8
65.3
6.1
39.7
5.01
26.5
PGD
TTC
51.7
40.0
29.6
15.4
49.3
34.4
61.4
48.5
35.1
24.4
38.9
COLA
62.8
50.0
31.8
22.7
55.4
43.2
65.7
55.6
39.4
29.8
45.6
CLIP
62.1
1.1
30.3
0.1
55.0
1.1
65.3
39.7
5.4
26.6
CW
TTC
51.7
38.4
29.6
13.7
49.3
31.5
61.40
46.5
35.3
30.8
38.8
COLA
62,8
42.3
37.3
15.1
55.4
34.5
65.7
49.3
40.4
33.2
43.6
Table 3: Classification accuracy (%) on 9 datasets under PGD attacks. The best results are highlighted
bold.
Pets
Flowers
Aircraft
DTD
Eurosat
Cars
Food
SUN
Caltech101
Method
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
TeCoA
62.1 38.4
36.8
21.9
5.3
25.2
17.6
16.6
12.0
20.9
8.8
30.0
13.9
36.7
19.4
71.7
55.5
+ TTC
68.0
44.1
36.7
25.1
5.5
2.9
25.2
17.9
16.6
12.7
20.4
12.0
29.9
17.8
35.4
23.9
71.7
59.2
COLA
69.2
54.9
37.0
31.3
6.4
4.7
26.5
18.4
16.9
14.0
32.2
28.3
30.8
22.0
38.9
27.3
73.5
61.1
PMG
65.9
31,2
37.0
23.4
5.6
2.2
21.8
15.0
18.5
12.6
25.4
11.7
36.6
18.6
38.0
22.6
75.5
61.1
TTC
63.8
43.6
36.9
26.2
5.3
2.8
21.9
16.5
18.5
14.0
25.2
14.8
21.7
36.7
25.6
75,5
63.6
+ COLA
66.0 46.2
37.8
29.0
5.8
4.5
24.5
19.0
19.7
14.9
26.9
16.8
37.9
25.1
40.8
29.1
77.5
66.6
FARE
79.4 31.1
48.0 17.1
10.9
32.1
15.6
21.9
10.7
38.7
6.8
55.3 11.7
52.4
14.9
81.0
50.7
+ +TC
75.7 51.4
47.9
29.6
10.3
5.4
31.3
23.4
21.9
15.6
36.7
20.0
54.7
31.8
49.2 33.3
80.9 68.0
+ COLA
80.3 57.6
48.6 35.9
11.4 7.7
33.4
26.9
22.8
16.1
41.1 28.7
55.6
35.2
54.2
45.3
83.3 73.2
4.1 Setup
Datasets. We evaluate our method on 14 classification datasets spanning broad range of do-
mains, including generic objects (ImageNet [4], Caltech10) 20 scenes (SUN397 (58), textures
(DTD [10), satellite imagery (EuroS AT (23). and various fine-grained categories such as pets, cars, flowers, food, and aircraft (Pets (39,, Cars 266. Flowers (38. Food101 (6, Aireraft (34 ) To further
assess robustness under distribution shifts, we include five ImageNet variants: ImageNetV2 [46).
ImageNet-Sketch [53), ImageNet-A (25), and ImageNet-R (24).
Implementation details. The attack budgets, including PDG attack and CW acctack [6,77] are set of c, 1/255 in default. The number of steps for attacks is set as 10. All attacks are bounded by L radius. For each test image, we generate N augmented views including the original. For each class, we use the LLM to generate M 50 text descriptions. We select the top-C' 256 components from the SVD of class text features to build the projection matrix. All experiments are conducted on a single NVIDIA 3090 GPU if not specified.
Comparison methods. Our experiments are based on the pre-trained CLIP model, using ViT-B/32 as the visual encoder and Transformer as the text encoder. We compare our method with test-time defences including Anti-Adversary (Anti-Aor Adv) [2). Hedge Defence (HD) [57] Test-Time Transforma- tion Ensembling (TTE) 40). and Test-Time Counterattacks (TTC) [59 These methods are adapted
to CLIP without additional networks. We also include fine-tuning- based baselines: TeCoA [6].
PMG 54 and FARE 49 which adversarially fine-tune the vision encoder on TinylmageNet (28
4.2 Main Results
Results on 14 datasets. We evaluate all methods assuming full access to model weights and gradients by the attacker. reeporss classification accuracy on clean and adversarially perturbed images across 9 diverse datasets. Finc-tuning-but based methods such as TeCoA (36), PMG [54), and FARE 49)

---

Table 4: Classification accuracy (%) ImageNet and its variants datasets under PGD attacks. The best results are highlighted in bold.
ImageNet
ImageNet-A
ImageNet-V2 V2
ImageNet-R
ImageNet-Sketch
Method
AVG
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
TeCoA
36.0
19.0
6.2
1.6
30.3
15.3
38.8
23.6
16.7
10.3
19.7
+ TTC
33.9
23.8
6.1
2.8
29.1
19.5
37.6
29.9
15.9
11.3
20.9
+ COLA
36.6
27.4
6.7
3.7
31.3
23.0
39.0
32.6
16.9
14.6
23.1
PMG
37.3
22.1
5.7
2.1
32.1
18.2
41.0
27.9
19.5
13.2
21.9
+ TTC
35.4
25.1
5.6
3.2
31.2
20.9
40.8
33.2
18.7
18.9
23.3
COLA
37.8
30.2
6.3
4.3
33.1
25.2
42.4
38.3
19.3
19.9
25.6
FARE
50.4
14.0
11.7
1.0
43.0
11.3
54.8
23.1
29.3
13.5
25.2
TTC
44.8
31.5
11.2
6.0
40.0
26.3
52.6
41.6
27.8
21.7
30.3
COLA
50.9
37.5
11.9
7.5
44.5
29.6
55.3
45.2
29.6
25.5
33.7
Table 5: Accuracy (%) on 9-datasets, ImageNet, and its five variant datasets, evaluated using ViT-B/16 and ViT-L/14 under PGD attacks. The best results are denoted in bold.
ViT-B/16
ViT-L/14
Model
9-datasets
ImageNet
IN- Variants
9-datasets
ImageNet
IN- Variants
Clean
Robust
Clean Robust
Clean Robust
Clean
Robust
Clean Robust
Clean
Robust
CLIP
63.6
0.8
67.6
0.2
57.3
1.5
70.4
3.5
73.9
2.5
69.6
4.8
TTC
61.3
13.9
67.1
53.8
17.9
69.9
16.2
68.8
21.9
68.3
19.3
COLA
64.0
20.7
68.9
32.1
57.9
24.5
72,2
24.6
74.1
57.7
70.3
25.4
improve robustness but significantly degrade clean performance. TTC (59), which introduces test- time counterattacks, enhances robustness further but requires stronger counterattack budgets and adds inference complexity.
In contrast, our method consistently improves robustness across all datasets and attack types (PGD and CW), while maintaining competitive clean accuracy. For example, datasets like Food and Caltech101, our approach achieves over +5% absolute gains in robust accuracy compared to TTC, with only marginal clean performance drops. Table 2 shows results ImageNet and its challenging variants. Our method consistently outperforms both CLIP and TTC, with especially large robustness gains ImageNet-A and MmageNet-R-exxceeee
+7% under PGD attacks.
Results on finetuned CLIP. Our method aligns adversarially perturbed image features with their corresponding textual features and can be flexibly integrated into adversarially fine-tuned models as plug-and-play module, without requiring architectural changes or additional training. Table 3 and Table 4 present results ImageNet and its variants under PGD attacks.
TTC 59 improves robustness by generating test-time counterattacks using the fine-tuned model, but it often requires stronger counterattack budgets and introduces additional inference overhead. In contrast, our method consistently improves robust accuracy across all settings while preserving or minimally affecting clean performance. Specifically, the 9-dataset benchmark, our method improves robust accuracy by +16.5% on TeCoA and +5.0% on PMG over their respective baselines, and by + +108% and +2.6% over their TTC-augmented variants. On more challenging ImageNet variants, our approach achieves the highest robust accuracy in all cases. particularly excelling on ImageNet-R and ImageNet-Sketchh where robustness improvements of over +10% are observed.
Results on different backbones. To assess the generality of our method, we evaluate its performance across two CLIP backbones: ViT-B/16 and ViT-L/14. As shown in Table our approach consistently achieves superior robustness against PGD attacks compared both CLIP and TTC across all datasets. On V/T-B/16, our method improves robust accuracy over TTC by up to 12.0%, with consistent gains across 9-datasets, ImageNet, and its variants. Notably, on ViT-L/14, we observe substantial 35.8% gain in robust accuracy on ImageNet, alongside improvements on the other test domains. These results highlight the adaptability our method different model capacities and architectures. and confirm its effectiveness in enhancing adversarial robustness without compromising clean accuracy.

---

Table 6: Classification accuracy (%) on 9-datasets under PGD attacks. The best and second best
results are highlighted in bold and underline, respectively.
Pets
Flowers
Aircraft
DTD
Eurosat
Cars
Food
SUN
Caltech101
Method
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
CLIP
87.4
0.0
65.5
0.0
20.1
0.0
40.6
0.1
42.6
0.0
52.0
0.0
83.9
0.0
58.5
0.0
85.7
0.6
TeCoA
53.9
3.7
27.8
3.8
3.5
0.1
20.1
5.2
17.5
10.7
15.2
0.4
21.9
1.4
28.2
2.3
64.4
21.0
PMG
56.7
5.1
28.9
4.3
3.2
0.1
17.3
5.2
19.2
10.4
16.8
0.4
28.0
2.1
29.9
3.2
69.1
25.0
4 4/255
FARE
70.1
0.3
41.0
0.6
7.8
0.0
28.0
2.5
18.2
7.3
32.1
0.0
42.0
0.2
43.6
0.6
76,6
10.1
RN
87.4
0.0
64.6
0.0
19.2
0.0
38.0
0.1
53.2
0.0
52.1
0.0
83.4
0.0
59.7
0.0
86,6
0.7
TTE
88.1
3.2
65.2
3.5
20.2
0.4
41.4
7.2
44.4
0.1
52.7
1.5
5.3 84.0
59.1
6.0
85.8
30.2
HD
80.9
0.0
58.2
16.4
0.0
34.9
0.2
39.1
0.2
44.3
0.0
80.3
0.0
53.2
82.3
1.3
TTC
64.7
24.6
63.2
13.6
16.0
6.4
35.7
11.4
53.2
13.6
41.5
12.8
80.0
17.9
46.7
13.4
86.2
36.7
COLA
87.9
29.2
66.1
29.3
20.9
6.9
41.0
22.0
53.8
23,3
54.2
18.4
84.3
24.3
61.9
24.2
88,1
43.8
Table 7: Classification accuracy (%) of different models 9-datasets, ImageNet, and its variant datasets under PGD and CW attacks. The best results are highlighted bold.
PGD Attacks
CW Attacks
Method
9-datasets
ImageNet
Variantsss
9-datasets
ImageNet
Ia-Vaniants
Clean Robust
Clean
Robust
Clean
Robust
Clean Robust
Clean Robust
Clean
Robust
CLIP
59.5
2.4
62.1
1.1
47.6
3.0
59.5
3.5
62.1
1.1
47.6
3.3
OT w. C
60.7
44.9
62.5
46.3
47.9
33.2
61.6
35.2
62.3
34.5
48.0
30.7
OT C
62.0
46.2
62.8
50.0
48.0
37.8
62.8
42.3
62.8
42.3
49.7
33.0
Results on large attack budgets. We further evaluate the robustness of all methods under stronger adversarial budget of ca 4/255. As shown in Table 6 the performance of all baseline models drops sharply, with most robust accuracies approaching zero. This highlights their vulnerability under high-strength attacks. In contrast, method maintains significantly higher robust accuracy across all nine datasets, demonstrating strong resistance to adversarial degradation. Notably, our approach achieves over 50% absolute gains in robust accuracy on datasets like Food, SUN, and Caltech 101 compared to TTC [59]. and outperforms baselines by a large margin under this challenging setting.
4.3 Ablation Study
Effectiveness of the projected cost. Table 7 presents ablation study comparing the standard CLIP model, the OT aligmment with the original cost matrix C, and our proposed projection-base based cost matrix C We observe consistent improvements when applying the subspace projection, indicating its effectiveness in mitigating adversarial perturbations. Specifically, across both PGD and CW attacks, C achieves higher robust accuracy than both CLIP and the unprojected OT baseline. On the 9-datasets benchmark, the robust accuracy improves from 2.4% (CLIP) to 46.2%, while clean accuracy is also preserved. Similar trends are observed ImageNet and its variants, with C yielding up to over 3% gain in robust accuracy over C under PGD attakes.
Effects of the number of augmentations. To investigate the sensitivity of our method to augmenta- tion strategies, we evaluate the effect of varying the number of image and class name augmentations on both clean and robust accuracy, shown in Figure 2 Increasing the number of image augmen- tations consistently enhances robustness, while the clean accuracy remains stable. However, the improvement becomes marginal when the number exceeds 5. A similar saturation effect is observed in class name augmentation, where performance gains plateau beyond 50 augmentations. These observations indicate that method is robust the choice of augmentation hyperparameters.
Effects of projection matrix construction. We study how varying the number of singular vectors C used to construct the projection matrix affects classification accuracy. As shown in Figure increasing C' steadily improves performance on both clean and adversarial examples across Caltech and ImageNet. The gains more prominent when C is small and gradually saturate beyond C = 200,
8

---

90
Out
Accurncy (16
60-
Clan
Oue
Accuracy (%)
60
Chia
85
Accuncy
Accuracy
80
40
75
70
20 30 40 60
20 70
Image Augroentation
Imsaee Aug mnnettttiiion
Name Augmenntatio
Class Name Angmentation
(a) Caltech101
(b) ImageNet
(c) Caltech101
(d) ImageNet
Figure 2: Accuracy (%) comparisons across Caltech and ImageNet datasets with varying the number augmentations. (a) and (b): Classification results under different numbers of image aug- mentations. (c) and (d): Classification results under different numbers of class name augmentations.
90
Accurscy (S)
60
Anaded Fomrs
Accuracy
50
Density
Density
85
Cla
80
30
100
200
300
400
500
100
200
300
400
300
0.1 Similurity Score
0.1
0.3
0.5
Principal Components
Principal Components
Similarity Score
(a) 1altech101.
(b) ImageNet.
(a) Attacked Similarity.
(b) Projected Similarity.
Figure 3: Accuracy (%) with different numbers Figure 4: Similarity distributions among original, of principal components the projection matrix. attacked, and projected features on ImageNet.
especially for clean samples. In contrast, robust accuracy improves more slowly, suggesting that while additional components better preserve semantic structure for clean inputs, they provide limited benefit under adversarial perturbations. Based this observation, fix C 256 in subsequent experiments to balance performance and efficiency.
To further understand the effectiveness of projection in Eq. distributions between image features and their corresponding text features under different conditions. As shown in Figure 4a adversarial perturbations significantly reduce the similarity between image and text features, indicating disrupted alignment. In contrast, 4Buuressssswss that projecting the attacked features onto the text-induced subspace effectively restores their similarity to the original level. This demonstrates that our projection eifectively corrects adversarial misalignment and strengthens semantic consistency across modalities, thereby enhancing robustness in classification.
analyze the similarity score
Running time. We compare the inference- efficiency of our method on ImageNet using CLIP ViT-B/32 with a batch size 128 on single NVIDIA 3090 GPU. As shown in Table 8 our method completes evaluation in 28 minutes, significantly faster than TTC (40 minutes) while achieving both higher clean (62.8% % vs. 51.7%) and robust (50.0% vs. 40.0%) accuracy. This efficiency stems from the training-fiee free nature of our approach, which avoids the costly iterative optimization required by TTC.
Table 8: Comparison of running time on ImageNet with ViT-B/32.
Accuracy
Model Running Time
Clean
Robust
CLIP TTC COLA
10min
62.1
1.1
40min
51.7
40.0
28min
62.8
50.0
5 Limitation and Conclusion
Limitation. While COLA substantially improves adversarial robustness, it still inherits potential biases from the pre-trained lasiunelanguage backbone [69 64 66). In particular, the text-induced subspace may encode sptaset- priors visual domains. Moreover, stronger defenses could provoke more adaptive attacks, suggesting the need for future research resilience under adaptive adversaries and fairness-award robustness.
limiting generalization to unseen linguistic
Conclusion. By enhancing robustness against adversarial manipulation, COLA contributes to safer multimodal systems, especially in high-stakes applications such as autonomous driving and medical imaging. We present COLA, training-free and theoretically grounded framework that improves the adversarial robustness of CLIP by addressing modality misalignment. COLA leverages subspace projection to restore global alignment and employs optimal transport refine local semantic consistency. By embedding projection into the OT cost computation, it maintains cross-modal

---

aligmment without retraining or architectural modification. Theoretical analyses show that COLA reduces cosine distortion and enlarges decision margins, thereby improving generalization. Extensive experiments across 14 benchmarks confirm that COLA consistently enhances zero-shot classification robustness while preserving clean accuracy.
Acknowledgments and Disclosure of Funding
This research is supported by the National Natural Science Foundation of Anhui (Grant No. 2508085MF143) and the advanced computing resources provided by the Supercomputing Cen- ter of the University of Science and Technology of China (USTC). Additional support was provided by the National Research Foundation, Singapore, under the NRF Investigatorship Award (NRF- NRF1100-004-0004).
References
[1] Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: compulational statistics, 2010.
[2] Motasem Alfarra, Juan C Pérez, Ali Thabet, Adel Bibi, Philip HS Torr, and Bernard Ghanem. Combating adversaries with anti-adversaries. In AAAI, 2022.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. ICML, 2017.
[4] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In IJCAI, 2021.
[5] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normmliized margin bounds for neural networks. In NeurIPS, 2017.
[6] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014.
[7] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, In IEEE Symposium on Security and Privacy, 2017.
[8] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. In ICLR, 2022.
[9] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based global reasoning networks. In CVPR, 2019.
[10] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In CVPR, 2014.
[11] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. TPAMI, 2016.
[12] Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In ICML, 2022.
[13] Marco Cuturi. Sinkhorn distances: 2013.
Lightspeed computation of optimal transport. In NeurIPS.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[15] Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In KDD, 2025.
[16] Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross- moosl alignment in CLIP. In ICLR, 2025.
10

---

[17] Junfeng Fang, Zac Bi, Ruipeng Wang. Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang. and Tat-Seng Chua. Towards neuron attributions in multi- molti-modal large language models. In NeurIPS, 2024.
[18] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. In ICLR, 2025.
[19] Junfeng Fang, Yukai Wang, Ruipeng Wang, Zijun Yao, Kun Wang, An Zhang, Xiang Wang, and Tat-Seng Chua. SafemIrm: Demystifying safety multi-modal large reasoning models. CoRR, abs/2504.08813,, 2025.
[20] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, 2004.
[21] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. In ICIR, 2017.
[22] Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Adaptive distribution calibration for few-shot learning with hierarchical optimal transport. In NeurlPS, 2022.
[23] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote, Sens., 12(7):2217-22) 2019.
[24] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -iittr-ddittiittionn generalization. In ICCV, 2021.
[25] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops, 2013.
[27] Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transductive and semi-supervised few-shot learning. In ICCV, 2021.
[28] Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 2015. 2015.
[29] Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng. James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vasion-language models, 2024.
[30] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[31] Lin Li, Haoyan Guan, Jianing Qiu, and Michael Spratling, One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In CVPR, 2024.
[32] Xiao Li, Wei Zhang, Yining Liu, Zhanhao Hu, Bo Zhang, and Xiaolin Hu. Language-driven anchors for zero-shot adversarial robustness. In CVPR, 2024.
[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2018.
[34] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. CoRR, abs/ 130.5111,,,,,, 2013.
[35] Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang, and Carl Vondrick. Adversarial attacks are reversible with natural supervision. In ICCV, 2021.
[36] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In ICLR, 2023.
11

---

[37] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In ICML, 2022.
[38] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
[39] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[40] Juan C. Pérez, Motasem Alfarra, Guillaume Jeanneret, Laura Rueda, Ali K. Thabet, Bernard Ghanem, and Pablo Arbeláez. Enhancing adversarial robustness via test-time transformation ensembling. In JCCVW, 2021.
[41] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, pages 355-607, 2019.
[42] Sarah M. Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
[43] Junxiang Qiu, Lin Liu, Shuo Wang, Jinda Lu, Kezhou Chen, and Yanbin Hao. Accelerating diffusion transformer via gradienl-optimizeed cache. CoRR, abs/2503.05156, 2025.
[44] Junxiang Qiu, Shuo Wang, Jinda Lu, Lin Liu, Houcheng Jiang, and Yanbin Hao. Accelerating diffusion transformer via error-optimized cache. CoRR, abs/2501.19243, 2025.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[46] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[47] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning, In ICML, 2020.
[48] Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In CVPR, 2023.
[49] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In JCML, 2024.
[50] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! NeurIPS, 2019.
[51] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
[52] Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning multi-mode token-level prompt alignment across modalities, 2023.
[53] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[54] Sibo Wang. Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In CVPR, 2024.
[55] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma. the margin theory of feedforward neural networks. 2018.
12

---

[56] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. 2020.
[57] Boxi Wu, Heng Pan, Li Shen, Jindong Gu, Shuai Zhao, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu, Attacking adversarial attacks as a defense. arXiv preprint urXiv:2106.04933 2021.
[58] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[59] Songlong Xing, Zhengyu Zhao, and Nicu Sebe. Clip is strong enough to fight back: Test-time counterattacks towards zero-shot adversarial robustness of clip. arXiv preprint arXiv:2503.03613. 2025.
[60] Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov- learning for graph partitioning and matching. NeurIPS, 2019.
[61] Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In ICML, 2021.
[62] Hongyang Zhang, Yaodong Yu, Jiantao Jíao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled princippeee trade-off between robustness and accuracy. In JCML, 2019.
[63] Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, and Jitao Sang. Adversarial prompt tuning for vision-language models. In ECCV, 2024.
[64] Beier Zhu, Jiequan Cui, and Hanwang Zhang. Robust fine-tuning of zero-shot models via variance reduction. In NeurlPS, 2024.
[65] Beier Zhu, Jiequan Cui, Hanwang Zhang, and Chi Zhang. Project-probe-aggregge: Efficient fine-tuning for group robustness. In CVPR, 2025.
[66] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. In NeurIPS, 2023.
[67] Xingyu Zhu, Shuo Wang, Jinda Lu, Yanbin Hao, Haifeng Liu, and Xiangnan He. Boosting few-shot learning via attentive feature regularization. In AAAI, pages 7793-7801, 2024.
[68] Xingyu Zhu. Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang. Zhicai Wang, Dongsheng Wang, and Hanwang Zhang. Dynamic multimodal prototype learning vision- language models. CoRR, abs/2507.03657. 2025.
[69] Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. In NeurIPS, 2024.
[70] Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision-language subspace projection for few-shot CLIP. In ACM Multimedia, 2024.
[71] Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring languaee models via augmentation, weighting, and transportation, 2024.
13

---

Contents
Introduction
2
Related Work
3
Method
Preliminaries
3.2
Cross Modality Alignment under Unified OT Framework
3.3
Theoretical Analysis
4
Experiments
Setup
6
Main Results
Ablation Study
5
I Limitation and Conclusion
Proofs
15
A. Proof of Cosine Similarity Distortion Bound
15
A.2 Proof of Margin Amplification
16
External Results
17
Algorithm
17
14

---

A Proofs
A.1 Proof of Cosine Similarity Distortion Bound
Let X1,X2 E Rd be clean feature vectors with ||x,|| 1. The adversarial features are denoted by
8; x; 8; ||+:|| S
where 5; 5j 5', with 8j u and 81 T u. Let (x;;) + 6i denote the projection of the adversarial feature onto the subspace U. The cosine similarity between adversarial features and projected features:
cOs(11, x2) (x1 + 61) (x2 + 82)
cos(TI())), = (x1 + +0)) (x2 + of) [ -
{x |
We define the cosine distortion quantities
A (0,o(81;;x)) cos((1; X2)1, An ((os( cos(x1, X2)]-
We now compare A and An. Using second-order Taylor approximation for small perturbations ||o:| < 1, we obtain:
cos ( (x X2 +
x
82
81 x2 + 8, 82) x, 6, - xT6, _1122 12)
[1],111(11
cos(TT(x)),
<<
x2 + x o) + (6)) [ X2 + (51)
(1 x || [ 13) - x3 of 1991)).
To simplify analysis, assume 61 = 52 5 and 8j = 5) = 6 Since E l and 01 1. 4, we have x. 81 and thus x, = x 6
Under this setting, both distortions simplify to:
A 2x [ x2 [x 511) + 0(2))
An
2x 8
2
x2) [x] 511 + (
Thus, the first-order terms A and An are identical. However, in the general case where || > the norm of the full feature x; is larger than that of its projection I(xx),. reducing the cosine similarity in the unprojected case.
Using Cauchy-Schwarz Schwarz and bounding terms:
2| | [1 + xx]xxl)
xx
Anl s 2| [I(1 + x21,,
|
16
6
Therefore, when the perturbation contains a non-zero orthogonal component (1001 || > 0), we have
A A
| |2
1
|
which shows that the cosine similarity distortion is strictly reduced by projecting the adversarial features onto the subspace U.
General case. When 5, * 62, let = x] x2 and l| = [T(x;) TTT (2). first-order Taylor expansion yields:
cos(X1, 2 x 52 xx 81 s(x] 8 +x, 8z). cos(II({)),
الهه + + 82ı
+x l1 (xx 8i1 x 821).
15

---

Hence, the projected distortion satisfies:
AT
S 8
spi
|Tx
[[x
where the first term reflects the semantic deviation between clean and projected subspaces, and the term accounts for asymmetric perturbations. Since projection removes orthogonal noise, the overall distortion ratio becomes:
An
1
<1,
1+
8
showing that projection consistently suppresses cosine similarity distortion even when perturbations differ direction or magnitude.
A.2 Proof of OT-Margin Amplification
We prove that the projected OT margin satisfies ,(C") 2 7(C), stated in the main text.
Proof. Since the projection IT(*)) x" + +oj removes the orthogonal perturbation 8f U, and each text prototype m 2 lies in the subspace U, the dot product is preserved:
(x*) T y (1I(x")) zn
Meanwhile, the norm of the adversarial feature satisfies:
| ||
||[((*))| + | 2 [I[()")
with equality if and only if 8T Therefore, the cosine similarities before and after projection
cos(x" z = (I({*)) z
cos(II(X*) ) = (II(*)) z
| |||
[ ||||,] ||
Since ||*'||
[I[(*))] II, we conclude:
|| | ||[(**)|
cos cos(II(x*), z.") = cos(x" 2.")
> cos(X" 22")
Hence, the projected cost matrix entry is smaller:
C (n, m) = cos(II()"), 2, S cos(x" (as(s*220 .,,) = Cy,n.m)
Given (n, m) s (n, we now compare the OT distances. The OT distance is defined as:
dor(P, Q: C.y) = min (T Cy), s.t. Ty = a, T,1N 1N = by
T,20
For any feasible transport plan Ty, we have (T Clll < (T C2). Let Tr = arg min Cy). Then:
dor(P, Q: c < TT C < (T C2) = dor (P. I.P.Q.CC. QyC.)
Define the OT distance reduction:
Ady dor (P. Q Cyy) dor (P. Q C 2
Next, we compare the cost reductions for the true class y* and a competing class Yneg- Note that:
| [IT(>>>] (
(
(n, - cff( (n, m) cos(TT(>*), x. cos(>". 2 = cos(x". ,.""
)
The multiplicative factor
| [[[["""]]|
0 is shared across all classes. Since cos(X" 22?')
>
cos(S") n 2. ), follows that:
(n, Cff (n,, m)) > Cynz (no, m) Cho (n, m).
16

---

Table Performance comparison under AutoAttack.
Model
9-datasets (Clean)
9-datasets (Robust)
ImageNet (Clean)
ImageNet (Robust)
CLIP
63.6
0.5
67.6
0.2
TTC
61.3
8.3
67.1
16.2
COLA
64.0
18.9
68.9
29.6
Algorithm 1 Pipeline of COLA
Input: Adversarial visual feature x with its N augmentations {*")"=1. and class textual embeddings {z," Jumm from L.M--genrrted descriptions.
KM and perform SVD: Z Global feature projection. Construct the textual embedding matrix Z
UΣV.. T. Extract top-C' principal components Uc = (.,:cc and project each adversarial image feature onto
the text-induced subspace: IT(*") = U
Local semantic distribution modeling. Represent the image and class as discrete distributions: P(x)
2n a" a"8(8" x). (2 Σn bg 6(2," z). The importance weights are normalized by prediction entropy: a" exp(h(("))) 65 & exp(hizz)) where h(-) defined Eq. (6).
Optimal transport alignment. For cach image-text pair (x", >))), compute the projected transport cost:
Cy (n, m) cos((1(X")) 2g Obtain the OT distance by solving:
dor(P(x (x):): C min (T Cy),, s.t. Tylı T IN by. 2,
Classification. Predict the label with minimal OT distance:
arg min dor (x))
k
Hence, for the respective optimal transport plans Ty and T
have:
"iees"
(T Cy- C T Mive
Cyiny
Cll
Vlng
Therefore:
Ady 2 Adwn
Finally, we analyze the margin difference:
>(C)) xCC) |dor (IP Q; "oog dor Q; Cynaa) |doT P: ( - dor(IP, Q: Cy-)
Ad
+ Ady = Ady Adyn 2 0.
Thus, 7(C (C), with equality if and only if 8" for all n.
B External Results
Analysis. As shown in Table 9 our method achieves the best robustness under AutoAttack, reaching 18. 18.9% on 9-datasets and 29.6% on ImageNet, surpassing CLIP (0.5% 0.2%) and TTC (8.3% 16.2 16.2%). These results show that the proposed subspace projection effectively filters non-semantic adversarial noise, while OT-based alignment restores image-text consistency.
C Algorithm
The overall procedure of COLA is summarized in Algorithm I which outlines the projection- based alignment and OT-based matching steps for adversarially robust inference.
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: COLA performs test-time cross-modality alignment by projecting adversarial features and matching them via optimal transport. The method is theoretically justified and empirically validated across diverse CLIP models and robustness benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes].
18

---

Justification:
The limitations have been discussed in Section ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We have provided the proof related to our method in Section A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the details our COLA framework in Section and included the implementation details in Section
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have uploaded the codes in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
public//uiidss/oddSuuuuuue
wttpsuumiiiiiiiossosoooioioieiee for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/nips.cc/publi///guu
CodeeSumiiii wwabbbmmsssiinii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have introduced the testing details in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Given the vision- languala models, the process of our method deterministic. Running multiple times will not introduce randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the GPU in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp////ooorrr...c//aaall/ll/ll
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impact on Section ??.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our method aims to improve model robustness, which poses no such risks to the best of our knowledge.
Guidelines:
22

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets used in this paper are publicly available and cited appropriately.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
23

---

Justification:
We do not involve such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our L.M policy http:://earrrppeecco/oofeeerrree for what should or should be described.
24


Reviews:
[{"review_id": "yD2VLs9Zse", "reviewer": "Reviewer_i8r4", "summary": "## Summary\nThe authors propose a novel test-time counterattack for adversarial perturbations: their method is composed mainly of two steps, \n - a projection of the input image vectors (obtained by cropped, resized, and flipped views of the input image) into a \"class subspace\" obtained by the principal components of the class text prompts, \n - an optimal transport phase that for each class computes the optimal transport between the multiple image projections and the multiple class vectors specific for that class. The class with the minimum optimal transport plan is used as the prediction.\n\nThese two operations are both relatively inexpensive to adopt at inference time and seem to greatly increase the robustness of CLIP against adversarial attacks.", "questions": "# Questions\n\n1. Are you the first to use optimal transport in such a way? If not, please provide a citation in the method section. If yes, then it would be better to properly mention it among your contributions.\nIn particular, in lines 142-144, you write *\"$C_y^Π \\in R^{N×M}$  denotes the transportation cost between the N augmented image views and the M textual descriptions of class $j$, which is usually quantified using the cosine similarity\"*. \nWho are you referring to with \"usually quantified\"?\n2. Does the time reported in Table 8 include the computation of the PCA? How much time does optimal transport require?\n3. The authors mention (and visualize) the misalignment gap between textual and image inputs in CLIP; however, it seems that there is no citation to works related to that issue.", "limitations": "To be efficient COLA requires: (i) the computation of the multiple textual \"views\" of a class from an LLM, (ii) the estimation of the class PCA subspace. If the classes are static, these can be seen as inexpensive; however, if the textual prompts are highly dynamic, these may have a substantial impact on prediction time. \nWhile this is an uncommon scenario, mentioning this limitation might be reasonable.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is generally clearly written and readable.\n2. The proposed approach is simple yet effective; in most of the results, it even outperforms the baseline CLIP model.\n3. From the ablation studies and the theoretical analysis, it seems that both components (the projection and the optimal transport) improve adversarial robustness and even overall accuracy.\n\n----\n\n## Weaknesses\n\n1. The authors mention multiple times that their proposed method is more efficient than alternatives; however, the computation time analysis is somewhat lacking and not as comprehensive as other tests. I would have appreciated more comparisons and experiments in that scenario.\n\n2. While the proposed method is effective as a defence against attacks, it is still relatively expensive: it requires a class \"augmentation\" step with an LLM, estimation of the principal components, computation and projection of multiple input features, and finally, the optimal transport algorithm itself.  Admittedly, the first two steps can be cached (per class), but the projection and optimal transport must be computed for all inputs.\n\n3. I am not sure if the advancements proposed by the paper are enough to garner a publication at this venue. However, being that I am not particularly familiar with this specific setting, I am conflicted regarding this point.", "was_revised": true, "final_justification": "Since the authors addressed my concerns in the rebuttal, I have decided to increase my score from \"borderline accept\" to \"accept\".\nThe issues I was concerned about were mainly regarding:\n - Lack of citation for certain statements in the article.\n - Relatively weak analysis in terms of runtime of different methods.", "rebuttal": "**Q1:** Computation time analysis and comparisons.  \n**A:** We have conducted additional experiments on running time, as shown in the table below. These results indicate that our method (COLA) not only requires less time compared to TTC but also achieves better performance.\n|Model| Dataset| Running Time | Clean Accuracy (%) | Robust Accuracy (%) |\n|--------------|-------------|--------------|---------------------|----------------------|\n| CLIP         | ImageNet   | 10min        | 62.1               | 1.1                  |\n| TTC          | ImageNet  | 40 min       | 51.7                | 40.0                 |\n| **COLA (Ours)** | ImageNet  | 28 min (Projection: 4 min, OT:8 min)      | 62.8             | 50.0                 |\n| CLIP         | Caltech101   | 2min        | 85.7               | 14.7                  |\n| TTC          | Caltech101  | 5 min       | 86.5                | 65.8                 |\n| **COLA (Ours)** | Caltech101  | 3 min (OT: 1min)      | 88.1             | 75.3                 |\n| CLIP         | DTD  | 0.5 min       | 40.6               | 3.0                  |\n| TTC          | DTD  | 3 min       | 37.0               | 27.3                 |\n| **COLA (Ours)** | DTD  | 1 min (OT: 0.5 min)       | 41.0               | 34.0   \n\n\n**Q2:** Computational cost for projection and optimal transport.  \n**A:** The projection step involves simple matrix multiplication between a $d$-dimensional input feature and the class-specific projection matrix, with a per-input cost of $\\mathcal{O}(d^2)$, which is negligible in practice (e.g., <0.01s per sample). For OT, its complexity is generally $\\mathcal{O}\\left(n^2 \\cdot \\log\\left(\\frac{1}{\\varepsilon}\\right)\\right)$, where $\\varepsilon$ is the regularization coefficient and usually set to 0.01. Importantly, the OT computation is performed in parallel across all classes, resulting in efficient runtime. As reported in the table in response to Q1, the time costs are 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minute on DTD.\n\n**Q3:** Explanation for OT.  \n**A:** To the best of our knowledge, we are the first to apply OT for improving the adversarial robustness of CLIP. Prior works have used OT in other tasks such as distribution calibration [17] and few-shot learning [22].  \nRegarding the cost metric, cosine similarity is commonly used in multimodal OT-based alignment. Prior works such as PLOT [8], AWT [56], and ALIGN [44] adopt it to measure semantic distances between visual and textual representations.\nWe have revised the manuscript accordingly.\n\n**Q4:** Details in Table 8.  \n**A:** The time reported in Table 8 includes the PCA computation. OT computation takes approximately 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minute on DTD, as shown in the table provided in response to Q1.\n\n**Q5:** Citation for CLIP modality misalignment.  \n**A:** We have revised Section 1 and Section 3.2 to include relevant citations. Li et al. [25] and Mao et al. [30] demonstrate that adversarial perturbations and low-level visual biases can disrupt the alignment between visual and textual features. More recent studies, such as [r1] and [r2], further characterize the modality gap in CLIP.  \n[r1] Stabilizing  Modality Gap & Lowering Gradient Norms Improve Zero-Shot Adversarial Robustness of VLMs, KDD 2025  \n[r2] Mitigate the Gap: Improving Cross-Modal Alignment in CLIP, ICLR2025\n\n**Q6:** Impact of dynamic prompt generation.  \n**A:** In CLIP-based classification, class names are fixed, and prompts are generated using simple templates. For example, CLIP uses “a photo of a [class name],” which is shared across all test samples. Thus, prompt generation incurs only a one-time cost per class and does not affect per-image inference time. While dynamic prompts per image could introduce overhead, such cases are rare and fall outside the scope of CLIP-based classification. We have included this limitation in the revision."}, {"review_id": "LHc1IsiLgB", "reviewer": "Reviewer_JTxh", "summary": "This paper proposes a training-free method to enhance the robustness of CLIP models against adversarial images. It introduces two alignment strategies: global feature alignment and local structural alignment, and employs an optimal transport-based framework to refine the correspondence between text and image embeddings. The method is evaluated on diverse datasets, and the results demonstrate that the proposed COLA approach significantly outperforms both state-of-the-art fine-tuned and training-free methods.", "questions": "1. In Figure 1(d), why does the Original CLIP have the lowest accuracy, while the Attacked CLIP shows the highest accuracy?\n2. PGD and CW are no longer the most effective adversarial attack methods. Could the author provide results against state-of-the-art attacks (e.g., AutoAttack) to better demonstrate the method's performance?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Theoretical analysis ensures the reliability of the proposed method.\n2. The introduced global feature alignment and local structural alignment respectively target cross-modal and intra-modal feature alignment, offering valuable insights.\n3. The experimental results are comprehensive and the performance is impressive.\n\nWeaknesses:\n1. The paper does not provide a clear framework diagram or pseudo-code, which makes it somewhat difficult for readers to fully understand the proposed method's workflow.\n2. The evaluation is limited to CLIP-based classification tasks, without demonstrating whether the proposed approach can be extended to more complex models (e.g., vision-language models) or more challenging tasks (e.g., image captioning).", "was_revised": true, "final_justification": "The authors' rebuttal has successfully addressed my concerns. The proposed method is highly innovative and demonstrates strong performance across multiple datasets. I believe this research makes a meaningful contribution to improving the robustness of multimodal models. Therefore, I recommend accepting the paper.", "rebuttal": "**Q1:**   Framework diagram or pseudo-code.  \n**A:** Thank you for the helpful suggestion. In the revised version, we have added a framework diagram in Section 3 to clearly illustrate the main components of our method, including subspace projection and OT-based alignment. We have also included pseudo-code in the appendix to describe the inference procedure in detail.\n\n**Q2:** Extending to more complex models or tasks.  \n**A:** Our current work focuses on CLIP-based classification tasks, which rely on computing similarity scores between visual features and predefined textual class embeddings.  Tasks like image captioning involve language generation, which falls outside the design scope of our OT-based framework.\nWe have conducted additional experiments using **SigLIP** and **EVA-CLIP**，two recent and stronger CLIP variants. As shown in the table below, our method consistently improves robustness under adversarial attacks, demonstrating its effectiveness across different vision-language model backbones.\n| Model        | Backbone  | 9-datasets (Clean) | 9-datasets (Robust) | ImageNet (Clean) | ImageNet (Robust) |\n|--------------|-----------|--------------------|----------------------|------------------|--------------------|\n| SigLIP         | SigLIP  | 65.6               | 1.5                  | 70.2             | 0.5                | \n| TTC          | SigLIP  | 62.3               | 15.8                 | 68.1             | 22.6               | \n| **Ours**     | SigLIP  | **65.0**           | **24.7**             | **70.9**         | **35.3**           | \n| EVA-CLIP           | EVA-CLIP  | 64.5              | 1.0                  | 68.8             | 0.4                | \n| TTC          | EVA-CLIP  | 61.2               | 13.7                 | 67.4             | 21.6               | \n| **Ours**     | EVA-CLIP  | **65.2**           | **18.9**             | **69.6**         | **33.7**           | \n\n\n**Q3:** Labeling in Figure1(d).  \n**A:** We thank the reviewer for pointing this out. The legend in **Figure 1(d)** was mistakenly swapped between \"Original CLIP\" and \"Attacked CLIP\". We have corrected this labeling error in the revised version.\n\n\n**Q4:** Results of autoattacks.  \n**A:** We have conducted additional experiments using **AutoAttack** on ViT-B/16. As shown in the table below, our method consistently improves robustness on both **9-datasets** and **ImageNet**, demonstrating effectiveness beyond specific attack types. We have included these results in the revision.\n| Model       | Attack Type   | 9-datasets (Clean) | 9-datasets (Robust) | ImageNet (Clean) | ImageNet (Robust) |\n|-------------|------------|--------------------|----------------------|------------------|--------------------|\n| CLIP        | autoattacks  | 63.6               | 0.5                  | 67.6             | 0.2                | \n| TTC         | autoattacks   | 61.3               | 8.3                 | 67.1             | 16.2               | \n| **Ours**    | autoattacks   | **64.0**           | **18.9**             | **68.9**         | **29.6**           |"}, {"review_id": "fK4hwnYLmq", "reviewer": "Reviewer_BF28", "summary": "This paper proposes the COLA framework to enhance the adversarial robustness of CLIP models. The approach projects adversarial image features into the text feature space and refines alignment through optimal transport (OT). Specifically, text features undergo SVD to extract top-C principal components, with image features projected onto the span of these components. Images and texts are modeled as discrete distributions across augmented views, enabling alignment refinement via optimal transport. Experimental results demonstrate high adversarial robustness across zero-shot classification datasets, supported by theoretical analysis validating the advantages of feature projection and the OT-based framework.", "questions": "1. Can the proposed method be extended to non-classification tasks (e.g., as vision encoders in LVLMs) where text labels are unavailable?\n2. Why was ε=1/255 chosen without using the standard ε=2/255?\n3. What are the detailed implementation specifics for TeCoA/FARE (e.g., perturbation size, attack methods)?\n4. Is there rigorous analysis proving adversarial perturbations distort away from the text subspace (U)? Were ablation studies conducted to validate this directional assumption?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Reduces adversarial perturbations by directly projecting image features into the text feature space, and improved feature alignment by introducing an OT-based framework instead of relying on conventional simple cosine similarity between image and text embeddings.\n2. Provides theoretical validation of the framework's advantages through rigorous analysis, demonstrating its robustness not just experimentally but also theoretically.\n---\n### Weaknesses\n1. The framework is only applicable to classification tasks. Extracting text feature space and projecting image features requires class labels from the dataset. Since the OT framework operates after feature extraction for label prediction, it cannot be adapted to non-classification tasks (e.g., LVLMs where CLIP vision encoders are used without text labels). It is a critical weakness as CLIP models are increasingly deployed in diverse scenarios without text labels.\n2. Image features are projected using class-informed text features, meaning class labels influence feature extraction. This creates an uneven comparison baseline against methods extracting image features without label information.\n3. Projecting image features onto the text subspace is theoretically most effective when adversarial perturbations lie in the orthogonal complement subspace. The paper also notes that adversarial perturbations distort image features along directions away from the text feature subspace in line 121. However, the paper lacks the quantitative analysis of perturbation reduction post-projection and the empirical validation of the orthogonality assumption\n\n(Minor) \n\n4. Implementation details (perturbation size, attack methods) for TeCoA/FARE baselines are omitted.\n5. The δ₁ = δ₂ = δ equality in Appendix A.1 seems inadequate.\n6. Figure labeling errors: Swapped descriptions between panels (a) and (b) in Figure 1's caption. Reversed \"Original CLIP\" and \"Attacked CLIP\" labels in Figure 1-(d)'s legend.", "was_revised": true, "final_justification": "The authors have provided detailed and thoughtful responses that satisfactorily address most of my initial concerns, particularly regarding (1) the validity of the orthogonality assumption, (2) the fairness of their feature extraction procedure, and (3) the implementation details for baseline methods such as TeCoA and FARE. Their theoretical clarifications and extended derivations in the appendix (e.g., the general case with mismatched perturbations) demonstrate rigor and help build confidence in the proposed projection framework.\n\nThat said, I still view the reliance on classification tasks with predefined labels as a non-trivial limitation, particularly in real-world deployments where CLIP is often used as a general-purpose visual encoder in open-ended tasks. Although the authors acknowledge this and suggest possible extensions (e.g., clustering-based pseudo-labels), such adaptations remain future work.\n\nOverall, I find the method to be technically solid, theoretically well-motivated, and empirically effective, albeit with scope limitations. Hence, I have raised my score to borderline accept.", "rebuttal": "**Q1:** Applicability beyond classification tasks.  \n**A:** Our method is specifically designed to enhance the adversarial robustness of CLIP in classification settings, where the text labels (i.e., class names) are the predefined categories used to classify images at inference time.\nWhile our current framework is designed for classification tasks with a predefined class set, we acknowledge the importance of extending it to open-ended settings. A promising direction is to extract pseudo-labels from captions via clustering or noun phrase parsing, enabling the construction of a latent text feature space without explicit labels. This would allow projection and OT-based alignment in more general scenarios, which we consider a compelling direction for future work.\n\n\n**Q2:** Label influence on feature extraction fairness.  \n**A:** We would like to clarify that CLIP-based classification is a closed-set problem, where all class names are predefined and available during inference. Our method projects image features into the subspace spanned by all class text embeddings, without using the ground-truth labels of test samples, thereby avoiding any label leakage. Moreover, the baselines we compare against (e.g., TTC, TeCoA) also rely on class text embeddings during inference, ensuring that all methods are evaluated under consistent and fair settings.\n\n\n**Q3:** Analysis of perturbation reduction and validation of orthogonality assumption.  \n**A:** We thank the reviewer for raising this important point. We provide both theoretical justification and empirical validation:\n- **Perturbation reduction after projection.**  \n  As shown in Equation (12) and formally proven in Appendix A.1, we demonstrate that projecting adversarial features onto the text subspace reduces semantic distortion:\n  $$\n  \\Delta_\\Pi \\leq \\Delta,\n  $$\n  where $\\Delta$ and $\\Delta_\\Pi$ denote the cosine deviation before and after projection, respectively. This is further supported by Figure 4, which shows that projected features exhibit significantly higher similarity to text embeddings than the adversarial ones.\n\n- **Validation of the orthogonality assumption.**  \n  We decompose adversarial perturbations into components parallel and orthogonal to the text subspace and find that the orthogonal component consistently dominates. For example, under PGD-1/255 on ImageNet, $\\left\\| \\boldsymbol\\delta_\\perp \\right\\| / \\left\\| \\boldsymbol\\delta \\right\\| \\approx 75.4\\%,$\n  This confirms that most adversarial perturbations lie outside the semantic subspace.\n\n\n\n**Q4:** Implementation details for TeCoA and FARE.  \n**A:**  We appreciate the reviewer’s suggestion. In our experiments, we did not re-implement TeCoA or FARE, but utilized the fine-tuned models released by TTC [51] to extract features. We have added the specific attack settings (including perturbation size, attack type, and evaluation protocol) used in the original TeCoA and FARE papers to the revised version.\n\n\n**Q5:** $\\boldsymbol\\delta_1 = \\boldsymbol\\delta_2 = \\boldsymbol\\delta$ assumption.  \n**A:** The assumption $\\boldsymbol\\delta_1 = \\boldsymbol\\delta_2 = \\boldsymbol\\delta$ was made for simplicity, to emphasize the core insight: **projection suppresses cosine similarity distortion under adversarial perturbations**. We have extended the analysis in the appendix to cover the general case where $\\boldsymbol\\delta_1 \\ne \\boldsymbol\\delta_2$. Specifically, under first-order approximation, the cosine distortion before projection becomes:\n$$\\Delta \\approx \\left| \\mathbf{x}_1^\\top \\boldsymbol{\\delta}_2 + \\mathbf{x}_2^\\top \\boldsymbol{\\delta}_1 - s(\\mathbf{x}_1^\\top \\boldsymbol{\\delta}_1 + \\mathbf{x}_2^\\top \\boldsymbol{\\delta}_2) + \\boldsymbol{\\delta}_1^\\top \\boldsymbol{\\delta}_2 \\right| + O(\\epsilon^2),\n$$\n\nand after projection:\n\n$$\n\\Delta_\\Pi \\leq |s - s_\\Pi| + \\epsilon \\left( \\frac{1}{\\|\\mathbf{\\Pi} \\mathbf{x}_1\\|} + \\frac{1}{\\|\\mathbf{\\Pi} \\mathbf{x}_2\\|} \\right).\n$$\n\nThis yields the bound:\n$$\n\\frac{\\Delta_\\Pi}{\\Delta} \\lesssim \\frac{1}{1 + |s|} < 1,\n$$\nThe result confirms that projection consistently mitigates cosine similarity distortion even under mismatched perturbations, with $s = \\cos(\\mathbf{x}\\_1, \\mathbf{x}2)$ and $s\\_\\Pi = \\cos(\\mathbf{\\Pi} \\mathbf{x}\\_1, \\mathbf{\\Pi} \\mathbf{x}\\_2)$.\n\n**Q6:**  Figure labeling errors.  \n**A:** We thank the reviewer for pointing out the labeling errors. We have corrected them in the revised version.\n\n**Q7:** Why was ε=1/255?  \n **A:** \nWe follow prior works such as TTC and TeCoA in using ε = 1/255 to ensure a fair and consistent comparison. We additionally conduct experiments under ε = 2/255 with ViT-B/16. Results are reported below and again confirm the superiority of our approach.\n| Model     | Attack Budget     | 9-datasets (Clean) | 9-datasets (Robust) | ImageNet (Clean) | ImageNet (Robust) |\n|-----------|-------------------|--------------------|----------------------|------------------|--------------------|\n| CLIP      | $\\epsilon=2/255$  | 63.6               | 0.7                  | 67.6             | 0.2                |\n| TTC       | $\\epsilon=2/255$  | 61.3               | 13.1                 | 67.1             | 19.6               |\n| **Ours**  | $\\epsilon=2/255$  | **64.0**           | **19.5**             | **68.9**         | **31.2**           |"}, {"review_id": "8cyxnctobe", "reviewer": "Reviewer_dXfe", "summary": "This paper addresses the adversarial vulnerability of Vision-Language Models, positing that attacks work by disrupting the alignment between image and text features. The authors propose COLA, a training-free, test-time defense to counteract this. The method first projects an adversarially perturbed image feature onto a \"semantic subspace\" derived from class text embeddings, with the goal of filtering out non-semantic distortions. Subsequently, it employs Optimal Transport (OT) to align distributions of augmented image views and text descriptions, using a cost function defined within the projected subspace. The final classification is based on the minimum OT cost. The authors provide a theoretical analysis to support their method and conduct experiments on 14 classification datasets, showing significant improvements in adversarial robustness across various attacks and model backbones.", "questions": "Please refer to the weaknesses section above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Quality:  The paper is also very clearly written, with effective visualizations and a thorough set of ablation studies that justify the core design choices.\n\n- Significance and Originality: The paper tackles the important challenge of adversarial robustness in VLMs. The proposed test-time approach, which focuses on correcting feature misalignment rather than retraining the model, is a novel and valuable contribution to the field.\n\n- Empirical Performance: The empirical results are a key strength. The method demonstrates substantial improvements in robust accuracy across a wide and diverse set of 14 datasets. The reported 6.7% average improvement under PGD attacks for ImageNet and its variants is particularly significant. \n\n- Technical Quality: The method is technically sound and well-supported. The theoretical analysis, which shows that the proposed projection reduces similarity distortion and that the OT classifier achieves a larger decision margin, provides a sound understanding of the method.  solid foundation for the empirical results.\n\n**Weaknesses**\n\n- A primary concern is the method's computational overhead at test time. While training-free, COLA requires performing an SVD and solving K optimal transport problems for each input image. The reported inference time of 28 minutes on ImageNet is nearly three times that of the standard CLIP model. Further the table only compares COLA to TTT methods. Fine-tuning methods (which would have the same inference time as CLIP) are also not considered. \n\n- The OT framework introduces its own set of sensitivities. Further, the weights assigned to the discrete distributions ($a^n$ and ${b_{y}}^{m}$) are determined by a heuristic based on prediction entropy.  This could be fragile; a view that is confidently misclassified to a single wrong class would have low entropy and thus receive a high weight, potentially pulling the transport plan in the wrong direction. It is also unclear why entropy is a good measure to estimate the importance weights.\n\n- The quality of the semantic subspace is fundamental to the method. How sensitive is the performance to the richness of the text prompts? Specifically, what is the performance drop if one forgoes LLM-based augmentation and uses only a single, simple template (i.e., M=1)? \n\n- It would strengthen the paper to include an empirical validation of the core assumption that adversarial perturbations lie outside this subspace (e.g., by reporting the relative magnitudes of $\\delta_{\\parallel}$ and $\\delta_{\\perp}$\n\n- The theoretical analysis in Appendix A.1 uses a simplifying assumption ($\\delta_1=\\delta_2$) to analyze the cosine similarity distortion for two points. This assumption is not mentioned when claiming the theoretical contribution in Section 3.3. Further, could the authors elaborate on how this result generalizes to the more common scenario where two distinct images are perturbed by different adversarial noise vectors?", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns, so I will increase my score to 4.", "rebuttal": "**Q1:** Computational overhead.  \n **A:** SVD in our method is performed only once before inference and can be cached, resulting in negligible overhead during test time. The per-image OT computations are parallelizable and efficient in practice. Overall, the additional cost remains moderate—for example, 8 minutes on ImageNet, 1 minute on Caltech101, and 0.5 minutes on DTD. While fine-tuning-based methods have similar inference speed to CLIP, they require substantial training overhead and thus fall outside the scope of our training-free setting.\n\n **Q2:**  Concerns over entropy-based weighting.  \n **A:** We agree that confidently misclassified views may receive lower entropy and thus higher weights. To examine this concern, we conducted an empirical analysis on the ImageNet dataset and found that the average weight assigned to such views is only 0.29, indicating that while these cases exist, they are non-dominant in the overall distribution. More importantly, the OT solver computes a global transport plan across the entire set, guided by the pairwise similarity structure between image and text distributions. Therefore, a single view cannot dominate the result due to OT’s normalization constraints. Entropy has been widely adopted in test-time adaptation and prompt tuning [56, r1, r2], as it directly reflects model confidence by quantifying output certainty. To assess its impact, we have compared entropy-based weighting with uniform weighting. As shown below, entropy-based weights consistently yield better robustness across datasets.\n\n[r1]Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, NeurIPS 2022  \n[r2]Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning, CVPR2023  \n| Method     | Pets (Clean) | Pets (Robust) | Flowers (Clean) | Flowers (Robust) | Aircraft (Clean) | Aircraft (Robust) | DTD (Clean) | DTD (Robust) | Eurosat (Clean) | Eurosat (Robust) | Cars (Clean) | Cars (Robust) | Food (Clean) | Food (Robust) | SUN (Clean) | SUN (Robust) | Caltech101 (Clean) | Caltech101 (Robust) |\n|------------|----------|----------|--------------|--------------|---------------|---------------|---------|---------|--------------|--------------|----------|----------|----------|----------|---------|---------|------------------|------------------|\n| CLIP      | 87.4     | 1.0      | 65.5         | 1.1          | 20.1          | 0.0           | 40.6    | 3.0     | 42.6         | 0.0          | 52.0     | 0.0      | 83.9     | 0.7      | 58.5    | 1.1     | 85.7             | 14.7             |\n| entropy-based   | 87.9     | 77.2     | 66.1         | 50.4         | 20.9          | 15.6          | 41.0    | 34.0    | 53.8         | 19.2         | 54.2     | 35.4     | 84.5     | 63.8     | 61.9    | 45.3    | 88.1             | 75.3             |\n| uniform weights   | 86.6     | 73.3     | 63.7         | 46.3         | 19.5          | 13.3          | 39.5    | 31.3    | 50.1         | 17.6         | 52.8     | 33.8     | 82.5     | 61.6     | 59.8   | 43.5    | 86.7             | 73.2             |\n\n\n**Q3:** Impact of using a single prompt.  \n **A:** The subspace projection step is relatively insensitive to prompt richness, as class names already encode essential semantics. However, OT operates on distributions, when only a single prompt is used ($M=1$), the textual side collapses to a point mass, and the OT objective no longer captures meaningful distributional alignment. Consequently, the transport plan becomes degenerate and fails to effectively model semantic matching across multiple views. We conducted an experiment using a single prompt and observed a significant performance drop.\n| Model       | Backbone   | 9-datasets (Clean) | 9-datasets (Robust) | ImageNet (Clean) | ImageNet (Robust) |\n|-------------|------------|--------------------|----------------------|------------------|--------------------|\n| CLIP        | ViT-B/16   | 63.6               | 0.8                  | 67.6             | 0.2                | \n| **Ours** (M=1)         | ViT-B/16   | 60.2               | 11.7                 | 64.3             | 18.5               |\n| **Ours** (M=50)   | ViT-B/16   | **64.0**           | **20.7**             | **68.9**         | **32.1**           | \n\n\n**Q4:**  Empirical validation of perturbation decomposition.  \n**A:** We have conducted an empirical analysis to validate the assumption that adversarial perturbations primarily lie **outside the text subspace**. Specifically, we decompose each perturbation into parallel ($\\boldsymbol\\delta_\\parallel$) and orthogonal ($\\boldsymbol\\delta_\\perp$) components with respect to the semantic subspace and compute their magnitudes. Results consistently show that $|\\boldsymbol\\delta_\\perp| \\gg |\\boldsymbol\\delta_\\parallel|$ across datasets and attacks. For example, on ImageNet under PGD-1/255, we observe: $||\\boldsymbol\\delta_\\perp|| / ||\\boldsymbol\\delta|| \\approx 75.4%$. This confirms that most adversarial perturbations lie in the orthogonal subspace, supporting our assumption.\n| Metric                    | Backbone | 9-datasets (Robust) | ImageNet (Robust) | ImageNet-Variants (Robust) |\n|---------------------------|----------|----------------------|--------------------|------------------------------|\n| $\\left\\lVert \\boldsymbol\\delta_{\\parallel} \\right\\rVert$ | ViT-B/16 | 0.451                | 0.472              | 0.416                        |\n| $\\left\\lVert \\boldsymbol\\delta_{\\perp} \\right\\rVert$     | ViT-B/16 | 0.736                | 0.754              | 0.783                        |\n\n\n**Q5:** Generalization beyond simplifying assumption $\\boldsymbol\\delta_1 = \\boldsymbol\\delta_2$.  \n**A:**  We adopt the simplifying assumption $\\boldsymbol\\delta_1 = \\boldsymbol\\delta_2 = \\boldsymbol\\delta$ in Appendix A.1 to simplify the derivation and highlight the key insight: **projection reduces cosine similarity distortion under adversarial perturbations**. Importantly, this conclusion still holds in the general case where $\\delta_1 \\ne \\delta_2$. The distortion before projection is approximated as:  \n$$\\Delta = \\left| \\cos(\\mathbf{x}_1 + \\boldsymbol{\\delta}_1, \\mathbf{x}_2 + \\boldsymbol{\\delta}_2) - s \\right| \n\\approx \\left| \\mathbf{x}_1^\\top \\boldsymbol{\\delta}_2 + \\mathbf{x}_2^\\top \\boldsymbol{\\delta}_1 - s(\\boldsymbol{x}_1^\\top \\boldsymbol{\\delta}_1 + \\mathbf{x}_2^\\top \\boldsymbol{\\delta}_2) + \\boldsymbol{\\delta}_1^\\top \\boldsymbol{\\delta}_2 \\right| + O(\\epsilon^2)$$  \nAfter projection onto the semantic subspace via matrix $\\mathbf{\\Pi}$:  \n$$\n\\Delta\\_\\Pi = \\left| \\cos(\\mathbf{\\Pi}(\\mathbf{x}\\_1 + \\boldsymbol{\\delta}\\_1), \\mathbf{\\Pi}(\\mathbf{x}\\_2 + \\boldsymbol{\\delta}\\_2)) - s \\right|\n\\leq |s - s\\_\\Pi| + \\epsilon \\left( \\frac{1}{\\|\\mathbf{\\Pi}\\mathbf{x}\\_1\\|} + \\frac{1}{\\|\\mathbf{\\Pi} \\mathbf{x}\\_2\\|} \\right)\n$$\n\nThis leads to:  \n$$\n\\frac{\\Delta_\\Pi}{\\Delta} \\lesssim \\frac{2\\epsilon}{2(1 + |s|)\\epsilon} = \\frac{1}{1 + |s|} < 1\n$$\nwhere $s = \\cos(\\mathbf{x}\\_1, \\mathbf{x}\\_2)$ and $s\\_\\Pi = \\cos(\\mathbf{\\Pi} \\mathbf{x}\\_1, \\mathbf{\\Pi} \\mathbf{x}\\_2)$. Thus, projection consistently reduces cosine distortion even when the adversarial perturbations differ."}]

Author rebuttal:

Meta-review: Contribution: it proposes a training-free method to enhance the robustness of CLIP models against adversarial images with two alignment strategies: global feature alignment and local structural alignment. It also employs an optimal transport-based framework to refine the correspondence between text and image embeddings. The method is evaluated on diverse datasets, and the results demonstrate that the proposed method significantly outperforms both state-of-the-art fine-tuned and training-free methods.

Ratings: the final scores are Accept (x2) and Borderline Accept(x2). Overall, the reviewers find the method to be technically solid, theoretically well-motivated, and empirically effective. So, it is a clear Accept.
Decision: Accept (spotlight)