Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices
Authors: Junhao Dong
Piotr Koniusz
Yifei Zhang
Hao Zhu
Weiming Liu
Xinghua Qu
Yew-Soon Ong
Venue: ICML 2025 spotlightposter (2025)
Keywords: Vision-Language Models
Adversarial Examples
Zero-Shot Classification
Primary area: deep_learning->robustness
TL;DR: We robustify VLMs by aligning entire adversarial simplices rather than individual adversarial samples with classifier scores of clean samples.

Abstract: Vision-Language Models (VLMs) such as CLIP excel at zero-shot classification due to large-scale pre-training but are vulnerable to adversarial examples. Adversarial fine-tuning robustifies zero-shot models by aligning prediction scores of individual adversaries with their clean counterparts, which typically overlooks intermediate adversarial samples along the adversarial trajectory crossing the decision boundary. Such intermediate adversaries and their vicinity produce informative representations capturing the decision boundary in detail. They can be improved by sampling adversarial candidates from simplices formed by joining two consecutive vertices on the adversarial trajectory and their clean counterpart. However, sampling simplices for adversaries is very costly. To train robust VLM, we overcome these limitations by Taylor expansion and formulating an upper-bound of alignment loss that depends on the Jacobian/Hessian obtained at clean samples. As regions between clean and intermediate adversarial samples capture a larger decision landscape, we robustify VLM by plausible adversaries from simplices by our closed-form formulation equivalent to infinite uniform sampling of the simplex. We obtain state-of-the-art robustness across 15 datasets and diverse vision-language tasks.

Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices
Junhao Dong* 12
Piotr Koniusz" 34
Yifei Zhang'
Hao Zhu
Weiming Liu
Xinghua Qu Yew-Soon Ong' 12
TeCuA
PMC
FARE
Abstract
ImageNNe
2+ 04.6.
Sample align
Vision-Language Models (VLMs) such CLIP excel at zero-shot classification due to large-scale pre-training but are vulnerable to adversarial ex- amples. Adversarial fine-tuning robustifies zero- shot models by aligning prediction scores of indi- vidual adversaries with their clean counterparts, which typically overlooks intermediate adversar- ial samples along the adversarial trajectory cross- ing the decision boundary. Such intermediate adversaries and their vicinity produce informative representations capturing the decision boundary in detail. They can be improved by sampling adversarial candidates from simplices formed by joining two consecutive vertices on the adversarial trajectory and their clean counterpart. However, sampling simplices for adversaries is very costly. To train robust VLM, we overcome these limi- tations by Taylor expansion and formulating an upper-bound of alignment loss that depends on the Jacobian/Hessian obtained at clean samples. As regions between clean and intermediate adver- sarial samples capture a larger decision landscape, robustify VLM by plausible adversaries from simplices by our closed-fome formulation equiva- lent to infinite uniform sampling of the simplex. We obtain state-of-the-an robustness across 15 datasets and diverse lasiun-anguuae tasks.
step
STL10
36.1
79.7
sample, natural sample
POAM 429
Mougp
(b) naive alignment
0.179
statistics align
Chestrigi44
735
$1A
step
J.
Palchen
Image Recogrition
Teat Ratrioall
adv. sample natural sample
Image Captioning
Medical Piagnods
Image Rerioval
(c) closed-form alignment
(a) robust accuracy
Figure I. Our AdvSimplex. Fig. Ia shows the zero-shot robust performance of our method (against adversarial samples) vs. ad- versarial fine-tuning approaches (TeCoA (Mao al., 2023). PMG (Wang et al., 2024), and FARE (Schlarmann et al., 2024)) across diverse downstream tasks, Fig. 1b shows the principle of sampling simplices formed from vertex x and consecutive intermediate ad- versaries x ò2,i and x + along the adversarial trajectory obtained with several steps of gradient ascent. However, sampling such simplices and aligning them individually with costly, Fig. Ic shows our model which is much faster as it computes closed-form E., used in our alignment formula based on Jacobian and Hessian (J, (Hz())).)
Goodfellow et al. (2015); Dong et al. (2024a) have shown their vulnerability to adversarial examples, which are low- level perturbations added to legitimate samples to elicit incorrect class predictions. Such adversarial vulnerabilities also affect Vision-Language Models (VLMs) (Zhang et al., 2022a; Zhao al., 2023a), posing concerns about deploying VLMs in real-life applications (Diaz-Rodrige et al., 2023).
1. Introduction
Despite significant advancements driven by Deep Neural Networks (DNNs) in various areas, Szegedy et al. (2014);
To counteract malicious adversaries, a growing body of re- search seeks to strengthen zero-shot adversarial robustness of VLMs through adversarial fine-tuning (Mao et al., 2023; Wang et al., 2024; Dong al., 2025), with a focus on CLIP- based architectures (Radford al., 2021). These methods predominantly align single adversarial predictions-deriveee from feature-level image-text cosine similarity-with ei- ther their benign counterparts or the ground-truth labels. However, such an alignment scheme overlooks the broader spectrum of underlying adversaries, especially intermediate
'Nayyang Technological University, Singapore CCentee for Frontier Al Research, IHPC, A*STAR, Singapore "Data61 & CSIRO, Australia o "Australian National Univer- sity *Zhejiang University. China "Bytedance, Singapore. *JD/PK junior/senior leads. PK is also in charge of theory. Cor- respondence to: Piotr Koniusz <piotr.konuszee data61. essiro.au> Yew-Soon Ong <asysong ntu.edu.sg>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
100
This is an image of [cal, dog, bird,... ...]
CLIP-T
cat
dog
bird
((eGG)) (FARE)
Rate
(TGGA)
prompts t
text tokenization
CE+ Softmax
image ar
8,, 6,,2 2,.....
Robust
Altack
86
CLIP-V
Parturbation Radiu (255.)
Partorbation Radius (255 s]
cantrnl Ep
(a) worst-case advorsary
(b) most transferable adv.
Figure 3. Evaluations on ImageNet for our AdvSimplex vs. TeCoA (Mao et al., 2023), PMG (Wang et al., 2024), and FARE (Schlar- mann et al., 2024). Fig. 3a shows robust accuracy against worst- case adversaries sampled from simplices formed as in Fig. la but at test time. The CE loss is used to choose the worst-case adver- sary. Fig. 3b shows the average transfer attack success rate for the most transferable adversaries. For each model, we choose the most transferable attack from the simplices of remaining models.
Figure 2. The pipeline of our AdvSimplex. -liss-wise prompts are formulated and passed via frozen CLIP-T (text branch in the blue box). Clean image x is passed via CLIP-V (vision branch the orange box). By the gradient ascent we obtain x + 8,,1.--. .......... x + 6a,m (intermediate) adversarial samples. The alignment process from Fig. lc is used to form simplices between vertex x and consecutive adversarial vertices pairs 82.1, x 85.++1) for i = I,..., 1. Next, efficient alignment is performed between x and all points on simplices to robustify the model.
the backbone to minimize
adversarial samples encountered along the adversarial tra- jectory obtained during iterative adversary generation (i.e.. the path that crosses the decision boundary). Although these intermediate adversaries and their variants encode rich infor- mation about the class boundaries, limited efforts have been made to incorporate them explicitly during adversarial fine- tuning due to additional computational cost. Consequently, such an oversight exposes VLMs to unforeseen adversaries.
min Σ
(()x(x), Exxxx -9 XEX
((()
(I)
Here, ((.--) can be cross-entropy (CE) loss, k= Ax| is the number of adversary candidates sampled from simplex associated with x. Thus, for dataset with 1M images, and = 10 and ascent steps 10, one obtains prohibitive 91 M images. Such naive setting is shown in Fig. 1b.
In this work, we exploit the disruptive effect of augmented or diversified adversaries (Wang et al., 2022a; Lu et al., 2023; Li & Spratling, 2023; Dong et al., 2024c; Gao et al., 2024). Li & Spratling (2023) explored data augmentation to improve adversarial diversity. Lu et al. (2023) obtained intermediate adversaries along the adversary generation tra- jectory to achieve cross-VI.M attacks. Gao et al. (2024) extended such mechanism to triangular region with the vertices of two consecutive adversarial samples and their clean counterpart. Wang et al. (2022a) exploited such geo- metric information search for adversarics in the black-box setting. However, the above works focus on conducting at- tacks with such adversarially diverse sets. Their associated computational cost makes them inapplicable to fine-tuning.
To train robust VLM, we overcome these limitations by a Taylor expansion of go (x around x and formulating an upper-bound of approximated alignment loss 22(-) that de- pends on the closed-form Σ, instead of naive computations of costly second-order matrix Σ, = Es, OA,6 We also require easily obtainable Jacobian and Hessian- vector product (J,,x), (H₂ (x)). p) of go(-) evaluated at a clean sample x. The entire pipeline of our model, called Adver- sarial Simplex (AdvSimplex), is shown in Figure 2. Fig. 1c illustrates our flosed-form formulation.
Fig. la demonstrates excellent zero-shot robustness of Ad- vSimplex against existing state-of-the-ar adversarial fine- tuning approaches on several benchmarks. Moreover, Fig, 3a shows an experiment where CE loss is used at the test time to select the most disruptive adversarial candidate sample from simplices sharing vertex x. Our approach is the most robust model against such adversaries. As TeCoA (Mao et al., 2023). PMG (Wang et al., 2024) and FARE (Schlarmann et al., 2024) perform poorly, this is an indirect proof that such "adversarial simplices" indeed contain adver- sarial samples. While their strength may differ, they clearly are harmful and thus can be used for robustification. Fig. 3b shows a similar experiment where the most disruptive candidate adversaries were sampled from simplices of all methods other than the tested method to verify cross-model
Thus, to effectively exploit the rich structure of decision boundaries and adversarial diversity of samples contained by simplices between a sample vertex and consecutive pairs of intermediate adversary samples, we depart from the tradi- tional wise alignment of prediction scores.
Specifically, Gao et al. (2024) sampled adversarial candi- dates from 2D simplices with vertices (x, x+6 x- for adversarial trajectory (with indices i = m-1) 1) ob- tained by the iterative gradient ascent, and used a small num- ber of such samples for attacks. However, explicit sampling of 2D simplices to robustify VLM is prohibitively costly. Further additional cost is due to passing such samples via
2

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
transferability. The figure shows that the attacks sampled from "adversarial simplices" are both strongly adversarial and highly transferable, highlighting their universality.
Table I. Summary of key symbols and notations with explanations.
Symbol
Explanation
X
Clean example.
6x
Adversarial perturbation. Predictions of the CLIP model.
In summary, our main contributions are as follows:
(e()
Jo(-)
Jacobian matrix of gel-) (Jacobians stacked for C' classes).
(H,())
Hessian matrix of go for class c.
1. In contrast to existing adversarial fine-tuning approaches that employ sample-wiss prediction alignment of ad- versaries to clean samples, we employ simplex regions formed from vertex x and consecutive adversarial pairs {(x + 6 6z,2, X + 6 62,++1); : 1,..,, m- m--1) } from the m- steps gradient ascent. While similar schemes were used to create attacks, due to their high computational cost due to sampling, they have not been used in robustification.
((-)
The Fuclidean alignment loss.
Q(-)
Upper bound of [i-) with the cross-product term.
((-)
Upper bound of [ti-) without the cruss product term.
counterparts or one-hot ground-truth labels. Given a set X, the standard adversarial fine-tuning approach (ie.. TeCoA (Mao al., 2023)) is formulated as minimax optimization:
2. To alleviate the computational burden of explicit sam- pling from "adversarial simplices", we derive an upper bound of alignment term !2() in Eq. (1) by the use of Taylor expansion, and a scalable upper bound that employs closed-form second-order statistic of points con- tained within simplices, equivalent to infinitely dense uniform sampling strategy. This alleviates the need to pass numerous candidate adversaries through the back- bone. Moreover, minimizing our upper bound is shown as minimizing the upper bound of robust risk.
min Ex
max Se
(go(( +.z), 52). v=)
where 6- an image-level adversarial perturbation con- strained within an -xomn ball of radius e, and ye repre- sents the class of x. The inner maximization of the cross- entropy loss generates adversarial examples by perturbing the prediction, while the outer minimization reduces the empirical risk over these adversaries. Following Mao et al. (2023); Wang et al. (2024), adversaries are generated by the iterative Projected Gradient Descent (PGD) (Madry et al., 2018), which updates perturbed input g(++1) =x ò2,t as:
3. We conduct experiments across 15 datasets and diverse scenarios (e.g., Fig. 1), showing that our method outper- forms state-of-the-arr adversarial fine-tuning approaches.
(V R ((o v.))]
X (((+1)= =Ia(xee))
x 4a
(4)
where denotes the step size, and sign(.) is the sign func- tion. Ils(x,c) ensures the perturbation remains within the -nom ball. Adversarial initialization begins with a ran- dom perturbation x0)) ~ x + 0.001 (O, After m iterations, the final adversarial sample & xmm) is obtained. The set of the final and intermediate adversarial samples is denoted as Ix {x()) m
2. Background
Related works are in Section 4. CLIP (Radford et al., 2021) enjoys great performance on zero-shot tasks. Its architecture consists of image and text encoders, parameterized by 0 and 0'. The image encoder fe X X Rd projects input images X' into d-dimensional feature space. Similarly, the text encoder for T +-RR" Rd maps input textual descriptions t E T into d-dimensional embeddings. By jointly encoding image-text pairs (x, t). CLIP aligns two modalities by the cosine similarity. The probability of assigning an image x to a specific category e E {l,...,C) C) is given as softmax:
fi=1'
Problem definition. Moving beyond traditional robust- ness evaluations on in-distribution adversarial examples (Croce et al., 2021), we address the more complex zero- shot robustness setting (Mao et al., 2023). In this sce- nario, adversarial examples are generated with unlimited access to data from previously unseen datasets during inference. The goal for defenders, including our ap- proach, is to maintain robustness against these novel security threats, despite having no prior exposure to such data. From a practical defense view, we assume that text prompts, stored multi-modal systems, remain unchanged during inference, so we do not attack text.
= exp (sim([J(x),,))))))) for (t.)))
(go(x))
(2)
Σ C i=1 exp (fos(f((x)) for (t.)))
where exp(-) is the exponential function, and sim(:.) represents the cosine similarity. Each text prompt t, =
" "[Context] J [CLASS]]" (e.g., "This is photo of a [CLASS.]") is tokenized and embedded by (-), serving as the alignment reference. The predicted proba- bilities of sample x for all C classes can be represented as go (x) = [[)(())) ... (oo (x) c]² E 1).
3. Proposed Method
Below, we introduce our proposed adversarial fine-tuning approach, enhancing zero-shot adversarial robustness. Our method leverages "adversarial simplices" formed from ver- tices (x, x x+6 for consecutive == 1,..., m-1
To enhance zero-shot robustness, adversarial fine-tuning adaptively integrates adversarial samples into the optimiza- tion process, aligning their predictions either with their clean
3

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
given the m-steps gradient ascent. Table lists common symbols and their explanations.
Here, vee(.) simply vectorizes the matrix-shaped input. As each argument of the above inner product contains (whh)* elements, this is prohibitive. Thus, we use upper-bound:
3.1. Upper-hounding s((x)
Σ 13(x, 6z)) 6xEAx
Below we start with Taylorexpansi of go (x+6x) around x, (e.g((++)) ==)(()()=)(()+++--)-))))())))(=) JJ (
1-n (115211 where (x) E R.xwh is the Jacobian matrix evaluated for ()(-) at vertex x. We also have C Hessian matrices
vec(ō ) @vec(8,))) Hy (x,
Ex
1..... (x)( E Rwhxwh where wh is width x height of an image. Notice that perturbations || Sc. Thus, we assume that for a sufficiently small 6, the remainder term O (11x=11)) of expansion around vertex x negligible. Moreover, observe that [g(x+ (5x)++,)000 5z)-g(x))12+ C
Σ
vec (0
Σ
vec(6_0) H, c)
6,EAx
5.EAx
Σ 626 (H1g())))
(10)
(x) + (H,, (x)) Sz1 )=1=_)
c= 2- so for a set Ax with
B_EAx
adversarial perturbations we obtain:
Σ,
=x
Σ
6, + 10 (H, 8m 12 (5)
Putting together Eq. (8) & (10), under negligible 0 19),
we readily obtain 2(xx) ((x))(((x))) {((x):
6
Σ
(
10 + 11 (x)g((x))) 8x 1 1
s2(;; E - (xx, (x(x (H, (x)()))) terms),
8, EAx
a(x,o,)
8(x,8o)
Gml
(Jz(x) 8z 18% (H₂ (x)x (x) 8,
(6)
C
f(x: ) 2(E [x e)) + (H, ()) c=1
(11)
7(x,62)
S E
B2EAx 8 2x 6z) + 1 5z).
(7)
Having developed two upper bounds of (x(). we now define a closed-form expression for Σn.
Eq. (7) is derived from inequality Jar + vl2 < 20al lal²) for p21 1 due to convexity. See Appendix C.I.
3.2. The oo-dense sampling of simplex (closed-form Σz)
The upper-bound in Eq. (7) does not solve our issue of aggre- gating over elements of the large set Ax. Notice (xx, 8z)=
To avoid aggregating empirical En over x elements of set A x, we propose the following theorem.
C (6.. (J,( Σ
(8.0% (J, (1,(
Theorem 3.1. The closed-form expression for Σ, E ppp"] over all p in simplex with vertices (x. y, z) is
As the inner product is linear in its argument, we have:
Σ = (x= (x+y+ T+ 12 (xxx= yy zzz
Σ 2a(xx6,)==2 6.EAx
6m)=22 2
Σ 5,0 (1)
(12)
EAEAx
Moreover, let Q be the number of vertices (Zı..... zQ) of simplex. The closed-fore expression for Σ, - E pp for higher-order simplices, e.g., tetrahedron (Q = 4 vertices) or pentachoron (Q =5 vertices) is given as:
Jg(x,e)
(8)
Σ.
We have now arrived at Eq. (8) which depends on ΣS (later to be replaced with closed-form Ex) and (x). The case of expanding 3(x, dz) is more complex and thus we will upper-bound this term. Notice that 8(x,6x)=
Σ Q(++))
Z +
1=1 z
Z
1=1
i=1
(13)
(z) E ( (5,5,((1((x)))
Proof. Parameterize point E Ruwh over vertices (x, y,
(2(x)
as p ox
2 0, 0,a+8+7=1 Expand
pp" T = (xxxx
and note that E pp
However, 2 is not linear in its arguments, yet can replace squaring by the Kronecker product @ as follows:
E a² xx xx y 2zz +E (xy +yx"""+x""/ )
E B[a7]] + )+ (yz +y For a uniform
(x 6..) (o)==(vver (vec(( vvec(( (x) ) (HL) (x)).
distribution on the simplex {a, 8, 2 0, a+8
we have [[]]=E[$] [[[]]=E[]]+ E[a?] " -E
(9)
6 E[a = =[an]] = =[[ 12. Substitute expectations
into the expansion to conclude the proof for Q = 3.
4

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
For higher-order simplices (i.e., Q > 3), one can parame- terize p = Σ -i=1 0:24, 03 2 0. Σ 0g = 1, and obtain: B[a.] - Φ. _
where for say the 0-1 loss ((.-), (g) (the natural risk) is the probability of misclassification on clean data, Reob(g) (g) (the robust risk) is the probability that the strongest perturbation in | < will cause misclassification. Risonnaary (g; e) quantifies the fraction of points "within é" of the classifier's s decision boundary, i... the set of points that can be flipped by a perturbation of size e. The prior knowledge also states that the following bound holds Reub (g) S Reat (g) Rooundaay (g: c)-
2 Q' and EL [aaaa = Q(+4)) i j for the underlying Dirichlet distribution. Then one pands EE [pp"" for pp" 1
Eq. (12) requires mere aggregation over four outer products of vectors. As long as >> 4, using the closed form is more efficient and equivalent to evaluating for x oc set Ax.
For efficiency, we use the Hessian-vector product (HVP) in i and 22, i.e., (Hg p) as follows:
For single adversarial simplex, our boundary risk becomes:
: Rioundary (g: Ax) == Px 330 Ax (x)
(E [pp" ] H = E [p" (Hg p)]
(14)
(20)
Q Σ
Q[++T
Σ T z
HHg- zi)) +
Zi
H
2
We can upper-bound this risk by our defined boundary counter risk, which not only captures decision flips on x but also counts in how many different ways x can be perturbed to cause a decision flip:
1=1
Thus, we set (E,, (H, (H,( ( ((Hg()))
p))
For simplex with Q =3 3 vertices (one vertex equals 0), this requires only three Hessian- vector products. One HPV evaluation costs the same as 2-4 Jacobian evaluations.
Reounter (g; Ax): -x EE g(x)+9(x++)
(21)
3.3. Our loss function
It is clear that Roounter Iz) S Reounter (g; Ax) where Is {20) )mm is the set of the final and intermediate adversarial samples of x. As the following holds
Following Fig. Ic, we have to align set of m-1 simplices with the Jacobian and Hessian statistics. Thus, our loss takes the form below:
Reob s Ruat (g) Rooundary (g; Ax)
(22)
min Σ (o)) + Σ (x) {2x; Ex,e), XEX
S Ruat (g)= + Reounter (g; Iz)
(23)
(15)
2 Raaa (g) + Reounter (g; Ax),
(24)
and Σx, is evaluated on vertices (0, dg. 8 12.+11) by Eq. (12), x 220 O controls the impact of minimizing the upper bound. Additionally, we can reweight the impact of each simplex by simple perturbance impact measure of inter- mediate adversarial vertex x0() - ~z,
we are optimizing the upper bound of the robust risk, taking into account counts of successful perturbations per x.
4. Related Works
(x(x) + (g0 - (90(x(1))
(16)
Prediction alignment. As a means of enforcing consis- tency between model outputs, prediction alignment is widely adopted in machine learning. Originally explored in knowl- edge distillation (Hinton et al., 2015), where the student network is trained to align its soft predictions with those of teacher, this concept has since been extended to various domains, including semi-supervised learning (Sohn et al., 2020), unsupervised learning (He et al., 2020), and domain adaptation (Tzeng et al., 2017). Noteworthy are also fea- ture alignment-based domain adaptation (Tas & Koniusz, 2018 8), few-shot detection and segmentation (Zhang et al., 2022b; Kang et al., 2023; Lu al., 2024), contrastive learn- ing (Zhang et al., 2025), and misalignment- based anomaly detection (Ding et al., 2025).
where 7= max (x))x;)) May (go (x) (x())) Bay
simply mea- sures the biggest perturbance for a batch B of samples x.
jEB
3.4. Bounding the robust risk
Generally, there exists an inevitable trade-off between nat- ural performance and adversarial robustness (Wang et al., 2024). Below, we study the implications of our design for robust risk (Zhang et al., 2019). Generally, the following three risks are known in adversarial learning:
Reat (9) : Ex Ex ((((x), vz)),
(17)
Rrob(g) == E. Exxxx max ((((x+5z) 0z), w)
(18) In the context of VLMs, prediction alignment plays a criti- cal role in promoting modality consistency and label-space agreement, where recent works leverage alignment losses to (19) improve cross-modal generalization (Jia et al., 2021; Yang
.10..1sc
Reoundary (g: e): : ==Ipx= 3d2: ||6,0 |e # g(x++xx))) 8m.)
5

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Table Zero-shot clean accuracy (%), Adversarial fine-tuning is conducted on ImageNet with evaluations across 15 datasets.
ImagoNet
CIFAR-10
CIFAR-100
SUN39T
Stanf.Cars
Food101
OxfordPet
Flower102
EuroSAT
Caltech101
Caltech256
STL1O
FGVC
PCAM
DTD
Method
Average
Standard CLIP
59.13
97.17
88.55
62.29
57.68
52.07
83.84
87.35
65.60
40.05
38.31
20.13
52.26
87.08
82.01
64.90
TeCoA
54.43
91.10
72.77
41.31
44.71
22.06
39.27
75.06
38.36
29.46
22.91
10.55
42.37
77.11
70.91
48.83
PMG-FT
51.33
90.70
73.05
42.04
44.40
27.72
42.61
75.39
39.37
29.02
20.32
11.45
47.22
80.08
71.01
49.71
FARE
50.94
93.90
81.98
56.25
49.94
42.73
63.30
81.59
53.29
34.27
21.53
14.66
45.04
85.72
75.03
56.68
AdvSimplex
61.28
96.30
87.14
59.05
53.61
43.86
69.83
84.71
56.28
37.08
23.27
16.52
48.57
87.33
78.65
60.23
Table Zero-shot robust accuracy (%).
Adversarial samples are created using the PGD-20 attack method with the image-level 2/255. Adversarial fine-tuning is conducted on ImageNet with evaluations across 15 datasets.
perturbation radius
ImageNet
CIFAR-10
CIFAR- 100
SUN397
Stanf.Cars
Food101
OxfordPet
Flower102 102
EutSAT
Caltech101
Caltech256
STL10
FGVC
PCAM
DTD
Method
Average
Standard CLIP
0.54
21,38
2.21
0.73
0.35
0.20
6.11
2.99
0.54
0.22
0.03
0.00
0.00
13.60
8.81
3.85
TeCoA
27.03
71.60
44.34
23.13
18.67
5.13
13.29
42.33
16.29
17.52
12.28
2.52
13.47
56.82
45.52
27.33
PMG-FT
26.20
71.91
45.72
23.41
18.87
6.89
14.62
43.14
16.72
18.02
12.59
2.73
21.39
58.25
46.21
28.44
FARE
24.57
77.28
52.91
30.37
17.85
9.66
18.20
46.26
18.46
19.53
10.24
2.82
23.36
62.95
49.59
30.94
AdySimplex
36.48
79.67
57.24
32.60
20.33
11.46
20.67
56.11
19.75
20.76
13.19
4.94
42.93
66.23
52.88
35.68
et al., 2022; Dong et al., 2025). Departing from conventional point- wise alignment, our method introduces robust align- ment over adversarial path simplices (sets of adversarial points), promoting stronger robust generalization.
ment that integrates a single adversarial counterpart per clean sample, overlooking the broader spectrum of plausible adversaries in the vicinity of the decision boundary, thus compromising zero-shot robustness against unforeseen ad- versaries. In contrast, we "incorporate" entire "adversarial simplices" into the robustification process.
Uni-modal adversarial robustness. The growing use of DNNs in both vision and language tasks (Khan et al., 2022: Wang et al., 2022b; Hu et al., 2024; Zhao & Zhang, 2024) has heightened awareness of their vulnerability to adver- sarial inputs, stimulating research on defense mechanisms (Bai al.. 2021; Aldahdooh et al., 2022: Xie & Yuille, 2020), among which adversarial training is very effec- tive paradigm. By iteratively optimizing models against worst-case perturbations, it enhances robustness under at- tacks (Madry et al., 2018; Zhang et al., 2019; Dong et al., 2024d;b). In this paper, we extend adversarial robustness in the context of multi-modal zero-shot generalization.
5. Experiments
Below, we provide our experimental configurations and present our comparisons between our AdvSimplex and other adversarial fine-tuning models across 15 datasets.
Datasets. We adopt the setup from Mao et al. (2023); Wang et al. (2024); Schlarmann et al. (2024), where CLIP is adver- sarially fine-tuned on the ImageNet training set (Deng et al., 2009). Then we assess the zero-shot results of the fine-tuned CLIP on the ImageNet val set and 14 novel datasets. We also investigate our method in medical image analysis and vision-text understanding. See settings in Appendix B.I.
Multi-modal adversarial robustness. The computational cost impedes scaling large VLMs such CLIP (Rad- ford et al., 2021). To address this limitation, adversarial fine-tuning (Mao et al., 2023), often leveraging Parameter- Efficient Fine-tuning (PEFT) (Jia et al., 2022; Zhou et al., 2022; Ni et al., 2024; Zhu et al., 2025), has attracted in- creasing attention. Mao al. (2023) introduced adversarial fine-tuning through text-guided contrastive learning, align- ing image-text embeddings for adversarial robustness. To mitigate the potential over-fitting to fine-tuning datasets, Wang et al. (2024) designed prediction-level regulariza- tion guided by natural CLIP, while Schlarmann et al. (2024) proposed an unsupervised adversarial framework. Despite their efficacy, existing approaches adopt point-wiee wise align-
Implementation details. Unless specified otherwise, we use CLIP (Radford et al., 2021) based on ViT-Base/32 (Dosovitskiy et al., 2021), as per studies (Mao et al., 2023; Wang et al., 2024; Schlarmann et al., 2024). For adver- sary generation during fine fine-tuning, we employ PGD (Madry et al., 2018) with m 10 iterations under the eoomnorm threat model, the perturbation radius e= 2/255 and the step size a 1/255. The weighting factor is set to X = 0.6. During evaluations, we assess the natural and robust per- formance under three strong white-box adversarial attacks: PGD (Madry et al., 2018) with 20 iterations, CW (Carlini
6

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Table Comparison of our method with other approaches, report-
Table 6. Comparison of our method with other approaches, report- ing average accuracy (%) on text-level and bi-level adversaries.
ing average accuracy (%) across diverse CLIP architectures.
Architecture
Method
Clean
PGD
CW
AA
Method
Text-Level Attacks
Bi-Level Attacks
TeCoA
48,83
27.33
26.80
25.75
BERT-Attack
GBDA
Co-Attack
SGA
ViT-B
PMG-FT
49.71
28.44
27.63
26.98
TeCoA
37.14
35.30
26.73
25.94
FARE
56.68
30.94
30.26
29.30
PMG-FT
37.61
36,46
28.11
27.85
AdySimplex
60.23
35.68
34.93
34,06
FARE
35.45
34.97
25.38
25.06
TeCoA
70.95
54.19
50.84
47.96
AdvSimplex
40.21
39.88
32.95
32.53
ViT-L
PMG-FT
68.67
55.75
$2.29
49.38
FARE
71.30
57.12
54.16
50.54
AdvSimplex
73.39
59.52
57.64
52.80
Table 7. Average accuracy (%) using adversaries of varying é for both fine-tuning and evaluations (ViT-B) with the VPT strategy.
TeCoA
43.25
23.20
22.53
21.74
ResNct-50
PMG-FT
45.11
24.68
23.72
23.05
Perturbation Radius
Method
Clean
PGD
CW
AA
FARE
45.43
24.27
23.19
22.24
AdySimplex
48.85
26.23
25.38
24.70
TeCoA
51.00
32.27
31.11
30.26
= 1/255
PMG-FT
52.64
33.09
32.10
30.83
FARE
52.75
32.69
31.58
30.64
AdvSimplex
55.08
34.96
33.75
33.16
Table Comparison of our method with other adv. fine-tuning approaches. Average accuracy (%) under diverse e are reported.
TeCoA
42.61
18.12
16.88
15.39
4 = 2/255
PMG-FT
42.11
19.26
17.68
16.47
Perturhation Radius
Method
PGD
CW
AA
FARE
42.81
18.98
17.46
16.35
AdvSimplex
44.70
22.57
21.13
20.28
TeCoA
38.10
36.62
35.83
q 1/255
PMG-FT
FARE
45.40
38.92
37.45
44.81
43.27
36.98
q = 3/255
PMG-PT
TeCoA
FARE
32.52
33.86
33.70
12.47
12.87
12.32
10.78
10.92
11.36
9.04
8.89
9.38
AdvSimpler
48.12
46.94
45.61
AdvSimplex
35.11
15.38
14,42
11.96
TeCoA
27.33
26.80
25.75
TeCoA
26.78
11.04
9.87
7.19
2/255
PMG-PT
28.44
27.63
26.98
q = 4/255
PMG-FT
23.57
11.73
10.01
7.26
FARE
30.94
30.26
29.30
FARE
26.17
11.49
10.32
7.53
AdvSimplex
35.68
34.93
34.06
AdvSimplex
29.23
14.25
12.63
10.22
TeCoA
17.90
17.51
17.08
= 3/255
PMG-FT
19.14
18.69
18.22
FARE
19.31
18.84
18.47
AdvSimplex
25.68
24.80
24.34
Adversarial fine-tuning w.r.t. diverse perturbation radii.
TeCoA
11.23
10.70
10.35
Table 5 shows the average clean and robust accuracy across 15 datasets w.r.t. various perturbation radii. Our AdySimplex consistently surpasses other adversarial fine-tuning methods when facing stronger adversaries in zero-shot settings.
9 = 4/255
PMG-FT
12.05
11.56
11.17
FARE
12.42
11.93
11.52
AdvSimpler
18.63
18.10
17.58
& Wagner, 2017), and Auto-Attack (AA) (Croce & Hein, 2020). All evaluations use adaptive attack schemes for fair comparison. See implementation details in Appendix B.2.
Robustness on text- and bi-level attacks. In addition to image-level attacks, we assess text-level and bi-level attacks. Table 6 reports robust accuracy under these settings. Text-level attacks are evaluated using BERT-Attack (Li et al., 2020) and Gradient- Based Distributional Attack (GBDA) (Guo et al., 2021), while bi-level adversaries are tested using Collaborative Multi-modal Adversarial Attack (Co-Attack) (Zhang et al., 2022a) and Set-level Guidance Attack (SGA) (Lu et al., 2023). Our AdvSimplex outperforms existing adversarial fine-tuning methods on all attack types.
5.1. Main Results
Evaluations across 15 datasets. We compare our AdySim-
plex with TeCoA (Mao et al., 2023), PMG-FT (Wang et al., 2024), and FARE (Schlarmann et al., 2024) in Tables 2 & 3, where we also provide zero-shot inference on additional 14 datasets, reporting natural performance and robustness against 20-step PGD attacks. AdvSimplex consistently out- performs other models in clean accuracy, with an average im- provement of 3.5%, thus approaching standard CLIP (Table 2). While the standard CL.IP has nearly zero adv. robustness (Table 3), our AdvSimplex enjoys an average improvement of 4.7% across all datasets in comparison to FARE.
Efficient fine-tuning with VPT. Fine-tuning the full param- eter space is computationally expensive for VLMs. Thus, we investigate adversarial fine-tuning with Visual Prompt Tuning (VPT) (Jia et al., 2022), parameter-efficient strat- egy with learnable parameters in the token embedding layer. Table 7 shows the zero-shot performance of AdvSimplex under various adversarial configurations, comparing it with other techniques that also use VPT. Our AdvSimplex, even when using VPT for efficiency, outperforms previous works.
Ad Adversarial fine-tuning of diverse architectures. Table 4 reports zero-shot results various clip architectures for clean samples and adversarial counterparts across three at- tack types (E= 2/255): PGD (Madry et al., 2018) (20 steps), CW (Carlini& & Wagner, 2017), and Auto-Attack (Croce & Hein, 2020). Our AdvSimplex outperforms other adversarial fine-tuning models under all architectures.
5.2. Extensions to Other Architectures and Tasks
BLIP: Vision-text Understanding. Beyond standard CLIP (Radford al., 2021), we further explore zero-shot robust-
7

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Table BLIP Extension for Vision-Text Understanding. Eval- uations clean and PGD-20 adversarial samples. TR and IR represent the recall@1 for text and image retrieval, respectively. CIDEr measures the similarity of a generated sentenee against set of ground truth sentences for image captioning evaluations.
Table 10, Average clean and robust accuracy (%) of our AdySim-
plex method vs. AdvTetrahedron across 15 datasets.
Configuration Clean
PGD)
AA
AdvSimplex
60.23
35.68
34.06
AdyTetrahedron
60.80
36,49
35.17
Image- Text Retrieval
Image Captioning
Method
Clean TR
Robust
Clean
Robust IR
Clean CIDEr
Robust CIDEr
Table 11, Average Performance (%) of our AdvSimplex method using different configurations of the derived upper bound f2
TR
IR
TeCoA
87.5
54.4
77.0
47.5
96.9
57.8
Contiguration
Clean
PGD
AA
PMG-FT
87.8
55.6
77.9
48.2
97.5
58.2
FARE
88.2
55.9
78.4
49.0
98.1
58.7
w/o cross product term (i)
59.97
34.92
33.19
AdxSimplex
91.7
58.7
80.9
51.6
99.5
63.4
w/ cross-product term (9)
60.23
35.68
34,06
Table 9. Medical CLIP Extension for Diagnosis. AUC score evalu- ations on both clean and PGD-20 adversarial samples.
Table 12, Ablation study of three components in our method for average clean and robust accuracy (%) on 15 datasets.
Method
ChestXray14
CheXpert
PadChest
i Re-weighting (Eg. (16))
Clean
PGD
CW
AA
Clean
PGD
Clean
PGD
Clean
PGD
55.79
30.24
29.50
28.86
TeCoA
0.674
0.526
0.857
0.685
0.602
0.483
59.45
34.38
33.64
32.96
PMG-FT
0.692
0,538
0,850
0.688
0.619
0.495
/
56.92
33.17
32.49
31.72
FARE
0.687
0.533
0.845
0.679
0.615
0.490
/
60.23
35.68
34.93
34.06
AdvStmplex
0.742
0.579
0.880
0.735
0.632
0.563
Higher-order simplices. According to Theorem 3.1 for simplices with Q > 3 vertices, i.e., a tetrahedron (Q = 4 vertices), Table 10 shows gains for (x, x+6 x+0 x+ 2x,1+2) for consecutive == 1......, m - 2 simplices from gradient ascent steps over AdvSimplex (Q- vertices).
ness under alternative VLM architectures and downstream tasks using BLIP (Li et al., 2022), which combines multi- ple Dision-language understanding tasks. We consider two cross-modal tasks: (i) image-text retrieval using Flickr30k (Plummer et al., 2015), and (ii) image captioning using No- caps (Agrawal et al., 2019). For adversarial fine-tuning, we adversarially optimize the Image-Text Contrastive (ITC) learning, Image-Text Matching (ITM), and Language Mod- cling (LM) modules (Li et al., 2021), instead of performing alignment in Eq. (3). Table 8 shows zero-shot results on clean samples and
Cross-product term in the upper bound f(xx). Recall that we have cross- pross-product term y (last term in approx. 2(x) in Eq. (6)) for a better estimation of the upper bound of the prediction gap. Table 11 analyzes its impact on ad- versarial fine-tuning. A more precise estimation with the cross-p poss--product term improves the zero-shot adversarial ro- bustness. The cross-term product is derived using Kronecker operations, resulting in third-order statistics. Such statistics also have closed-form solution, and the dimensionality of tensors is sensor-sketched to keep calculations fast (Wein- berger et al., 2009). See Appendix C.3 for derivations.
adversarial attacks (20 steps) with c= 1/255, where AdvSimplex enjoys great adaptability.
Medical CLIP: Medical Diagnosis. Below, we investigate AdySimplex in the medical imaging domain, where adver- sarial threats may affect computer-aided diagnostics (Zhao et al., 2023b). We employ a radiology-oriented CLIP vari- ant under the CheXzero paradigm (Tiu et al., 2022) with a ViT-B backbone, again applying adversarial fine-tuning. We measure zero-shot performance on three multi-label ra- diology benchmarks: ChestX-ray 14 (Wang et al., 2017), CheXpert (Irvin et al., 2019), and PadChest (Bustos al., 2020). We report the Area Under the Curve (AUC) on clean and adversarial samples, where adversaries are generated via 20-step PGD under c= 1/255. Table 9 shows that our method consistently obtains higher AUC scores than prior approaches in clean and adversarial samples. Our method enjoys superior robustness on PadChest, which includes 192 disease categories and numerous uncommon pathologies.
Impact of each module. Below we ablate two key com- ponents of AdvSimplex: (i) Ω and (ii) weighting in Eq. (16). Table 12 shows that our baseline (first row) follows the surrogate optimization of robust risk (i.e., TRADES (Zhang et al., 2019)) by extending the point-wise clean- adversarial prediction alignment with intermediate adver- saries. Despite its simplification, our haseline approach al- ready achieves competitive performance compared to prior adversarial fine-tuning methods. Incorporating "adversar- ial simplices" yields further gains in both clean and robust accuracy. Our adaptive re-weighting emphasizes that not every simplex is equally adversarial.
Accuracy-robustres trade-off. Striking balance be- tween natural performance and adversarial robustness is known from uni-modal adversarial learning (Zhang et al., 2019; Dong et al., 2023a). Below, we explore it in the multi-
5.3. Further Analyses
Below, we analyze the effectiveness and generalizability of our AdvSimpeex across diverse settings.
8

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Table 14, Average robust accuracy (%) for closed-form "adver- sarial simplices" vs. sampling from "adversarial simplices" (70 samples per simplex) evaluated on attack samples with index num- ber i= 6, 8, 10, 12, 14 generated via PGD-20.
Clean Accuncyy (50
Robuat Acouncy (%i)
Robust Accurae (18)
X
Intermediate Adv. Sample Index (Attack)
Optimization Strategy
37
Asheourid Berpl
8-th
10-th
12-th
14-th
Coees/ftm Upper Bound
51
Trade-off Factor
07
Average Training Time per Epoch (min)
200
400
900
1000
Closed-form simplex
Sampled simplex
38.61
35.66
37.89
35.18
34.84
37.22
36.50
34.47
34.19
36.13
(a)
(h)
Figure+ 4. Evaluations of the average performance (%) across 15 datasets. (a) Tradc-off' between zero-shot clean and robust accuracy by tuning A. (b) Comparison of our closed-form upper bound with adversarial sampling at varying sample sizes (the circle radius indicates the number of samples; for "closed-form" radius is oo).
Table 15. Average robust accuracy (%) on worst-case and most transferable adversaries sampled from "adversarial simplices".
Optimization Strateyy
2/255
4/255
WorssClas
Transfor
Woest-Case
Transfo
Sumpled simplex
85.
36.86
40.00
16.92
21.15
33.38
Closed-forma simplex
35.68
39.51
43.38
18.63
23.72
35.92
Table 13, Average Performance (%) of our proposed method with diverse re- weiieghtting strategies for adversarial triangular regions.
Re- Reighigg Strategy
Clean
PGD
AA
against adversarial attacks of various step numbers.
Uniform Weighting (No Weights)
59.45
34.38
32.96
Performance w.r.f sampled "adversarial simplices". In addition attacks along the generation path, we also eval- uate the adversarial robustness against both the worst-case and the most transferable adversaries sampled from "ad- versarial simplices". Following the setup from Figure 3, worst-case adversaries are obtained from the target CLIP model, and the most transferable adversaries from three other CLIP models. Table 15 shows that our derived closed- form upper-bound model enjoys greater robustness against adversaries from "adversarial simplices".
Linear Weighting (i/m)
58.74
35.12
33.48
Adaptive Weighting (Ours)
60.23
35.68
34.06
modal CLIP and zero-shot scenario. We analyze the effect of hyper-parameter X, which controls the relative weighting of clean sample classification vs. adversarial-clean prediction alignment. Figure 4a shows that increasing À enhances ad- versarial robustness yet reduces clean accuracy. Conversely, lowering À improves zero-shot performance on benign in- puts at the cost of reduced robustness. Such a trade-off stems from the optimization of natural and boundary risks.
6. Conclusion
Performance of sampling vs. closed-form solution. Recall that we introduce an efficient upper bound derived with a closed-form solution replacing sampling "adversarial sim- plices". Below, we compare our "'upper-oound closed-form model" against explicit sampling of "adversarial simplices" for the standard alignment loss. Figure 4b shows that al- though increasing the sampling amount leads to a gradual improvement in adversarial robustness, it stabilizes around 70 samples and saturates around 100 samples, posing sub- stantial computational training time of 13.6 hours per epoch. In contrast, our AdvSimplex requires merely 4.1 hours per epoch while attaining comparable robustness.
Motivated by our analysis of the robustness degradation against underlying adversaries from "adversarial simplices", we have uncovered that the point-wise prediction alignment in robust VLMs leads to weak robustness generalization. Thus, we have explored recent attack strategies to formulate simplices between clean vertex x and consecutive adversar- ial samples on the gradient ascent path. While sampling such simplices is prohibitive, and aligning such adversarial candidate points also prohibitive, one may reformulate the problem by minimizing an upper bound of the align- ment loss. Our upper bound employs closed-form statistics obtained from the vertices of simplices, the Jacobian and Hessian matrices. We only pass clean samples via the en- coder, reducing time complexity, and we achieve "infinite sampling" effect with our formulation during fine-tuning.
Re-weighting mechanisms for "adv. simplices". Below we compare the use of weights from Eq. (16) with (i) uni- form weighting vs. (2) linear weighting (i/m) that place greater emphasis on adversaries from later iteration steps. Table 13 shows that our Eq. (16) outperforms other variants.
A Acknowledgements
Closed-forme sampled "adversarial simplex". Below, we analyze the average robust accuracy of the closed-form "adversarial simplices" vs. sampled "adversarial simplices" (70 samples per simplex). We attack both methods with an index adversary from PGD-20. Table 14 shows that the closed-form solution enjoys greater zero-shot robustness
This research is supported by National Research Foundation, Singapore and Infocomm Media Development Authority un- der its Trust Tech Funding Initiative, the Centre for Frontier Artificial Intelligence Research, Institute of High Perfor- mance Computing, A Atarr and the College of Comput- ing and Data Science at Nanyang Technological University.

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Any opinions, findings, conclusions, recommendations expressed in this material are those of the author(s) and do not reffect the views of National Research Foundation, Sin- gapore, and Infocomm Media Development Authority. Piotr Koniusz and Hao Zhu are supported by CSIRO's Science Digital.
els: A review and experimental comparison. Artificial Intelligence Review, 2022. 6
Bai, T., Luo, J., Zhao, J., Wen, B., and Wang, Q. Recent ad- vances in adversarial training for adversarial robustness. In International Joint Conference on Artificial Intelli- gence, IJCAI, 2021. 6
Impact Statement
Bergstra, J., Yamins, D., and Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pp. 115-123. PMLR, 2013. 17
The advancement of Vision- Language Models (VLMs) has revolutionized zero-shot classification and a series of vision- language understanding tasks. However, the deployment of these models in real-world applications necessitates robust defenses against (unforeseen) adversarial attacks, which can significantly undermine their performance and trust- worthiness. Our proposed adversarial fine-tuning method addresses this critical security vulnerability by enhancing the robustness of VL.Ms against adversarial samples, ensur- ing more reliable performance in diverse scenarios. The improvement in zero-shot adversarial robustness can further lead to several positive outcomes as follows:
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6- 2014, Pro- ceedings, Part VI 13, pp. 446-461. Springer, 2014. 15
Bustos, A., Pertusa, A., Salinas, J.-M., and De La Iglesia- Vaya, M. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66: 101797, 2020. 8, 16
1. Research Advancement:
Our findings the corre- lation between adversaries on simplex and adversarial robustness provide insights for the broader research community related to robust cross-modal learning.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017. 6, 16
2. Enhanced AI Safety: By improving the robustness of foundational VLMs against unforeseen adversarial attacks, our method contributes to the development of more secure Al systems that can be safely deployed in sensitive applications, such autonomous driving, healthcare diagnostics, and security surveillance.
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber,
J., Tsipras, D.,, Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness. arXiv preprint arXiv:
2019. 15
Chen, J., Dong. J., and Xie, X. Mind the trojan horse: Image prompt adapter enabling scalable and deceptive jailbreaking. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2025. 15
3. Social Trust:
As AI systems become increasingly in- tegrated into daily life, ensuring their reliability and robustness is important. Our improvement of adver- sarial robustness on foundational models further helps build socictal trust in Al technologies.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 360033006163 00006-331,,00 2014. 15
References
Addepalli, S., Jain, S., et al. Efficient and effective aug- mentation strategy for adversarial training. Advances in Neural Information Processing Systems, 35:1488-1501, 2022. 17
Coates, A., Ng. A., and Lee, H. An analysis of single- layer networks in unsupervised feature learning. In Pro- ceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011. 15
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D.,, Parikh, D., Lee, S., and Anderson, P. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8948-8957, 2019. 8, 16
Croce, and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020. 7, 16
Aldahdooh, A., Hamidouche, W., Fezza, S. A., and Déforges, O. Adversarial example detection for dnn mod-
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein,
10

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
M. Robustbench: standardized adversarial robustness benchmark. In Proceedings of the Neural Information Processing Systems, 2021. 3, 15
Dong, J., Koniusz, P., Chen, J., Wang, Z. J., and Ong, Y.-S. Robust distillation via untargeted and targeted intermedi- ate adversarial samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28432-28442, 2024c. 2
Deng, J., Dong, W., Socher, R., Li, L.-J,, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248- 255. leee, 2009. 6, 15
Dong. J., Koniusz, P., Chen, J., Xic, X., and Ong. Y.-S. Adversarially robust few-shot learning via parameter co- distillation of similarity and class concept learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28535-285444 2024d. 6
Diaz-Rodríguez, N., Del Ser, Coeckelbergh, M., de Prado, E., and Herrera, F. Connecting
M. L., Herrera- the dots in trustworthy artificial intelligence: From ai principles, ethics, and key requirements to responsible ai systems and regulation. Information Fusion, 99:101896, 2023. 1
Dong. J., Wang, Y., Xie, X., Lai, and Ong, Y.-S. General- izable and discriminative representations for adversarially robust few-shot learning. IEEE Transactions on Neural Networks and Learning Systems, 2024c. 17
Ding, D., Wang, L., Zhu, L., Gedeon, T., and Koniusz, P. Learnable expansion of graph operators for multi- modal feature fusion. In The Thirteenth International Conference on Learning Representations, 2025. 5
Dong. J., Koniusz, P., Qu, X., and Ong, Y.SS. Stabilizing modality gap lowering gradient norms improve zero- shot adversarial robustness of vlms. In Proceedings of the 31 ACM SIGKDD Conference Knowledge Discovery and Data Mining V. 1, pp. 236-247, 2025. 1,
Dong. J. and Xie, X. Visually maintained image disturbance against deepfake face swapping. In 2021 IEEE Interna- tional Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2021. 15
Dosovitskiy. A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, ICLR, 2021. 6, 15
Dong, J.. Wang, Y., Lai, J.-H., and Xie, X. Improving adver- sarially robust few-shot image classification with general- izable representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022. 17
pp.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop. pp. 178-178. IEEE, 2004. 15
Dong, J., Moosavi-Dezfooli, S.-M., Lai, J., and Xie, X. The enemy of my enemy is my friend: Exploring in- verse adversaries for improving adversarial training. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2023a. 8
Gao, S., Jia, X., Ren, X., Tsang, I., and Guo, Q. Boosting transferability in vision-language attacks via diversifica- tion along the intersection region of adversarial trajectory. In European Conference on Computer Vision, pp. 442- 460. Springer, 2024. 2
Dong, J., Wang, Y., Lai, J., and Xie, X. Restricted black-box adversarial attack against deepfake face swapping. IEEE Transactions on Information Forensics and Security, 18: 2023b. doi: 10.1109/TFF..22233566777
15
Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In ICLR, 2015. 1
Dong, J., Yang, L., Wang, Y., Xie, X., and Lai, J. To-
ward intrinsic adversarial robustness through probabilis- tic training. IEEE Transactions on Image Processing, 32: 1862-38722 2023c. doi: 10.1 S0t.TT...2..3200533.3 17
Griffin, G., Holub, A., and Perona, P. Caltech-256 object category dataset. 2007. 15
Dong, J., Chen, J., Xie, X., Lai, J., and Chen, H. Survey on adversarial attack and defense for medical image analysis: Methods and challenges. ACM Computing Surveys, 57 (3):1-38, 2024a. 1
Guo, C., Sablayrolles, A., Jégou, H., and Kiela, D. Gradient- based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP, pp. 5747- 5757, 2021. 7, 16
Dong. J., Koniusz, P., Chen, J., and Ong, Y.-S. Adversarially robust distillation by reducing the student-teacher vari- ance gap. In European Conference on Computer Vision pp. 92-111. Springer, 2024b.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
on Computer Vision and Pattern Recognition, pp. 9729- 9738, 2020. 5
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009. 15
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019. 15
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang. J. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioin- formatics, 36(4):1234-12)) 2020. 16
Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Ad- vances in neural information processing systems, 34: 9694-9705, 2021. 8, 16
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint WwXiv:153.025533 02,31, 2015. 5
Hu, Z.,, Liu, Y., Zhao, J., Wang. S., Wang, Y., Shen, W., Gu, Q. Luu, A. T., Ng. S.-K., Jiang, Z., and Hooi, B. Lon- grecipe: Recipe for efficient long context generalization in large language models, 2024. 6
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning, pp. 12888-12900. PMLR, 2022. 8, 16
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y.. Ciurea-Ilcus. S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpan- skaya, K., et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33, pp. 590-597, 2019. 2019. 8, 16
Li, L. and Spratling, M. W. Data augmentation alone can improve adversarial training. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Li, L., Ma, R., Guo, Q. Xue, X., and Qiu, X. BERT- ATTACK: adversarial attack against BERT using BERT. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing EMNLP, pp. 6193-6202, 2020. 7, 16
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q.. Sung. Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916 PMLR, 2021. 5
Lu, C., Liu, Z., and Koniusz, P. OpenKD: Opening prompt diversity for zero- and few-shot keypoint detec- tion. In 18th European Conference on Computer Vi- sion, pp. 148-165, Berlin, Heidelberg, 2024. Springer- Verlag. ISBN 978-3-031-726544 978-3-031-72655-- 9.
Jia, M., Tang, I.., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022. 6, 7, 16
doi: 10.1007/
Johnson, A. E., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., and Horng, S. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Sci- entific data, 6(1):317, 2019. 16
Lu, D., Wang, Z., Wang. T., Guan, W., Gao, H., and Zheng, F. Set-level guidance attack: Boosting adversarial trans- ferability of vision- language pre-training models. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 102-111, 2023. 2, 7, 16, 17
Kang, D., Koniusz, P., Cho, M., and Murray, N. Dis- tilling self-supervised vision transformers for weakly- supervised few-shot classification & segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023. 5
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. 3, 6, 16
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft, arXiv preprint arXiv: 130655330 2013. 15
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., and Shah, M. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1141 2022. 6
Mao, C., Geng, S., Yang, J., Wang, X., and Vondrick, C. Understanding zero-shot adversarial robustness for large- scale models. In The Eleventh International Conference on Learning Representations 2023. 1, 3, 6, 15, 16
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object rep- resentations for fine- fine-graing grained categorization. In Proceed- ings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013. 15
12

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Ni, Y., Zhang. S., and Koniusz, P. PACE: Marrying general- ization in parameter-efficient fine-tuning with consistency regularization. In Globerson, A., Mackey, L., Belgrave, D.. Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp. 61238-61266. Curran Associates, Inc., 2024.
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng. A. Y., and Rajpurkar, P. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12):1399 1406, 2022. 8, 16
Tzeng, E., Hoffmann, J., Saenko, K., and Darrell, T. Adver- sarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 7167-7176, 2017.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant cnns for digital pathol- ogy. In Medical Image Computing and Computer As- sisted Intervention Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp- 210-218. Springer, 2018. 15
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012. 15
2018: 21st International
Wang, S., Zhang, J., Yuan, Z., and Shan, S. Pre-trained model guided fine-tuning for zero-shot adversarial robust- ness. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2024. 1, 2, 3, 5, 6,7, 15, 16
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C.,, Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-tur coainnto-phhee to-phrase correspondences for richer image-to-sentence models. Proceedings of the IEEE international conference on computer vision, pp. 2641- 2649, 2015. 8, 16
Wang. X., Peng, Y. L., Lu, Z.,, Bagheri, M., and Sum- mers, R. M. Chestx-ray8: Hospital-scall chest x-ray database and benchmarks
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763. PMLR, 2021. 1, 6,
weakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-210900 2017. 8,
7, 15
Wang, X., Zhang, Z., Tong, K., Gong. D., He, K., Li, Z., and Liu, W. Triangle attack: query-efficient decision- adversarial attack. In European conference on compuler vision, pp. 156-174. Springer, 2022a. 2
Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Ro- bust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In Proceedings of the 41 International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 33685-43700 PMLR, 21-27 Jul 2024. 1, 2, 6, 7, 15, 16
Wang. Y., Zhang, Q., Lai, J., Xie, X., and Dong, J. Learn- ing bi-directional feature propagation with latent layout modeling for group re-identification. In 2022 26th Inter- national Conference on Pattern Recognition (ICPR), pp. 903-913. IEEE, 2022b.
Sohn, K., Berthelot, D., Carlini, N., Zhang. 7z, Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.-L. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural informa- tion processing systems, 33:596-608 6086666087770 2020.
Weinberger, K., Dasgupta, A., Langford, J., Smola, A., and Attenberg, J. Feature hashing for large scale multitask learning. Proceedings of the 26th annual international conference on machine learning, pp. 11 1113-1121 2009. 8
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. In ICLR, 2014. I
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralha, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010. 15
Tas, Y. and Koniusz, P. CNN-based action recognition and supervised domain adaptation on 3d body skeletons via kernel feature maps. In British Machine Vision Confer- ence 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, pp. 158. BMVA Press, 2018.
Xie, C. and Yuille, A. L. Intriguing properties of adversarial training at scale. In 8th International Conference on Learning Representations, ICLR, 2020.
13

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L.,, Zeng. B., Chilimbi, T., and Huang. J. Vision-language pre-training with triple contrastive learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision 2022. 5
Zhao, Z., Liu, Y., Wu, H., Li, Y., Wang, S., Teng, L., Liu, D., Li, X., Cui, Z., Wang, Q. et al. Clip in med- ical imaging: A comprehensive survey, arXiv preprint arXiv:2312.0735, 2023b. 8
and Puttern Recognition, pp.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning prompt for vision-language models. International Jour- nal of Computer Vision (IJCV), 2022.
Yu, L., Zhang, H., and Xu, C. Text-guided attention is all you need for zero-shot robustness in vision-language models. In Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. 18
7hu, H., Zhang, Y., Dong, J., and Koniusz, BiLoRA: Almost-orthogonal parameter spaces for continual learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 6
Zhang, H., Yu, Y., Jiao, J., Xing, E.. El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019. 5, 6, 8, 17
Zhang, J., Xu, X., Han, B., Niu, G., Cui, I.., Sugiyama, M., and Kankanhalli, M. Attacks which do not kill training make adversarial learning stronger. In International con- ference on machine learning, pp. 11278-11111 1287. PMLR, 2020. 18
Zhang. J., Yi, Q.. and Sang. J. Towards adversarial attack on vision-late Lanou-la pre-training models. In Proceedings of the 30th ACM International Conference on Mullimedia, pp. 5005-5013, 2022a. 1, 7,7, 16
Zhang, S., Murray, N., Wang. L., and Koniusz, P. Time- rEversed DiffusioN tEnsor Transformer: A New TENET of Few-Shot Object Detection. In 17th European Confer- ence on Computer Vision, pp. 310-328, Berlin, Heidel- berg, 2022b. Springer-Verlage ISBN 978-3-031-20043-4, 18. 5
doi: 10. 10.11077/97---031-22000
Zhang, Y., Zhu, H., Yang. M., Liu, J., Ying, R., King, I, and Koniusz, P. Understanding and mitigating hyperbolic dimensional collapse in graph contrastive learning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD '25, pp. 1984-1995, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 97984007 12456. doi: 10. 10.11445/6696223309999... 5
Zhao, J. and Zhang, X. Large language model is (multilingual) compositional relation reasoner. In First Conference on Language Modeling, 2024. 6
Zhao, Y., Pang, T., Du, C., Yang, X., Li, C., Cheung, N.- M. M., and Lin, M. On evaluating adversarial robustness of large vision- laiguaee models. Advances in Neural Information Processing Systems, 36, 2023a.
14

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Appendices.
A. Limitations.
Although method yields substantial gains in zero-shot adversarial robustness, supported by both empirical results and theoretical analysis, we acknowledge two primary limitations that suggest avenues for future improvements:
1. Computational Overhead: While our
strategy and subsequent optimization via the closed- upper save time during fine-tuning over naive sampling of simplex, this cost would still benefit from further reduction. It is worth noting that the inference stage does not incur extra overhead relative to previous methods.
2. Implicit Trade-off Using PEFT:
In this work, language models. According to Tables 5 and 7, it is evident that although Parameter- Efficceett Fine-Tuning (PEFT) provides improved training efficiency, it can also degrade zero-shot performance on both clean and adversarial data compared to full fine-tuning. This performance-efficieney trade-off' affects all adversarial fine-tuning methods, and we anticipate that future advancements in PEFT will help minimize the robustness gap compared to full fine- one-tuning.
primarily concentrate adversarial full fine-tuning of vision-
3. Positive Applications of Adversarial Examples: While our work primarily focuses on defending against adversarial examples and mitigating their associated threats, adversarial examples can also serve constructive purposes. For instance, they are widely used as robustness benchmarks to evaluate model reliability under distribution shifts (Carlini et al., 2019; Croce et al., 2021), and as tools for privacy protection (Dong & Xie, 2021; Dong et al., 2023b; Chen et al., 2025). A more comprehensive treatment of these positive applications is beyond the scope of this work, but we consider an important direction for future research.
B. Experimental Configurations.
In this section, we provide a comprehensive overview of the experimental configuration used throughout our work, including both dataset specifications for adversarial tin-tunnin/eevaluation and the implementation details of our proposed method.
B.1. Dataset descriptions
We follow prior works (Mao et al., 2023; Wang et al., 2024) to adversarially fine-tune the CLIP model on the ImageNet training split (Deng et al., 2009) and evaluate its performance on the ImageNet validation set, as the ground-truth labels for the test set are not publicly available. Additionally, we assess the CLIP model on 14 diverse zero-shot datasets, covering a broad spectrum of image recognition tasks. Collectively, these 15 datasets encompass:
General Image Classification; ImageNet (Deng et al., 2009). STL-10 (Coates et al., 1), CIFAR-10/100 (Krizhevsky et al., 2009), Caltech-101 (Fei-Fei et al., 2004), and Caltech-256 (Griffin et al., 2007).
Fine-Grained Classification: FGVC Aircraft (Maji et al., 2013), Flower102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), Oxford-IIIT Pets (Parkhi et al., 2012), and Stanford Cars (Krause et al., 2013).
Domain-Speciffe Classification: Describable Textures Dataset (DTD) (Cimpoi al., 2014), EuroSAT (Helber et al., 2019), and PatchCamelyon (PCAM) (Veeling al., 2018).
Scene Recognition: SUN397 (Xiao al., 2010).
We use standard data pre-processing (image resizing to the resolution of 224 x 224 and center-cropping) during adversarial fine-tuning to ensure the consistency with prior works (Mao et al., 2023; Wang et al., 2024; Schlarmann et al., 2024). In addition to the datasets used for zero-shot classification, we further incorporate vision- languae understanding and medical imaging datasets, with additional details provided in the following section.
B.2. Implementation details
Standard Setup. Following established adversarial fine-tuning works (Mao et al., 2023; Wang et al., 2024), we adopt the CLIP model (Radford et al., 2021) with the ViT-Base/32 backbone (Dosovitskiy al., 2021). We use the SGD optimizer
15

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
with momentum factor of 0.9, a batch size of 512, and an initial learning rate of 10-3 (scheduled via cosine decay) to optimize the image encoder of the CLIP model. For Garameter-Efficien Fine-Tunnig (PEFT) with Visual Prompt Tuning (VPT) (Jia et al., 2022), we insert learnable tokens of size 100 into the vision branch of CLIP and use a learning rate of 40, For adversary generation during the fine-tuning stage, we employ PGD (Madry et al., 2018) with m = 10 iterations under the -norm threat model, setting the perturbation radius e = 2/255 and the step size or = 1 1255, unless stated otherwise. The weighting factor is set to À = 3.0. All robustness evaluations are performed using adaptive attack schemes to ensure a fair comparison. All the experiments were run on eight NVIDIA H100 GPUs.
Evaluation Protocol. In line with prior studies on adversarially robust CLIP (Mao al.. 2023; Wang et al., 2024; Schlarmann et al., 2024), we evaluate method with both clean samples and three strong white-box adversaries: 20-step PGD (Madry et al., 2018) 8), the CW attack (Carlini & Wagner, 2017), and Auto-Attack (AA) (Croce & Hein, 2020), focusing mainly on adversarial images to align with real-world defense applications. We additionally assess robustness against text-level adversaries (BERT-Attack (Li et al., 2020) and GBDA (Guo et al., 2021)) and bi-level adversaries from both image and text branches (Co-Attack (Zhang et al., 2022a) and SGA (Lu et al., 2023)), as described in the main text.
BLIP Extensions. To examine zero-shot robustness on downstream vision-language understanding tasks, we incorporate the BLIP framework (Li et al., 2022), which unifies vision-language tasks via bootstrapped pre- rre-training. Specifically, we consider two cross-modal tasks: (i) image-text retrieval using the Flickr30k dataset (Plummer et al., 2015), and (ii) image captioning using the Nocaps dataset (Agrawal et al., 2019). We conduct adversarial min-max optimization of the Image- Text Contrastive (ITC), Image-Text Matching (ITM), and Language Modeling (L.M objectives following (Li et al., 2021) with our approach as an addition optimization term to obtain a robust version of BLIP. We then evaluate its robustness using iterative PGD for both image-text retrieval (maximizing the ITM loss) and image captioning (maximizing the LM loss). We generate and evaluate adversarial samples using the perturbation configuration
= 1/255.
Medical CLIP Extensions. Beyond evaluations natural images, we further investigate robustness in medical imaging using specialized CLIP model (ViT-B/16) B/16) trained via CheXzero (Tiu et al., 2022) on chest radiographs and medical reports from the MIMIC database (Johnson et al., 2019). The text encoder of the CLIP model is based on BioBERT (Lee et al., 2020), tailored for biomedical language. We evaluate zero-shot AUC on ChestX-ray 14 (Wang et al., 2017), CheXpert (Irvin et al., 2019), and PadChest (Bustos et al., 2020) under 20-step PGD attacks with the perturbation radius of 1/255.
C. Further Analyses
C.1. Derivation of bound in Eq. (7)
In Eq. (5) & (6), elements after the sum can be written as (vā + +VQ+1VB)= a =
where 7 VaB and
3 2 0. Using the known inequality (a + b)p < 2a + b²) and setting p = 2, = Va. b 2 JVB, we obtain (Va+ + \B)² < 2a + g. This upper bound eliminates the 7 term, which is why it does not appear in Eq. (7). This result corresponds á in Eq. (11). For completeness, we evaluate the term with y in Appendix C.3, but is computationally expensive due to operating in (R() instead of Rwh.
C.2. Hyper-parameter analyses
38.0
Clean Accuracy (%)
60.5
35.5
Accuracy Accuracy (%))
60.0
35.0
Clean Accuracy
59.5
59.5
34.5
88.0
34.0
58.5
33.5
58.0
33.0
58.0
33.0
57.5
32.5
57.0
128
256
512
1024
2048
32.0
57.0
20
32.0
Count Sketching Size
Count Sketching frequency
(h)
Figure 5. dlyeer-paramee (Count sketching size d' and refreshment frequency of count sketching matrices) sensitivity of our adversarial fine-tuning method on average clean and (Auto-Are Attoo)) robust accuracy (%) across 15 datasets in the sero-shot setting.
16

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
Beyond our analysis of the loss weighting factor described in Figure 4a in the main text, we further analyze how other hyper-parameters influence our adversarial fine-tuning method, especially using diverse configurations of the count sketching for accelerating the inner product computation.
As shown in Figure 5, we present both clean accuracy and (Auto-Attack) robust accuracy under various hyper- parameter configurations. Note that the setting of all the hyper-parameteer is obtained through the Hyperopt package (Bergstra et al., 2013) for a 25-iteration hyper-parameter search on a 1% subset of the ImageNet training set to ensure consistency and was subsequently applied across diverse adversarial fine-tuning scenarios. Notably, appropriate choices of the count sketching haper-parameters enable balanced trade-off between natural performance and adversarial robustness in the zero-shot setting.
C.3. The cross-product term of the upper bound ((x).
The aggregation over the cross-terms 7x, 5z) can be expanded follows:
Σ 6.EAx
( (J(() 8z, 6 (x)xx))c
62. (x)))ee) (6.02. (H₂(x))a)
(25)
6. E,EAx c=1
Σ dz 2 (520)) Σ Jz(x)) 8 vec((H((x))b))
(26)
6, E,E
P8 @ ved
ved P Σ (Jz(x)) (ec((((((())))))))
6,EAx
(27)
where P PERe'l'xue xich is the projection matrix used by the count sketching, and w/h' &wh.
C.4. Set-level augmentation variant of AdySimplex
Recall that SGA (Lu et al., 2023) introduces adversarial perturbations in every adversary generation step (gradient ascent step) around the intermediate adversary v', which continues the ascent step from their mean (μ * v'). Similar strategies have also been demonstrated to be effective in enhancing adversarial robustness (Addepalli et al., 2022; Dong et al., 2023c; 2024e). Thus, for the same it is possible to generate three different adversarial trajectories. Let consider SGA variant of our AdvSimplex by using the simplex formed from the final 3 adversaries of 3 adversarial paths per sample dubbed as AdvSimplex-SGa: Furthermore, we consider using the final adversarial example generated via SGA and generating 3 adversarial paths as in AdySimplex-SGa and use the final 3 adversaries directly for robustification. The average results across 15 datasets are shown in Table 16.
Table Average Performance (%) of our AdvSimplex method with the set-level augmentation.
Configuration
Clean
AA
AT-SGA (standard: one adversary per x)
56.84
30.08
AT-SGA (3 adversaries per x)
57.31
30.93
AdvSimplex-Se SGA (simplex on 3 Adversaries per x)
58.75
32.40
AdySimplex
60.23
34.06
C.5. KL-divergence alternative to the Euclidean basedd alignment
Prediction alignment via the KL-divergence has been widely used in the context of adversarially robust learning in previous studies (Zhang al., 2019; Dong al., 2022). Thus, in this section, we also consider KL--divergence alternative to our AdvSimplex which does not use the Euclidean distance. Specifically, expansion log g(x + 5x) - log gxx) z JioR(ot)) (x)ox + 1 (o) o2 where p = 1 x 10-5 is added for the numerical stability of log. Hence, we can obtain (xL(x) = 1/ ES K LLgg(x)ll(( + 6x)) x --<(((())) (g(x)).
replace 2(x) by using a different Taylor
[(xx (o c)) + 2 (Ex, (H (Hoggooo)) (x))c)}, where fx is the analytical mean of simplex, and Σx is from Theorem
17

---

Improving Zero-Shot Adversarial Robustness in VLMs by Closed-form Alignment of Adversarial Path Simplices
3.1. The average results across 15 datasets are shown in Table 17.
Table 17. Average Performance (%) of our AdvSimplex built with the KL-divergence.
Configuration
Clean
AA
KL-divergence in Eq. (1)
59.28
32.96
AdvSimplex-Ki (using 9xL instcad of !(()))
59.13
33.20
AdvSimplex
60.23
34.06
C.6. Comparison with TGA-ZSR
TGA-ZSR (Yu et al., 2024) designs text-guided attention enhance zero-shot robustness of VLMs. Below we evaluate and compare our AdvSimplex with TGA-ZSR on image-level, text-level, and bi-level adversarial attacks. The average results across 15 datasets are shown in Table 18.
Table 18. Average Performance (%) of our AdvSimplex method vs. TGA-ZSR.
Image-level Attacks
Text-level Attacks
Bi-level Attacks
Method
Clean
PGD
AA
BERT-Attack
GBDA
Co-Attack
SGA
TGA-7SR
57.54
31.15
30.41
38.07
37.32
29.30
28.58
AdvSimplex
60.23
35.68
34,06
40.21
39.88
32.95
32.53
C.7. Analysis of intermediate adversaries along the adversary generation trajectory
As the gradient ascent optimization during adversary generation uses a fixed gradient step size and the decision boundary of VLMs is non-linear, the adversarial path is thus not a perfect ascent. Specifically, intermediate adversarial samples along the trajectory may sometimes be stronger than final adversarial samples. In addition, these intermediate adversaries enjoy diversity in adversarial directions. To validate the dynamic weighting behavior described in Eq. (16), empirically investigate whether the final-step adversary at iteration consistently receives higher weight (wm) than its intermediate counterparts. We note that the inequalities wm 2 4m 4m-1 2 Wm-2, and wim-2 2 Wm-3 hold only 83%, 79%, and 73% times on average, respectively. This observation supports our claim that the final-step adversary is not necessarily more adversarial than adversaries from earlier iterations, justifying the need for adaptive weighting across intermediate steps.
Inspired by the insights of Friendly Adversarial Training (Zhang et al., 2020), which emphasizes the role of weak adversaries in shaping robust decision boundaries, we analyze the impact of using simplices constructed from adversarial examples different steps.
Table 19, Average Performance (%) of our AdvSimplex using simplices formed by subsets of intermediate adversary indices.
Indices of Used Adversarial Simplices
Clean
AA
1-3 (early steps)
59.79
32.00
1-5 (early steps)
60.37
32.49
6-10 (late steps)
59.14
33.38
8-10 (late steps)
58.65
33.06
1-10 (all)
60.23
34.06
Table 19 reveals that early adversarial steps (e.g., steps 1-3) improve clean accuracy, likely due to their strong correlation with the original input and their augmentation effect. In contrast, late steps (e.g., steps 8-10) contribute more to adversarial robustness. Notably, combining intermediate steps across the entire adversarial trajectory (1-10) achieves the best trade-off, yielding the highest clean and robust accuracy, suggesting that mid-level adversaries offer complementary benefits in balancing robustness and generalization.
18


Reviews:
[{"review_id": "LAoXyMtjK6", "reviewer": "Reviewer_EyNB", "summary": "This paper tackles the challenge of making vision-language models (VLMs) more robust to adversarial attacks in zero-shot classification. Existing methods try to improve robustness by sampling adversarial examples along the decision boundary, but they come with high computational costs. To address this, the authors propose a more efficient approach using a closed-form formulation based on Taylor expansion. This allows them to approximate adversarial alignment loss without expensive second-order computations while still capturing key adversarial perturbations along the trajectory. Their method effectively simulates infinite uniform sampling over simplices, offering a computationally efficient way to strengthen VLMs against attacks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. EyNB\nWe thank Rev. for constructive feedback.\n### **1. I checked key proofs...they appear to be correct.**\n\nThank you.\n\n### **1b.** Why $\\gamma$ disappears in Eq. 7? (for Rev. XcU1)\n\n- In Eq. 5 \\& 6, elements after the sum can be written as $(\\sqrt{\\alpha}+\\frac{1}{2}\\sqrt{\\beta})^2=\\alpha+\\frac{1}{4}\\beta+2\\cdot\\frac{1}{2}\\gamma$ where $\\gamma=\\sqrt{\\alpha\\beta}\\;,\\alpha,\\beta\\geq 0$.\n- Use known inequality $(a+b)^p \\le 2^{p-1}(a^p+b^p)$ and set $p=2$ so $(a+b)^2 \\le 2^{1}(a^2+b^2)$.\n- Substitute the first eq. into the inequality: $(\\sqrt{\\alpha}+\\frac{1}{2}\\sqrt{\\beta})^2\\le 2\\alpha +2\\cdot\\frac{1}{4}\\beta=2\\alpha +\\frac{1}{2}\\beta$ and that is why $\\gamma$ does not appear in Eq. 7.\n - This result corresponds to $\\bar{\\bar{\\Omega}}$ in Eq. 10 which is an upper bound of Eq. 6.\n - Note we evaluate in supp. material (**C.2**) Eq. 9 includes $\\gamma$ but is very costly due to operations in $\\mathbb{R}^{(wh)^2}$ *vs.* $\\mathbb{R}^{wh}$.\n\n### **1c. To further showcase our work, below we have additional extensions of Theorem 3.1.**\n\n- **Theorem 3.1 (extended)** can be extended to higher-order simplices, e.g., **tetrahedron ($Q=4$ vertices)** or **pentachoron ($Q=5$ vertices)**:\n\n  Let $Q$ be the number of vertices (${\\bf z}_1,\\ldots,{\\bf z}_Q$) for a simplex. The closed-form expression for ${\\bf\\Sigma}_x$ is given as: \n\n  $\\mathbb{E}[{\\bf pp}^T]=\\frac{1}{Q(Q+1)}\\bigg[\\sum\\_{i=1}^Q{\\bf z}\\_i {\\bf z}\\_i^T + \\Bigl(\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)\\Bigl(\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)^T\\bigg]$.\n\n  Proof follows steps in paper, e.g., parameterizing ${\\bf p}=\\sum\\_i^Q\\alpha_n{\\bf z}\\_i, \\\\;\\alpha\\_i\\geq 0,\\\\; \\sum\\_i^Q\\alpha\\_i=1$ and noting that:\n  - $\\mathbb{E}[\\alpha_i] = \\frac{1}{Q}$\n  - $\\mathbb{E}[\\alpha_i^2] = \\frac{2}{Q(Q+1)}$\n  - $\\mathbb{E}[\\alpha_i\\alpha_j] = \\frac{1}{Q(Q+1)},\\\\;i\\neq j$\n \n  for the underlying Dirichlet distribution. Then we expand $\\mathbb{E}[pp^T]$ where $p p^T = \\sum\\_{i,j=1}^Q \\alpha\\_i\\alpha\\_j {\\bf z}\\_i{\\bf z}\\_j^T$.\n\n### **1d** We have also a theoretical extension linking Theorem 3.1 above with the **Hessian-vector product (HVP)** needed for fast evaluation of Eq. 10. \n\nEvaluating Hessian even with **functorch** is slow. In our paper, **we used HPV which never evaluates Hessian explicitly.** HPV computes very fast $({\\bf H\\_g\\cdot v})$ instead.\n\n- To take advantage of it, Theorem 3.1 includes:\n\n    $\\mathbb{E}[{\\bf p}^T (H\\_g\\cdot{\\bf p})]=\\frac{1}{Q(Q+1)}\\bigg[\\sum\\_{i=1}^Q{\\bf z}\\_i^T(H\\_g\\cdot{\\bf z}\\_i) + \\Bigl(\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)^T\\Bigl(H\\_g\\cdot\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)\\bigg]$.\n\n- Thus, in Eq. 10, we can substitute $\\langle {\\bf \\Sigma}\\_x, (H\\_g({\\bf x}))\\_c\\rangle^2=\\Big(\\mathbb{E}[{\\bf p}^T ((H\\_g({\\bf x}))\\_c\\cdot{\\bf p})]\\Big)^2$. For a simplex with three vertices (one is $0$), this requires three HPV evaluations.\n\n### **2. Alternative to Euclidean alignment in Eq. 1.**\n\nWe replace $\\Omega({\\bf x})$ in Eq. 1 with a KL-divergence variant integrated with Theorem 3.1. To this end:\n- we use a different Taylor expansion $\\log g({\\bf x}+{\\bf\\delta}\\_x)-\\log g({\\bf x})\\approx J\\_{\\log (g+\\rho)}({\\bf x}){\\bf\\delta}\\_x + \\frac{1}{2}\\bigl[{\\bf\\delta}\\_x^T(H\\_{\\log (g+\\rho)}({\\bf x}))\\_c\\\\,{\\bf\\delta}\\_x\\bigr]_{c=1}^{C}$ where $\\rho=1e^{-5}$ is added for the numberical stability of $\\log$.\n- then $\\Omega\\_{KL}({\\bf x})=\\frac{1}{\\kappa}\\sum\\nolimits\\_{{\\bf\\delta}\\_x\\in\\Delta\\_\\mathcal{X}} KL\\big(g({\\bf x}) || g({\\bf x}+{\\bf\\delta}\\_x)\\big)\\approx-\\sum\\_{c=1}^C (g({\\bf x}))\\_c\\cdot \\Big[  \\big\\langle{\\bf\\mu}\\_x, \\mathcal{J}\\_{\\log (g+\\rho)}({\\bf x},c) \\big\\rangle+\\frac{1}{2}\\big\\langle{\\bf{\\Sigma}}\\_x, (H\\_{\\log (g+\\rho)}({\\bf x}))\\_c \\big\\rangle \\Big]$\n  \n  where ${\\bf\\mu}\\_x$ is the analytical mean of simplex, and ${\\bf{\\Sigma}}\\_x$ is from Theorem 3.1.\n\n### **3. Results for extensions.**\n\n|Method|Avg. Clean|Avg. AA|Train Time per Epoch (hours)|\n|-|-|-|-|\n|KL Div. in Eq. 1 (sampling)|59.28|32.96|3.5|\n|AdvSimplex-KL|59.13|33.20|3.0|\n|SGA (standard: one adv. per ${\\bf x}$)|56.84|30.08|1.3|\n|SGA (3 adversaries per ${\\bf x}$)|57.31|30.93 |1.4|\n|AdvSimplex-SGA (simplex on 3 adv. per ${\\bf x}$)|58.75|32.40|2.1|\n|**AdvSimplex**|60.23|34.06|2.9|\n|**AdvTetrahedron**|60.80|35.17|2.7|\n\nIn the table above:\n- **AdvSimplex-KL** uses our KL div. $\\Omega\\_{KL}({\\bf x})$ instead of bounds of Euclidean $\\Omega({\\bf x})$.\n- **AdvTetrahedron** uses our extended Theory 3.1 for tetrahedron ($Q=4$). We take ${\\bf x}$ and additional consecutive 3 vertices instead of 2 as in AdvSimplex ($Q=3$).\n- **AdvSimplex-SGA** uses simplex formed from the final 3 adversaries of 3 adversarial paths per sample ${\\bf x}$. As SGA perturbs each intermediate adversary by noise, running gradient ascent 3x produces 3 distinct paths.\n- **SGA (3 adv.)**: we generate 3 adv. paths as in AdvSimplex-SGA and use the final 3 adversaries directly for robustification.\n- **SGA (standard)**: we use the final adversary from a single adv. path for robustification.\n\nSGA:\n> Set-level Guidance Attack..., ICCV'23.", "claims_and_evidence": "All claims in the paper are clearly stated and supported by theoretical proofs.", "methods_and_evaluation_criteria": "The proposed method is well-motivated and effectively addresses the computational challenges of adversarial fine-tuning for vision-language models (VLMs). The use of a closed-form formulation based on Taylor expansion makes sense for improving efficiency while maintaining robustness. The evaluation criteria are appropriate, with experiments conducted diverse tasks and datasets, which provides a strong empirical results for assessing the method’s effectiveness.", "theoretical_claims": "Regarding the theoretical claims, the paper provides clear mathematical derivations to support its approach. The Taylor expansion and the closed-form approximation of adversarial alignment loss are well-structured and logically follow from the given assumptions. I checked the key proofs related to the approximation and alignment loss formulation, and they appear to be correct.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and analyses, which appear well-structured and aligned with the paper’s objectives. The evaluation spans 15 datasets, providing a diverse testbed for assessing the robustness of the proposed method. The comparisons with prior adversarial fine-tuning approaches are relevant, and the reported 10× speed-up adds an important efficiency perspective.", "relation_to_broader_scientific_literature": "This paper builds on prior work in adversarial robustness for vision-language models (VLMs) and extends research on efficient adversarial training methods. Traditional adversarial fine-tuning methods, such as those explored by Wang et al. and Lu et al., have highlighted the importance of intermediate adversarial examples along the adversarial trajectory for improving robustness. However, these approaches rely on costly second-order computations, limiting their scalability. The proposed method addresses this limitation by leveraging Taylor expansion to approximate adversarial alignment loss in a closed-form solution, eliminating the need for expensive sampling of adversarial simplices.", "essential_references_not_discussed": "The paper covers the essential references relevant to its topic, and I did not find any major prior work that was overlooked.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atxDER9Tk2", "reviewer": "Reviewer_td6r", "summary": "This paper proposes a new adversarial fine-tuning method for VLMs that enhances zero-shot adversarial robustness by leveraging adversarial simplices formed from a clean image and consecutive intermediate adversaries along the gradient ascent path. The proposed method employs a Taylor expansion of the model's prediction function to derive a closed-form upper bound on the alignment loss. This closed-form solution, which approximates infinite uniform sampling within the simplex, is then integrated into a combined loss function that balances standard classification loss with an adaptive adversarial alignment term, which achieves SOTA zero-shot adversarial robustness and computational efficiency across 15 datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. td6r\nThank you for the constructive feedback. **Kindly also note additional Theory/Results in Resp. to Rev. EyNB**.\n### **1. Theoretical derivations well-grounded/experiments support claims.**\nThank you. We have also a generalization of our Theorem 3.1 to simplices with more vertices, e.g.,  **tetrahedron ($Q=4$ vertices)** or **pentachoron ($Q=5$ vertices)**.\n\\\n\\\n**Kindly see Resp. 1 to Rev. EyNB for details.** Below we provide a list of our extensions.\n\n  Let $Q$ be the number of vertices (${\\bf z}_1,\\ldots,{\\bf z}_Q$) for a simplex. The closed-form expression for ${\\bf\\Sigma}_x$ is given as: \n\n  $\\mathbb{E}[{\\bf pp}^T]=\\frac{1}{Q(Q+1)}\\bigg[\\sum\\_{i=1}^Q{\\bf z}\\_i {\\bf z}\\_i^T + \\Bigl(\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)\\Bigl(\\sum_{i=1}^Q {\\bf z}\\_i\\Bigr)^T\\bigg]$.\n\nTable below shows further gains from our extended theorem:\n\n|Method|Avg. Clean|Avg. AA|\n|-|-|-|\n|Our **AdvSimplex**|60.23|34.06|\n|Our **AdvTetrahedron**|**60.80**|**35.17**|\n  \n### **2. Comparison with [1]**\n> [1] Text-guided attention is all you need for zero-shot robustness in vision-language models, NeurIPS 2024.\n\nTGA-ZSR [1] designs text-guided attention to enhance zero-shot robustness on VLMs. We will cite [1].\n\\\n\\\nBelow we evaluate/compare our AdvSimplex with [1] on image-level, text-level, and bi-level adversarial attacks. The average results across 15 datasets are below:\n\n|||*Image-Level Attacks*||*Text-Level Attacks*||*Bi-Level Attacks*||\n|-|-|-|-|-|-|-|-|\n|**Method**|**Clean**|**PGD**|**AA**|**BERT-Attack**|**GBDA**|**Co-Attack**|**SGA**|\n|TeCoA|48.83|27.33|25.75|37.14|35.30|26.73|25.94|\n|PMG-FT|49.71|28.44|26.98|37.61|36.46|28.11|27.85|\n|FARE|56.68|30.94|29.30|35.45|34.97|25.38|25.06|\n|$\\color{red}\\text{TGA-ZSR [1]}$|57.54|31.15|30.41|38.07|37.32|29.30|28.58|\n|**AdvSimplex**|**60.23**|**35.68**|**34.06**|**40.21**|**39.88**|**32.95**|**32.53**|\n\n**AdvSimplex significantly outperforms other methods.**\n\n### **3. Sensitivity w.r.t. $\\lambda$.**\n- $\\lambda$ controls the impact of the upper bound of $\\Omega({\\bf x})$ when aligning adversarial predictions with clean predictions.\n- Figure 4a (main paper) shows that increasing $\\lambda$ enhances the adversarial robustness at the cost clean accuracy. Lowering $\\lambda$ improves zero-shot clean acc. but reduces adv. robustness.\n- Such a trade-off stems from the optimization the natural and boundary risks.\n\nAverage results across 15 datasets w.r.t. $\\lambda$ are below:\n - **Uniform weighting**: $\\omega_i=1/(m-1)$ is const. for all $i=1,\\ldots,m-1$.\n - **Adaptive weighting**: as per Eq. 13 (main paper)\n\n|Re-weighting Strategy|$\\quad\\lambda\\quad$|Clean Accuracy|AA-Robust Accuracy|\n|-|-|-|-|\n|Uniform|0.5|59.67|32.38|\n|Uniform|**0.6**|**59.45**|**32.96**|\n|Uniform|0.7|59.06|33.64|\n|Uniform|0.8|58.72|34.09|\n|Uniform|0.9|58.29|34.26|\n|Adaptive|0.5|60.41|33.35|\n|Adaptive|**0.6**|**60.23**|**34.06**|\n|Adaptive|0.7|60.02|34.75|\n|Adaptive|0.8|59.78|35.10|\n|Adaptive|0.9|59.04|35.39|\n\nIn our paper, $\\lambda=0.6$ in all experiments.\n\n### **4. Formatting issue.**\nWe will fix this. It appears tables jumped between pages unintentionally.\n\n### **5. Trade-off analyses between closed-form vs. sampling.**\n- In addition to Figure 4b, we present further results (clean accuracy, adv. robust accuracy (AA-Robust Accuracy), training time per epoch):\n\n|Method|No. samples per simplex|Clean Accuracy|AA-Robust Accuracy|Train Time per Epoch (hours)|\n|-|-|-|-|-|\n|Sampling|3|57.42|28.86|1.4|\n|Sampling|10|57.63|30.37|2.6|\n|Sampling|30|57.98|31.98|5.1|\n|Sampling|50|58.55|33.17|8.0|\n|Sampling|70|59.07|33.69|10.3|\n|Sampling|100|59.61|33.92|14.0|\n|Sampling|200|59.87|34.07|25.5|\n|Sampling|300|60.18|34.20|36.2|\n|Our **closed-form simplex (10 steps)**|$\\infty$|60.23|34.06|2.9|\n|Our **closed-form simplex (5 steps)**|$\\infty$|58.75|33.27|1.6|\n|Our **closed-form simplex (3 steps)**|$\\infty$|58.36|32.19|1.2|\n|Our **closed-form tetrahedron (10 steps)**|$\\infty$|60.80|35.17|2.7|\n|Our **closed-form tetrahedron (5 steps)**|$\\infty$|59.12|33.64|1.4|\n|Our **closed-form tetrahedron (3 steps)**|$\\infty$|58.67|32.70|1.1|\n\n- Increasing the number of sampling adversaries from simplex yields gains in robustness at the substantial computational cost increase.\n- Our closed-form solution effectively balances robustness and computational efficiency, providing **competitive clean and robust performance at ~$10\\times$ reduced computational cost.**\n\n### **6. Extension to image classification.**\nIn our paper, we also have:\n- **Vision-text understanding.** Table 7 is **image-text retrieval (Flickr30k)** and **image captioning (Nocaps)**\n- **Medical diagnosis.** Table 8 shows results on ChestX-ray14, CheXpert and PadChest.\n\nFor more results, table below is single-modal image classification built on TRADES with Wide-ResNet-28-10 (CIFAR-10/100):\n||CIFAR-10||CIFAR-100||\n|-|-|-|-|-|\n|**Method**|**Clean**|**AA**|**Clean**|**AA**|\n|PGD-AT|83.79|56.30|58.13|26.65|\n|TRADES|85.17|57.09|59.48|26.99|\n|**AdvSimplex**|**87.25**|**58.64**|**61.30**|**28.57**|", "claims_and_evidence": "Yes, the claims made in the submission are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "This paper aims to improve the zero-shot adversarial robustness of VLMs. Clean and robust accuracy used in this paper are widely acknowledged as the golden metrics in zero-shot robust classification problems. 15 benchmark datasets used in this paper are widely acknowledged to evaluate the zero-shot capabilities of CLIP. All the baseline methods are fine-tuned on ImageNet, which is a relatively large-scale dataset in this field.", "theoretical_claims": "The proof is overall correct. In particular, I checked the derivation of Theorem 3.1 on the closed-form second-order statistics for points uniformly sampled in a simplex, as well as the bounding argument in Section 3.1 (where the paper moves from the exact alignment loss to its Taylor-based upper bound). The closed-form expression for the uniform simplex statistic (Theorem 3.1) is standard and appears correctly applied.", "experimental_designs_or_analyses": "The experimental design is very sound from my perspective: this paper uses 15 datasets and multiple attack methods for evaluations (e.g., PGD, C&W and AutoAttack) and all evaluations use adaptive attacks. In addition, this paper systematically measures both zero-shot clean and robust accuracy across different model architectures and perturbation budgets. The authors also provide ablation studies and sensitivity analyses that help support the validity of their choices of hyperparameters. Regarding the section on baseline methods, I notice that a recently published paper [1] can be considered as one of the baseline methods in this paper as well since it also claims that it achieves the SOTA performance in zero-shot robustness.\n\n[1] Text-guided attention is all you need for zero-shot robustness in vision-language models, NeurIPS 2024.", "relation_to_broader_scientific_literature": "The paper builds upon a substantial body of work in adversarial training and fine-tuning. For example, Wang et al. (2022) and Lu et al. (2023) emphasize the importance of intermediate adversarial samples along the gradient ascent path, as well as Gao et al. (2024) who introduced the concept of forming simplices with consecutive adversaries. However, previous approaches struggled with the computational costs of sampling such simplices. This paper advances the previous methods by using Taylor expansion and deriving a closed-form solution that captures the infinite sampling behavior within a simplex. This approach not only aligns with established ideas in robust optimization but also addresses practical scalability concerns, thereby contributing a novel and computationally efficient method to the field.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\n1.This paper proposes an interesting fine-tuning method that could largely improve the zero-shot performance and, at the same time, significantly reduce computational cost.\n\n2.Theoretical derivations are well-grounded and the experimental results support the claims.\n\nWeaknesses:\n\n1.The experiment lacks a sensitivity analysis of the parameter $\\lambda$ used in Eq. (13).", "other_comments_or_suggestions": "I noticed that Table 5 and Table 6 overlap in the current layout. I would suggest adjusting their formatting to avoid overlapping.", "questions_for_authors": "1.The adaptive re-weighting mechanism in Eq. (13) appears crucial to your loss formulation. How sensitive is the method’s performance to the choice of $\\lambda$?\n\n2.In your comparisons, the closed-form alignment is shown to be computationally more efficient than explicit sampling. Could you provide further quantitative analysis on the trade-off between computational overhead and adversarial robustness when using closed-form versus sampled adversarial simplices?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ZwwMgNuf1", "reviewer": "Reviewer_XcU1", "summary": "This paper aims to enhance the robustness of CLIP for zero-shot image classification. \nIt emphasizes that existing defense methods often disregard intermediate adversarial samples along the trajectory, which are found to be beneficial in this study.\nThe proposed method, AdvSimplex, uses an efficent method to statistically align clean sample and adversarial samples from the trajectory, instead of sampling adversarial examples from the trajectory area and aligning them with a clean sample one by one.\nExperiments on ImageNet demonstrate improvements in both accuracy and robustness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. XcU1\nThank you for the constructive feedback. **Kindly also note additional Theory/Results in Resp. to Rev. EyNB**.\n### **1. Compare Cost. 10x speedup.**\n- Below are **training times for FARE, TeCoA, PMG-FT**.\n- The *10x speed-up* is for our closed-form \"infinite\" sampling from adv. simplex *vs.* \"naive\" sampling (see below *Sampling 300* samples per simplex *vs.* AdvSimplex (10 steps)).\n\n|Method|Clean|AA|Training Time per Epoch (hours)|\n|-|-|-|-|\n|TeCoA|48.83|25.75|1.3|\n|PMG-FT|49.71|26.98|2.1|\n|FARE|56.68|29.30|1.7|\n|Sampling (300 samp.)|60.18|34.20|36.2|\n|**AdvSimplex (10 steps)**|60.23|34.06|2.9|\n|**AdvSimplex (5 steps)**|58.75|33.27|1.6|\n|**AdvSimplex (3 steps)**|58.36|32.19|1.2|\n|**AdvTetrahedron (10 steps)**|60.80|35.17|2.7|\n|**AdvTetrahedron (5 steps)**|59.12|33.64|1.4|\n|**AdvTetrahedron (3 steps)**|58.67|32.70|1.1|\n\n- For **the same time budget (~1.6h), AdvSimplex (5 steps) outperforms FARE by ~2\\% and ~4\\%** in the clean and robust accuracy (AA)\n- For 1.7x time budget, **AdvSimplex (10 steps) gets ~3.5\\% and ~4.7\\% gains over FARE** (clean and robust acc. (AA)).\n\n- **Our method does not use additional modules/parameters:** the inference time is consistent with other VLMs.\n\n- **AdvTetrahedron** uses a generalization of our Theorem 3.1 to simplices with more vertices, e.g.,  **tetrahedron** ($Q=4$ vertices). **Kindly see Resp. 1 to Rev. EyNB for detailed information.** \n\n### **2. Why Weak Adversaries can Robustify? 2b. Clean accuracy gains. 2c. SGA does not use intermediate adv. 2d. Novelty using adv. trajectory.**\n\n> [a] Set-level Guidance Attack..., Lu et al. (2023), ICCV\n\n> [b] Boosting Transferability in Vision-Language Attacks..., Gao et al.(2024), ECCV\n\n- SGA [a] in every ascent step generates few perturbations around intermediate ${\\bf v}'$ and continues ascent step from their mean ($\\bf\\mu\\neq v'$). Thus, for the same ${\\bf x}$ generated 3 traj. will differ.\n- [b] forms triangles between ${\\bf x}$ and consecutive 2 intermediate steps ${\\bf v}\\_i$ and ${\\bf v}\\_{i+1}$ but does not use noise. But Fig. 1b [b] shows triangles capture multiple trajectory routes.\n\nAs gradient ascent uses fixed gradient step and the decision boundary is non-linear, **the adversarial path is not a perfect ascent**. Intermediate adv. samples:\n- **may be sometimes stronger than final adv. samples**\n- **enjoy diversity in adversarial directions.**\n  \\\n  \\\n  We verify this:  **weights $\\omega\\_i$ in Eq. 13 (main paper) for the final-step adversary $m$ is not always more adversarial than intermediate adversary $m-1$**:\n\n  |$\\omega\\_m\\geq\\omega\\_{m-1}$|$\\omega\\_{m-1}\\geq\\omega\\_{m-2}$|$\\omega\\_{m-2}\\geq\\omega\\_{m-3}$|\n  |-|-|-|\n  |83%|79%|73%|\n\n*Friendly Adversarial Training (ICML'20)* notes that weak adv. help obtain robust decision boundary.\n\nBelow we also show that simplices from:\n- late intermediate steps lead to greater adversarial robustness\n- early intermediate steps lead to greater clean accuracy\n- **mid intermediate steps further boost clean+adversarial accuracy**\n\n|Indices of Used Adv. Simplices|Clean|AA|\n|-|-|-|\n|8-10 (Late Steps)|58.65|33.06|\n|6-10 (Late Steps)|59.14|33.38|\n|1-10 (All)|**60.23**|**34.06**|\n|1-5 (Early Steps)|60.37|32.49|\n|1-3 (Early Steps)|59.79|32.00|\n\n**2b.** Early adversaries improve clean acc. as they are more correlated with clean ${\\bf x}$ and act as sample augmentation.\n\n**2c.** To validate Rev. assumption that only final adversaries matter, per ${\\bf x}$ we generate 3 adv. paths as SGA (paths differ due to noise injection). **We build a simplex only from final adv. points of 3 paths (AdvSimplex-SGA)** per ${\\bf x}$: this outperforms vanilla SGA but is worse than **AdvSimplex**.\n \n|Method|Avg. Clean|Avg. AA|\n|-|-|-|\n|SGA (standard: one adv. per ${\\bf x}$)|56.84|30.08|\n|SGA (3 adversaries per ${\\bf x}$)|57.31|30.93 |\n|AdvSimplex-SGA (simplex on 3 adv. per ${\\bf x}$)|58.75|32.40|\n|**AdvSimplex**|60.23|34.06|\n\n**2d.** Our main contribution is **\"infinite\" sampling+alignment but adv. simplex can be generated in many ways** (our strategy, SGA-like strategy, etc.) Unlike SGA [a] and [b] we are the first to use adv. simplices from trajectory for robustification.\n\n### **3. $\\gamma$ missing in Eq. 7**\n\n**See Resp. 1b to Rev. EyNB.**\n\n### **4. Results differ from other papers.**\n\n- [Schlarmann, 2024] reports on a sampled subset of ImageNet (5000 samples) on **ViT-L** in their Table 4\n- We report on **ViT-B** in our Tables 1+2 following TeCoA and PMG settings (Mao, 2023; Wang, 2024)\n\n  **To address Rev.'s concern, we extend our Table 3 to ViT-L in Resp. 6 to Rev. srgi.**\n\n### **5. Compare Batch Sizes.**\n\nIn paper, we use same experimental setup (BS: 512) for TeCoA, PMG, FARE and ours for fairnesss.\nSee **Resp. 7 to Rev. srgi**\n  \n### **6. Auto-Attack for Each Dataset.**\nSee **table in Resp. 8 to Rev. srgi.** We can post ViT-L AA in discussions.\n\n### **7. Extend to image classification.**\nSee **Resp. 6 to Rev. td6r**\n\n### **8. Formatting/abstract.**\nSee **Resp 1-4 to  Rev. srgi**", "claims_and_evidence": "- In the abstract, the authors claim, “We obtain state-of-the-art with 10× speed-up”. \n    - However, the computational cost is never comparable to that of state-of-the-art methods. \n\n- Adversarial training fundamentally involves solving a min-max optimization problem. It is unclear why using weaker adversarial examples from the trajectory would enhance robustness. The authors cite [Gao et al., 2024], whose work focuses on improving the transferability of adversarial examples, which involves sacrificing attack strength on the surrogate model. However, the relevance of transferable adversarial examples to adversarial training remains insufficiently validated, since using such weaker attack is suboptimal in min-max optimization.\n\n[Gao et al. 2024] Gao, S., Jia, X., Ren, X., Tsang, I., and Guo, Q. Boosting transferability in vision-language attacks via diversifica- tion along the intersection region of adversarial trajectory. ECCV 2024", "methods_and_evaluation_criteria": "- The numbers are very different from those of existing papers. For example, in [Schlarmann et al. 2024], the clean zeros-shot accuracy on ImageNet is over 70% for all FARE, TeCoA, and original CLIP; however, Table 1 shows much lower numbers. Also, the robust accuracy is much lower. Why?\n    - [Schlarmann et al. 2024] Schlarmann, Christian, et al. \"Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models.\" ICML 2024\n\n- Table 2: The PGD-20 results as the main results are not convincing. Auto-Attack results are reported only for the average of datasets. This lacks information for readers.", "theoretical_claims": "- I don’t get how Eq.7 is derived. How did $\\gamma(x, \\delta_x)$ disappear? Are the notations correct? If $\\alpha(x,\\delta_x) = || J_g(x) \\delta_x ||^2_2$, why is it doubled in Eq.(7)??? Sorry if I misunderstand anything.", "experimental_designs_or_analyses": "Experimental designs are mostly aligned with existing work. However, there are some concerns, which are mentioned in \"Methods And Evaluation Criteria.\"", "relation_to_broader_scientific_literature": "The idea of leveraging adversarial trajectory in adversarial training may be new. It could also be applied to image classification or other tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weakness:\n- The presentation quality is not optimial.\n  - The caption of Table 6 overlaps with the table.\n  - I don’t think it’s good practice to include extensive mathematical formulations in the abstract. At the very least, using multiple notations without explanation is problematic.", "other_comments_or_suggestions": "- P.2, Line.90: “Lu et al. (2023) obtained intermediate adversarial samples along the adversary generation trajectory to achieve cross-VLM attacks.\" \n    - This seems to be a wrong citation? Set-level Guidance Attack (SGA) does not use generation trajectory.", "questions_for_authors": "- Is the idea of using adversarial trajectory novel? If yes, do you think it can be applied to image classification task, not only VLMs?\n- Why does the clean accuracy improve with the proposed method?\n- Is the training time per epoch longer than FARE or TeCoA?\n- This method uses a much larger batch size of 512 compared to 128 of TeCoA and FARE.\n    - Does this method require a large batch size?\n    - Do TeCoA/FARE results improve with larger batch sizes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TDndEyaR3", "reviewer": "Reviewer_srgi", "summary": "In this work, the authors focus on a topic, robustness of VLM (CLIP). The authors propose to solve a challenge in adversarial fine-tuning, and propose to use Taylor expansion to enlarge the dicision space, which can make the model more robust.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Rev. srgi\n\nWe thank Rev. for the constructive feedback. **Kindly also note additional Theory/Results in Resp. to Rev. EyNB**.\n\n### **1. Abstract.**\n\nThank you. We have removed now math notations:\n\n```\nVision-Language Models (VLMs), e.g., CLIP, excel at zero-shot classification due to large-scale pre-training yet are vulnerable to adversarial examples. Adversarial fine-tuning robustifies zero-shot models by aligning prediction scores of individual adversaries with their clean counterparts, which typically overlooks intermediate adversarial samples along the adversarial trajectory crossing the decision boundary. Such intermediate adversaries and their vicinity offer informative representations of the decision surface, which can further be improved by sampling adversarial candidates from simplices formed by joining vertex with consecutive vertices on the adversarial trajectory. However, sampling simplices for adversaries is prohibitively costly. To train robust VLM, we overcome these limitations by Taylor expansion and formulating an upper-bound of alignment loss that depends on Jacobian/Hessian obtained at clean samples. As regions between clean and intermediate adversarial samples capture a larger decision landscape, we robustify VLM by plausible adversaries on simplices by our closed-form formulations equivalent to infinite uniform sampling of the simplex. We obtain state-of-the-art robustness across 15 datasets and diverse vision-language tasks.\n```\n\n###  **2. Separate Section for Related Works & Concise Intro.**\n\n**Absolutely. This is very easy to achieve.**\n\nWe will provide a separate detailed related work section in the revised version, including:\n- alignment schemes (plus necessary equation).\n- single-modal adversarial attacks/robustness\n- multimodal adversarial attacks/robustness for a better understanding.\n\n### **3. More Explanations of Section 3.**\n\n- We first discuss the upper bound of the prediction alignment between clean and adversarial examples derived by Taylor expansion in Section 3.1\n-  In Section 3.2, we propose a theorem to avoid empirical aggregating/sampling from adversarial simplices. In other words, we provide a closed-form solution for sampling from adversarial simplices.\n- The overall loss function is shown in Section 3.3.\n- We further demonstrate that our method also bounds the robust risk in Section 3.4.\n\nWe will add the summary table of symbols:\n\n|Symbols|Explanations|\n|-|-|\n|$\\mathbf{x}$|Clean example|\n|$\\bf{\\delta}_{\\mathbf{x}}$|Adversarial perturbation|\n|$g_\\theta(\\cdot)$|Prediction of CLIP|\n|$J_{g}$|Jacobian matrix|\n|$H_{g}$|Hessian matrix|\n|$\\Omega$|Vanilla upper bound of the Euclidean prediction distance|\n|$\\bar{\\Omega}$|Upper bound with the cross-product term|\n|$\\bar{\\bar{\\Omega}}$|Upper bound without the cross-product term|\n\n### **4. Formatting Issues.**\n\nWe apologize. We will fix Eq. 10 and one table that jumped between pages breaking formatting.\n\n### **5. Code Release.**\n\n**We stress that our code, model weights, and setup will be publicly available.**\n\n$\\color{blue}\\text{General Additional Results:}$\n### **6. Additional results (ViT-B vs. ViT-L).**\n\nWe extend our Table 3 (ImageNet), showing the average acc. across 15 datasets on ViT-B and ViT-L:\n|Architecture|Method|ImageNet Clean|ImageNet AA|Avg. Clean|Avg. AA|\n|-|-|-|-|-|-|\n|ViT-B|TeCoA|54.43|25.19|48.83|25.75|\n|ViT-B|PMG-FT|51.33|24.94|49.71|26.98|\n|ViT-B|FARE|50.94|23.78|56.68|29.30|\n|ViT-B|**AdvSimplex**|61.28|32.26|60.23|34.06|\n|ViT-L|TeCoA|73.61|61.14|70.95|47.96|\n|ViT-L|PMG-FT|73.92|60.63|68.67|49.38|\n|ViT-L|FARE|72.51|57.20|71.30|50.54|\n|ViT-L|**AdvSimplex**|76.87|64.09|73.39|52.80|\n\n### **7. Additional results (Batch Size).**\n\nFor completeness, we present results with two different batch sizes, **128 and 512** to show our approach performs best.  The average results across 15 datasets on ViT-B are below:\n\n|Batch Size|Method|Avg. Clean|Avg. AA|\n|-|-|-|-|\n|128|TeCoA|48.70|25.62|\n|128|PMG-FT|49.45|26.74|\n|128|FARE|56.22|28.95|\n|128|**AdvSimplex**|59.86|33.54|\n|512|TeCoA|48.83|25.75|\n|512|PMG-FT|49.71|26.98|\n|512|FARE|56.68|29.30|\n|512|**AdvSimplex**|60.23|34.06|\n\nOur method does not require a large batch size, but large batch size can enhance a bit all methods.\n\n### **8. Additional results: Auto-Attack Results for Each Dataset.**\n\nThe auto-attack results ($\\epsilon$=2/255) of ViT-B corresponding to Table 2 is below:\n\n|Method|ImageNet|STL10|CIFAR-10|CIFAR-100|SUN397|Stanf.Cars|Food101|OxfordPet|Flower102|DTD|EuroSAT|FGVC|PCAM|Caltech101|Caltech256|Avearge|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|TeCoA|25.19|69.38|42.01|21.40|16.58|4.09|12.51|40.87|14.96|16.07|11.19|1.68|12.06|54.53|43.74|25.75|\n|PMG-FT|24.94|69.98|43.28|21.48|16.70|6.04|13.57|41.06|15.68|17.21|11.89|2.16|19.57|56.16|44.92|26.98|\n|FARE|23.78|75.40|49.76|28.35|16.29|8.32|17.04|44.61|17.03|18.39|8.58|2.25|21.41|60.14|48.16|29.30|\n|**AdvSimplex**|32.26|77.82|55.64|30.99|18.80|9.45|19.87|53.67|18.90|19.85|12.52|4.41|40.53|64.85|51.34|34.06|", "claims_and_evidence": "The claims are supported by some math derivation.", "methods_and_evaluation_criteria": "The paper uses a lot of quantitative results to demonstrate the effectiveness of this work.", "theoretical_claims": "This work includes one theorem (3.1), which is easy to understand.", "experimental_designs_or_analyses": "The work is all SOTA across all the tables of quantitative result, which is a large improvement.", "relation_to_broader_scientific_literature": "There is no need to discuss broader scientific literature.", "essential_references_not_discussed": "The discussion of related works are sufficient.", "other_strengths_and_weaknesses": "Strength:\n\n1. The quantitative results are a large improvement against baselines.\n\n2. The figures are of high quality.\n\n3. The paper provide theoretical analysis of the proposed theory.\n\nWeakness:\n\nAlthough the idea of the paper may be good, the writing of this work is not good, which make this work not very suitable for publication.\n\n1. I think the writing of abstract needs to be improved. It should convey the key idea of the paper, instead of listing complex math notation. I suggest moving the math notation into the method section and just use natural language to describe the key idea of this work.\n\n2. The related work should be in a seperate section instead of put in the introduction section. Also, the introduction is too long and too redundant.\n\n3. The section 3 is too intensive with math. I think there should be some explanation of notations and some text for why you are writing this line of equation.\n\n4. Small errors: Equation (10) is out of the space. Table 6 is overlapped with other tables.", "other_comments_or_suggestions": "I am willing to raise the score after effective rebuttal.", "questions_for_authors": "Will you open-source the code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an innovative closed-form alignment of adversarial path simplices to further improve the zero-shot classification performance of a pre-trained VLM. The main contributions of this paper come from two sides: 1) it is novel to consider using path simplices for the adversarial robustness and 2) solid theoretical justifications make this paper effective in more scenarios.

In the initial reviews, two reviewers find some confusing parts within the paper (experiments and presentation). Then, the authors did a great job in addressing these concerns, and all reviewers agreed to accept this paper in the end.
Decision: Accept (spotlight poster)