Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models
Authors: Junhao Dong
Cong Zhang
Xinghua Qu
Zejun MA
Piotr Koniusz
Yew-Soon Ong
Venue: NeurIPS 2025 spotlight ()
Keywords: Superalignment
weak-to-strong generalization
sdversarial robustness
vision-language model
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose the first robust weak-to-strong generalization framework to elicit robust knowledge from a strong student VLM in an unsupervised scheme.

Abstract: Numerous well-established studies have demonstrated the superhuman capabilities of modern Vision-Language Models (VLMs) across a wide range of tasks. However, growing is the doubt about the continuing availability of reliable high-quality labeling (supervision) from human annotators, leading to stagnation of the model's performance. To address this challenge, ``superalignment'' employs the so-called weak-to-strong generalization paradigm, where the supervision from a weak model can provide generalizable knowledge for a strong model. While effective in aligning knowledge for clean samples between the strong and weak models, the standard weak-to-strong approach typically fails to capture adversarial robustness, exposing strong VLMs to adversarial attacks. This inability to transfer adversarial robustness is because adversarial samples are normally missing in the superalignment stage. To this end, we are the first to propose the weak-to-strong (adversarial) robustness generalization method to elicit zero-shot robustness in large-scale models by an unsupervised scheme, mitigating the unreliable information source for alignment from two perspectives: alignment re-weighting and source guidance refinement. We analyze settings under which robustness generalization is possible. Extensive experiments across various vision-language benchmarks validate the effectiveness of our method in numerous scenarios, demonstrating its plug-and-play applicability to large-scale VLMs.

Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models
Junhao Dong! Cong Zhang", Xinghua Qu', Zejun Ma²,
Piotr Koniusz5<7 and Yew-Soon Ongl.2 Nanyang Technological University, CFAR, IHPC, A*STRR, TTikTok, Singapore, *Bytedance, Data61 University of New South Wales, Australian National University
Abstract
Numerous well-established studies have demonstrated the superhuman capabilities of modern Vision- Languuge Models (VLMs) across wide range of tasks. However, growing is the doubt about the continuing availability of reliable high-quality labeling (supervision) from human annotators, leading to stagnation of the model's performance. To address this challenge, "superalignment" employs the so-called weak-to-strong generalization paradigm, where the supervision from a weak model can provide generalizable knowledge for a strong model. While effective in aligning knowledge for clean samples between the strong and weak models, the standard weak-to-strong approach typically fails to capture adversarial robustness, exposing strong VLMs to adversarial attacks. This inability transfer adversarial robustness is because adversarial samples are normally missing in the superalignment stage. To this end, we are the first propose the weak-to-strong (adversarial) robustness generalization method to elicit zero-shot robustness in large-scale models by an unsupervised scheme, mitigating the unreliable information source for alignment from two perspectives: alignment re-weighting and source guidance refinement. We analyze settings under which robustness generalization is possible. Extensive experiments across various vision-language benchmarks validate the effectiveness our method in numerous scenarios, demonstrating its plug-and-play applicability to large-scale VLMs.
Introduction
Vision-Language Models (VLMs) enjoy remarkable prediction capabilities, frequently surpassing human performance diverse multimodal tasks, ranging from zero-shot recognition multimodal reasoning [60, 42, 49] to task unlearning [24]. Despite these outstanding achievements, they suffer from an emerging issue: models become highly capable, human annotators fail to reliably guide and/or evaluate their outputs, especially given the complexity and ambiguity inherent vision- language paired data due to inherent limitations in cognitive capacity and scalability [7, 4J. This fundamental limitation has prompted the super-intelligence alignment (superaligmment) [41], a conceptualized framework align (future) superhuman models with human values and goals based on limited human supervision. One seminal approach toward superalignment generalization [5], which explores supervision from weak (lightweight) teacher model to guide strong (large-scale) student model, enabling the student to generalize beyond the original limitations of its supervision source (reference model) on unforeseen data.
weak-to-stron
However, while existing weak-to-strong generalization methods focus on distilling knowledge from a teacher model, we demonstrate that robustness, a distinct form of knowledge, cannot be transferred
"The research work was done during Junhao's internship at Tik Singapore. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

into VLM even when guided by robust teacher'. This limitation inherently exacerbates the vulnerability to adversarial examples-orignnal images with added imperceptible artifacts leading to erroneous predictions with high confidence [65]. In addition to single-modal backbones [21, 20, 23, 15, 22, 14), these vulnerabilities are especially critical in VLMs, where the correctness of language reasoning heavily relies on visual inputs: poisoned images may compound errors and propagate catastrophic failures through the model's output, fundamentally compromising its reliability for real-world deployment [72, 51, 46, 47].
Through systematic analysis, we are the first to identify the root cause of robustness transfer failure presenting as mismatch in learning objectives arising from the data employed in weak-to-stronp gen- eralization models. Our analysis in Section 3.2 reveals that integrating adversarial examples- rather than their clean counterparts into learning can induce robustness in the strong student model, even without explicit supervision from an adversarially robust teacher. Motivated by this insight, we propose the first adversarially robust weak-to- reak-to-strong generalization framework, termed Ady-W28, designed to elicit robust knowledge from the strong student model an unsupervised scheme. Specif- ically, our method mitigates unreliable supervision signals from the weak teacher model through two complementary mechanisms: alignment re-weighting and source guidance refinement. To quantify the reliability of guidance of the weak teacher, we introduce an instance-wist re- se-weighting mecha- nism based on prediction entropy, adaptively modulating the emphasis placed on robust knowledge alignment between the teacher and student VLMs. Additionally, in order to further reduce reliance on potentially erroneous guidance signals, we design an adaptive refinement scheme for prediction reference, generating benign inputs that can be considered as improved guidance for robust weak-to- strong generalization. Our theoretical analysis establishes two core claims: (i) low prediction entropy enforced the student promotes larger classification margins and robustness, and (ii) alignment toward refined source guidance strictly improves robustness generalization during superalignment.
Extensive experiments conducted across diverse network architectures and evaluation scenarios demonstrate that our proposed Adv-W2S framework achieves superior zero-shot classification perfor- mance in both clean and robust accuracy compared to state-of-the-ar adversarial fine-tuning methods. Moreover, we show that the robustness induced by Adv- W2S can effectively transfer to a broad array of downstream vision- vision-language tasks-includin reasoning- -via a aoug-and-play replacement of the original vision encoder with our robustly aligned encoder. Our empirical analyses further reveal that the in-distribution robustness initially obtained via unsupervised alignment can be further improved through supervised fine-tuning. To our best knowledge, this work represents the first systematic exploration of robust superalignment, providing a promising pathway toward building robust and aligned foundation VLMs for artificial general intelligence applications.
hallucination mitigation, and thain-of-thouue including image captioning, visual question answering,
Our core contributions are summarized as follows:
1. We reveal that standard teak-to-strong generalization schemes fail transfer VLM robustness. Through systematic analyses, we identify- for the first the absence of adversarial examples in alignment objectives of weak-to-strong generalization.
the root cause of this failure is
2. contrast to adversarial fine-tuning from scratch, we propose Ady-W2S, the first adversarially robust weak-to-stron to-t-to--trong generalization framework to elicit robust knowledge from weak student VLM by an unsupervised scheme. We also investigate unreliable source guidance mitigation from two complementary perspectives: alignment re-weighting and source guidance refinement. Theoretical analyses characterize the conditions under which robustness generalization is possible.
3. We conduct extensive experiments across 20 datasets spanning diverse vision-language tasks (including visual question answering and captioning) across various scenarios, demonstrating that our Ady consistently outperforms state-of-the-art adversarial fine-tuning approaches in terms of natural performance and zero-shot robustness, while supporting plug-and-play integration.
2 Related Works
Foundation VLMs. Vision-language pre-training learns joint visual-textual representations to improve downstream task performance [73]. CLIP [60] pioneered large-scale vision- language con- trastive learning, enabling zero-shot transfer to a wide range of vision tasks. Subsequent foundation
'Empirical evidence for our motivation provided in Section 3.2.

---

VLMs, e.g. BLIP [43], OpenFlamingo [2], and LLaVA [49, 48], have demonstrated strong multi- modal understanding and generalization. Our study concentrates on improving the robustness of the widely adopted CLIPP model, generalizing robustness to other foundation VLMs via a plug-and-play replacement of their vision encoders with our robust counterpart. To mitigate computational costs and potential overfitting associated with full fine-tuning, we explore Parameter-Efficien Fine- Tuning (PEFT) [36, 35, 56, 76, 74] within our robust weak-to-strong generalization framework.
Weak-to-strong generalization for superalignment. Inspired by the challenge of aligning super- human models via weaker supervision (superalignment), Burns et al. [5] first explored fine-tuning large-scale models using weak-model supervision. This paradigm, termed tion, improves the generalizability of the strong student model via a weak teacher model, distinct from standard knowledge distillation [34]. Lang et al. [40] established the error bound for weak supervision, highlighting the significance of pseudolabel correction. Our study explores prediction- based pseudolabel refinement by generating more benign inputs for improved supervision via an inverse procedure of adversarial generation. Subsequent works studied the inherent reliability of weak teacher supervision [29, 30J. However, prior works focus single-modal vanilla knowledge transfer for specific tasks. In contrast, we address an underexplored problem: robust weak-to-strong generalization for VLMs, generalizing natural and robust knowledge across diverse multimodal tasks.
generaliza-
Adversarial robustness of VLMs. The increasing security risks posed by adversarial examples [37, 70] have spurred extensive research on defense schemes for VLMs [63, 75, 3, 17, 18, 16, 19]. Adversarial fine-tuning, a leading defense paradigm, augments training data with adversaries based on naturally pre-trained VI.Ms, e-g., CLIP [60]. Mao et al. [55] first introduced adversarial fine-tuning within a contrastive learning framework to align adversarially perturbed image embeddings with their text counterparts. Schlarmann et al. [63] developed an unsupervised adversarial fine-tuning strategy by preserving the features of the original CLIP model. Unlike prior adversarial fine-tuning approaches that exclusively rely on training data, we focus on a weak-to-strong generalization framework guided by a weak VLM to elicit the strong robustness generalization capability from a strong VLM across diverse downstream tasks, offering a promising solution toward the superalignment challenge for foundation VLMs.
3 Robust Weak-to-strong Generalization for Superalignment
Below, we propose Adv-W2S, the first robust weak-to-strong generalization framework to elicit zero- shot robustness from large VLMs across a variety of langunla tasks without extra supervision.
3.1 Revisiting Adversarial Fine-tuning and Weak-to-strong Generalization
Adversarial Fine-tuning. CLIP [60] consists of an image encoder fo, IR and a text encoder fo. T T- Rd parameterized by 0 (r), These encoders map image-text pairs (x, t) into d- dimensional features. Classification is performed via computing the probability that input x belongs to class CE (1....,C) C) via softmax over the image-text feature cosine similarity:
exp cos( fo, (x), for (tc)))
(x(x) Ee=11 (cos( (So, (x), Sor (t.))))
where exp(-) and cos(.) denote the exponential and cosine similarity functions. Each text prompt is
tokenized by 9()) and embedded by for(),, denoted as t. = = [Context][CAASS_ "). ").A A typical
template is "This is photo of [CLASS,-]". We define p(x) = (x)................ PC as the prediction for input x across C' categories. Given image-text pairs D. standard adversarial fine-tuning (TeCoA) [55] is framed as a minimax optimization to improve CLIP robustness:
PC (x) T T [0, 1
min E (x.O)D
max LCE 11
+ (cE(p(x+0), +6),
(2)
where ((c) = (
(c- C) (0, is one-hot label for class c, and Len is the Cross-Entropy (CE) loss. The adversarial example x e lies within an -nom hyperball of radius around x. Further details about adversarial learning are in Appendix A. CLIP parameters optimized by empirical risk on ad adversaries, generated via Projected Gradient Descent (PGD) [6].
(++1) = ll (9) + a sign (C
(3)
3

---

where s-gn(-) represents the sign function, and the scalar a denotes the step size. The projection operator (xx)) restricts the adversary in lo -nom hyperball with e-radius around x. The starting point is randomly initialized x ~ - 0000000- 1) with the final adversary 8= x=x(m)) after m steps.
Weak-to-strong Generalization. In analogy to superalignment, Burns el al. [5] proposed weak- to-strong generalization via fine-tuning a strong student model using soft-label supervision from a weak teacher. We extend this idea to VLMs, denoting the weak teacher VLM as [fo (-), ()) for (-)
and the strong student VLM as (), fof())) the prediction of the teacher and student on input x. The standard weak-to-strong generalization approach is formulated as an auxiliary confidence loss:
(+), Based on Eq. (I), let (x) and ps(x) denote
CAuxConf (1 8) LCE (Ps(x))),,)))))))))))))))))))) pT(()) + B (CE(ps(x).
(4)
where M(-) produces one-hot labels based on strong model predictions, and 3 controls the trade-off between teacher-student prediction alignment (first term) and student self-refinement (second term).
Problem definition. Unlike standard robustness evaluations that assume in in-distribution adversar- ial attacks [1]], we study more challenging zero-shot robustness scenario [55]. where adversaries originate from unseen distributions during inference. Under practical defense conditions, we presume textual prompts are fixed and safeguarded, as they typically reside within multimodal systems and are thus not subjected to manipulation. Within this demanding zero-shot robustness context, we further explore how weak-to-stron generalization contributes to improved robustness elicitation across different vision-language model settings and downstream- generalization.
3.2 Can Vanilla Weak-to-strong Generalization Elicit Robustness?
While vanilla weak-to-strong general- ization improves natural performance in single modality tasks [5], its effec- tiveness in enhancing VLM robust- ness remains underexplored. We in- vestigate whether the vanilla weak-to- strong generalization scheme (Eq. (4)) can elicit robustness from the strong student model. As shown in Table 1. standard weak-to-strong generaliza- tion primarily improves natural perfor- mance, but fails to yield robustness in large-scale student VLMs. This con-
Table I: Performance (%) of fine-tuned teacher CLIP models, and their weak-to-strong generalizations, where robustness
is w.r.t. Auto-Attack [12]. Vanilla-W2S and Adversarial-W2S,
denote vanilla and adversarial weak-to-strong generalization.
ImageNet
Avg. 13 Datasets
Method
Clean
Robust
Clean
Robust
Natural Fine Tunnnn Robust Fine-Tuning
76.06
0.00
58.64
0.01
64.96
39.74
48.27
32.92
Natural FT
Vanilla-W W25
79.54
0.00
65.50
0.26
Robust FT
Vanilla-W2
72.72
5.60
55.37
9,20
Natural FT
Advasarial-W2N
76.10
50.82
63.28
40.97
Robust FT Advesarial-W22
74.95
51.97
62.57
41.70
trasts with vanilla knowledge distillation, where robustness has been successfully transferred from robust teachers [27]. Intriguingly, even substituting the weak guidance with an adversarially robust VLM (e.g., TeCoA [55]) does not facilitate robustness elicitation. We attribute this to a mismatch in model capacity and learning objectives. Representations from a limited-capacity VLM trained on clean data poorly generalize to high-capacity VLM tasked with handling unforeseen adversaries.
To validate our claim, we explore an adversarial variant of vanilla weak-to-strong generalization by shifting the data perspective of the strong student VLM to adversarial samples (colored in red):
CAdv-AuxConf (1 B) Lce (ps(ps( + 8), px + B Lce (ps(x + 8), M(Ps(x))) (5)
Integrating adversaries into weak-to-strong generalization elicits robustness even with guidance from a non-robust weak VLM (Table 1). The necessity of employing adversaries arises from their ability to represent worst-case input distributions, enabling the strong student VLM to better mimic the weak teacher's ss robust behavior and its own self- seff-knowledge refinement. Thus, we focus on adversary-driven weak-to-strong generalization for transferable robustness. To mitigate potentially unreliable supervision from weak teacher VLMs, we investigate two complementary mechanisms: alignment re-weighting and source guidance refinement, supported by theoretical analyses in the following sections.

---

3.3 Entropy-guided Uncertainty Re-weighting
Recall that in soak-to-ttrong generalization, the trade-off factor balances teacher-student alignment and student self-refinement, However, / is either fixed or follows predefined warm-up schedule, neglecting the potential prediction errors from the weak teacher model. Thus, we propose an adaptive re- re-weighting mechanism guided by the teacher's prediction uncertainty, quantified via entropy:
Teacher entropy. We define teacher prediction entropy as ++(x)=-> Ee [x) [x) log [PT(x)],
capturing uncertainty in an unsupervised scheme: higher entropy Hr(x) indicates greater uncertainty (predictions are close to uniform distributions), whereas low entropy reflects more confident predic- tions. By leveraging prediction entropy reliability quantification, we adaptively emphasize the low-entropy guidance from the teacher model, enhancing robust knowledge superalignment.
We seek an instance-wise re- re-weighting function w(-) that maps prediction entropy weights while preserving relative differences in uncertainty across samples. To avoid uncertainty scale distortions from ad hoc normalization, is required to be monotonically increasing in Hr (so that more uncertain examples receive higher weight). A suitable choice is a soft normalization:
(x(x)
w(x) (x(x) + AHH
where KH is obtained by an adaptive scaling strategy the median entropy of the teacher's prediction over the dataset: KH = median [[Hr(x)|x E D)). ) The median entropy serves as a robust central tendency measure against outliers, allowing confident teacher predictions to guide alignment, while uncertain cases prioritize the student self-refinement. Robust weak-to- to-soong generalization (Eq. (5)
can be reformulated by replacing fixed B with entropy -uiided weighting w(x) per input x.
Theorem (Adapted from [25]). Let the weights of the classifier, i.e., in our case these are textual VIM embeddings denoted as 4b_- = (te), and the vision feature extractor be given by Φ(.). Let H[]]
be the Shannon entropy. Then for the conditional entropy over predictions holds true that:
1lx(x) 2 log(C)- (-(p-- ((x), L....., 4c)]
2 max;=1.....c 1|2,112
Proof. See [25].
Hence, with fixed textual embeddings (denominator), lower prediction entropy tightens the model selection space for the vision encoder in VLMs by favoring models with larger weights. The vision encoder is thus constrained to more stable optimization trajectory and is less prone to overfitting.
Definition (Classification margin in VLMs). Consider VLM of the form (Φ(-), ( (1..)..). where 6(x) E Ied is the vision feature for input x, and each vb. E Fd a text embedding representing class c. For input x with ground-true label y. the classification margin is defined as:
((lx)) == 1by, o(x) max 1b? (xx).
(8)
cF
The input x is correctly classified as y when the classification margin (v/x)> > 0, with larger 7(lxx) indicating a wider separation from the nearest decision boundary.
Theorem 2. Let (o(-). {&b (e))) be the same VLM setup as in Definition 1. Assume o()) L- Lipschitz continuous w.r.t. the input norm, i.e., (lx(x)
L||x all x, E X. Suppose input x satisfies Theorem I (iee, low prediction entropy enforces large feature norm [[x(x)]12) Under the mild alignment assumption that o(x) is not close-to-ore taas-ttoorthhoonall to the ground-truth textual embedding by of class y. increasing the vision feature norm [[0(x) l|2 expands the margin (x)). If 7(x) >0, O, the prediction for (x+ 65) cannot be altered away from the ground-truth label y for any perturbation d with:
7(x)
|10| 110112 < L maxx 11 2/ 2by112
Proof. See Appendix C.I.
Combined Theorems 1 & 2, and Definition indicate that low-entropy prediction enforces a large vision feature norm, which in turn amplifies the classification margin. Under mild Lipschitz assumption, this margin translates to robustness against input perturbations. In other words, model with highly confident predictions naturally places each sample far from the decision boundary in feature space, making it less susceptible to small adversarial or noisy modifications.
5

---

3.4 Teacher Guidance Refinement via Inverse Adversarial Examples
Beyond re-weighting uncertain teacher predictions, we investigate an adaptive prediction refinement scheme to reduce reliance on erroneous supervision. In other words, we focus on generating more benign inputs to improve guidance for robust weak-to-strong generalization model. Instead of generating adversaries that reduce confidence (i.e., loss-input gradient ascent), we consider an inverse procedure by reversing the gradient to maximize the likelihood the neighborhood region of clean samples. Thus, this input perturbation enhances teacher prediction confidence. Such refined samples highlight class-specific features, offering more reliable guidance for the robust weak-to-strong generalization framework.
Inverse adversarial perturbation formulation. Given an input x, we aim to generate benignly perturbed sample x ==x+ō within an foc bounded region (|j bm S E), which maximizes the teacher's prediction confidence. Formally, this inverse adversarial example is defined by:
x(++1) [I x - a- sign (iign(Vvsne (e (MI
M(PT
(10)
Equivalently, the perturbation ò shifts the teacher's prediction pT (X) closer to a one-hot distribution based on the teacher-derived pseudolabels Mxpr(x)). This process contrasts standard adversary generation Eq. (3) by reinforeing, rather than undermining, the teacher's initial decision, creating an "inverse adversary". Consequently, the resulting inverse adversary confidence and refines teacher supervision, supporting robust weak-to-strong generalization. We reformulate LCB (x 6). (x)) into its refined counterpart LCE ( (x (x+ 6). (x)) by aligning adversarial predictions with high-confidence teacher outputs. To justify this refinement, Theorem formalizes the unique role of inverse adversaries as effective guidance distinct from standard inputs.
Ô enhances class-specific
Theorem 3. Let PT (x) and Ps(x) denote the softmax predictions over C' categories w.r.f the teacher and student VLM during weak-to-strony reatono- generalization, respectively. Suppose = x + is an adversarial example for student, while x = x + ô is inverse adversarial example for the teacher with high confulence in the ground-truth class. Then the following cross-entropy inequality holds:
(ce(ps(8),
2 Lce (ce(pp(x),Pr))
(11)
where (cE(q.P)=->))) - (g) is the cross-entropy of w.r.t. target p. Proof. See Appendix C.2.
Theorem 3 shows unique and beneficial property of using our proposed unsupervised inverse adversaries as guidance during robust veak-to-strong generalization. Reducing the prediction gap between py (&) and ps(() also enhances the standard teacher-student prediction alignment for clean samples, thus enhancing natural generalization to unseen data.
Objective function. We formulate Adv-W2S as a simple min- min-max optimization by replacing the learning objectives in standard weak-to-strong generalization (Eq. (4)) with adversarial and inverse adversarial examples. Under the unsupervised setting, we conduct both adversarial perturbation and inverse adversarial perturbation ò generation based on the pseudolabels from the teacher VLM:
8 arg max Lce (Ps(x + 8), (x)) and ò = =aarg arg min [|
LCE + M)p (12)
||8|1__
We can thus form adversarial example & = and its inverse adversarial counterpart x based on the maximization above. By reformulating the adversarial variant of weak-to soek-tt-strong generalization (Eq. (5)) with our entropy- entropy-guided uncertainty re-weighting (Eq. (6)), our Adv-W2S minimizes:
C Adv- (1-w(x)) LeE (ps (x), (x)) + w(x) Les (P)
(13)
During the inference stage, we directly use the CLIP model fine-tuned via our Ady-W25 method for robustness evaluations and further cross-task generalization without modifying VLM architectures.
4 Experiments
In this section, we compare our Ady-W2S with state-of-the-ar adversarial fine-tuning approach across various downstream tasks and scenarios. Below, we detail our experimental configurations.

---

Table 2: sero-shot clean and robust accuracy (%). Adversarial learning is conducted on ImageNet
with evaluations across 14 datasets. Adversaries are generated via Auto-Attack with radius c= 2=2/255.
ImageNNt
CIFAR-10
CIFAR-100
Suat.Car
Callcch 101
OxfondPer
Flower102
EuroSAT
ImageNor-R
ImageNerss
STL.10
FOVE
PCAM
DTD
Eval.
Method
Average
Robest Clean Acc.
17.17
TeCoA
80.7
15.00
FARE [69]
80.26
49
Adv-W2S
75.84
98.48
91.64
91.11
70.68
49.57
24.96
63.51
85.37
59.58
68.75
Stundard IP
0.02
0.00
0.00
0.00
0.00
0.01
0.09
TeCoA [55)
61.90
18.62
70.57
68.
27.27
26.17
[69]
60.02
37.14
5.79
47
62 24
45.08
45.74
FARE (71)
61.46
86,
16.56
35.44
21 60
68
26
11.37
47
60
45. 19
Adv-W2S
58.30
90.49
69.56
41.08
29.62
73.03
73.24
34.33
29.63
11.06
7.53
48.76
65.32
45.43
48,38
Table Average accuracy (%) of diverse CLIP
Table 4: Average accuracy (%) of diverse when fine-tuning and testing w.r.t CLIP w/ ViT-L.
backbones with perturbation radius e= 2/255.
Backbone
Method
Clean
Robust
Radius
Method
Clean
Robust
TeCoA [55]
43.40
23
TeCoA [55]
58.90
38.07
PMG [69]
46.34
22.48
PMG [69]
61.72
39.40
ResNet101
FARE [63]
47.88
16.61
c= 3/255
FARE [63]
63.55
37.17
TGA [7]
45.29
23.18
TGA [71]
59.65
38.59
Ady-W2S
50.14
25.93
Ady-W2S
64.79
40.85
TeCoA [55]
49.46
33.41
TeCoA [55]
56.25
32.53
PMG [69]
53.57
33.05
PMG [69]
58.82
33.87
ViT-B/16
FARE [63]
54.09
30.54
<= 4/255
FARE [63]
60.26
32.02
TGA [
51.32
33.07
TGA 771
56.76
32.86
Adv-W25
55.42
35.20
Ady-W2S
61.56
35.19
Datasets. In line with prior works [55, 63], we conduct adversarial learning on the ImageNet training set [13], with zero-shot classification evaluations on its test set and other 13 datasets. We further explore downstream task generalization across various datasets w.r.t. image captioning, visual question answering, object hallucination, and science question answering (see Appendix B.
Implementation details. Unless specified otherwise, we adopt CLIP [60] with the ViT-Large/14 14 architecture, as previous studies [55, 63]. During adversarial weak-to-strong generalization, we conduct adversary generation via 10-step PGD with perturbation radius = 2//255 and step size 1/1/2 in an unsupervised scheme. In analogy to superalignment, we consider relatively weak teacher of the ViT-Base/32 architecture, pre-trained on the ImageNet training set using TeCoA [55]. Zero-shot robustness is assessed under Auto-Attack [12], an ensemble adversarial attack method for reliable evaluations. We achieve downstream task generalization based on two large-scale VLM frameworks, LLaVA 1.5 7B [48] and OpenFlamingo 9B [2], by replacing vision encoders with our robust counterparts, For faimess, evaluations are under adaptive attacks. Details are in Appendix B.2.
4.1 Main Results (Zero-shot Classification)
Zero-shot classification performance. Table 2 compares Ady-W2S with ttate-of-the-art adversarial fine-tuning methods using CLIP ViT-L. We report both clean and robust accuracy (Auto-Attack [12)) across 14 datasets. Our Ady-W2S achieves the best zero-shot performance with an average improvement of -5% in clean accuracy and -5.8% in robustness. While in-distribution accuracy on ImageNet drops slightly due to the unsupervised learning scheme [63], Section 4.3 shows this gap can be mitigated by incorporating a supervised objective into robust weak-to- yeak-to-strone to-stoong generalization.
Robustness across diverse CLIP backbones. In addition to robustness with VIT-L, we here apply our Adv A2s method on ResNet101 and ViT-Base/16 based on the weak teacher VLM of ResNet50 and ViT-Base/32, respectively. Table 3 shows that our Ady-W2S consistently outperforms other adversarial learning approaches in both average clean and robust accuracy across 14 datasets.
Adversarial learning with diverse perturbation radii. Beyond the default perturbation radius e 2/255, we explore adversaries of larger -nom perturbation radii - 3/ 3/255, 4/255) during both fine-tuning and robustness evaluations for fair comparisons. As shown in Table 4, we observe that our Ady-W2S W2S surpasses other methods across diverse perturbation radii in zero-shot scenarios.
7

---

Table 6: Zero-shot transfer performance on image captioning (CIDEr score) and VQA (accuracy %)
Captioning
Visual Question Answering
VLM Type
Method
COCO
Plickrilk
TexiVQA
VQAv2
Vizwiz
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Clean
Robust
Standard CLII
112.3
2.9
74.7
1.0
34.8
0.0
74.5
0.0
34.4
2.3
TeCoA [35]
96.7
45.1
55.2
24.0
23.8
12.8
66.2
35.7
42.5
29.6
PMG [69]
103.1
52.8
63.2
28.4
27.6
14.0
68.4
35.1
41.0
27.6
LLaVA 1.5
FARE 163
108.5
47.9
67.4
24.5
30.5
14.7
70.3
14,5
41.9
25.3
TGA [
101.3
50.6
61.9
27.8
27.1
14.6
67.3
35.0
42.8
28.0
Ady-W2S
110.8
55.3
72.6
30.7
32.5
16.4
72.9
37.3
45.3
33.8
Standard CLIP
78.8
1.5
58.7
22.3
47.7
0.0
17.7
3.3
TeCoA [55]
73.0
29.6
47.4
13.7
17.3
2.4
46.1
23.8
17.
4.0
PMG [69]
76.2
31.2
52.0
16.5
17.7
2.8
47.0
24.0
16.9
4.2
OpenFlamingo
FARE [63
77.9
$2.7
53.5
15.9
18.8
2.2
46.7
21.8
17.2
3.8
TGA 7
74.2
30.5
51.8
16.0
19.0
2.7
46.2
23.0
18.0
2.5
Ady-W2S
80.0
34.3
56.9
17.1
20.5
3,9
47.4
25,3
18.2
5.3
Extension with adversarial PEFT. Fully fine-tuning large-scale VLMs typically introduces significant com- putational costs. We therefore extend adversarial VLM learning with LoRA [35]. PEFT strategy using trainable low-rank matrices for efficient adaptation. We report both clean and robust accuracy in the zero-shot setting of our LoRA- AeRRedd Adv-W2S as well as other approaches with the LoRA strategy (see Table 5). The results indicate that our Adv- Adv-W2S can still outperform other approaches, even with the LoRA strategy for efficiency.
Table 5: Acc. (%) w.r.t. diff. for fine- tuning and testing (ViT-L) with LoRA.
Radius
Method
Clean
Robust
TeCoA [55]
55.22
26.54
PMG [69]
56.82
27.02
3/255
FARE [63
57.84
23.85
TGA [7]
55.88
26.66
Adv-W2S
59.21
29.13
TeCoA [55]
49.84
19.87
PMG
[69]
52.08
20.07
4/255
FARE [63]
53.20
19.09
TGA [7]]
51.13
19.83
Adv-W2S
54.28
21.91
4.2 Zero-Shot Downstream Task Generalization
Image captioning extension. We here extend A2v to image captioning by replacing the vision encoders of large-scale VLMs (e.g., LLaVA and OpenFlamingo) with our robust versions. We report the CIDEr score [66] for both COCO and Flickr30k datasets (see Table 6). We observe that our Adv-W2S achieves the best captioning performance in terms of both clean and adversarial examples compared to other adversarial learning approaches. In addition to quantitative results, we further provide image captioning visualizations against unforeseen adversaries in Figure 1 in Appendix F.
Visual Question Answering (VQA) extension. Table 6 reports VQA accuracy [1] across three standard VQA datasets using different VLMs. Our Adv-W2S method receives large gain in zero- shot robustness while maintaining comparable natural performance with standard CLIP. Note that our method can even outperform standard CLIP in clean accuracy on Vizwiz, leading to lossless robustness enhancement. The corresponding visualizations are provided in Figure 2 in Appendix F.
Object hallucination extension. Foundation VLMs are vulnerable to object hallucinations (i.e., erroncously recognizing objects that do not exist in inputs) [61]. POPE [44] served as an ob- ject hallucination evaluation benchmark based on VQA across diverse question sampling sce- narios (details Appendix B.3). We extend ad- versarial VLM learning with hallucination eval- uations in Table (see also visualizations in Figure in Appendix F). Notably, our Ady-W25 shows
Table 7:
Hallucination evaluations (Fl-score) us- ing POPE for adversarial VLM learning (ViT-L).
Method
POPE Sampling
Avg. Score
Random
Popular
Adversarial
TeCoA [55]
79.8
79.
75.2
78.0
PMG (69)
81.7
80.9
76.3
79.6
FARE 63)
82.2
81.5
78.6
80.8
IGA [71]
80.4
79.8
76.0
78.7
Adv-W2S
85.6
84.9
$1.0
83,8
reduced hallucination rates, benefiting from our inherent regularization to over-confident or overly aligned predictions by balaneing teacher-student alignment with model uncertainty.
Science question answering extension w/ CoT.
Table 8: CoT eval. (Acc.) using science question answering for adversarial VLM learning (ViT-L).
Chain of Thought (CoT) has been widely ex- plored to elicit intermediate reasoning steps from large-scale VLMs by guiding them to gen- erate multi-step rationales before producing final answer [8]. Science Question Answering [52] has been recognized as standard evalua- tion benchmark for CoT due to its large quantity of multi-option questions with multimodal con- texts from the science curriculum. As shown in Table 8, we extend diverse adversarial VLM
Setting
Method
Temperature
Avg. Ace.
0.0
0.1
0.2
TeCoA [SS]
$1.4
$1.6
50.0
$1.0
[69]
Stundard
FARE T63
52.2
52.4
TGA [71]
52.1
51.9
51.8
Ady-W2S
53.8
53.9
53.6
53.8
TeCoA [S5]
67.4
67.6
67.1
67.4
Single
PMG [69]
68.2
68.0
67.7
68.0
Choice
FARE [63]
67.7
Adv-W2S
69.3
69.5
69.0
69.3
8

---

learning methods to science question answering across different settings to evaluate their CoT capabil- ity. Further details of diverse configurations are in Appendix B.3. Our method consistently achieves the best science question answering performance across diverse settings, which demonstrates that our robust weak-to-strong generalization potentially enhances the CoT capability of large-scale VL.Ms. Visual examples are in Figure 4 in Appendix F.
4.3 Further Analyses (Why Adv-W2S is Effective)
Below we conduct systematic analyses of our proposed Adv-W2S framework and its component modules to justify its efficacy and generalization capability across diverse configurations.
Impact of component modules. We analyze the con-
Table 9:
Ablation study of key compo-
tribution of two main components of our Adv-W2S: (i)
nents in our Adv-W2S for average clean and robust accuracy (%) on 14 datasets.
Entropy-guided Uncertainty Re-weighting (EUR) from Eq. (6), and (ii) Inverse Adversarial Refinement (IAR) in Eq. (10). Table 9 reports the average clean and robust accuracy across 14 datasets in the zero-shot setting. We adopt the adversarial variant of weak-to-stiee to-strong generalization from Eq. (5) as the baseline (first row in Table 9). Enforcing instance-wiss re-weighting to balance prediction aligmment and self-refinement contributes to im- proving both natural performance and adversarial robustness. Refining the teacher guidance with inverse adversarial examples also enhances natural generalization to unseen data, yielding further gains in the zero-shot accuracy.
EUR
IAR
Clean
Robust
63.45
42.43
2 3
/
66.42
47.35
/
67 57.09
46.79
68.75
48.38
Impact of teacher-student setups. In the stan- dard weak- to--ttoog generalization paradigm, the weak teacher model is typically pre-trained using task-specific supervised learning under natural -dversarrsar conditions. To provide further insights into the teacher-student config- urations employed in our adversarial weak-to- strong generalization (Adv-W2S) framework, we summarize the key setups (i) whether the weak teacher is pre- are-trained under adversarial or natural conditions, and (ii) whether an unsupervised adversarial fine-tuning (i.e., FARE [63]) is employed during the initial warm-up stage of Adv-W2S. Table 10 shows that an adversarially pre-trained teacher with unsupervised adversarial fine-tuning for student warm-up enjoys greater zero-shot robustness.
Table 10: Average performance (%) of our Adv- W2S method with diverse teacher-student setups.
Teacher
Student
Clean
Robust
Nat. Pre- Trained
w/o Adv. Warm-up
68.05
46.79
Adv. Pre-Trained
wlo Adv. Warm-up
67.53
47.40
Nat. Pre- TheiTrained
w/ Adv. Warm-up
47.02
Adv. Pre- Trained
w/ Adv. Warm-up
68.75
48.38
Impact of adversary generation schemes. Be- low, we analyze the average zero-shot classifi- cation performance of diverse adversary gener- ation schemes (i.e., objective functions for gen- erating adversarial examples +-) in Table
Table 11: Average accuracy (%) of our Adv-W2S with diverse adversary generation schemes.
Adversary Generation
Clean
Robust
Student Feature Deviation
67.17
46
Student Prediction Deviation
65. 98
47.20
11. We observe that adversary generation by maximizing the leacher-student prediction gap enforces a better robustness transfer in the context of weak-to-strong generalization. More details are in Appendix D.
Teacher-Student Prediction Distance
68.75
48.38
Impact of inverse adversary generation schemes. In addition to diverse adversary gen- eration strategies, we also investigate a range of inverse adversary generation approaches (details
Table 12: Average accuracy (%) of Adv-W2S with
diverse inverse adversary generation schemes.
Inverse Adversary Generation
Clean
Robust
Pseudo-Margee Minimization
68.19
47.82
in Appendix D.2) in Table 12. The results in- dicate that using the pseudolabel cross-entropy loss leads to inverse adversaries of more benign guidance for robust weak- tor--tto-ttonon generalization.
Prediction Entropy Minimization
67.93
47.34
Pseudolabel Cross-Entropy 68.75
48.38
Auxiliary ground-truth supervision. Recall that we primarily focus on ro- bust teak-to-strong generalization in an unsupervised scheme following the standard setup [5]. Despite its im- proved zero-shot performance, its in- distribution performance on the fine- tuned dataset is still lower than super-
Table 13: Clean and robust accuracy (%) of our Adv-W2S
with/without an auxiliary ground-truth supervision.
Method
ImageNet
Avg. 13 Datasets
Clean
Robust
Clean
Robust
TeCoA [55]
80.00
61.74
60.14
41.84
Unsupervised Adv- A2S
75. 84
58.30
68.20
47.62
Supervised Ady-W2S
80.56
64,08
65.72
45,25
vised adversarial fine-tuning at the same VLM backbone. Thus, we investigate the underlying effect

---

of appending an auxiliary ground-truth supervision in our Adv-W2S method (see Table 13). Details of this auxiliary branch are in Appendix E. We observe an inherent trade-off between nn-distribution (zero-shot) performance other datasets.
performance on ImageNet and
Hyper-parameter sensitivity analyses We further provide systematic analysis of key hyperparam- eters involved in our proposed Adv-W2S method in Appendix G.
5 Conclusions
Motivated by analysis of standard weak-to-ste stronn generalization paradigm in eliciting VLM robustness, we have uncovered that neglecting adversarial examples in alignment objectives leads to robustness degradation, even when leveraging source guidance from adversarially pre-trained VLM. Thus, we have investigated an adversarial adaptation of standard weak-to-strong generalization, explicitly integrating adversarial examples elicit robust knowledge in unsupervised scheme. Recognizing that supervision from a weak-capacity VLM may be inherently unreliable, we introduce two complementary strategies: uncertainty-base alignment re-weighting and source guidance refinement via inverse adversarial examples. Further theoretical analyses characterize the robustness elicitation efficacy of method, demonstrating enhanced generalization against subtle adversarial or noise modifications.
Acknowledgments
This research is supported in part by National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative, partly supported through the Al Singa- pore Programme under the project titled "AI-based Urban Cooling Technology Development" the Centre for Frontier Artificial Intelligence Research, Institute of High Performance Computing, A Sttar, and the College of Computing and Data Science at Nanyang Technological University. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, and Infocomm Media Development Authority. Piotr Koniusz is supported by CSIRO's Science Digital.
No. -ISG33-TC-22244-1----050
References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zimieck, and Devi Parikh. Vqa: Visual question answering. Proceedings of the IEEE international conference computer vision, pages
2015. 8, 18
[2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023. 3, 7, 18
[3] Rishika Bhagwatkar, Shravan Nayak, Pouya Bashivan, and Irina Rish. Improving adversarial robustness in vision-language models with architecture and prompt design. In Findings of the Association Computational Linguistics: EMNLP 2024, pages 17003-17020, 2024. 3
[4] Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:22 2022. 1
[5] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-stronn generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning. ICML, 2024. 1, 4,
[6] Paul H Calamai and Jorge Moré. Projected gradient methods for linearly constrained problems. Mathematical programming, 39(1):33-116, 1987. 3
10

---

[7] Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575. 2018.
[8] Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth A survey of chain of thought reasoning: Advances, frontiers and future. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), ACL, pages 1173-1203, 2024.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pages 3606-3613, 2014. 17
[10] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
17
[11] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. 4
[12] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In International conference on machine learning. pages 2206-2216 PMLR, 2020. 4, 7. 18
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009. 7, 17
[14] Junhao Dong, Piotr Koniusz, Junxi Chen, and Yew-Soon Ong. Adversarially robust distillation by reducing the student-teacher variance gap. In European Conference on Computer Vision, pages 11. Springer, 2024.
[15] Junhao Dong, Piotr Koniusz, Junxi Chen, Jane Wang, and Yew-Soon Ong. Robust distillation via untargeted and targeted intermediate adversarial samples. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 28432-28442, 2024. 2
[16] Junhao Dong, Piotr Koniusz, Liaoyuan Feng, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21 21037-21047, October 2025. 3
[17] Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I, KDD '25, page 236-247, New York, NY, USA, 2025, Association for Computing Machinery. 3
[18] Junhao Dong. Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Improving zero-shot adversarial robustness in vision-languss alignment of adversarial path simplices. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 14061-14078. PMLR, 13-19 Jul 2025.
models by closed-form
[19] Junhao Dong, Jiao Liu, Xinghua Qu, and Yew-Soon Ong. Confound from all sides, distill with resilience; Multi-objective adversarial paths to zero-shot robustness. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 624-634, October 2025. 3
[20] Junhao Dong, Seyed-Mohsen Moosavi-Dezfood Jianhuang Lai, and Xiaohua Xie. The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11467824463 June 2023. 2

---

[21] Junhao Dong, Yuan Wang, Jian-Huarg Lai, and Xiaohua Xie. Improving adversarially ro- bust few-shot image classification with generalizable representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9025-9034, 2022. 20222 2
[22] Junhao Dong, Yuan Wang, Xiaohua Xie, Jianhuang Lai, and Yew-Soon Ong. Generalizable and discriminative representations for adversarially robust few-shot learning. IEEE Transactions on Neural Networks and Learning Systems, (000))580-55999 2024. 2
[23] Junhao Dong, Lingxiao Yang, Yuan Wang, Xiaohua Xie, and Jianhuang Lai. Toward intrinsic adversarial robustness through probabilistic training. IEEE Transactions on Image Processing, 32:3862-3872, 2023. 2
[24] Junhao Dong, Hao Zhu, Yifei Zhang, Xinghua Qu, Yew-Soon Ong, and Piotr Koniusz. Machine unlearning via task simplex arithmetic, In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[25] Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum-entropy fine grained classification. Advances in neural information processing systems, 31, 2018. 5
[26] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pultern recognition workshop, pages 178-178. IEEE, 2004.
17
(27] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3996-4003,
2020. 4
[28) Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017. 17
[29] Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03349, 2024.
[30] Yue Guo and Yi Yang. Improving weak-to-strone to-strong generalization with reliability alignment. arXiv preprint arXiv:2406. 19032, 2024. 3
[31] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3608-3617, 2018. 17
[32] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
17
[33] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021. 17
[34] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1rXXv:1503.000 02531, 2015. 3
[35] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR, 2022. 3, 8, 18
12

---

[36] Menglin Jia, Luming Tang, Bor- Ca-- Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022. 3
[37] Jachyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung. Qifan Wang. Fuli Feng, Lifu Huang, and Madian Khabsa. Roast: Robustifying language models via adver- sarial perturbation with selective training. In Findings of the Association for Computationa Linguisties: EMNLP 2023, Singapore, December 6-10, 2023, pages -444-3444, 2023. 3
[38] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 354-561, 2013. 17
[39] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[40] Hunter Lang. David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to- strong generalization. Advances neural information processing systems, 37:46837-468880 2024. 3
[41] Jan Leike and llya Sutskever. Introducing Superalignment. OpenAl Blog, 2023. 25
[42] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- -mage pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 PMLR, 2023.
[43] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 18980. PMLR, 2022.
[44] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 292-305, 2023. 8, 18, 24
[45] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014. 17
[46] Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. survey of attacks on large vision-language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.074003 2024.
[47] Daizong Liu, Mingyu Yang. Xiaoye Qu, Pan Zhou, Xiang Fang. Keke Tang, Yao Wan, and Lichao Sun. Pandora's 's box: Towards building universal attackers against real-world large vision- language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[48] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction wning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages (6296-26306, 2024. 3. 7, 18
[49] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
1,3
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, ICLR, 2019. 17
[51] Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng, Set-level guidance attack: Boosting adversarial transferability of vasion-language pre-training models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 102-111, 2023. 2
13

---

[52] Pan Lu, Swaroop Mishra, Tanglin Xia. Liang Qiu, Kai- Chang, Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022. 8, 17,
[53] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR, 2018. 17
[54] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5155 2013. 17
[55] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero- shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations. 2023. 4, 7, 8, 9, 16, 17.23 23
[56] Yao Ni, Shan Zhang, and Piotr Koniusz. PACE: marrying the generalization of PArameter- efficient fine-tuning with consistency regularization. In The Thirty-eighth Annual Conference
on Neural Information Processing Systems, 2024. 3
[57] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphies & image processing, pages 722-729, IEEE, 2008. 17
[58] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3508 IEEE, 2012. 17
[59] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-pheen correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015. 17
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021. 1, 2, 3, 7, 16, 17
[61] Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. comprehensive survey of hallucination in large language, image, video and audio foundation models. In Findings of the Association for Computational Linguisties: EMNLP, pages 1 11709- 1 11724, 2024. 8
[62] Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-moda founda- tion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pages 3677-3685, 2023. 18
[63] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision- lision-language models. In International Conference on Machine Learning, pages 7, 8. 9, 16, 17, 18
PMLR, 2024.
[64] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh. and Mareus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019. 17
[65] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In JCLR, 2014. 2
[66] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4566-4575, 2015. 8, 18
14

---

[67] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: s/ International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part 11, pages 210-218. Springer, 2018.
[68] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 00506-10518, 2019. 17
[69] Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024. 7, 8, 16
[70] Xilie Xu, Keyi Kong. Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan S. Kankan- halli. An LLM can fool itself: A prompt-based adversarial attack. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024. 3
[71] Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. In Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, NeurIPS, 2024. 7,8, 17
[72] Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre- training models. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1005-5013, 2022. 2
[73] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-languar models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 2
[74] Yifei Zhang, Hao Zhu, Junhao Dong, Haoran Shi, Ziqiao Meng, and Han Yu Piotr Koniusz. CrossSpectra: exploiting cross-layer smoothness for parameter-efficient fine-tuning. In The
Thitt Annual Conference on Neural Information Processing Systems, 2025. 3
[75] Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang. Pip: Detecting adversarial examples in large vision- languag models via attention patterns of irrelevant probe questions. In Proceedings of the 32nd ACM International Conference Multimedia, pages 11175-11183, 2024.
[76] Hao Zhu, Yifei Zhang, Junhao Dong, and Piotr Koniusz. BiLoRA: almost-orthogonal parameter spaces for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25613-25622, June 2025. 3
15

---

Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models (Supplementary Material)
Junhao Dong Cong Zhang', Xinghua Qu', Zejun Ma', Piotr Koniusz$67 and Yew-Soon Ong!. NNanyang Technological University, CCAR, IHPC, STARR,, TTikTok. Singapore, *Bytedance, Data6 LCCIIO, A Univeessity of New South Wales, Australian National University
Abstract
This supplementary material provides additional theoretical, algorithmic, and em- pirical details to support our main paper. We first present comprehensive overview of adversarial fine-tuning schemes in lision-language models (Appendix A) and detail our experimental configurations, including dataset descriptions, implemen- tation details, and downstream task extensions such as image captioning, visual question answering. object hallucination, and scientific reasoning (Appendix B). Appendix C offers formal proofs for our theoretical statements. We further elabo- rate on diverse objective functions for adversary and inverse adversary generation (Appendix D). Details with respect to the auxiliary supervised branch are provided in Appendix E, and the hyperparameter sensitivity analyses are in Appendix G. Finally, visualizations and extended qualitative results are provided in Appendix F, and broader impact and limitations are discussed in Appendix H.
A Further Adversarial Fine-tuning Schemes
We have analyzed the standard adversarial fine-tuning scheme (TeCoA) [55] Eq. (2), and here provide more details regarding other adversarial fine-tuning schemes in the context of Vision- Language Models (VLMs) for a more comprehensive background introduction.
PMG [69]. Motivated by the inherent overfiuting of TeCoA [55] with generalization degradation, PMG [69] leveraged the prediction-level guidance from the vanilla pre-trained VLM with regularization of clean samples for the target model to conduct adversarial fine-tuning, as follows:
min ((x,)) max LeE (0x + +A1 LxL (Peng (x)) x)(xx=)))))))) + (X(x)))))((XX (14) 0i
[1
where LKL. represents the Kullback-Leibler divergence, and Porig denotes the prediction of the vanilla pre-trained CLIP model [60]. A and X2 are the corresponding loss weighting factors.
FARE [63]. To enhance the robustness generalization capability across diverse vision-language tasks, Schlarmann et al. [63] proposed an unsupervised adversarial fine-tuning approach, dubbed FARE, to adversarially optimize feature-level discrepancies in an unsupervised scheme:
min (x.e))) 0.
max ||
[anig(X) - f(x + 5)
(15)
where /()) denotes the image encoder the CLIP model for fine-tuning, while forg (-) is the image encoder of the vanilla pre-trained CLIP model as the frozen reference.
"The research work was done during Junhao's internship at TikTok Singapore.
Corresponiing author.
16

---

TGA [71]. Motivated by the empirical observation that adversarial perturbations induce shifts in text attention maps, TGA [71] has been proposed to incorporate test-guided attention into adversarial fine-tuning, improving the zero-shot adversarial robustness of VLMS:
min ((x,)) max (p) + + (xxr g(x+ 8] 12 e(x)]| ||
(16)
where g(-) extracts the text-guided attention map. While the original TGA mechanism relies on the global [CLS] token available in ViT-based CLIP models to compute text-guided attention maps, this approach not directly applicable to ResNet-basee CLIP architectures due to the absence of an explicit global pooling token. To address this limitation, adapt the TGA computation by operating on the final spatial feature maps extracted from the ResNet-100 101 backbone (as shown in Table 3).
B Experimental Configurations
In this section, present comprehensive overview of the experimental setup for our Adv-W2S framework, including dataset descriptions, implementation details, and the methodology for extending approach diverse vision-language downstream tasks.
B.1 Dataset Description
In line with prior adversarial VLM learning works [55, 63], we conduct adversarial weak-to-strong generalization on the training of ImageNet [13] and evaluate its classification performance on the validation set of ImageNet, as the ground-truth labels for the test set are not publicly available. We further evaluate the zero-shot classification performance across broad spectrum of image recognition
tasks, encompassing 13 datasets:
Natural Object Recognition: STL-10 [10], CIFAR-10/100 [39], and Caltech-101 [26].
Fine-Grained Recognition: Stanf. Cars (Stanford Cars) [38], Oxford-IIIT Pets [58], Flower102 [57], and FGVC Aircraft [54].
Texture Recognition: DTD (Describable Textures Dataset) [9].
Remote Sensing Classification: EuroSAT [32].
Medical Image Diagnosis: PCAM (PatchCAMelyon) [67].
Robust Classification: ImageNet-RRenndition) [33] and ImageNet-S(ketch) [68].
In addition zero-shot image classification, we further evaluate zero-shot transfer performance range of image-text downstream tasks, including image captioning and Visual Question Answering (VQA), assessing both natural performance and adversarial robustness. For image captioning. evaluations are conducted on the COCO [45] and Flickr30k [59] datasets. For VQA, we assess performance on TextVQA [64], VQAv2 [28], and VizWiz [31]. Object hallucination evaluated on the COCO dataset [45], while Chain-of Thought (CoT) reasoning is assessed on the SCIENCE QA benchmark [52], large-scale multi-choice dataset comprising multimodal science questions with detailed explanations and diverse domain coverage.
B.2 Implementation Details (Zero-shot Classification)
Standard setups. In line with the configurations used in prior studies [55, 63], we adopt the CLIP model [60] with the ViT-Large//4 architecture for the strong student model, unless stated otherwise. In analogy to superalignment, we consider adversarial weak-to-strong generalization from a weak-capacity teacher CLIP model of the ViT-Base/32 architecture that is adversarially pre-trained on the ImageNet training using TeCoA [55]. For robust weak-to-strong generalization of other CLIP architectures (see Table 3 in the main text). we consider the relatively strong-capan student model of 10s and Vas-Bas with the corresponding weak- capaicap teacher model of ResNet50 and ViT-Base/32, respectively. During adversarial
generalization, we conduct adversary generation via 10-step PGD [53] with perturbation radius = 2/255 and step size 1/255 in an unsupervised scheme (see Eq. (12)). We set the inverse adversarial perturbation radius as ē = 2/255. For network parameter optimization, we adopt the AdamW optimizer [50] with momentum coefficients (0.9, 0.95). The adversarial weak-to-strong generalization is optimized with
17

---

a cosine annealing learning rate schedule with a linear warm-up up to the maximum learning rate of 1 x 10-5 for epochs. During the warm-up period. we also integrate the objective function FARE [63] (See Appendix A) with a tiny weighting factor of Awarm-up = 0.2. For Parameter-Fificien Fine- Tuning (PEFT) extension of our Adv-W2S method in Table 5, we adopt the Low-Rank Adaptation (LoRA) [35] framework, applied specifically to the attention modules.
Evaluation protocol for zero-shot classification. Following [63], we evaluate the classification performance in terms of clean samples and their adversarial counterparts. Specifically for adversary generation during evaluations, we adopt Auto-Attack (AA) [12] with the maximum perturbation radius of e 2/255 unless specified otherwise. For fairness, all the robustness evaluations are conducted under adaptive attacks. Note that zero-shot classification with CLIP is performed by computing the cosine similarity between image embeddings and a set of text embeddings corresponding to category-specific prompts. The input image is then classified by choosing the label whose text. embedding has the highest similarity to the input image representation.
B.3 Downstream Task Extensions
We achieve downstream task generalization based on two large-scale VLM frameworks, LLaVA 1.5 7B [48] and OpenFlamingo 9B [2], by replacing their vision encoders of the ViT-Large/14 architecture with our robust counterparts. We further explain specific modifications and evaluation protocols for individual vision-language task extensions below.
Configurations for image captioning extensions. The CIDEr score [66] adopted to measure the similarity of a generated sentence against a set of ground-truti sentences for image captioning
evaluations in this paper. For adversary generation in the context of image captioning, we conduct APGD attacks [12] with 100 steps against each ground-truth caption w.r.t. each image following [63]. After each attack step, we compute the CIDEr scores and exclude samples from further attacks their scores fall below our predefined thresholds, 10 for COCO and 2 for Flickr30k, representing less than 10% of the baseline (LLaVA) performance. This selective strategy allows us to allocate computational resources more effectively by concentrating on samples that retain relatively high scores. the final stage, we conduct an adversarial attack with the perturbation that previously resulted in the lowest CIDEr score and using the corresponding ground-truth reference.
Configurations for visual question answering extensions. For VQA tasks, we report VQA accuracy [1] and adopt a similar evaluation scheme to image captioning, without using prediction score threshold. Note that we select the five most frequent ground-truth answers from the ten available annotations. We further conduct targeted adversarial attacks based on the text prompt strings "Maybe" and "Word". Such an evaluation protocol is also aligned with [62, 63] with APGD attacks.
Configurations for object hallucination extensions. For object hallucination evaluations, we focus on the POPE benchmark [44] convert the standard hallucination evaluation into binary classification task by prompting LVLMs with basic yes-or-no short questions about the probing objects in three object sampling scenarios:
Random sampling: Objects absent from the image selected random.
Popular sampling: From the set of objects not present in the current image, we choose the top-k most common objects across the entire dataset.
Adversarial sampling: We rank all candidate objects by their co-occurrence frequency with ground-truth objects and choose the top-k among those not appearing the input image.
Configurations for chain of thought extensions. We assess on the ScienceQA test split based on the LLaVA VA framework. During inference, we follow two prompting/evaluation regimes: standard and single-choice. In the standard setting, the target VLM first produces an unrestricted chain- of-thought (CoT) rationale and is then re-prompted to extract its final answer token, emulating free-form scientific reasoning. While in the single-choice setting, we append the instruction "Answer with the option's 's letter from the given choices directly." " to the initial prompt, foreing the model to emit exactly one of (A, B, C, DJ. We then test both scenarios w.r.t. different temperatures. Note that in Table 8 in the main text, the single-choice setup typically achieves better performance than the standard case. Constraining the decoder to output only the option label can potentially climinate major eiror source present in the standard CoT pipeline. The classifier-style
18

---

prompt concentrates probability mass on four symbols, reducing exposure to sampling noise and temperature-indune induuce drift that can corrupt the final answer token.
C
Theoretical Analyses
C.1 Proof of Theorem
We now detail the proof of Theorem 2 that a large feature-space norm can yield a large classification margin, which in turn leads to robustness against small input perturbation.
Theorem 4 (Theorem from the main text), Let (o(.), ba ))=)) be the same VLM setup as in Definition I. Assume (6.) is L-Lipschitz continuous w.r.f. the input norm, iee., 1 (x(xx) || x||x-'|| for all x, x E. Suppose input x satisfies Theorem I (i.e., low prediction entropy enforces large feature norm [6(x)]l2). Under conditions where correct prediction
Definition I holds,
increasing the vision feature norm [lo(x)l22 expands the margin (x)). If (f)(x) O, the prediction for (x+8) cannot be altered away from the ground truth label y for any perturbation de with:
7(x)
|| S 1181/2 < L max 11 ,2ll2
(17)
Proof. We consider the VLM (o(:), (10)) (. where v. are ez-nomaaliedd, and L is the Lipschitz constant of w.r.t. the input norm:
|(x(x) - 6(x')l2 S -||x-xX|] for all x, x' E X.
Such Lipschitz continuity commonly holds under standard smoothness or regularity assumptions or can be enforced by spectral normalization in neural network layers, or locally by aligning adversarial samples with their clean and/or inverse adversarial counterparts at the feature level (we indeed have such an alignment realized by cross-entropy in our loss).
For an input example x with ground category y {1, (l...., .,. C).. define the classification margin:
(v)x) = 41 oxx) max 5. 6(x).
c/y
If (y)()))) > O, then x is correctly classified by the ground- ground-truth truth label y.
Part (i): Large feature norm implies larger classification margin. By Theorem 1, low prediction entropy forces [íø(x)]l2 to be sufficiently large, unless text embeddings tb become unbounded. Note that the textual embeddings (prompts) are fixed and safeguarded, they typically reside within multimodal systems and are thus not subjected to manipulations. Under the mild alignment assumption that d(x) not close-to-orthogonal to the ground- -otun textual embedding &b of class y. we examine the logit gap:
V5 [x) o(x) (b,- -.) + o(x) -1lb, vellall0xx)]1 cos (0w.)):
where fy.c is the angle between vectors (*b,, - *b.) and ((x). Thus, increasing the vision feature norm [xx( amplifies each logit gap provided cos (0g,c) remains bounded away from zero. Hence, the overall classification margin:
(x) = min (abb, wb)) Tx(x)
also grows as 1ø(x)]12 increases, leading to a wider separation from the nearest decision boundary.
Part (ii): Margin implies robustness. A standard argument in margin- based classification shows that if (x) is large, then no small input perturbations flip the predicted label. Formally, if > > 0. to flip the label from ground-truth y some e + y,, one needs:
$T. (x + 5) 2 vb o(x 5) -
(vb. - -b,,),,)) ,b,,) [Ø(x + 6) [xx)] 2 -x()).
According to the
inequality, one can obtain:
(+b. w,,) 10(x + 6) d(x)]| < 11 2b,,, 1,,112ll0(xx + 6) 6(x)]12.
19

---

Since Φ is L-Lipschitz, L.e., | + 6) - o(x)ll - 2| Therefore, we have:
(ve w,y,) 10(x
8) o(x) 2 - -1 121124 ||0||2.
Hence, if c0 I ||ö||2 <
1.110-- the above inequality cannor hold, preventing a label flip.
7(x)
Taking the worst-case class c 7 y via max l1 - 2 establishes the radius:
7(x)
L maXeoa y 1 V/yl2
Thus, no perturbation ||öl| below that threshold alter the predicted label from y.
C.2 Proof of Theorem 3
Theorem 5 (Theorem 3 from the main text). Lef PT (x) and ps(x) denote the softmax predic- tions over C categories w.r.. the teacher and student VLM during weak-to-stront generalization, respectively. Suppose x = X + is an adversarial example for student, while & x + 5 is an inverse adversarial example for the teacher with high confidence on the ground-truth class. Then the following cross-entropy inequality holds:
Cce (Ps(Ps()), (T) (x)) 2 Lce(ps(x).
(18)
Proof. Recall that Lcr(q. p) = - Pt log(4;) is the cross-entropy of w.r.t. the target dis- tribution p. Let teacher's prediction on inverse adversaries pT [p f2, f2, PCc. student's prediction on adversaries (*) [@t, (2,, 92.... ....c), and student's prediction on clean samples Ps(x) = [4i 02, "a,,,,,,,,,, "9c]. The cross-entropy difference in Eq. (18) can be written as:
Lcu (r - (cu(g,p7(x)) Σ ps log(()) + Σ vs log(u:) = E
Dr log
96 ) #
(19)
where == = ps()) and q == px(x). The inequality in Eq. (18) thus equivalent to:
Σ
p. log
2
By construction, & is inverse adversarial example for the teacher with high confidence in category 1. We assume the high-confidence prediction w.r.t. the inverse adversary #ı >> pú for index >> 1. In addition, let an adversarial example against the student model, which means that its prediction q reduces the probability of the ground-truth class 1 by some positive amount K1 >> 0 and increases the probability of each wrong category i by K; > O. Thus, we obtain:
à Q1
q = qi Ki, i i= 2.... C.
This adversarial shift captures the typical adversarial behavior moving mass away from the ground- truth class 1 to the remaining classes.
We thus examine the sum:
C
Σ pe log
d: V.
= in log
21
+
Dr log
9t qi-Mi
i=2
Define:
((R)...,KK)) = #1 log
p.
log
qi gitto
91 -i
We show that for K; > 0. ((s)
2 0. Its derivative w.r.t. Ki is:
0g
P1
ag and for i > 1, 3x,
p qi Ka
0k
91
Ki
20

---

Hence, the Hessian is diagonal with entries:
gg
P1
0g
ps
Q1 m1 0,
> 0,
2).
0k1
dn
(g: + kz)2
Thus, G is strictly convex in the vector k. By the first-order property of convex functions, for any R, 6", we get:
((k) 2 g(x²) + Vg(Kk) (R-=") n²).
Evaluating ~^ 0. Set (0, 0, ..... 0); then we get:
((0) = p log
+
p. log
= O.
1=2
Moreover,
V9(0)=
P1
P2
Pc
qc
Hence, for any K (Ki,,..., Kc),
0g Org =0= M1
((x) 0+
q:
Ensuring non-negativity of Eq. (19). Since #1 >> pr for i 1, we may assume P1 and f S for i 1. (This aligns with the intuition that the teacher distribution over the inverse adversarial example PT (X) is heavily concentrated on the ground-truth class whereas ps(x) is less confident.) Thus, we get the following
A1 < Kis i i= .....,..
Kl
Therefore,
G(k) 2 = 0,
1=2
implying E log (1)) 2 0. Revisiting Eq. (19) shows that the cross-entropy difference is non-negative. Hence, obtain:
Ler (q. (T))) 2 Ler (q. (x))
= (ps(ps()), pT()) (cI(ps(x).
which completes the proof of Eq. (18).
D Objective Functions for Adversary and Inverse Adversary Generation
In this section, we provide several alternatives for both adversary and inverse adversary generation in an unsupervised manner during our proposed Adv-W2S learning.
D.1 Adversary Generation in Weak-to-strong Generalization
In addition generating adversarial examples by maximizing teacher-student prediction distance in Eq. (12), we here also consider two adversary generation schemes solely based on the student VLM:: (i) feature-level deviation between clean and adversarial examples, and (ii) prediction-level deviation between clean and adversarial examples, as follows:
Student feature deviation (ōs-Feat adversarial perturbations):
8 arg max
|| + Ōs.Feat), fxx)
(20)
where fs(:) denotes the image encoder of the student CLIP model during fine-tuning.
Student prediction deviation (8g-Pred as adversarial perturbations):
j = arg max
LCE (Cs(pssx 85-Pret), ps(x)).
(21)
|[8mmall 1l <e
21

---

Clean Captioning
Adversarial Captioning
CLIP: group of people are skiing on mountain.
CLIP:
A toy car is placed on on top
Adv. Image 2/255
Clean Image
TeCoA: men walking mounfain.
TeCoA: A across a tile floor.
standing
PMG: group of people skiing mountain.
PMG: Two young girls are walking across the street.
FARE: group of people are skiing on mountain.
FARE: A group of people standing on snowy mountain.
TGA: skiing mounfain.
black and white photo of fan sitting on desk
Adv-W2S: A group of people skiing on snowy mountoin.
Adv-W2S: A group of people skiing snowy mountain.
Clean Captioning
Adversarial Captioning
CLIP: Two elephants standing
CLIP: A heart made out of dog food and dog laying
next to each
Adv. Image 2/255
Clean Image
TeCoA: elephonts walking a field.
TeCoA: A big group of elephants move across field.
PMG: Two elephants standing field.
PMG: Two zombie elephants sitting driveway.
FARE: Two elephants standing next to each other.
FARE: Three elephants walking across a niver.
TGA: couple elephants in the bush.
TGA: turtle sitting on a stage.
Adv-W2S: Two elephants walking together field.
Adv-W25: Two elephants standing in niver.
Figure I: Visual examples (clean and adversarial samples) for image captioning when using the LLaVA 1.5 framework with the robust vision encoder of diverse adversarial VLM learning methods.
Question: Is there hotel in this area? type establishment big sign left
Question: there hotel this area?
What type astablishment Adv.
big blue sign on the
Clean Image
Clean VQA
Adversarial VQA
(Yes, Hotel)
CLIP: (Word, Maybe)
TeCoA: (Yes, Barber shop)
TeCoA: (No, Barber shop)
PMG: Hotel)
PMG: (Yes, Restourant)
FARE: (Ves, Hotel)
(Yes, Borber shop)
TGA: (Yes, Bor)
TGA: (Ne, Bar)
Adv-W2S: (Yes, Hotel)
Adv-WZS: (Yes, Hotel)
Question: How much does this test cost?
Question: How much does this test cost?
Clean Image
Clean VQA
Adv. Image 2/255
Adversarial VQA
CLIP: $19.99
CLIP: $3000
DNA
PMG: $19.99
DNA
PMG: E15
TEST
FARE: $19.99
TEST
FARE: $100
Adv-W2S: $19.99
Adv-W2s: $19.99
Figure 2: Visualizations (clean and adversarial samples) for visual question answering based on
LLaVA 1.5 with the robust vision encoder of diverse adversarial VLM learning methods.
D.2 Inverse Adversary Generation in Weak-to-strong Generalization
Recall that in the main text, focus inverse adversary generation based the cross-entropy
loss with reference to the one-ho vector based on the teacher's 's prediction (xps(x)) (see Eq. (12)). We further consider two other types of inverse adversary generation in unsupervised scheme: (i)
pseudo-margin minimization, and (ii) prediction entropy minimization, as follows:
Teacher pseudo-margl minimization (ÔT.M as adversarial perturbations):
ò
8IM = arg
min 8
max p (x + dPm) max 인 (x 5F:)]]
(22)
where j arg max, o 8) denotes the index of the maximum prediction.
22

---

Question: Is there person the image?
Question: Is there traffic light the image?
Image
POPE Hallucination VQA
Image
POPE Hallucination
TeCoA: No
TeCoA: No
FARE: Yes
FARE: Yes
TGA, Yes
TGA: No
Adv-W2S:
Adv-W2S:
Question: Is there dog in the image?
Question: Is there bus in the image?
Image
Hallucination VQA
Clean Image
Hallucination
TeCoA: Yes
ToCoA: No
FARE: Yes
FARE: No
TGA: Yes
TGA: Yes
Adv-W2S:
Adv-W2S:
Figure 3: Visual examples for POPE hallucination evaluations when using the LLaVA 1.5 frame- work with the robust vision encoder of diverse adversarial VLM learning methods.
Teacher prediction entropy minimization (Ór-PredEP as adversarial perturbations):
ò OT.PredFP = arg
min
max HT(x + 8T..redEF)]
| @
(23)
where H-(-) denotes the prediction-level entropy of the teacher VLM.
E
Auxiliary Ground-truth Supervision
Recall that we primarily concentrate on robust weak-to-strong generalization in an unsupervised scheme (see Adv-W2S method in Eq. (13)). Despite its improved zero-shot performance as shown in Table 2. its corresponding in-distribution performance on the fine-tuned dataset is still lower than standard supervised adversarial fine-tuning (e.g., TeCoA [55]) at the same VLM backbone. To mitigate such an in-distribution performance drop, we investigate an auxiliary branch of ground-truth supervision for our standard Ady-W2S method (Eq. (13)):
L
W-Tny Adv-AuxConf Adv-AuCConf + A LCR (ps(x 5)) (()),
Sup-Was W-Iny
(24)
where ÀSup = 0.5 denotes the weighting factor for the auxiliary supervised loss. The corresponding hyper-parameter analysis is shown Figures 5c & 5d. The trade-off effect of such an auxiliary supervision is demonstrated in Table 13.
F
Visualizations of Image Captioning and Visual Question Answering
We here provide visualizations of diverse adversarial learning approaches (including our proposed Adv-W2S method) in the context of image captioning and visual question answering.
Image captioning. As shown in Figure 1, we can observe that our Adv A2W method facilitates an adversarial invariance when confronted with unforeseen adversaries, generating correct and high- quality captioning results for both clean samples and their adversarial counterparts. While most baselines exhibit severe semantic drift or hallucinated content under ad adversarial perturbations, such as describing toy cars, turtles, or zombie elephants, our method, Adv-W25 maintains robust and semantically consistent captions across both scenarios. These results highlight the superior robustness of Ady-W2? in preserving grounded vasual-language alignment under distributional shift.
Visual question answering. Figure illustrates the robustness of the LLaVA framework using the robust vision encoder w.r.t. diverse adversarial VLM learning approaches under clean and adversarial conditions. For clean images, most methods provide consistent and correct answers. However, under adversarial perturbations (e = 2/255), other comparative approaches exhibit severe degradation, producing hallucinated even incorrect answers, c.g., listing the price "$3000" or
23

---

Clean Images
Science
Question:
Which property do these objects
PMG: A. Blue
Blue
FARE: A, Blue
Hard
Sticky
Hard
Adv-W2S: A, Blue
Question: Which ocean highlighted?
Ocean
PMG: B. The Atlontic Ocean
Incian Ocean
The Atlontic Ocean
FARE:
The Pacific Ocean
The Pacific Ocean
The Southem Ocean
Adv-W2S: A. The Indian Ocean
Slippery
Question:
Which property do these objects Slippery
PMG: B. Transporent
FARE: Yellow
Transparent
Yellow
Slippery
Adv-W2S: C. Yellow
Trane wid
pibllagpt
Figure 4: Visual examples for science question answering when using the LLaVA 1.5 framework with the robust vision encoder of diverse adversarial VLM learning methods.
20
202
Acconvo Arcunny Ph)
305
Factor
Aucllary Superviion
Ausilary Supervaton Factor
(a) E (Avg.)
(h) Aucamuu (Avg.)
(c) Amp (In-Distribution)
(d) Amup (OOD)
Figure 5: Hyper-parameter (inverse perturbation radius é, warm-up factor Awarmups and auxiliary su- pervision factor Awp) sensitivity of Adv-W2S. (a) Varying averaged across datasets. (b) Different
Awarmup setups averaged across 14 datasets. (c) Sensitivity to Xuu ImageNet iin-ditribbttio).. (d)
Sensitivity to Auup averaged over the remaining 13 -i--t---istribution (OOD) datasets.
misdentifying a hotel as a "barber shop". In contrast, our Adv-Wi W2S method consistently preserves correct and semantically grounded answers, demonstrating strong zero-shot robustness against adversarial perturbations in real-world VQA scenarios.
POPE object hallucination. The visualizations of the POPE benchmark [44] in Figure 3 illustrate how our Ady-W2S method suppresses object-level hallucinations the adversarial evaluation scenario. Note that previous adversarial VI.M learning works tend to over-rely on language priors: they answer "Yes" whenever the question mentions a common object (person and traffic light) and "No" when it mentions unlikely (dog baseball field), regardless of the actual pixels. However, our proposed Adv-W2S the only method that consistently avoids hallucinating objects that are absent missing objects that are present.
Science question answering. Figure focuses on multi-ste nalti-step reasoning under the ScienceQA protocol [52]. Although all methods observe the same inputs, other adversarial VLM learning approaches collapse to the most linguistically co-occurring option (e.g., Atlantic Ocean for map highlighting the Indian Ocean). However, our Ady-W2S maintains the correct answer even when it contradicts the majority of peer models. The result supports our claim that adversarial weak-to-strong generalization not only reduces single-token hallucinations but also strengthens higher-order inference pipelines.

---

G Hyper-parameter Sensitivity Analyses.
A key practical advantage of our Adv-W2S method has only few mild-impact hyper- hyper-parametert apart from the standard parameter optimizer settings, Adv-W2S introduces only (i) the inverse adversarial perturbation ? that controls adversary strength in Eq. (12), (ii) the warm-up factor Xwarnup that anneals the adversarial loss in the early epochs, and (iii) the auxiliary supervision factor Asup in Eq. (24). Notably, manual pass-balancing terms are required, as all three scalars modulate self-contained components and therefore have bounded influence.
We adopt the same inverse adversarial perturbation radius configuration the standard adversary setup for consistency. We can also observe that increasing mildly strengthens both clean and robust accuracy up to É = 2, beyond which returns saturate. A minor weighting for the adversarial warm-up facilitates improving natural performance and adversarial robustness. The analyses in Figures 5c & 5d further justify the underlying trade-off between in-distribution robustness and out-of-distribution robustness. We therefore choose balanced value of Amap = 0.5 in our standard setting, which gives near-optimal ImageNet accuracy while limiting OOD performance degradation.
H Broader Impact and Limitations.
H.1 Broader Impact
The advancement of VLMMs has significantly revolutionized zero-shot learning across multimodal tasks. However, their real-world deployment raises serious concerns regarding their inherent vulnerability to adversarial attacks, which can severely compromise reliability and trustworthiness in high-stakes applications. Our proposed adversarial weak-to-strongg generalization (Adv-W2S) framework directly addresses this concern by improving the zero-shot adversarial robustness of VLMs across diverse multimodal tasks in an unsupervised scheme.
Our method also aligns with the emerging objective of superalignment, which arises from the widening gap between model capabilities and the scalability human supervision [41]. As models increasingly exceed human performance, reliance weak teacher models to guide stronger student models becomes scalable alternative (analogy of superalignment). Our work advances this paradigm by showing that adversarially guided supervision, rather than solely natural signals, can more effectively elicit robustness in langua-an settings.
The improvement carries several broader impacts:
Research advancement. Our theoretical and empirical analyses of entropy-guided uncertainty re-weighting and inverse adversarial examples provide more insights into robust knowledge superalignment in cross-modal settings, potentially inspiring future work in robust cross-modal learning and robust AI with foundation models.
Al safety. By strengthening the robustness of foundational models against unforeseen adversarial examples, our method lays the groundwork for more secure and trustworthy Al systems applicable diverse vision recognition and understanding tasks.
Public trust. As Al technologies become embedded in everyday decision-makirn improving ro- bustness under distribution shift and adversarial scenarios contributes to building public confidence in their safe and responsible deployment.
H.2 Limitations
While our method significantly enhances adversarial robustness, it presents several limitations, yet we also try to address/mitigate these limitations in our paper:
In-distribution performance trade-off. The unsupervised nature of our Adv-W2S can lead to a slight degradation in in-distribution (c.g.. the ImageNet dataset for fine-tuning) performance. However, we show that this can be mitigated by incorporating supervised loss term into our optimization, allowing the model balance zero-shot performance and in-distribution performance more effectively (see the last analysis in Section 4.3).
Dependence on teacher reliability, The effectiveness of weak-to-strong generalization inherently depends on the quality of the weak teacher's 's predictions. When the teacher exhibits high uncer-
25

---

tainty or spurious behavior, it may misguide the student. This is common challenge to all the weak-to-strong generalization approaches. To somehow mitigate this, our proposed framework introduces entropy-guides re- we-geinht and inverse adversarial refinement to reduce reliance on low-confidence supervision (see Sections 3.3 and 3.4).
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly convey the key contributions and scope of the paper with a clear motivation for robust weak-to-strong generalization. The claims are consistently supported by theoretical analyses and extensive experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix H.2.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: The paper provides complete theoretical results, with all assumptions clearly stated alongside each theorem. Full formal proofs included in the appendix, and intuitive sketches are offered in the main text to aid understanding.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed experimental setups are provided in Appendix B. For each experi- ment/anaylsis, we also provide its corresponding configuration for clarity.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Detailed experimental setups for reproducing results are provided in Appendix B. All the datasets used in the paper are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //niss. wwtpcccpppcc.aecccccccccccccetlt http://pwwggggpps/////eeseeaeael (oodeSubmiisiiooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Full details of training and evaluations are provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeat each experiment five times under different random seeds and report the mean performance. Consistent trends across runs indicate the robustness of our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The required computing resources our setting are provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //nurr//////r.....//// http://errppp..pppbiiiiccoc////t httpppuiicicciiccatteleeeeeeeeee ines?
Answer:
[Yes]
Justification:
We have thoroughly checked the NeurIPS Code of Ethics for our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader positive impacts are discussed in Appendix H.I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the external resources we used are properly mentioned and credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
32

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttps/////euu../eeeeee www.urrps...ooooffeeeeeeeeeeeeee /00t////eenneee/225/////// 20LL/LLM for what should or should be described.
33


Reviews:
[{"review_id": "NTN0mCwvo8", "reviewer": "Reviewer_uQC8", "summary": "This paper studies the problem of weak-to-strong (W2S) generalisation for vision-language models (VLMs) in the context of adversarial robustness, a capability that standard W2S approaches fail to transfer. The authors identify the root cause as a mismatch in the data used for alignment: while clean examples suffice for natural accuracy, they fall short for robustness. \n\nTo address this, the paper proposes Adv-W2S, an unsupervised, adversarially guided W2S generalisation framework that: 1) generates adversarial examples for the strong student and inverse adversarial examples for the weak teacher; 2) re-weights teacher guidance using an entropy-based uncertainty score; 4) aligns the student on adversarial inputs to the teacher’s refined predictions. The paper provides theoretical analysis (Theorems 1-3) to elucidate how low-entropy supervision encourages larger feature norms and margins, which underpin improved robustness of their proposed method. Extensive experiments on 20 datasets across diverse tasks, including zero-shot classification, image captioning, VQA, and hallucination show that Adv-W2S consistently improves both clean accuracy and adversarial robustness over strong baselines. Comprehensive ablations further validate the effectiveness of Adv-W2S.", "questions": "-\tHow was the one-hot labels based on strong model predictions derived in Eq. 4)? (I.e., how was the threshold determined to derive the one-hot labels given the predictions of the student model?).  \n-\tAll reported robustness results are against $\\ell_\\infty$ PGD/AutoAttack with $\\epsilon \\leq 4/255$. It remains unclear how Adv-W2S performs under other perturbation types, such as larger $\\epsilon$, $\\ell_2$-norm attacks, or text-prompt-based adversaries.\n-\tSince $κ_H$ in Eq.6) is set to the median teacher entropy, what happens when teacher predictions are close to uniform?\n-\tCan Adv-W2S handle open-set classes unseen by the teacher? How is $ \\mathbf{p}_T (x)$ defined in that case?\n-\tIt seems like Theorem 2 assume the image encoder $\\Phi$ is globally $L$-Lipschitz under the $\\ell_2$ norm. How does this impact in practice when the method is implemented with a standard CLIP backbone, which is not Lipschitz-controlled.\n-\tSeveral typos found: e.g., ‘Robutness’ in Section 3.2 Heading, ‘satifies’ in Line 202, ‘psedolabels’ in Line 222.", "limitations": "The authors have adequately addressed the limitations. Some suggestion to consider is how the presented theoretical analysis relate practically.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n-\tThis paper demonstrates a clear problem formulation and empirical evidence, where vanilla W2S cannot transfer robustness - even with a robust.  \n-\tAdv-W2S is simple yet novel: coupling adversarial and inverse-adversarial views with entropy-based curriculum elegantly addresses noisy teacher signals.\n-\tA practical plug-and-play replacement vision encoder for downstream VLMs.\n-\tExtensive experiments and ablation empirically validate the efficacy of Adv-W2S across a wild range of tasks and benchmarks.\n-\tThe paper is clearly structured and easy to follow.\n\nWeaknesses:\n-\tThe authors have presented several theoretical results, it is suggested to provide intuitions following each result to improve clarity.\n-\tTable 11 shows that maximising the teacher-student gap outperforms student-only perturbations, but no intuition is provided beyond “better transfer”.\n-\tTheorem 2 establishes an $\\ell_2$ robustness radius of $ \\gamma / L_{\\max} || \\Psi_c - \\Psi_y ||_ 2$, however, it seems like the empirical evaluations rely on $\\ell_\\infty$ attacks. The theoretical bound may not be applicable to the evaluation setting without further clarification.\n-\tIt is suggested to provide computational cost analysis.\n-\tWriting can be improved (See questions).", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer uQC8\nWe thank the reviewer for constructive feedback.\n\n### **1. (Weakness-1) Intuitions of theoretical results for clarity.**\n\nAbsolutely. We will include clearer and intuitive explanations in the revision. Our theoretical results are intended to explain **why entropy-guided re-weighting and inverse adversarial refinement improve robustness transfer**. Intuitive remarks will be added after each theorem:\n\n- Theorem 1 shows that low prediction entropy (high certainty prediction) enforces a larger vision feature norm, whereas norms of features for high entropy of prediction can be low. Therefore, the model demonstrably is able to filter high-confidence samples from low-confidence samples. This is somewhat similar to the organization of feature space in hyperbolic encoders, where a low norm of features indicates high uncertainty of predictions, and a high norm indicates low uncertainty of predictions [a].\n- Theorem 2 builds on Theorem 1, showing that larger feature norms lead to wider classification margins, improving robustness since small perturbations are less likely to change predictions.\n- Theorem 3 explains that the loss between adversarial student and inverse adversarial teacher upper bounds that of the clean student, implying that improvements on adversarial samples also benefit clean predictions.\n\nThese clarifications directly link the theory to our design choices, improving readability without sacrificing rigor.\n\n[a] Nickel et al., Poincare embeddings for learning hierarchical representations, NeurIPS 2017\n\n### **2. (Weakness-2) Intuition of maximising teacher-student gap (Table 11).**\n\nThe rationale for maximizing the teacher–student prediction gap is to align adversary generation with weak-to-strong generalization:\n- **Target worst-case disagreements**: Perturbations focus on where student predictions deviate most from the teacher, forcing learning from challenging cases.\n- **Consistent optimization**: This keeps inner (adversary generation) and outer (parameter optimization) steps aligned toward reducing the gap rather than optimizing different objectives.\n- **Selective guidance**: The student aligns with reliable teacher predictions while refining uncertain ones. In other words, rather than fully imitating the teacher, the student should maximize agreement with the teacher on stable predictions while relying on their own refinement when the teacher is uncertain. Optimizing adversaries that maximize the teacher–student gap directly exposes these disagreement regions, allowing the student to correct them and inherit both natural performance and robustness more effectively than student-only perturbations, thereby improving transfer performance.\n\nWe will add this explanation in the revised manuscript for clarity.\n\n### **3. (Weakness-3) Theorem 2 provides an $\\ell_2$ robustness bound, but evaluations use $\\ell_\\infty$ attacks.**\n\nThe $\\ell_2$ bound in Theorem 2 was presented for notational simplicity, yet the Lipschitz continuity argument is norm-agnostic and directly extends to $\\ell_\\infty$. Specifically, robustness bounds can be converted via standard norm inequalities (e.g., $\\lVert\\delta\\rVert_\\infty\\leq\\lVert\\delta\\rVert_2$), yielding $\\lVert\\delta\\rVert_\\infty < \\frac{\\gamma(x)}{L\\max_{c\\neq y} \\lVert \\Psi_c - \\Psi_y \\rVert_2}$. Therefore, our theoretical framework is fully compatible with the $\\ell_\\infty$-based robustness. We will clarify this generality in the revision to ensure consistency between theory and $\\ell_\\infty$-based evaluations.\n\n### **4. (Weakness-4) Computational cost analysis.**\n\nWe report the average accuracy (%) across 14 datasets with the training time per epoch in the table below. We also show that reducing the adversary generation step to 5 further enhances the efficiency while preserving most of the natural performance and robustness.\n\n|Method|Clean|Robust|Time|\n|-|-|-|-|\n|TeCoA|61.56|43.26|3.1 h|\n|PMG|64.46|45.74|4.3 h|\n|FARE|65.50|42.97|3.8 h|\n|TGA|62.11|45.19|4.0 h|\n|**Adv-W2S (10 Steps)**|**68.75**|**48.38**|**5.3 h**|\n|**Adv-W2S (5 Steps)**|**66.49**|**47.13**|**2.9 h**|\n\nThis demonstrates that Adv-W2S can maintain state-of-the-art robustness with substantially lower computational overhead and scales well to efficiency-sensitive settings.\n\n### **5. (Question-1) How was the one‑hot label in Eq. 4 derived from strong model predictions?**\n\nIn Eq. (4), $M(\\cdot)$ denotes a simple **argmax operation** that converts the student’s soft prediction into a one-hot label by selecting the index of the maximum logit. No additional threshold is used. The one-hot label directly corresponds to the predicted class with the highest probability. We will clarify this notation in the revision.\n\n### **6. (Question-2) Robustness Beyond $\\ell_\\infty$ Attacks (larger $\\epsilon$, $\\ell_2$-norm attacks, or text-prompt-based adversaries)**\n\nWe report average accuracy (%) of larger $\\ell_\\infty$ perturbation $\\epsilon$  in the table below:\n|$\\epsilon$|Method|Clean|Robust|\n|-|-|-|-|\n|6/255|TeCoA|52.31|25.78|\n||PMG|55.23|27.12|\n||FARE|55.63|26.35|\n||TGA|53.10|26.57|\n||**Adv-W2S**|**58.25**|**30.18**|\n|8/255|TeCoA|47.45|20.88|\n||PMG|50.21|22.51|\n||FARE|50.94|21.59|\n||TGA|47.75|21.13|\n||**Adv-W2S**|**54.83**|**26.07**|\n\nWe also validate the robustness against $\\ell_2$ perturbations ($\\epsilon=128/255$)  \n|Method|Clean|Robust|\n|-|-|-|\n|TeCoA|59.26|41.57|\n|PMG|62.20|42.95|\n|FARE|62.90|42.17|\n|TGA|60.11|41.83|\n|**Adv-W2S**|**66.33**|**46.28**|\n\n**Robustness against text-prompt-based adversaries can be found in Response 1 to Reviewer 7iVd**. We show that our Adv-W2S method consistently outperforms others under diverse perturbation types/scenarios.\n\n### **7. (Question-3) What happens when teacher predictions are close to uniform?**\n\nWhen teacher predictions are uniform (high entropy), the weighting function in Eq. (6) naturally degrades toward a **constant weighting (0.5)** across all instances. In this case, the teacher cannot provide informative guidance. If teacher predictions are close-to-uniform but contain useful information, weights can be post-processed by sigmoid with sharp slope to boost the teacher's signal. In our experiments, the weak teacher is adversarially fine-tuned and typically exhibits **non-uniform predictions**, so this scenario rarely occurs in practice. We will clarify this behavior in the revision.\n\n### **8. (Question-4) Can Adv-W2S handle open-set classes unseen by the teacher? How is $p_{T}(x)$  defined in that case?**\n\nIn our current formulation, teacher–student alignment during training assumes a **shared label space**, where $p_T(x)$ is defined as the teacher’s predictions over this space via the cosine similarity between the image feature of $x$ and its corresponding text feature set of diverse categories. This enables efficient alignment without ground-truth labels, consistent with CLIP-style training.\n\nFor an open-set scenario during training, where class candidates may be unknown or unbounded, we envision three possible extensions:\n\n- **Large-scale virtual label space**: Following the original CLIP pretraining approach, we can construct $p_T(x)$ by computing cosine similarity between the image feature of $x$ and a large set of text embeddings (e.g., descriptive phrases). While feasible, this approach incurs a higher computational cost. Using a subset of image properties can somehow enhance the computational efficiency.\n- **Prototype-based virtual labels**: A more efficient variant is to cluster the text embedding space into representative prototypes, then define $p_T(x)$ via cosine similarity between image features and these prototypes, forming pseudo-label distributions for alignment.\n- **Feature-level alignment**: Alternatively, adversarial weak-to-strong generalization can be adapted to perform directly at the feature level, potentially with a projector to resolve dimensionality differences between weak and strong models.\n\nWhile our current experiments focus on shared-label training, these strategies highlight that our framework can be extended to open-set training scenarios; we will clarify this point in the revision.\n\n### **9. (Question-5) Assumption of Global Lipschitzness in Theorem 2.**\n\nThe global $L$-Lipschitz assumption was given for analytical clarity. Naturally, local Lipschitzness in the vicinity of datapoints ${\\bf x}$ is sufficient for our margin–robustness argument. In Eq. (13), we align predictions of perturbed $\\hat{\\bf x}$ with predictions of ${\\bf x}$, which implicitly teaches CLIP student local Lipschitzness on the data, as $w({\\bf x})>0$ is in practice. However, as we assume Lipschitzness on the feature rather than the predictions, we verified results by imposing a Lipschitz regularization term with the equation: \n\n$$ \\eta \\\\,\\Big(\\lVert\\Psi(x)-\\Psi(x')\\rVert_2 - L\\lVert x-x' \\rVert_2\\Big)^2 + \\beta\\lvert L \\rvert, $$\n\nwhich enforces bounded variations in feature space. This approach directly optimizes the Lipschitz equation and yields similar robustness trends (Table below), confirming that the theoretical assumption is not restrictive.\n\n|Method|Clean|Robust|\n|-|-|-|\n|Adv-W2S (w/ Lipschitzness)|68.29|48.43|\n|Adv-W2S|68.75|48.38|\n\nMoreover, our analysis focuses on explaining the **mechanistic relation between margin and robustness** rather than requiring exact constants. Although standard CLIP backbones are not explicitly Lipschitz-controlled, we empirically estimated the local gradient norm of ViT-L/14 on 1,000 random ImageNet samples and observed bounded values around **0.22–0.29**. These small values indicate **local Lipschitz behavior** around natural data, making the approximation reasonable and consistent with the robustness trends confirmed by our experiments.\n\n### **10. (Weakness-5) Writing. (Question-6) Several typos.**\n\nWe will thoroughly proofread the manuscript and correct all typographical errors to improve overall clarity in the revised version."}, {"review_id": "9w5QNb9BfV", "reviewer": "Reviewer_7iVd", "summary": "In this work, the weak-to-strong robustness generation problem is studied, and the key finding is that the strong teacher VLM is also vulnerable to adversarial attack as the training objective is not attack-aware. To this end alignment re-weighting and reference refining are proposed to enhance the weak-to-strong alignment and improve the model's adversarial robustness. In detail, besides using a fixed weights $\\beta$ to balance the tradeoff between teacher-student alignment and student self-refinement, a teacher entropy to measure the uncertainty is proposed and used to guide the adaptive weights. The proposed approach is validated in multiple datasets with multiple tasks like linear probing, image caption, VQA, hallucination, etc.", "questions": "- This work is mostly focusing on the perturbation based adversarial attack to the vision encoder, which is fine. But one follow up question would be: Can this approach be generalized to other adversarial attacks? E.g, the perturbation based attack is also studied in text only language models [1]. From principal, it seems this entropy based approach can be generalized to this case. Another possibility is extend the approach to adversarial attack that's not perturbation based, like prompt based attack [2]\n\n- Another hypothesis is that stronger VLM based model will be more robust, so it would be good to see how robust the more recent VLM models are, like the Qwen VL and llava-onevision, or llava 1.6(next).\n\n\n[1] Kim, Jaehyung, et al. \"RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training.\" EMNLP (Findings). 2023.\n[2] Xu, Xilie, et al. \"An LLM can Fool Itself: A Prompt-Based Adversarial Attack.\" ICLR. 2024.", "limitations": "Yes, limitations are discussed in appendix.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The paper is well formated and written, with both theoretical analysis and experiments.\n- Sufficient experiment coverage, the proposed method is validated in multiple vision-language tasks, like zero-shot classification, image caption, VQA, and hallucination.  \n- The pretrained vision encoder is validated on multiple open sourced base models, like llava 1.5 and openflanmingo.\n\n### Weakness\n- The studied adversarial attack seems to be limited: 1) only studied visual encoder. 2) limited to perturbation based attack\n- Did not test with more recent VLM models like llava 1.6 and Qwen VL", "was_revised": true, "final_justification": "Given the comprehensive response with additional discussion and evidence showing the effectiveness of the approach on 1) Language models and 2) more recent VLMs with stronger performance. I updated my rating accordingly to reflect these discussions", "rebuttal": "# Response to Reviewer 7iVd\nWe thank the reviewer for constructive feedback.\n\n### **1. (Weakness-1) (Question-1) Generalization beyond perturbation-based visual adversarial attacks (e.g., text perturbation attacks, text prompt attacks).**\n\nWe thank the reviewer for this question. While our main experiments focus on perturbation-based attacks on the visual encoder (dominant in multimodal robustness studies), our approach is not restricted to this setting.\n\n- **Connection to text-based perturbations**: Work [1] studies continuous embedding-level perturbations for LLMs, while [2] addresses discrete text modifications to achieve prompt-based attacks. Importantly, both types of text-based adversarial attacks are applicable for evaluating the robustness of our method as well as other adversarial fine-tuning approaches.\n- **Extended evaluation**: To demonstrate generality, we additionally evaluate robustness against text perturbation [1] and prompt-based attacks [2]; results are summarized below:\n\n|Method|Clean|Robust Acc. (Text Perturbation Attack)|Robust Acc. (Text Prompt Attack)|\n|-|-|-|-|\n|TeCoA|61.56|41.22|39.97|\n|PMG|64.46|42.37|40.82|\n|FARE|65.50|40.28|38.58|\n|TGA|62.11|41.54|40.11|\n|**Adv-W2S**|**68.75**|**45.30**|**44.06**|\n\nThese results confirm that **our Adv-W2S generalizes beyond image-level perturbations** and remains robust under diverse modalities. We will add this discussion and cite [1,2] in the revised manuscript.\n\n[1] Kim, Jaehyung, et al. \"RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training.\" EMNLP (Findings). 2023. \n\n[2] Xu, Xilie, et al. \"An LLM can Fool Itself: A Prompt-Based Adversarial Attack.\" ICLR. 2024.\n\n\n\n### 2. **(Weakness-2) (Question-2) Evaluation on recent VLMs (LLaVA 1.6 (next), Qwen VL)**\n\nWe thank the reviewer for this helpful suggestion. To address it, we extend our evaluation to **recent stronger VLMs** on Qwen VL 7B and LLaVA 1.6 (Next) 7B, both using the ViT-L vision encoder for fair comparison. Results on COCO captioning (CIDEr) and VQAv2 (VQA accuracy) are summarized below. Adv-W2S continues to provide significant robustness gains while maintaining competitive clean performance, indicating the method generalizes well to recent VLM architectures.\n\n|Model (7B, ViT-L)|Method|COCO Captioning (Clean/Robust)|VQAv2 (Clean/Robust)|\n|-|-|-|-|\n|**Qwen VL**|Standard|116.2/4.3|78.5/1.1|\n||TeCoA|102.5/49.1|72.8/36.4|\n||PMG|108.4/53.0|74.1/37.3|\n||FARE|109.0/48.6|75.2/35.7|\n||TGA|105.7/51.3|72.5/36.9|\n||**Adv-W2S**|**114.8/57.7**|**77.9/40.5**|\n|**LLaVA 1.6 (Next)**|Standard|118.7/5.1|80.2/1.8|\n||TeCoA|105.4/51.3|74.6/37.8|\n||PMG|110.1/54.8|75.5/38.6|\n||FARE|111.5/50.8|76.0/36.9|\n||TGA|107.3/53.5|74.8/37.5|\n||**Adv-W2S**|**116.9/60.2**|**79.8/42.3**|\n\nThese results show that Adv-W2S retains its robustness advantage on newer, stronger VLM backbones, scaling effectively while maintaining competitive clean accuracy. We will add this discussion and highlight that the robustness improvements persist even as model capacity increases."}, {"review_id": "X11RuT1wG8", "reviewer": "Reviewer_m9zw", "summary": "The paper proposes a new framework called Adv-W2S (Adversarial Weak-to-Strong) to improve the robustness of VLMs against adversarial attacks. It makes several key contributions: it provides theoretical analysis that explains the shortcomings of traditional weak-to-strong generalization methods in terms of robustness, attributing this issue to the lack of adversarial samples in the learning objective; it introduces a new method based on adversarial sample generation, allowing robust knowledge to be extracted from strong student models in an unsupervised way; and it validates the effectiveness of Adv-W2S through extensive experiments, showing its superior performance in zero-shot classification tasks as well as its applicability to various downstream tasks such as image captioning, visual question answering, and COT reasoning. These contributions offer valuable insights for addressing current vulnerabilities in the security and robustness of VLMs.", "questions": "When I first started reading the paper, a question came to mind: The term \"superalign\" usually refers to the field of AI alignment, where the focus is on aligning highly intelligent, complex, or enhanced models with human values and intentions. It primarily concerns how to ensure that superhuman-level AI systems (such as models that perform beyond human capabilities) remain consistent with human values and objectives.\n\nCLIP clearly does not fall into this category. However, in the later experimental sections, the paper integrates CLIP into large-scale VLMs such as LLAVA, which partially addresses this issue. It would be helpful if the authors could clarify this point more explicitly at the beginning of the paper. However, this does not detract from the fact that this is still a good paper.\n\nFurthermore, based on this point, since the authors likely have sufficient GPU resources, expanding the experiments to even larger models would better reflect the \"superalignment\" scenario.\n\nother comments see weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n① The writing is excellent, and the paper is logically structured.\n\n② New method design: The authors propose a weak-to-strong generalization framework called Adv-W2S, based on adversarial sample generation, which can extract robust knowledge from a strong student model in an unsupervised manner.\n\n③ Comprehensive experimental validation: The authors conduct extensive experiments across multiple network architectures and evaluation scenarios, demonstrating the effectiveness of Adv-W2S.\n\nWeaknesses：\n\n① The paper introduces the parameter β in the CLIP weak-to-strong formulation (i.e., Equation (4)) . However, there is limited discussion later in the paper regarding the specific value of β. It would be beneficial to include a brief analysis or ablation study on how β affects the experimental results.\n\n② The authors could refer to the PGR formulation used in Burns' work to evaluate the robust generalization performance of their method. Additionally, introducing a \"strong ceiling\" — that is, showing the upper bound of performance when directly training the strong model — would make the robustness comparison more convincing.", "was_revised": true, "final_justification": "This article is well-developed in terms of topic selection, experimental analysis, and findings. The rebuttal also addressed my questions, and I believe it is a good paper.", "rebuttal": "# Response to Reviewer m9zw\nWe thank the reviewer for constructive feedback.\n\n### **1. (Weaknesses-1) Analysis and impact of the parameter $\\beta$ in the CLIP weak-to-strong formulation (i.e., Eq. (4)).**\n\nWe thank the reviewer for raising this point. The trade-off hyperparameter $\\beta$ controls the balance between **weak-to-strong teacher-student alignment** and **self-prediction consistency** in Eq. (4) and is similarly applied in our adversarial baseline (Eq. (5)). In our experiments, $\\beta$ is selected via 1% of the training data and either fixed or linearly warmed up. To further clarify the role of $\\beta$, we report the performance of the baseline adversarial weak-to-strong formulation (Eq. (5)) under different fixed $\\beta$ values in the table below:\n|$\\beta$|Clean|Robust|\n|-|-|-|\n|0.3|64.10|41.65|\n|0.5|63.45|42.43|\n|0.7|63.27|41.96|\n\nHowever, a fixed $\\beta$ cannot adapt to varying teacher reliability, particularly for uncertain or hard samples. To address this, our final method introduces **Entropy-guided Uncertainty Re-weighting (EUR)** (Eq. (6)), which adaptively scales $\\beta$ based on teacher confidence, leading to our unified objective in Eq. (13). We report average clean and robust accuracy across 14 zero-shot datasets to quantify this effect:\n\n|Method|Clean|Robust|\n|-|-|-|\n|Baseline (Eq. (5))|63.45|42.43|\n|Adv-W2S w/o EUR|67.09|46.79|\n|Adv-W2S|**68.75**|**48.38**|\n\nThese results demonstrate that **entropy-based adaptive weighting notably improves robustness generalization** while also enhancing clean performance. We will incorporate this clarification and the ablation results into the revised manuscript.\n\n### **2. (Weakness-2) PGR evaluation from Burns et al. [a]**\n\nWe thank the reviewer for this helpful suggestion. We here estimate the \"strong ceiling\" by performing robust fine-tuning of the strong model directly on each downstream dataset, which approximates the upper bound of achievable robustness. This provides a clearer perspective on the gap between robust weak-to-strong generalization in the zero-shot setup and task-specific robust training. To better contextualize our results, we consider the Performance Gap Recovered (PGR) [a] in both clean performance and adversarial robustness across diverse tasks. The table below summarizes the **PGR results (%)** of our method (average classification accuracy against clean/adversarial examples across zero-shot datasets, captioning on the COCO dataset, and VQA on the VQAv2 dataset):\n\n|Task|Clean PGR|Robust PGR|\n|-|-|-|\n|Classification|86|79|\n|COCO Captioning|81|75|\n|VQAv2|83|78|\n\nThese results indicate that **Adv-W2S recovers the majority of the performance gap** to the strong ceiling in both clean and robust settings, validating its effectiveness in achieving robust weak-to-strong generalization without task-specific training. We will include this analysis in the revised manuscript.\n\n[a] Burns et al., Weak-to-strong generalization: Eliciting strong capabilities with weak supervision (ICML 2024).\n\n### **3. (Question-1) Clarifying “superalign” terminology.**\n\nWe thank the reviewer for this observation. We agree that \"superalign\" is commonly used in AI safety to describe aligning superhuman-level models with human values. Our work uses the term in the weak-to-strong generalization sense, following Burns et al. [a], where the goal is to transfer robustness from a weaker model to a stronger one — an analogous alignment challenge in multimodal robustness.\n\n- We will revise the introduction to explicitly acknowledge the conventional **meaning of \"superalign\"** and clarify our intended usage. \n- We will highlight that our method starts from CLIP (a foundational vision-text model) and extends to large-scale VLMs such as LLaVA, emphasizing that our formulation addresses robustness transfer rather than human-value alignment. \n- This clarification will be added early in the paper to prevent confusion and situate our contributions within the broader literature.\n\n[a] Burns et al., Weak-to-strong generalization: Eliciting strong capabilities with weak supervision (ICML 2024).\n\n### **4. (Question-2) Scalability to larger model backbones**\n\nTo validate scalability, we here extend experiments to larger CLIP backbones (ViT-H/14 and ViT-g/14), which are ~3$\\times$ larger than ViT-B/16 and widely used in recent VLMs. Average zero-shot classification accuracy (%) across 14 datasets under perturbation radius $\\epsilon=2/255$ are summarized below:\n|Backbone|Method|Clean|Robust|\n|-|-|-|-|\n|ViT-L->**ViT-H**|TeCoA|66.95|48.63|\n||PMG|68.96|50.31|\n||FARE|70.55|50.25|\n||TGA|65.91|49.14|\n||**Adv-W2S**|**72.89**|**53.27**|\n|ViT-H->**ViT-g**|TeCoA|67.41|49.20|\n||PMG|69.71|51.95|\n||FARE|71.62|51.68|\n||TGA|66.99|50.44|\n||**Adv-W2S**|**73.94**|**54.09**|\n\nOur method maintains consistent gains on both clean and robust accuracy, with especially strong improvements in robustness as model capacity increases. We will include this analysis and also reference **robustness results of other large-scale VLMs in other vision-language tasks reported in Response 2 to Reviewer 7iVd**."}, {"review_id": "fioGiqqcSP", "reviewer": "Reviewer_zCJ1", "summary": "This paper introdces the robustness of vision-language models in the context of weak-to-strong generalization. The authors reveal that weak student models cannot provide robustness by simple distillation, and emphasize the importance of incorporating adversarial samples during training. In addition, this paper proposes Entropy-Guided Uncertainty Re-Weighting and Teacher Guidance Refinement via Inverse Adversarial Examples for improving robustness of student model.", "questions": "As describe in Weakness, I wonder how EUR compares to the adaptive weighting approach proposed in [1]. Specifically, what are the unique benefits of EUR, and how does the model's performance change when EUR is substituted with the adaptive weight from [1]?\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024).", "limitations": "Yes, the authors report the limitations and broader impacts in the Appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Clear and Logical Writing.\n2. To the best of my knowledge, this is the first paper to address the robustness of vision-language models specifically in the context of weak-to-strong generalization.\n3. The theoretical justification that increasing the teacher model’s confidence leads to greater robustness in the student model is highly logical, and the proposal of inverse adversarial examples is well-reasoned.\n\nWeaknesses:\nIn weak-to-strong generalization loss function, replacing the constant balancing factor $\\beta$ with an adaptive parameter was already explored in [1], which balances the weight utilizing the confidences of both the teacher and student models. I wonder what the advantages of EUR compared to the adaptive weight [1] and how the performance would change if EUR were replaced by the adaptive weight [1].\n\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024).", "was_revised": true, "final_justification": "This paper presents a novel study on the robustness of superalignment in Vision-Language Models — a topic that has not been explored in prior research — and supports its claims with both theoretical analysis and extensive experimental evidence. My concerns have been satisfactorily addressed during the discussion.", "rebuttal": "# Response to Reviewer zCJ1\nWe thank the reviewer for constructive feedback.\n\n### **1. (Weakness-1) Comparison of EUR with adaptive weighting in [1]**\n\nWe thank the reviewer for pointing this out. While [1] introduces an adaptive weighting based on **confidence** for weak-to-strong generalization, it focuses solely on single-modal vision models. Our work addresses robust weak-to-strong generalization for multimodal Vision-Language Models (VLMs), and our Entropy-guided Uncertainty Re-weighting (EUR) extends beyond raw confidence by leveraging prediction entropy, theoretically justified via Theorems 1 & 2 (feature norm $\\leftrightarrow$ classification margin $\\leftrightarrow$ robustness). In contrast, [1] provides only empirical justification.\n\nTo quantify the impact, we also implement the adaptive confidence weighting from [1] as a variant and compare it with our EUR below (average over 14 zero-shot datasets):\n|Method|Clean (%)|Robust (%)|\n|-|-|-|\n|Confidence Re-Weighting [1]|67.17|46.92|\n|Entropy Uncertainty Re-Weighting (EUR)|**68.75**|**48.38**|\n\nOur Adv-W2S method with EUR achieves higher robustness while also improving clean accuracy, validating the benefits of entropy-based adaptive scaling. We will incorporate this comparison and clarification into the revision.\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024).\n\n### **2. (Question-1) Unique benefits of EUR beyond [1].**\n\nThe empirical comparison of EUR with the adaptive weighting in [1] is reported above. Here, we emphasize **the theoretical contributions unique to EUR**: \n- [1] balances teacher–student alignment purely via **prediction confidence** yet lacks a theoretical link to generalization/robustness properties.\n- Our EUR is built on **prediction entropy**, which we theoretically show (Theorems 1 & 2) to correlate with **feature norm** and **classification margin**, thereby providing a direct mechanism for robustness improvement. This connection enables a principled adaptive weighting scheme: low-entropy predictions (confident teacher) lead to stronger alignment, while high-entropy predictions defer to student self-refinement, improving robust weak-to-strong generalization.\n\nThis theoretical grounding explains why EUR improves both clean and robust performance, as confirmed by the empirical results. We will add this discussion to the revised manuscript.\n\n[1] Guo, Jianyuan, et al. \"Vision superalignment: Weak-to-strong generalization for vision foundation models.\" arXiv preprint arXiv:2402.03749 (2024)."}]

Author rebuttal:

Meta-review: There is a general consensus among the reviewers that the paper, in its current form, is suitable for publication. After careful consideration of the feedback and the paper itself, the recommendation is to accept this submission.
Decision: Accept (spotlight)