Tug-of-War No More: Harmonizing Accuracy and Robustness in Vision-Language Models via Stability-Aware Task Vector Merging
Authors: Junhao Dong
Xinghua Qu
Cong Zhang
Sua Qi Rong
Nguyen Duc Thai
Wenbo Pan
Xinfeng Li
Tongliang Liu
Piotr Koniusz
Yew-Soon Ong
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Model
Task Vector
Trade-Off
Robustness
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose the first model merging framework based on task vectors to reconcile natural performance and robustness without repeated fine-tuning.

Abstract: Foundation Vision-Language Models (VLMs) excel across benchmarks yet remain vulnerable to adversarial attacks. While adversarial fine-tuning improves robustness, attaining a desirable clean–robust performance trade-off typically requires costly hyperparameter searches with multiple retraining runs. A promising alternative is to merge task vectors (i.e., parameter displacements from pre-trained models) to balance accuracy and robustness without retraining. However, we find that naive task-vector merging produces a near-linear trade-off, as it equally weights all coordinates and fails to distinguish weights that aid both objectives from those that create conflicts. To overcome this limitation, we propose a prediction stability-aware merging framework that composes task vectors from off-the-shelf naturally and robustly fine-tuned VLMs. Our key insight is that prediction stability serves as a proxy for cross-objective compatibility, enabling us to favor perturbation-invariant parameters while attenuating those with high cross-objective impact. Specifically, we estimate per-parameter stability from gradients under both objectives, building complementary masks that retain jointly stable coordinates while suppressing counterpart-sensitive ones. We further refine these masks along adversarial parameter trajectories, with steps weighted by a prediction-sensitivity index.  Our theoretical analysis shows that the masks provably contract first-order cross-objective interference, and the prediction criticality index tracks curvature, biasing the merge toward flatter minima and better generalization. Extensive experiments across benchmarks and scenarios demonstrate our method consistently achieves superior clean–robust trade-offs over prior approaches, with the learned balance transferring effectively to downstream tasks.

Published as a conference paper at ICLR 2026
TUG-OF-WAR No MORE: HARMONIZING ACCURACY AND ROBUSTNESS IN VISION-LANGUAGE MODELS VIA STABILITY-AWARE TASK VECTOR MERGING
Junhao Dongl2, Xinghua Qu', Cong Zhang', Qi Rong Sua', Nguyen Duc Thai' Wenbo Pan*, Xinfeng Li", Tongliang Lius, Piotr Koniusz Yew-Soon Ong!2 'Nanyang Technological University, CCnttre for Frontier AI Research, IHPC, A STAR,, 'Bytedance, CCity University of Hong Kong, University of Sydney, Universitty of New South Wales, Datu61VCSSIO junhao00 38ntu. sg
ABSTRACT
Foundation Vision-Language Models (VLMs) excel across benchmarks yet re- main vulnerable to adversarial attacks. While adversarial fine-tuning improves robustness, attaining desirable clean-robust performance trade-off typically re- quires costly hyperparameter searches with multiple retraining runs. A promis- ing alternative is to merge task vectors (i.e., parameter displacements from pre- trained models) to balance accuracy and robustness without retraining. How- ever, we find that naive task-vector merging produces a near-linear trade-off, as it equally weights all coordinates and fails to distinguish weights that aid both objectives from those that create conflicts. To overcome this limitation, we pro- pose prediction stability-aware merging framework that composes task vectors from off- fff-the-shelle naturally and robustly fine-tuned VLMs. Our key insight is that prediction stability serves as proxy for cross-ohjective compatibility, enabling us to favor serturbationiinvaiian cross-objective impact. Specifically, we estimate per-parameter stability from gra- dients under both objectives, building complementary masks that retain jointly stable coordinates while suppressing counterpart-sente fine these masks along adversarial parameter trajectories, with steps weighted by a cnndiction-sensitvviee index. Our theoretical analysis shows that the masks prov- ably contract first-order cross-objective interference, and the prediction criticality index tracks curvature, biasing the merge toward flatter minima and better gen- eralization. Extensive experiments across benchmarks and scenarios demonstrate our method consistently achieves superior clean-robust trade-off's over prior ap- proaches, with the learned balance transferring effectively to downstream tasks.
parameters while attenuating those with high
ones. We further re-
INTRODUCTION
Despite redefining multimodal learning across diverse tasks, foundation Vision-Language Models (VLMs) like CLIP (Radford et al., 2021) remain alarmingly vulnerable under adversarial attacks (Zhang et al., 2022; Zhao et al., 2023). Even subtle input perturbations can trigger huge performance drops, undermining their reliability in practice and posing severe security risks (Huang et al., 2025).
Bridging the gap between natural performance and robustness is thus essential for the safe and widespread deployment of VLMs. Previous efforts primarily focused on adversarial fine-tuning, where adversarial examples adaptively integrated into training enhance robustness (Mao al., 2023; Schlarmann et al., 2024; Dong et al., 2025b;c;e). However, extensive empirical evidence indi- cates that even increasingly larger and advanced multimodal architectures continue suffer from persistent accuracy-robustned
trade-off' (Wang et al., 2024). Rather than resolving this fundamental tension, most existing approaches rely on exhaustive hyperparameter searches and costly retraining to find acceptable compromises, limiting the scalability and efficiency of robust VLM solutions.
"Corresponding authors: Xinfeng Li (Ixfmakeit gmmaillcmmmmm.mmmm.mmmmm Yew-Soon Ong (asysong @ wnt...du.ssp....

---

Published as a conference paper at ICLR 2026
Given recent progress in parameter- spacce model merging, which combines fine-tuned models with- out joint training (Wortsman et al., 2022b; Ilharco et al., 2023), a compelling question arises: Can model merging extend beyond similar tasks reconcile the inherent conftict between natural perfor- mance and adversarial robustness? However, our initial investigation reveals that vanilla task-vector merging of the vision encoder in VLMS yields near-linear clean-robust trade-off with sweet point, as empirically shown in Section 3.2. To gauge feasibility and diagnose the issue, we examine directional compatibility by comparing gradients of the two objectives the respective fine-tuned CLIP models. Figure reports gradient sign agreement and gradient cosine similarity between nat- ural and adversarial losses, where higher values indicate a more similar update direction to preserve clean accuracy while improving robustness. To test whether any observed alignment is merely local, we evaluate both local gradients (at the fixed parameters) and neighborhood gradients averaged within an fz-ball of radius thereby probing stability under small parameter changes. Although this analysis reveals a degree of aligmment, it also shows that the alignment remains modest and degrades with a larger attack radius, evidencing growing directional conflict and motivating more selective, stability-awand merging strategy over naive uniform addition.
Motivated by the need to resolve the parameter-level conflicts, we, for the first time, propose novel model merging framework based on task vectors (i.e., parameter differences between fine- tuned and pre-trained models) derived from off-the-shelf naturally and adversarially fine-tuned models, named PredIction STability-awar mOdel. mErging (PISTOLE), to reconcile nat- ural performance and robustness without repeated fine-tuning by selectively fusing compatible knowledge. Specifically, our PISTOLE estimates per-parameter stability under the natural and robust objectives from gradient magnitudes and builds complementary, gradient-informed masks that retain coordinates stable for both objectives while down- weighttgg those that the counterpart would strongly change. These masks are applied to the respective task vectors prior to mixing. To better capture local (oss-parameter
Local Cracket
Gradienntooo Simttrttt
Local Gradart
geometry, we refine the masks by ac- cumulating gradients along adversar- ial parameter trajectories, with steps weighted by a prediction-sensiivviyy index that quantifies how even small parameter perturbations affect the output. Furthermore, we provide the- oretical analyses demonstrating that these masks contract cross-objes coos-objeettive first-order interference and that the sensitivity index tracks curvature, steering the merge toward flatter, more generalizable regions and yicld- ing a stronger clean-robust trade-off.
Gradient Sign Agruement
Nagitoohood Gradent
boighbrtoood Credken
0.18
0:10-
0.14
0:15
Perturbation Radius (255
Perturbation Radius (255-)
(a) Gradent Sign Agreement
(b) Gradient Cosine Similarity
Figure 1: Gradient alignment on ImageNet between natu- rally and adversarially fine-tuned CLIP: (a) sign agreement and (b) cosine similarity for local gradients and neighbor- hood gradients (aggregated within an fz-ball) across pertur- bation radius. Alignment degrades with attack strength, mo- tivating selective (not uniform) parameter merging.
Through comprehensive experiments, we demonstrate that our PISTOLE consistently achieves state- of-the-art trade-offs between natural performance and robustness compared to existing methods across diverse datasets, architectures, and scenarios. Furthermore, we show that our obtained uccuracy-robustreg trade-off effectively transfers to a spectrum of downstream rision-language tasks, including captioning, visual question answering, hallucination mitigation, and reasoning, sim- ply through a plug-and-play replacement of the vision encoder with the robustly merged encoder.
Our core contributions summarized as follows:
1. We systematically explore the feasibility of pera merging conflicting objectives (nat- ural performance and robustness) via empirical gradient analyses.
2. To address this trade-off without costly fine-tuning, we introduce PISTOLE, a novel prediction stability-aware model merging framework that leverages gradient-informed stability masks and multi-step adversarial parameter perturbations for precise parameter re- re-weighting.
3. We provide theoretical analyses proving that PISTOLE identifies parameter-sensitive predictions in high-curvature regions, guiding selective merging for improved accuracy-robusstees trade-offs.
4. We conduct extensive experiments to demonstrate the efficacy and generalizability of PISTOLE across tasks and scenarios, scaling without incurring additional fine-tuning costs.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Trade-off's in foundation VLMs. Foundation VLMs (e.g., CLIP (Radford al., 2021), LLaVA (Liu et al., 2024), OpenFlamingo (Awadalla et al., 2023)) achieve strong zero-shot transfer via large-scale image-text pre-training, yet core tensions constrain practical deployment: size vs. efficiency (Vasu et al., 2025), specialization vs. generalization (Zang et al., 2024), and fairness (Luo et al., 2024). Among these, the trade-off between adversarial robustness and natural performance remains partic- ularly challenging, with theory on competing objectives (Zhang et al., 2019). In this work, we target this trade-off in foundation VLMs, seeking to enhance robustness while preserving much natural performance.
gains in adversarial robustness often degrade clean performance, consistent
Parameter-space model merging. Beyond prediction ensembles (Yang et al., 2023), parameter merging combines knowledge from VLMM without retraining (Wortsman et al., 2022b;a). A key ap- proach leverages task vectors (i.e., parameter differences between fine-tuned and pre-trained VIMSs (aharco al., 2023; Ortiz-Jimenez et al., 2023), providing flexible mechanism for merging knowl- edge. Recent works like Ties-Merging (Yadav et al., 2023) proposed resolving interference between merged models by identifying parameter conflicts, while AdaMerging (Yang et al., 2024) introduced adaptive merging strategies for multi-task task learning. These methods overlook parameter turbations and local loss geometry. We thus complement this line by using crudient-informe masks and adversarial parameter trajectories to account for sensitivity and curvature during merging.
per-
Adversarial robustness foundation VLMs. Adversarial robustness in VLMs remains a critical problem, with recent works (Mao et al., 2023; Schlarmann et al., 2024) mainly pursuing adversarial fine-tuning by integrating adversarial examples into training to bolster robustness. However, these methods often erode clean accuracy and require heavy hyperparameter tuning and costly retraining, limiting scalability. We instead merge off- the-shelf naturally and robustly fine-tuned VLMs via task vectors, using crudient-informed stability masks and adversarial parameter trajectories to trace gradient paths during merging, reconciling accuracy and robustness without additional fine-tuning.
3 PREDICTION STABILITY-AWARE MODEL MERGING
Below, we propose PISTOLE, the first task vector- model merging method to address the accuracy-robustness trade-off without costly adversarial fine-tuning, generalizing across tasks.
3.1 Background: ADVERSARIAL
AND TASK VECTORS
CLIP. As milestone of multimodal learning, CLIP (Radford et al., 2021) employs two modality- specific encoders: an image encoder fe. X pd and a text encoder fe, : + IR whose outputs reside in a shared d-dimensional embedding space. For an input image and set of class prompts (t)..... ) constructed from templates (e.g., "This is a photo of [CLASS.c] ), the prediction is obtained via the cosine similarity between the visual feature and each textual feature: exp ( (cos(fe, (x), (t.))
(x; Or, 0T)
Df exp ( (cos (x), for (ter))
where exp(-) denotes the exponential function, and cos(-, -) computes the cosine similarity between two embeddings that have been lg normalized. The prediction vector w.r.t. the CLIP parameter set 0 = 0, 0r] across C' categories is written as (x = [pr (x; 0, ,p(xx 0, 0r)
Standard adversarial fine-tuning (TeCoA). In consistent with the standard adversarial training paradigm (Madry et al., 2018), TeCoA (Mao et al., 2023) enhances CLIP robustness against fx- norm adversarial attacks by solving the following minimax optimization problem:
min E (x,o)DD 0i
max LCs LCE(Po(x + o). Se
where e. [1(c 1),.... 1(c C) E 1) is the one-hot label for class e, and Lee denotes the cross-entropy loss. The inner maximization is approximated by the m-step Projected Gradient Descent (PGD) (Calamai & Moré, 1987) on the negative loss function:
& (1+1)= ((++))[ll(xx)))
( +a sign LC (po(2()) ).e.))
(3)
initialized with (0) = 0.000 (O, I). Here, a represents the step size, sign(.) is the element- wise sign function, and (-) denotes the projection onto the lo ball of radius e. Further details of other adversarial fine-tuning approaches are in Appendix B.
3

---

Published as a conference paper at ICLR 2026
Task vectors for multi-task adaptation. Given downstream task T; with data D,, fine-tuning a pre-trained VLM 9, yields task-specific parameters 0, and the task vector (Ihharco et al., 2023) (parameter displacement) T; 0; On. Task vectors compose for vision encoders: for {T, {T:) the M aggregate Tidi = اك' Ti defines the merged model add = On + À Tadd, with the scalar tuned on a validation set. This simple addition typically attains competitive performance across the M tasks.
3.2 CAN VANILLA TASK- VECTOR ADDITION AID TRADE-OFF?
Weight-space interpolation reveals an InD-shift sweet point. Following WiSE-FT (Wortsman et al., 2022b), let 0, denote a pre-trained zero-shot VLM and PPT its VLM fine-tuned on a reference (in-distribution, InD) dataset, e.g. ImageNet. Evaluating on both the reference dis-
Costtttttttttn (Saits tom impelted Arturay
tribution and shifted distributions (e.g.. natural variants and subpopulations), the linear interpolation OwISE (1 A) 0 + X 0pr, E [0. I], typically traces a Pareto- like curve with an interior X that preserves high InD ac- curacy while improving accuracy under distribution shift
A for both buttie betion and sunaf sistr buthe ecouracy
(pecs rngjiblee cird
Space
Shot
Naturalyy Fine- Fine-Tuned CLIP Medil
(see Figure 2). This simple weight-space averaging with a sweet point (improved trade-off) motivates viewing pa- rameter operations (including task-vector arithmetic) a wigght alternative to exhaustive hyperparameter sweeps when balancing specialization the reference data against generalization shifted data.
Parameter Space Ensemble A 0.5
10
In- Imagehist Accuracy (S]
Figure 2: Parameter-space ensembling of pre-trained and fine-tuned CLIP.
Vanilla addition fails to balance natural performance and adversarial robustness. Let Pmn and Pnb be natu- rally and robustly (adversarially) fine-tuned models with task vectors Thar - -nm On and Trob- Oroh- Og. Under the naive interpolation (A) = -- (1 A) Tix + Trob, Ae [0, 1], clean and adversarial accuracies vary almost lin- early and antagonistically, producing near straight line between endpoints with no pronounced interior optimum (Figure 3). The issue is equal weighting: it ignores which coordinates align or conflict across objectives, so robust- ness gains come by eroding clean accuracy at roughly constant rate. Gradient analyses (Figure 1) indicate that compatible and conflicting directions co-exist, motivat- ing a prediction stability-awe -aware, selectively re-weighted merge that preserves consensus coordinates while attenuating counterpart-sensitive ones.
The track-oft setwern performance plong the parceater space estarble carea
Rolust Finetuning
In seeet pain be fouid
Farmmeee Space Ensembees
Naturaly
(Adeersarally) Tured
Average Robust Acsuracy (P)) |Peturtation Radus 2055)
Figure 3: Parameter ensembling of nat- urally and robustly fine-tuned CLIP.
3.3 GRADIENT- STABILITY RE-WEIGHTING
Vanilla task vector merging typically treats all coordinates equally, reproducing the near-linear clean-robus. trade-off of oaan (A). In contrast, Figure shows only modest aligmment that degrades with attack strength, indicating mix of compatible and conflicting coordinates across the natural and adversarial objectives. We therefore build complementary masks that (i) preserve coordinates the counterpart objective deems stable and (ii) attenuate coordinates where the counterpart exhibits large gradient magnitude (i.e., it would strongly update those weights in an opposing direction).
Aggregated gradients and layer-wise scaling, Raw per-batch gradients noisy and differ in scale across layers; without normalization, a few high-variance tensors dominate the mask. We first accumulate expected gradients for the two objectives and then normalize them per layer:
Enai = ((x,))) V Ve... Lce(pe... e(c)))
grob = E (x,c))= Ven. (pe(Pe... (x), e(c)]] (4)
where & is an adversarial sample for d obtained by Eq. (3). For each objective index s E (ntt, rob} and each layer let g!!) denote the gradient tensor of layer 1. We define the per-layer normalization:
Norm (g!") = max g
E [0, 1] (g
(() = Noorm (g."))
(5)
+
4

---

Published as a conference paper at ICLR 2026
[[righht
Conficts drynkl shored
Irafficercy occus
Foot
rightee Pesk Wetkr
Bub
Masking
suppress
cehembranch
dewt-weighted
(a] Naive Linear Marge
(o) PESTOLE
Mos
Mub
Figure 4: (a) Naive linear merge uniformly adds natural and robust task vectors, ignoring conflicts and yielding a near-linear trade-off. (b) PISTOLE merges with complementary, gradient-informe masks, suppressing conflicts and preserving shared directions for better clean-robust balance.
where small perturbation > 0 and temperature 7 E [0, 1] are to control dynamic range. Stacking across layers yields Beat = concat, (!!)) and Eoob concat, (1)) both in [0, 1j
Complementary stability masks. Gradients generally indicate which coordinates each objective would change, which means that large magnitudes flag parameters that are sensitive for that ob- jective. To avoid reintroducing antagonism during merging, we suppress coordinates that the other fine-tuning objective wishes to change most, while preserving those it considers stable. Concretely, for 8 E {nat, rob} and each layer I, let gs E [0, ijshipe be the normalized gradient magnitudes from Eq. (5), and write g. concat, (1) [0, 1jd. We convert these into complementary masks:
Meat
(1 _rob)" M
(1 (1-gnat)
(6)
where k 2 1 sharpens selectivity. To guarantee a user-controlled stability budget, we apply per- layer quantile cap, M + min is tq..) and M f- min (Min.. (q))), where end (resp., (il) is the q-quantile of M (resp., Mo..,). This attenuates the layerwise top-q most counterpart- sensitive coordinates, thereby upper-bounding first-order cboss-objectivee interference.
Theorem 1. Let g := -eoCnr(O)) (0)le=e, and Bi VoCnu(()) o=), Let M := (1-
and Mor = (1 (1-Emar)" )" be the uncapped complementary masks from Eq. (6), with k 2 1 and 8. =00 defined by Eq. (5). For each layer I, let tggl be the q-quantile of (Ma and tyg the q q-quantile of (MPt)) (empirical quantiles on layer entries)." I Define the capped masks layerwise by (Mma):= == min( (Maa)c ), und 1) and (Mmt)) == min( (ain((Mr))) (ot)) (ag1) Set Pnat == maxt qet and Pmbb maxi 1nll Then for any 5 E TRd,
[gnar Maa 5) S Pmat Il |01|2
[g M 5) S Pub | ll 1:0 2-
Moreover, if is increased (i.e., sharpening ")) any of the layerwise caps 1g tgg are decreased, the right-hanne sides in Eq. (7) are monotone nonincreasing.
Proof. The full proof is provided in Appendix D.I.
Corollary 1 (Worst-case first-order contraction vs. uniform addition). For task vectors Taor Onot 0, and T = 0 0y (g M @ Trob) S Pnot ll || S lig || T and the symmetric bound holds swapping (nat, rob). Hence, complementary masking contracts worst-case cross-objective first-order interference by factors Peats Prob S relative to unmasked mixing.
Theorem formalizes a first-order non-interference guarantee: when any displacement is filtered by the counterpart's 's mask, the first-order increase of the other objective's 's loss is bounded by tunable multiplicative factor Σ 1. The factors Pont and Prob depend only on layerwise caps (through their maxima) and respond monotonically: larger к (sharper masks) or tighter caps reduce p and thus strengthen attenuation. Practically, this means masked combinations curve the otherwise near-linear clean- dlean-robust trade-off of uniform addition by suppressing coordinates that the counterpart objective marks as sensitive, while preserving jointly stable coordinates.
AAy standard definition of the empirical q-quantile with g (0. 1] suffices. We here consider that quantiles monotone under component-wee decreases.
5

---

Published as a conference paper at ICLR 2026
3.4 TRACING ADVERSARIAL PATHS IN PARAMETER SPACE
The complementary masks in Section 3.3 are built from single-point gradient magnitudes at nO Omb), which capture first-order instability but can miss nearby high-curvature pockets where sensitivity spikes. To refine these stability estimates without retraining, augment them with ad- versarial parameter perturbation: we trace short parameter updating trajectories in a small neigh- borhood of each fine-tuned solution and aggregate gradients along these paths. Intuitively, coordi- nates that remain stable under worst-direction parameter nudges are safe to keep, whereas coordi- nates of large gradients along these trajectories fragile and should be attenuated during merging.
Adversarial parameter trajectories. For each objective & E (nat, rob), define the Frobenius ball r I 17 r1[(011F) } with radius factor 7) > 0. Starting from 0!¹² 0s, we perform K projected gradient ascent steps in parameter space that follow locally worst-case directions:
Vo. {A: |||
(c))) x-0
g ( t Io. + B )
(8)
F +
where 8 >> is the step size, and Ll, Vo, (.) projects onto Ve,. We take Xam =x (clean inputs) and Xyob =x (adversarial inputs), so that Lus and Luo are evaluated under their respective input regimes.
Path-integrated gradients. To calibrate the anth-integrated gradients with local sensitivity, we in- troduce the Prediction Criticality Index (PCI): a curvature-awe aware scalar that measures how fragile a prediction is to small parameter-space perturbations. We use PCI to weight steps along the adver- sarial parameter trajectories, so high-curvature (fragile) regions contribute more to the accumulated gradient, while flat, confidence-saturatio regions are deemphasized. Formally:
Definition (Prediction Criticality Index (PCT)). Let e be the model parameters and p(x; 0) [0, 1 be the prediction for an input x. For a fixed radius factor 1j > 0, define the parameter-level Given A sampled isotropically and with zero mean from Va (e.g., uniform in the hyperball). Its covariance is denoted as o²L,,
Frobenius perturbation ball Vo A E R ||||| S 0]| p
where the per-coordinate second moment is σ² == A [|]] r² e : The equality on the right follows directly from the radius normalization for any isotropic, zero-mean law supported the ball Vg. For the ground-truth class c (I,..., C}, we define PCI as follows:
Po(x; A) P-.pp 0)
PCI(x, c, 0) EaeV.
(x 0)
large PCI w.r.t. clean/adversarial examples indicates that prediction confidence is highly sensi- tive small parameter changes (fragile knowledge), while small value reflects robustness. We therefore accumulate path inth--interrted gradients:
G, (xx,)))
Σ PCI(X,, c, VoL. (po(x)))) ==0
E roat, (10)
o=0!!)
We then normalize per layer in Eq. (5) to obtain scores g: g [0, 1]d, and form path-refined complementary masks via MPath (1 -11-3h g.pall)* and M = (1 pathy followed by the same
per-layer caps as in Section 3.3. The final merged displacement keeps the stable parts:
TA))) A (M C That) + (1 - A) (M path rob
C Trob), OPISTOLE(A) = 0, + (A). (11)
Empirically, varying À with (PISTOLE(A) bends the trade-off of uniform addition, yielding interior points that outperform naive mixing. Figure an overview of our PISTOLE method compared with naive linear merging. See Appendix E for pseudocode. We next formalize the curvature link that motivates our PCI-based weighting.
frontier beyond the near-linear
Theorem 2. Let PCT be defined as in Definition I, where (x; 0) > 0 is twice continuously differ- entiable in a neighhorhood of O. Given (0) = (x: 0), for sufficiently small n, we have the following approximation:
x² (r. (0))
PCI(x, c)
+ 0
(12)
2
P. (x 0)

---

Published as a conference paper at ICLR 2026
Table 1: Zero-shot accuracy of diverse adversarial learning methods evaluated 14 datasets. Met-
rics: Clean, Robust (AutoAttack, -oomm = 2/255) Accuracy, and Clean+Robust Sum.
ImaupoNet
CHARR 10
CIPAR-10
Cileeb11
OxfordPa
Flower
EUDSAT
ImsgeNees No
ImageNes.S
STL10
ROVE
PCAM
DTD
Eval.
Method
Average
CLIP
74.90
95
77.91
17
87
Clean
leC
(0)
05. 40
N6 88
61
44
43
45
80
80.3
000
58
30
40
1%
61 1.5
58
FARE
73.96
08. 25
90
24
76
66.80
85.
89
75
16.
54
22
50.02
56.86
65.50
IOTA
N0.26
06.55 15-
88
81
81
49
45.9
50
30
14
29.01
8D
3X
62
26.20
70.08
62.34
Stansled CLIP
0.00
0.02
0.00
0.00
0.00
0 90
9.00
000
0.00
0.00
0.03
001
0.01
0.00
0.01
Robest
51 74
86
21
01
15
18
38 30
26
2900
63
14
44.50
43.20
FARE
43.56
85
82
89
23
11
10
67 T0
39
56
17
3635
12.97
PISTOLE
61.89
66.71
39.83
28.36
72.83
71.30
33.42
28.84
6,61
63,03
4588
47.65
Standard CLIP
74.90
09 55
V5 20
31 MN
77 91
N1
93.33 21
S5-21
31
K3
19.70 TU
7340
Sum
PMR
13.88
185
102
155.8
(39
86.0
21.00
26.43
$7.05
145
102 00
100
FARE
166
ON
30
153
19.98
07,83
140.53
104
30
PISTOLE
142.71
187.55
157.54
108.01
96.71
189003
162.65
103.50
80.6
41.91 33.03
110.66
148.21
105.76
116.9
Table Average accuracy (%) of diverse CLIP
Table 3: Avg. accuracy (%) of diverse e when fine-tuning and testing w.r.t. CLIP w/ ViT-L.
backbones with perturbation radius ==2 = 2/255.
Backbone
Method
Clean
Robust
Sum
Radius
Method
Clean
Robust
Sum
TeCoA
66.95
48.63
115.58
TeCoA
58.90
38.07
96.97
PMG
68.96
50.31
119.27
PMG
61.72
39 9..40
101
VIT-H/14
FARE
70.55
50.25
120.80
=33/555
FARE
63.55
37.17
100.72
49
15.005
59.65
38.59
98.24
PISTOLE
73.61
52.45
126.06
PISTOLE
65.09
40.57
105.66
TeCoA
48.83
25.75
74.58
TeCoA
56.25
32.53
88.78
PMG
49.71
26.98
76.69
PMG
38.82
33.87
92.69
VIT-B/32
FARE
56.68
29.30
85.98
4/255
FARE
60.26
32.02
92.28
LGA 57.54
31.15
88.69
TOA
56.76
32.86
89.62
PISTOLE
60.71
33.58
94.29
PISTOLE
62.37
34.94
97.31
Proof. The full proof is provided in Appendix D.2.
Theorem 2 reveals that the PCI is large when the prediction confidence lies region of large Hessian trace (high curvature), whereas a small PCI characterizes flat, confidence-saturated zones. High-PCI samples (i.e., those most sensitive to parameter perturbations) are up-weighted in the accumulated gradients, directing the merge to address fragile prediction modes that would otherwise dominate post-fusion error. The sensitivity masks suppress features that are unstable in one model while retaining their more robust analogs in the other, fostering synergistic blend of natural performance and adversarial robustness.
EXPERIMENTS
In this section, we compare our PISTOLE method with state-of-the-art adversarial fine-tuning ap- proaches across different scenarios and downstream vision-language tasks.
Datasets. Task vectors are obtained by natural and adversarial fine-tuning on ImageNet-Ik (Deng al., 2009). We report zero-shot classification on its test set plus 13 additional datasets, and assess transfer on captioning, visual question answering, hallucination, etc. (details in Appendix C.I).
Implementation details. Unless stated otherwise, the base VLM is CLIP with ViT-L/14 encoder, following cobust-CLIP practice (Mao et al., 2023; Schlarmann et al., 2024). The natural VLM is trained by ERM on clean data; the robust VLM follows PMG (Wang et al., 2024) with 10-step PGD (p c 2/255, step a 1/255). We form task vectors (Tmat Trob) and merge them via PISTOLE with default mixing À 0.2. Zero-shot robustness is measured with AutoAttack (Croce & Hein, 2020). For downstream transfer, we replace the vision encoder in LLaVA-1.5-7B and OpenFlamingo-99 with our merged encoder while keeping other components fixed. Evaluations use adaptive attacks for faimess. Additional configurations appear in Appendix C.2.
4.1 MAIN RESULTS ON ZERO-SHOT CLASSIFICATION
Zero-shot classification. Table summarizes clean and robust accuracies for CLIP ViT-L/14 on 14 evaluation sets. In addition to reporting each metric separately, we include scalar trade-off score Sum (Clean+Robust) to capture overall performance. PISTOLE delivers the strongest zero-shot results, improving mean clean accuracy by ~5% and mean robust accuracy by ~5.8% over state- the adversarial fine-tuning baselines. On in-distribution ImageNet, our trade-off is marginally better than alternatives, which we attribute to the robust component obtained via PMG (Wang al.,
7

---

Published as a conference paper at ICLR 2026
Table 5: Zero-shot transfer image captioning (CIDEr score) and VQA (accuracy %).
Image Captioning
Visual Question Answering
VyM
Method
COCO
Flickr30k
TexVQQ
VQAv2
Vizwiz
Clean
Robust
Sum
Clean
Rohnet
Sum
Cless
Roiras
Sum
Clean
Robust Sum
Clean
Rohnet
Sum
Standard CLIP
112.3
2.9
115.2
74.7
14
34.8
54.8
74.5
74.5
20.4
IeCoA
96.7
45.1
141.8
55.2 2
24,0
19.2
23.8
12.3
36.5
66.2
35.7
101.9
43.5
29.6
72.1
LLaV
FARS
108
479
136.4
67
25-
91.
27.6
14.7
254
70
35.
1048
41 9
0
TGA
101
50.6
15L.
B1.
80.1
14.6
41-7
61.
35.0
102.3
42.8
70.8
30.5
33.1
32.3
Standard CLIP
73.8
1,5
80,3
58.7
0.5
59.3
22-3
0.0
22-3
17.7
0.0
47.7
12.
3.3
21.0
OpenfPlamingo
FARE
PMG
76 3
27.9
11-2
32.7
1074
L10.6
12.0
33.5
15.9
13-1
69
68.5
17.7
18.8
28
21.0
203
47.
16.7
34.0
21.8
68.5
71.0
4-2
31.1
27
TKD
3.5
2004
PISTOLE
80.7
34.0
114.7
57.2
16.9
74.1
21.2
4.1
25.3
47.8
24.9
72.7
19.4
5.7
25.1
2024) used the merge. Section 4.3 further examines how substituting different robustly fine-tuned VLM components can alter the relative trends.
Robustness generality across backbones. Beyond ViT-L/14, we evaluate our PISTOLE method with ViT-H/14 and ViT-B/32 based on the clean/robust task vectors of the identical CLIP archi- tecture. As summarized in Table 2, the method consistently surpasses previous adversarial VLM learning approaches in both average clean and robust accuracy across the same 14 datasets.
Robustness across varying perturbation radius. We further stress-test robustness by increasing the Lo budget beyond the default e = 2/255, considering E 3/225.. 4/255 for both fine-tuning and evaluation to ensure parity. Results in Table indicate that our PISTOLE maintains its lead across these stronger threat models in the zero-shot setting.
Task vector merging with the PEFT ex- tension. To reduce the high computational cost brought by full parameter fine-tuning, Parameter-Efficit Fine- Fine-Tuning (PEFT) strate- gies (e.g., LoRA (Hu et al., 2022) based on learnable low-rank matrices for efficient adap- tation) were typically paired with adversarial fine-tuning. We thus extend prior adversarial VLM learning methods and our task vector- based merging approach with LoRA. We report both clean and robust accuracy associated with their sum for our LoRA-based based PISTOLE method against other LoRA-enabled baselines in Table 4. Even under this efficiency regime, our PISTOLE typically achieves the best trade-off.
Table 4: Average accuracy (%) w.r.f. different e for fine-tuning and testing (ViT-L) w/ LoRA.
Radius
Method
Clean
Robust
Sum
TeCoA
55.22
26.54
81.76
PMG
56.82
27.02
83.84
3/255
FARE
57.84
23.85
81.69
TGA
55.88
26.66
82.54
PISTOLE
59.47
29.35
88.82
TeCoA
49.84
19.87
69.71
PMG
52.08
20.07
72.15
= 4/255
FARE
53.20
19.09
72.29
TGA
31.13
19.83
70.90
PISTOLE
55.06
21.78
76.84
4.2 ZERO-SHIE SHOT DOWNSTREAM TASK TRANSFER
Transfer to image captioning. We here evaluate zero-shot task transfer to image captioning by swapping the vision encoder in LL and OpenFlamingo with our PISTOL.E-mereed encoder. Ta- ble 5 (Left) reports the CIDEr score (Vedantam et al., 2015) on COCO and Flickr30k. PISTOLE attains the strongest scores both clean inputs and under adversaries for better trade-off, outper- forming adversarial learning baselines. Qualitative visualizations are in Figure 6 (Appendix F).
Transfer to Visual Question Answering (VQA). Table 5 (Right) also summarizes VQA accu- racy across three standard benchmarks. PISTOLE consistently increases the sum relative to prior baselines, primarily by delivering sizable zero-shot robustness gains while keeping natural accuracy essentially intact (and even higher on VizWiz). These results indicate a better accuracy compromise in the zero-shot regime. Qualitative examples in Figure 7 (Appendix F).
Transfer to object hallucination. To probe hallucination (i.e., erroneously recognizing ob- jects that do not exist in inputs) (Sahoo et al., 2024), adopt the POPE benchmark (Li et al., 2023) with its standard guestion-sampling pro- tocols (Appendix C.3). Results in Table indi- cate that PISTOLE consistently lowers halluci- nation rates versus competing adversarial learning schemes. We attribute this to our stability-
Table 6: POPE hallucination benchmark (F1-
score) with ViT-L using three sampling protocols.
POPE Sampling
Method
Avg. Score
Random
Popular
Adversarial
TeCoA FARB
75.2
817
80.9
70
823
81
78.6
80
TOA
80.4
79.8
76.0
78.7
PISTOLE
84.6
83.7
80.8
83,0
masking, which dampens over-confident, brittle features and favors parameters that remain reliable under perturbations. Qualitative cases are in Figure 8 in Appendix F.
8

---

Published as a conference paper at ICLR 2026
Transfer to science question answering w/ Chain-of-Thought (CoT), We assess CoT reasoning on the ScienceQA benchmark (Lu et al., 2022). Across multiple prompting and VLM settings (Appendix C.3), our PISTOLE achieves the best overall accuracy (Table 7), suggesting that stabilizing the vision backbone from diverse knowledge sources improves the robustness of multi-step reasoning. Illustrative examples are in Figure 9 in Appendix F.
Table 7: CoT eval. (Acc.) using science question answering for adversarial VLM learning (ViT-L).
Temperature
Method
Avg. Acc.
0.0
0.1
0.2
TeCoA
51.4
51.6
50.0
51.0
PMG
510
52.0
6
TARE
52.
52.4
52.2
55
5
PISTOLE
54.1
53,9
54.2
54.1
4.3 FURTHER ANALYSES (WHY PISTOLE IS EFFECTIVE)
In this section, we conduct series of controlled ablations of our PISTOLE method and its compo- nents to justify its effectiveness and generalizability across different scenarios.
Effect of PISTOLE components. We here quantify the contributions of three core modules in our PISTOLE: (i)
Table Component ablations for PIS- TOLE. We report mean clean and robust
Gradient-Informed Stability Mask (GISM) in Eq. (6), (ii) Re-weighting of Prediction Criticality Index (PCI) in Definition I, and (iii) Adversarial Parameter Trajectory (APT) in Eq. (8). Table 8 reports zero-shot results aver- aged over 14 classification benchmarks. As reference, the first row (baseline) applies vanilla task vector merging in O.am()) in Section 3.2. Enforcing
accuracy (%) averaged on 14 datasets.
GISM
PCI
APT Clean
Robust
Sum
66.57
45.69
113.47
/
68
0633
46.47
114.83
07.64
47.11
114.75
/ 69.24
47.65
116.89
stability masks simultaneously improves natural performance and robustness. Augmenting it with PCI further lifts the trade-off by prioritizing fragile predictions. Incorporating adversarial parameter trajectory improves robustness by refining sensitivity estimates beyond single-oiin gradients.
Effect of natural knowledge components. We vary the source of the "natural" task vector while fixing the robust task vector in our PIS- TOLE method. As shown in Table 9, replacing the pre-trained (zero-shot) VLM with natu- rally fine-tuned one yields a stronger clean-robust trade-off. Intuitively, natural empirical risk min- imization contributes task-calibrated shifts that better align with the robust objective's consensus direction, which our stability masks preserve while suppressing antagonistic coordinates.
Table 9: Avg. performance (%) of PISTOLE
with diverse task vectors for natural knowledge.
Natural Knowledge Source
Clean
Robust
Sum
Pre-Trained (Zero-Shot) VLM
67.10
47.83
114.93
Naturally liie--tuee VLM
69.24
47.65
116.89
Effect of robust knowledge compo- nent. We ablate the source of the robust task vector while fixing the natural one, instantiating PISTOLE with diverse adversarial fine-tuning methods (summarized in Appendix
Table 10: Avg. performance (%) of our PISTOLE with di- verse task vectors for robust knowledge.
Robust Knowledge Source
ImageNet
Avg. 13 Datasets
Cleun
Robust
Sum
Cleun
Robust
Sum
43.52
FARE, PMG (Our Sctup)
75.86
62:02
137.88
69.47
44.91
80.82
61.89
142.71
68.35
46.55
114.90
TeCoA yields the strongest in-distribution (Im- ageNet) robustness but transfers less favorably under shift, while FARE improves OOD robustness yet lags on ImageNet. Our setup offers the most balanced performance, producing the best trade-off. objective, which preserves features aligned with the
B). From Table 10, we observe a consistent pattern:
We attribute this its medietion-regulariin natural objective. Our stability masks retain these while suppressing antagonistic coordinates.
Curvature analyses. We quantify loss parameter curva- ture along the local update (gradient) direction for ad- versarial inputs in Figure 5, which plots this directional curvature while changing the merging weight A. Vanilla mixing exhibits consistently higher curvature, especially at a larger radius, whereas PISTOLE lowers curvature throughout, indicating smoother, more stable landscape under parameter nudges. We observe that À 0.2
0.85
Loss-Parameter Curvature
Vanila Task Vector Merging
0.80
Our PISTOLE
0.75
0.70
0.65
0.60
(our operating point) attains the lowest curvature and si- multaneously yields the best trade-off. This behavior aligns with our theorems: PCI weighting prioritizes high- curvature regions (Theorem 2). and the complementary masks provably contract cross-objective first-order inter-
0.55
0.01
0.05
0.1
0.2
0.3
0.4
Merging Weight
Figure 5:
Analyses of loss- parameeerr curvature on ImageNet adversaries.
ference (Theorem 1). jointly steering the merge toward flatter, dettee--ennralizingg solutions.

---

Published as a conference paper at ICLR 2026
Extended analyses. More analyses are in Appendix G, including task-vector re- re-weighting ablations, hyperparameter analyses, and cost comparisons, all of which corroborate the efficacy of PISTOLE.
CONCLUSION
Motivated by our gradient-similartyy analyses between natural and robust VLMs, we introduced PISTOLE, a prediction stability-aware task-vector merging framework that composes off-the-shelff natural and robust VLMs without retraining. PISTOLE forms complementary. mudieent-informe' masks and refines them along adversarial parameter trajectories, weighting steps by a curvature- linked prediction criticality index. Our theorems bound cross-objective interference and show that this index tracks Hessian trace, explaining why the merge gravitates toward flatter, more general- izable regions. Empirically, PISTOLE consistently improves the clean-robust trade-off across 14 datasets and scenarios, and transfers as drop-in encoder to downstream tasks. Rigorous ablations and curvature analyses validate each component and quantify its impact the clean-robust frontier.
ACKNOWLEDGMENT
This research/project partly supported by the National Research Foundation, Singapore under its partly by the MTI under its
AI Singapore Programme (AISG Award No: AISG3-RP-2022 Al Centre of Excellence for Manufacturing (AIMfg) (Award W25MCMF)14), and the College of Computing and Data Science, Nanyang Technological University.
ETHICS STATEMENT
This work studies model robustness and parameter-space merging for vision-language models. Our experiments use publicly available datasets and model checkpoints under their original licenses. No personally identifiable or sensitive data is introduced. We evaluate robustness with standard adversarial attacks stress-test models in defensive setting. As with most robustness research, there is potential dual use: insights that improve defenses could also inform stronger attacks. To mitigate misuse,
paper reports evaluations and ablations strictly for benchmarking and does not target real users or deployed systems. We encourage responsible release and deployment practices, including clearly labeling merged checkpoints, documenting training/merging procedures, and validating safety filters when models are adapted. From an environmental perspective, parameter- space merging substantially reduces computational costs compared with repeated adversarial fine- tuning. We are not aware of disparate-impact risks unique our method beyond those inherited from the underlying datasets and models. Nonetheless, we recommend auditing downstream applications for distribution shift and faimess where appropriate.
REPRODUCIBILITY STATEMENT
We organize the paper and appendix enable end-to-end replication using publicly available models and datasets. Complete experimental configurations, including datasets, preprocessing, evaluation splits, and attack settings, are centralized in Appendix C. Dataset coverage and zero-shot evaluation suites are detailed in Appendix C. The task-vector construction and merging/evaluution protocol (backbones, default hyperparameters, and attack parameters) appear Appendix C.2, while down- stream transfer setups for image captioning, VQA, object hallucination, and ScienceQA CoT specified in Appendix C.3. Theoretical assumptions and complete proofs are provided in Appendix D. Hyperparameter choices, sensitivity studies, and search ranges are summarized in Appendix G.2. Tables and figures in the main text refer back to these sections so that all reported clean and robust accuracies can be reproduced under the stated configurations. We stress that our code and check- points will be publicly available to facilitate independent verification and reuse.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015. 18
10

---

Published as a conference paper at ICLR 2026
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint M.Xiv:2308.01390 2023. 3,
Paul H Calamai and Jorge Moré. Projected gradient methods for linearly constrained problems. Mathematical programming, 39(1):93-116, 1987. 3
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014. 17
Adam Coates, Andrew Ng, and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the Sourteenth international conference on artificial intelli- gence and statistics, pp. 215-223, JMLR Workshop and Conference Proceedings, 2011. 17
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter-fres free attacks. In International conference on machine learning, pp. 2206-
2216. PMLR, 2020. 7, 17, 18
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scart scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leee, 2009. 7,
Junhao Dong, Piotr Koniusz. Junxi Chen, and Yew-Soon Ong. Adversarially robust distillation by reducing the 92-111. Springer, 2024a. 17
variance gap. European Conference on Computer Vision, pp-
Junhao Dong, Piotr Koniusz, Junxi Chen, Xiaohua Xie, and Yew-Soon Ong. Adversarially robust few-shot learning via parameter co-distillation of similarity and class concept learners. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28535- 28544, 2024b. 17
Junhao Dong, Yuan Wang, Xiaohua Xie, Jianhuang Lai, and Yew-Soon Ong. Generalizable and discriminative representations for adversarially robust few-shot learning, IEEE Transactions on Neural Networks and Learning Systems, 36(3):5480-544 -4990,599,,-0 2024c. 17
Junhao Dong, Piotr Koniusz, Liaoyuan Feng, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 21037- 21047, October 2025a. 17
Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 236 236-247,44 2025b. 1
Junhao Dong, Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. In Forty-second International Conference on Machine Learning. 2025c.
Junhao Dong, Jiao Liu, Xinghua Qu, and Yew-Soon Ong- Confound from all sides, distill with resilience: Multi-objective adversarial paths to zero-shot robustness. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 624-634, 2025d. 17
Junhao Dong, Cong Zhang, Xinghua Qu, Zejun Ma, Piotr Koniusz, and Yew-Soon Ong. Robust superalignment: Weak-to-strone robustness generalization for vision-language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025e.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEFE, 2004. 17
11

---

Published as a conference paper at ICLR 2026
Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the in vqa matter. Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017, 17
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018. 17
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222, 2019. 17
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021. 17
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR, 2022. 8, 17
Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang. Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502. 2025.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learming Representations, ICLR, 2023. 2, 3, 4, 25
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 561, 2013. 17
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 17
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 292-305, 2023. 8, 17, 18
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014. 17
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- fion, pp. 26296-26306, 2024. 3, 17, 18
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, ICLR, 2019. 17
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-252 2022. 9, 17, 18
Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, al. Fairelip: Hamesssin faimess in vision- vision-language learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12289-12301, 2024. 3
12

---

Published as a conference paper at ICLR 2026
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR. 2018. 3, 17
Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:/306.5157 2013. 17
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero- shot adversarial robustness for large-scale models. The Eleventh International Conference on Learning Ceppresentationns/CLA 2023. 1, 3, 16,
Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp- 722-729. IEEE, 2008, 17
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, (6:66676:667766671 66754, 2023. 3, 25
David Osowiechi, Mchrdad Noori, Gustavo Vargas Hakim, Moslem Yazdampanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizace, Ismail Ayed, and Christian Desrosiers. Watt: Weight average test time adaptation of clip. Advances in neural information processing systems,
2024. 25
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012. 17
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting egion-to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp. 2641-2649, 2015. 17
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
8748-8763. PMLR, 2021. 1, 3, 16, 17
Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A comprehensive survey of hallucination in large language, image, video and audio foundation models. In Findings of the Association for Computational Linguistics: EMNLP, pp. 11709-
11724, 2024.
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CV International Conference on Computer Vision Work- shop, pp. 3677-3685, 2023. 18
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In International Conference on Machine Learning, pp. 13685-43700 PMLR, 2024. 3, 7, 16,
17, 18
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019. 17
Pavan Kumar Anasosalu Vasu, Fartash Faghri. Chun- Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastylm: Efficient vision encoding for vision language models. Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 19769-19780. 2025. 3
Ramakrishna Vedantam, C Lawrence Ziticck and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015. 17
13

---

Published as a conference paper at ICLR 2026
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Cnrrvention-MICCAA 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pp. 210-218. Springer, 2018. 17
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019. 17
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2024. 1, 7. 16, 17
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lope Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbllthh, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time. In International conference on machine learning, pp. 23965-23998 PMLR, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971 2022b. 3,4
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in Neural Information Processing Systems, 33:2958-2969, 2020. 17
Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, and Xiao Xiang Zhu. Geolangbind: Unifying earth observa- tion with agglomerative vision-language foundation models. arXiv preprint arXiv:2503.0633 06312, 2025. 25
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models, Advances in Neural Information Processing Systems, 36:7093-7115, 2023. 3, 24
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. The Twelfth International Conference on Learning Representutions, ICLR, 2024. 3, 24
Yongquan Yang, Haijun Lv, and Ning Chen. A survey on ensemble learning under the era of deep learning, Artificial Intelligence Review, 56(6):5545-553) 5589, 2023. 3
Lu Yu, Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness in vision-language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, NeurIPS, 2024. 16
Yuhang Zang, Hanlin Goh, Joshua M. Susskind, and Chen Huang. Overcoming the pitfalls of vision- language model finetuning for OOD generalization. In The Twelfth International Conference on Learning Representations, ICLR. 2024. 3
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Bl Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision- -sionnlanguuag pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5005- 5013, 2022. 2012. 1
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai- Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36:54111-54145 2023.
14

---

Published as a conference paper at ICLR 2026
Minghao Zhu, Zhengpu Wang, Mengxian Hu, Ronghao Dang, Xiao Lin, Xun Zhou, Chengju Liu, and Qijun Chen. Mote: Reconciling generalization with specialization for visual-language to video knowledge transfer. Advances in Neural Information Processing Systems, 37:55403-55424, 2024. 25
15

---

Published as a conference paper at ICLR 2026
A APPENDIX SUMMARY
This appendix provides background adversarial fine-tuning for VLMs (Section B), followed by full experimental configurations (Section C). We then present the complete theoretical analyses (see Section D) with the algorithmic specification of our PISTOLE method. We supply qualitative visualizations of zero-shot transfer under clean and adversarial inputs (Section F). Extended studies report ablations re-weighting. hyperparameter sensitivity, computational cost comparisons, and also the performance comparison with standard task vector merging approaches are listed in Section G. Finally, we conclude with limitations and broader impact of our work in Section H with an LLM usage declaration in Section I.
B DIVERSE ADVERSARIAL FINE-TUNING SCHEMES
Below, we provide details regarding adversarial fine-tuning schemes in the context of Vision- Language Models (VLMs) for a more comprehensive background introduction.
TeCoA (Mao et al., 2023) [In- In--istribbutin Robustness]. Given image-text pairs D, standard ad- versarial fine-tuning (TcCoA) (Mao al., 2023) is framed minimax optimization improve adversarial robustness of the CLIP models:
min E (x,o)
max |1
(13)
PMG (Wang et al., 2024) [OOD Robustness). Motivated by the inherent over-fitting of TeCoA (Mao et al., 2023) with generalization degradation, PMG (Wang et al., 2024) leveraged the prediction-level guidance from the vanilla pre-trained VLM with regularization of clean samples for the target model to conduct adversarial fine-tuning, as follows:
min (x..)~ max LeE (p) 18
(14)
+ A1 CKI
where LKL represents the Kullback-Leiblee divergence to align predictions, and Porig denotes the prediction of the vanilla pre-trained CLIP model (Radford et al., 2021). λι and A2 the corre- sponding loss weighting factors.
FARE (Schlarmann et al., 2024) [OOD Robustness]. To enhance the robustness generalization capability across diverse vision-language tasks, Schlarmann et al. (Schlarmann et al., 2024) pro- posed unsupervised adversarial fine-tuning approach, dubbed FARE, to adversarially optimize feature-level discrepancies in an unsupervised scheme:
F(x +0) [)][[]]
min E (x.c)DD
max [|
F Fori
(15)
where F(-) denotes the image encoder of the CLIP model for fine-tuning, while F Fori(()) is the image encoder of the vanilla pre-trained CLIP model as the frozen reference.
TGA (Yu et al., 2024) In-Distributtin Robustness). Built on TeCoA (Mao et al., 2023), Text- Guided Attention (TGA) adversarially fine-tunes the image encoder while aligning text- text-guided at- tention maps, computed by correlating per-patch visual tokens with frozen text embedding of the class prompt (from the original CLIP). It (i) pulls the target model's attention on adversarial images toward the original model's clean-image attention and (ii) keeps the target's clean-image attention close to the original, aiming to boost robustness with eroding marginal clean accuracy.
C FULL EXPERIMENTAL CONFIGURATIONS
This section details datasets used, task vector merging/evaluation settings, and downstream transfer protocols used in experiments and analyses with our PISTOLE method.
16

---

Published as a conference paper at ICLR 2026
C.I DATASET DESCRIPTION
Following prior work on robust VLMs (Mao et al., 2023; Schlarmann et al., 2024; Dong et al., 2025d;a), train on the ImageNet- Ik training split (Deng al., 2009) and report classification on the validation split (test labels are unavailable). Zero-shot classification is further evaluated on 13 datasets covering:
Natural objects: STL-100 (Coates al., 2011), CIFAR-10/100 (Krizhevsky et al., 2009), Caltech-101 (Fei-Fei et al., 2004).
Fine-grained: Stanford Cars (Krause et al., 2013), Oxford-IIT Pets (Parkhi et al., 2012), Flowers- (Nilsback & Zisserman, 2008), FGVC-Aireraft (Maji et al., 2013).
Textures: DTD (Cimpoi al., 2014).
Remote sensing: EuroSAT (Helber et al., 2019).
Medical: PCAM (Veeling al., 2018).
Robustness variants: ImageNet-R (Hendrycks al., 2021) and ImageNet-S (Wang et al., 2019).
We also assess zero-shot transfer on series of downstream image-text understanding tasks: image captioning on COCO (Lin al., 2014) and Flickr30k (Plummer al., 2015); Visual Question An- swering (VQA) on TextVQA (Singh et al., 2019). VQAv2 (Goyal et al., 2017), and VizWiz (Gurari al., 2018) 2018); object hallucination evaluation via POPE (Li et al., 2023); and multimodal Chain-of- Thought (CoT) reasoning on ScienceQA dataset (Lu et al., 2022).
C.2 IMPLEMENTATION DETAILS (ZERO-SHOT IMAGE CLASSIFICATION)
Default setup. Unless noted, we use CLIP (Radford et al., 2021) with ViT-L/14 image encoder. In line with standard task arithmetics (Ilaacco et al., 2023), all the task vectors are the parameter differences of the vision encoder between fine-tuned and pre-trained CLIP models. The natural VLM trained by Empirical Risk Minimization (ERM) on clean data, while the robust VL.M follows PMG (Wang et al., 2024). Adversarial examples are generated by 10-step PGD (Madry et al., 2018) at fo radius ε = 2/255 and step size a = 1/255. We form task vectors (Teat, Trob) and merge them via PISTOLE with default mixing factor À = 0.2 (Eq. (11)) and the mask sharpness factor k= 2 (Eq. (6)) Following adversarial weight training in single-modal architectures (Wu et al., 2020; adopt the adversarial parameter radius factor 1 backbone studies (Table 2). we consider both a large architecture CLIP ViT-H/14 and a lightweight one CLIP ViT-B/14. For fine-tuning the CLIP models, Hutter, 2019) (betas (0.9, 0.95)). cosine schedule with linear warm-up to x 10 10-5 over 2 epochs. For PEFT experiments (Table 4), we use the LoRA (Hu et al., 2022) scheme specifically on attention blocks. All experimental runs and empirical analyses use eight NVIDIA H100 GPUs.
Dong et al., 2024a),
10- (Eq. (8)). For
optimize with AdamW (Loshchilov &
Evaluation protocol. Following prior adversarial learning works (Mao et al., 2023; Schlarmann et al., 2024; Dong et al., 2024c:b). we report clean accuracy and robust accuracy under AutoAttack (AA) (Croce Hein, 2020) with 2=2/255 unless specified. Note that AA is an ensemble adversarial attack of diverse scenarios for practical reliability assessment, All the robustness evaluation results are based on adaptive attacks for fair comparison. Zero-shot classification follows standard CLIP inferencceeevaluation protocol: cosine similarity between image features and class-prome text features, selecting the nighest-scoring prediction index as the result.
C.3 DETAILS OF DOWNSTREAM TASK EXTENSIONS
For downstream vision-language task evaluations, we replace the ViT-L/14 vision encoder in LLaVA-L.5-7B (Liu et al., 2024) and OpenFlamingo-99 (Awadalla et al., 2023) with merged encoder with a better clean-robust trade-off. Note that all other components remain fixed. Below, provide detailed explanations about each vision-language task and its corresponding setup.
Downstream task extension to image captioning. We evaluate the CIDEr score (Vedantam et al., 2015) COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) using LLaVA-115570 and OpenFlamingo-991 with our merged vision encoder swapped in. Adversarial examples are generated with APGD (Croce& & Hein, 2020) under an f~o perturbation budget of < =2/ 2/255, running 100 steps
17

---

Published as a conference paper at ICLR 2026
per inage-reference pair following (Schlarmann et al., 2024), After each step, we recompute CIDEr and early-stop that example if the score falls below 10 (COCO) or (Flickr30k). For reporting, we apply the adversarial perturbation that achieved the lowest CIDEr score across references.
Downstream task extension to visual question answering. For evaluations, we report standard Visual Question Answering (VQA) accuracy (Antol et al., 2015), selecting the five most frequent answers among the ten annotations for each example. Adversarial inputs are crafted with APGD (Croce & Hein, 2020) at the radius of = 2/255, using targeted perturbations steered by prompt strings (e.g., MMybe'' Model decoding and text prompts follow each framework's 's default setup unless stated otherwise.
'Word' ) as in (Schlarmann & Hein, 2023; Schlarmann et al., 2024).
Downstream task extension to object hallucination (POPE evaluation benchmark). We assess object hallucination with POPE benchmark (Li al., 2023) by issuing binary (yes/no) queries about candidate objects under three standard sampling strategies: Random (uniform over absent objects), Popular (top-k most frequent absent objects), and Adversarial (top-k absent objects with highest co-occurrence with present objects). Following (Li et al., 2023), we report the Fl score and also average the score across sampling strategies for a single summary metric.
Downstream task extension to Science question answering with thain-of-thoughe We evaluate on ScienceQA (Lu et al., 2022) using I.L.VA-1.5-7B (Liu et al., 2024) with our PiSTOLFE-meree VIT-L/14 vision encoder swapped in and all other weights frozen. Each instance provides image, a question, and four options (A, B, C, We adopt a chain-of-thought protocol: the model first prompted to "think step by step" to produce free-form rationale, then a short follow-up prompt elicits a single final option token. Inputs follow LLaVA defaults (single image and single-turn dia- logue). Decoding uses the temperature T {0.0, 0.1, 0.2). Note that the last explicit token among A/B/C/D is taken as the prediction, with fallback to option-text matching if needed. We report accuracy at each T and the mean across temperatures (Table 7).
D FULL THEORETICAL ANALYSES
D.1 PROOF OF THROREM 1
In this section, we provide the complete proof of Theorem 1, establishing first-order attenuation of cross-objective interference under layerwise complementary masking and its monotone dependence on the mask hyperparameters.
Theorem (Theorem 1 from the main text). Let Snat = VoCmu(()) l= and glob ==
VoLmb (0) Let More == (1 11-g-m _ma)" and M == (1- be the uncapped comple- mentary masks from Eq. (6). with 2 1 and Es 10, defined by Eq. (5). For each layer I, let wwb be the q-quantile of (Mar)): and (.) the q-quantile of (Mer)): (empirical quantiles 'ny
on layer entries)? Define the capped masks layerwise by (Mnai): min ( (MMMa) egg 1) and (Mot)) (ainnMM (o taal)) Set Pmar == tius and Prob := maxx q..1 Then for any E Rd.
tmb
(g Mjab ② 8)1 S Paut ll ||01|2.
(g M 3 8)
S Prob 11g.% l2 110112- (16)
Moreover, if is increased (i.e., sharpening - (1- or any of the layerwise caps ggt yob are decreased, the right-hana sides in Eq. (16) monotone nonincreasing.
Proof. We here prove the first bound, while the second is identical with roles swapped.
Layerwise lo control of the capped mask. According to the definition of the capping, for every layer I and every index in that layer's 's index set IT.
(M,b)) < yaat
Consequently, if we write m := Mjeb and use the layer partition { then
| = max |m.] = max max m, S max a Paat-
(17)
EEI,
(
2 Any standard definition of the empirical q-quantile with g (0. 1] suffices. We here consider that quantiles monotone under component-wee decreases.
18

---

Published as a conference paper at ICLR 2026
Bounding the masked displacement in lz. For any vector 8,
m
81
Σ m of I
|
Σ
8 |m|
|
Taking square roots and invoking Eq. (17) yields
M
c 6||2 I || ||1|| S Pnut ||0||2.
(18)
First-order inner-product control. By Cauchy-Schwarz,
(gnat: Maa C 5) - || llz Mab @ 8112.
Combining with Eq. (18) gives
(g MoO C 8)
I Phat || l1z |6
which is the first inequality in Eq. (16).
Monotonicity in x and caps. Consider the pointwise map dx = (1 (1-u)* on u E [0, 1]. Since + 6x (u) is nonincreasing for every fixed [o, 1], increasing makes the pre-cap masks M and nut component-wis wise no larger. Quantiles are monotone under component-wise decreases: if a S b element-wise then the empirical q-quantile of is that of b. Therefore n.d and aod are nonincreasing in K. The capping operation (v + min(v, f1)) is also monotone nonexpansive (it cannot increase any coordinate). Thus both masks after capping nonincreasing in K, and Prot max; th and Prob = max; 1. Finally, explicitly decreasing any aq or g further shrinks the corresponding mask entries and hence cannot increase the right-hand sides in Eq. (16).
D.2 PROOF OF THEOREM 2
Theorem 4 ([Theorem 2 from the main text). J Let PCI be defined as in Definition 1, where (x 0) > 0 is twice continuously differentiable in a neighborhood of 0. Given H.(0) == (x, for sufficiently small 17, we have the following approximation:
PCI(x, 0) o² Tr(H. O
(19)
(x:
Proof. Throughout the proof, we take all the expectations with respect to the isotropic and zero- mean perturbation A E Ve at the parameter level introduced in Definition 1. The covariance of A is IT, where the per-coordinate second moment is の² = AE V |||A]
Given the mapping 0 + (x 0) is C2 by definition. The second-order multivariate Taylor's expansion around 0 with Lagrange form remainder yields:
(x 0 A -x; 0)
Vop. (x; A
(a)
(20)
(O) +0(|
Recall that isotropy and zero mean imply E[A] 0. Thus, the first-order term (a) vanishes un- der expectation |E] Vo (x; 0 A] 0. For the quadratic term (b), we apply ([ATH.(A)A) =
( AA = HH. ()) for isotropic covariance. Hence, we obtain:
(x 0 A) (x: 0) 2 +
(21)
Taylor's theorem bounds the truncation emror in Eq. (20) by O(1 A 11). Under the isotropic law in Vo. we have EE| A = do², so Hölder's inequality with exponents (3/2, 3) gives:
E
3/2 3/2 (do2)3/2 3/2 03
(22)
Formally, for any t, the empirical CDFs satisfy Fa(t) 2 Fb(t), hence inf(t Falt) 2 9) inf(t: Fb(() q).
19

---

Published as a conference paper at ICLR 2026
Algorithm PredIction STability-aware mOdeL mErging (PISTOLE)
Input: natural robust CLIP models (fe.. fe,a) and their vectors (Tma, Treb); dataset D {(x, J input-PGID steps mn and step size a; paramcier-rajjettoyy steps K and step size 8; parameter-hall radius factor mask temperature and sharpness K; small >>> O; trade-off X; quantile cap
1: Initialize accumulated gradients: Gmz + 0, Grob +
2: while not at the end of task vector merging do
Sample (x, c) DD and x +x x 0.001 NN 1)
4: for 1,... 1,...,77 17 do
PGD to obtain adversarial input for robust branch (x(-1)), e(c)))
5:
$(0) < (b-1)) + sign V git-1) Les (po. $(1 (Bxx
6:
end for
8:
Set & 4- ( Xaal +- x, Nach +
9:
for E (nat, rob) do
Adversarial parameter trajectories
10:
0.0h 0.; Va, t (A
| I m
11:
for 0 K 1 do
12:
g - VoL. (po e(c)) 0=
13:
g/ <|(||/] +))
14:
(0." + u) 0 I1o. vo. (0.)
15:
G. G. PCI(xx, 0 -g
16:
end for
17:
end for
18:
end while
Per-layer normalization and path-refined masks:
goach + Normo G.I)" for E (nat,rob): M (1 S10b go MMa (1 goalll
20: quantile cap g is specified then
Apply per-layer caps: M + min (Maah rob
21:
M + min (Maah rub.e:
mu (.
rob.!
22:
end if
Re-weight task vectors and merge:
23:
Tin + M @ Thas T + puth
Tru. T" + X Tin + (1 A) Trb
24: return merged vector T' and parameters OPSTTOL Ou T
Hence, the expectation of the remainder term is O(o²), validating the order claimed in Eq. (21).
Subtracting (x(x; 0) from Eq. (21) and dividing by Po(x; O) 0 (keeping terms up to O(o²))
yields:
Pe (xxx++)) - 0) 2² Tr (H
(x +0()3).
(23)
(x(x;
For sufficiently small o², the leading term dictates the sign, and the outer absolute value in the definition of our proposed PCI (Definition 1) therefore keeps the magnitude and removes the sign. Consequently, obtain:
PCI(x, c, 0)= o² Tr (o) +((0))
(24)
(x 0)
establishing the quadratic estimation Eq. (19).
E PISTOLE: FULL ALGORITHMIC SPECIFICATION
This section instantiates the procedure described in the main text (ef. Sections 3.3 and 3.4). The rou- tine first constructs robust inputs for the robust branch via the PGD adversary generation scheme, then traces short adversarial parameter trajectories around each fine-tuned solution. Along these
20

---

Published as a conference paper at ICLR 2026
Clean Captioning
Adversarial Captioning
CLIP: man is skateboarding in par king lot.
CLIP: man riding motorcycle the road.
Adv. Image
Clean Image
TeCoA: man riding a surfboard on the beach.
2/255
TeCoA: person sitting desk using loptop.
PMG: man skatebaarding on the strest
PMG: boy riding a skateboard on the street while holding o dog.
car.
FARE: man skateboards with a car behind.
FARE: Skater rolls through traffic as cars approach from behind.
person snowboarding on snowy slope.
A group people sit sitting around table in office.
PISTOLE: A man is skateboarding parking lot.
PISTOLE: man is skateboarding parking lot.
Clean Captioning
Adversarial Captioning
CLIP: bunch of skis are lined up
CLIP: parking lot filled with cars sun.
rack.
Adv. Image 2/255
Clean Image
TeCoA: pile of skis and ski poles are leaning against a metal pole.
TeCoA: A pile of bicycles are
leaning against a metal fence.
Many pairs of poles are leaning against rock.
group of people walking through snowy village street.
FARE: pile of skis and ski pales leaning against wall.
FARE: Several snowboords are stacked together the ground.
TGA: A rack of fishing rods near
TGA: A row of sur fboards stonding
lake.
upright on sondy beach.
PISTOLE: bunch skis arranged neatly in racks outdoors.
PISTOLE: Skis and snowboards organized neatlyy in racks.
Figure 6: Image captioning under clean and adversarial inputs (Eoc -norm perturbation c 2/255) using LLaVA-1.5 coupled with vision encoders from the compared methods COCO. PISTOLE maintains semantic consistency across perturbations, whereas alternatives often drift or hallucinate.
Question: What being edvertised?
Question: What being advertisedf
Image
VQA
Image 2/255
CLIP: Tecnage mutont sinja turties
S
None
TeCoA: Nons
PMG: Teenage mufont ninjo turtles
Turtles
FARE: Techege matont nitja turtless
FARE: Tarkeys
TGA: None
TGA: None
PISTOLE: Teenage mutont ninje turtles
PISTOLE: Teenage mutont ninje turtles
Question: What's 's the title of the book to the very top right?
Question: What's the titte of the book to the very Top right?
Clean Image
Clean VQA
Adv. Image = 2/255
Adversarial VQJ
CLIP: Frezen stor
CLIP: None
TeCoA: Shopaholic
TeCoA: Shopaholic
PMG: Frozen stor
PMG: Frezen
FARE: Sloeping where they fol
FARE: Sleeping where they fol
TGA: Sleeping
TGA: None
PISTOLE:
PISTOLE
Figure 7: Qualitative visual question answering under clean samples and their adversarial counter- parts (e (c=2) 2/ 2/255) TextVQA, using LLaVA-1.5 with different robust vision encoders. PISTOLE preserves correct answers across both conditions, whereas baselines often drift or abstain.
trajectories, it accumulates PCI-weighted gradients, converts them to per-layer normalized sensitiv- ity scores, and forms complementary, path-refined masks. Finally, it re-weights the natural/robust task vectors and merges them with mixing coefficient X.
In practice, the method is training- free and efficient: small step counts suffice; e stabilizes normal- ization; (y, x) tune mask dynamic range and selectivity; and the quantile cap g enforces a first-order non-interference budget. Unless otherwise stated, we apply our PISTOLE method to the vision encoder parameters, but the specification in Algorithm is architecture-agnstiic.
21

---

Published as a conference paper at ICLR 2026
Question: Is there car the image?
Question: Is there a dining toble in the image?
Image
Hallucination VQA
Image
POPE Hallucination VQA
TeCoA: Yes
TeCoA: Yes
FARE: No
FARE: No
TGA: Yes
TGA: Yes
No
PISTOLE: No
Question: Is there person the image?
Question: Is there bus in the image?
Image
Hallucination VQA
Image
POPE Hallucination VQA
TeCoA: No
TeCoA: No
FARE: No
FARE: No
TGA: No
TGA No
Yes
PISTOLE: Yes
Visual examples POPE hallucination benchmark using LLaVA 1.5 with vision coders from different adversarial VLM learning schemes. PISTOLE reduces yes/no hallucinations and maintains pixel-grounded responses under perturbation.
Figure 8:
Input Images
Science QA
TeCoA: 8. November
Question: Which month has the lowest average precipitation in London?
July
July
FARE: C. February
November February
November
PISTOLE July
TeCoA: Bilberry
Question: Which of these orgatisms contains matter that was once part of the sedge?
PMG: A. Arctic fox
A. Arctic fax Barret-ground caribou Bilerry
TGA: 8, Barnen Barren-ground caribou
Arctic fox
Plaret
Vokare Orn't
Parmar conpeetinn
Question: following statement about our solor system true or false?
618 20
True
Wenus
9.29 10
PMG: A True
volume of Mors than ten times as large us Mercury's.
True
20
TGA: True
True
PISTOLE False
Necture
Figure 9: Qualitative examples for science question answering w/ CoT using LLaVA-1.5 with dif- ferent robust vision encoders. Our PISTOLE method maintains evidence-consisten answers under perturbations, while alternatives often drift toward prior-biased choices.
F VISUALIZATIONS OF ZERO-SHOT TRANSFER TO DOWNSTREAM TASKS
In this section, we present qualitative comparisons across adversarial learning baselines and our task vector merging method (PISTOLE) for zero-shot transfer across diverse downstream vision- language tasks, e.g., captioning and visual question answering, under clean inputs and adversaries.
Zero-shot transfer to image captioning. Figure shows that equipping LLaVA-1.5 with PIS- TOLE's merged encoder yields captions that remain semantically stable from clean to adversarial images (f.o- -nom perturbation 2/22255. Competing encoders frequently drift across domains or
22

---

Published as a conference paper at ICLR 2026
Clean Ancuracy (N0
Robust Accuracy (%)
Accuracy (5g
Olean Accuracy (%)
Clean Accuracy
68.0
48.0
20.0
45.0
67.6
47.0
47.3
Merging Weight
Sharpness
Merging Weight
(a) (Avg.)
(b) (Avg.)
(c) (Avg.)
Figure 10: Hyper- parameter (merge weight X, mask sharpness factor K, and parameter-trajectory radius scaling factor 1j) sensitivity analyses of PISTOLE.
hallucinate objects/atrributes under perturbation, while PISTOLE preserves object grounding and scene ciananticss-videeen of stronger invariance input attacks and distribution shift.
Zero-shot transfer to visual question answering, Figure illustrates two TexIVQA cases in which PISTOLE maintains answer consistency from clean adversarial inputs (e.g., "Teenage Mutant Ninja Turtles" and "Frozen star"), while competing encoders frequently switch to incorrect strings or "None". These qualitative trends align with our quantitative gains associated with higher sum of clean and robust VQA accuracy shown in the main text.
Zero-shot transfer POPE object hallucination. We here illustrate POPE object hallucination cases across its question-samplinng protocols in Figure 8. The baseline encoders frequently follow language priors, thus answering "Yes" for common objects and "No" for unlikely ones, regardless of image evidence. In contrast, our PISTOLE method aligns decisions with the pixels, lowering both false positives (hallucinated objects) and false negatives (missed objects), and preserving consistent judgments between clean and adversarial inputs.
Zero-shot transfer to science question answering w/ CoT. Figure 9 contrasts predictions on rep- resentative ScienceQA items. Under adversarial perturbations, prior robust CLIP variants frequently select high-prior but visually unsupported options, whereas PISTOLE remains aligned with the chart/diagram context and the underlying facts, yielding stable, sidence-ggoundee answers. These cases suggest that stabilizing the vision backbone with PISTOLE curbs brittle pattern pattern-matchin matching and supports more reliable multi-step reasoning.
G EXTENDED ANALYSES OF PISTOLE
G.1 EFFECT OF TASK VECTOR RE- WEIGHTING
We compare three re-weighting strategies for task vector merging: i) no mask (uniform ad- dition), (ii) a random mask matching the lay- erwise sparsity, and (iii) inddient-infommed sta- bility masks in Eq. (6) Table 11. Uniform ad- dition yields the weakest trade-off, reproducing the near-linear clean-robust antagonism. Random masking offers small gains by incidentally prun- ing conflicts but lacks guarantees. contrast, our sradient-infommed strategy consistently achieves the best trade-off, bending the frontier toward interior optima by suppressing counterpart-sensstive coordinates. This aligns with Theorem and Corollary which proves that complementary masks contract cross-objective first-order interference, unlike other schemes.
Table 11: Avg. performance (%) of our PISTOLE with diverse task vector re-weighting schemes.
Re- Re-Weighting Scheme
Clean
Robust
Sum
No Mask
66.57
44.54
111.11
Random Mask
67.32
44.95
111.27
Gradient-Informer Mask
69.24
47.65
116.89
G.2 HYPERPARAMETER SENSITIVITY ANALYSES.
In this section, we study the effect of three core hyperparameters in our PISTOLE, the merge weight (Eq. (11)), the mask sharpness factor (Eq. (6)), and the (Eq. (8)), while holding all other settings fixed. Figures 10 plot both the average clean and robust accuracy across the 14 evaluation datasets.
radius scaling factor
23

---

Published as a conference paper at ICLR 2026
Merge weight A. Sweeping À from 0.1 to 0.3 reveals the expected trade-off: clean accuracy in- creases with larger weight on the natural task vector, while robustness decreases roughly monoton- ically. The frontier is bowed (not linear), yielding
interior optimum of the Clean+Robust Sum near À = 0.2, which we adopt as default. Trends are stable across seeds, indicating that À primarily operating point rather than introducing instability.
sets the eecuracy-robustren
Mask sharpness factor s. Sharpening the complementary masks by increasing K. (Éq. (6)) improves clean accuracy and raises robust accuracy up to an interior peak, after which robustness plateaus or dips slightly. The trade-off is maximized near K = 1.5, which we use by default to attenuate cunnteraart-snsiiiii coordinates without over-pruning
radius scaling factor We can observe that controls the neighborhood explored by the adversarial parameter trajectory: too small under-explores the loss geometry, while too large drifts off the manifold. As shown in the attached sweep, both clean and robust accuracy peak a moderate radius n= ==1x 10-3 It can also be seen that smaller parameter-level perturbation radii yield limited gains, and larger ones degrade the trade-off.
G.3 COMPUTATIONAL COST COMPARISONS
Training-free merging. PISTOLE
Table 12:
Computational cost (training time) comparison
operates on off-the-shelf naturally
between PISTOLE and other adversarial learning methods.
and adversarially fine-tuned VLMS
Method
Clean
Robust
Sum
Time
and performs a one-shot merge: we estimate stability masks from a small calibration split (no epochs of weight updates), then apply element-wise re- weighting to the two task vectors and compose the final encoder. In contrast, prior adversarial VLM learning approaches run full opti-
TeCoA
61.56
43.26
104.82
6.2 hours
PMG
64.46
45.74
110.20
8.6 hours
FARE
65.50
42.97
108.47
7.6 hours
TGA
62.
45.19
107.30
8.0 hours
PISTOLE
69.24
47.65
116.89
0.8 hours
mization loops with inner PGD steps and (often) model forward propagation, incurring substantial GPU time. To make costs comparable across methods, we count gradient evaluations and auxil- iary forwards. As shown in Table 12, we can observe that our PISTOLE method attains the best clean-robust trade-off while being ~8-11x faster than prior adversarial fine-tuning baselines.
Complexity analysis. Let No be the size of the (small) calibration split, m the number of PGD steps for adversarial inputs, and K the number of parameter-trajecton steps. PISTOLE does: (i) m input-gradient evaluations to generate adversarial inputs for the robust branch (same inner loop as standard adversarial training), and (ii) for each branch, parameter-gradies evaluations along the adversarial parameter trajectory, each weighted by PCI. Thus, the total number of gradient evalu- ations to estimate the masks is O(Ne(m + 2K)). run once over a calibration split with no weight updates and multi-epoch optimization loop. In contrast, adversarial fine-tuning over E epochs on the full training set of size N has complexity O(EN((m + 1)), since each iteration both runs PGD and performs parameter update. In practice, E is large (multiple epochs), while K is small constant, and N₂ << N, so the overall cost of PISTOLE is substantially lower even though we do multiple forward-backward passes during calibration.
G.4 COMPARISON WITH STANDARD TASK VECTOR MERGING METHODS
Recall from Section 2 that Ties-Merging (Yadav et al., 2023) enforces sign- consistent sparsification and AdaMerging (Yang et al., 2024) learns per-parameter weights for multi-task settings. Table 13 compares our PISTOLE with these stan- dard parameter-space merging methods that combine the same naturally and ro- bustly fine-tuned CLIP models in the iden- tical configuration. proving the Across 14 datasets, method attains the best clean and robust accuracy, im- trade-off over naive task vest-r addition, Ties-Merging and AdaMerging. We attribute these gains to modeling parameter-space sensitivity and local loss geometry: PISTOLE
Table 13: Average performance (%) over 14 datasets for merging the same natural and robust CLIP models under the identical configuration.
Task Vector Merging
Clean
Robust
Sum
Vanilla Merging
66.57
44.54
111.11
Ties-Merging
67.91
46.27
114.18
AdaMerging 68.23
46.52
114.75
PISTOLE 69.24
47.65
116.89
24

---

Published as a conference paper at ICLR 2026
Glan Image
Notural Kroneladgg Sourin
Adi: Image c 2/202
Adversatal Koowelagge Source
Adv: 22255
PISTOLE (Dur))
Quedaee What Her tow??
OD mrdda th tee? Anawet
What Faee?
Asswer:2:3
Figure 11: On an adversarially perturbed image, both the robust source model and PISTOLE output a wrong answer, despite the natural model giving a partially correct answer on the clean input, illustrating failures that merging cannot fix.
uses gradient-informed complementary masks and refines them along adversarial parameter trajec- tories account for sensitivity and curvature during merging perging-factorrs overlooked by prior methods.
G.5 FURTHER EXPLANATION OF THE "TRAINING-FRER ASSUMPTION
Our intent is emphasize that PISTOLE performs additional gradient-base based optimization for each downstream task once such source models are available. First, this assumption aligned with common practice in the task-vector/modellediing literature Ilharco et al. (2023); Ortiz-Jimenez et al. (2023), where methods start from already fine-tuned checkpoints and apply post-hoc parameter- space operations. In the VLM ecosystem, high-quality natural and robust checkpoints (e.g., CLIP- /OpenCLIP-styis models and their adversarially trained/fine-tunee counterparts) are increasingly released and reused as off-the-shelf backbones. PISTOLE is designed precisely for this regime: given pre-existing natural and adversarial models, we can cheaply obtain continuum of merged models with improved clean-robust trade-offss without any further training.
Second, even when pair of natural/robust models must be trained once, this one-time cost is amor- tized over many downstream tasks/domains. In contrast, standard adversarial fine-tuning typically re-optimizes the model for each new target task. While our PISTOLE exhibits generalizable robust- ness across diverse downstream Aision-language tasks.
G.6 ANALYSIS ON FAILURE CASES
Figure 11 illustrates typical failure mode that PISTOLE cannot fix. Our method operates by inter- polating and masking between a natural and an adversarially trained model in parameter space, so can only reshuffle how much each endpoint contributes. When both source models systematically fail on certain patterns (e.g., rare classes or heavily shifted domains), the merged model likewise produces wrong answers and may even accumulate errors, as seen in the adversarial example where both the robust source model and PISTOLE are incorrect despite the natural model being partially correct.
G.7 BROADER PARAMETER SPACE CONTEXT (MORE DISCUSSION)
Beyond classical model soups and linear task-vector addition, several recent VLM-specific meth- ods also operate directly in parameter space. WATT Osowiechi et al. (2024) adapts CLIP under domain shift via test-time updates followed by weight averaging of the adapted parameters, im- proving test-time robustness. GeoLangBind Xiong et al. (2025) trains remote-sensing VLM and uses progressive multimodal merging strategy to aggregate knowledge from multiple visual backbones within a single VLM. MoTE Zhu et al. (2024) adds temporal experts top of VLM and employs weight-merging regularization in parameter space to enhance the trade-off. Our work is complementary: instead of merging across domains, backbones, or experts, PISTOLE performs stability awarr merging between natural and adversarially trained vision encoders to reconcile clean accuracy and robustness.
H LIMITATIONS AND BROADER IMPACT.
H.1 BROADER IMPACT
PISTOLE targets a central safety concern in foundation VLMs: robustness to adversarial perturba- tions. By reconciling clean accuracy and robustness through training-free, plug-and-play merge
25

---

Published as a conference paper at ICLR 2026
of off-the-shelf natural and robust task vectors, our approach make multimodal systems liable in downstream applications (captioning. VQA, hallucination mitigation, and scientific QA). The method is compute-efficien requiring only short calibration pass and no weight updates, thereby lowering the environmental and financial cost of robustness compared to full adversarial fine-tuning. Because PISTOLE composes existing checkpoints rather than collecting also cases reproducibility and facilitates community vetting. At the same time, improved ro- bustness should be paired with standard safeguards (bias audits, red-teaming, and attack-aware evaluation) to ensure equitable performance across subpopulations and responsible deployment.
data, it
H.2 LIMITATIONS
While PISTOLE shows strong empirical gains and formal guarantees, several limitations remain. We note them alongside why their impact is limited or how partially mitigate them.
Dependence on off-the-shelf task vectors. PISTOLE assumes access to natural and robust fine-tuned VLMs to form task vectors. In practice, this minor constraint: high-quality CLIP- family checkpoints (natural and adversarial) are widely available in open-source repositories, and our method is agnostic the specific recipe used to produce them. Moreover, Section 4.3 (Tables 9& 10) shows robustness to the choice of source models, and our masks provably attenuate cross- objective interference (Theorem 1).
Inheritance of upstream biases. Merging cannot remove biases present in the component models and may propagate spurious correlations. Our gradient-informed masks down-weight counterpart-senntee sensiiiee (often brittle) coordinates, which empirically reduces hallucination and im- proves grounding, but it does not replace fairness auditing. We report Doject-halllcciatiin and reasoning improvements, while broader bias assessments are a valuable direction for future work.
Scope of architectures and tasks. Most experiments use CLIP-like encoders and open- vocabulary classfication/cappioonnggg tasks, coverage is not exhaustive (e.g., video, speech-vision). The merge is model-agnostic and only requires gradients for calibration, and our curvature results (Figure 5, Theorem 2) suggest applicability beyond the tested settings/application..
Although we show transfer across backbones and
I LLM USAGE DECLARATION/DISCLOSUNN
We used Large Language Model (LLM) (e.g., ChatGPT-5) solely for polishing: grammar, wording, and LaTeX phrasing. The LLM did not generate ideas, methods, experiments, analyses, or results. All technical content and claims were authored and verified by us. Outputs were reviewed and edited by the authors, and all citations/equations were checked manually. No proprietary or non-public data is provided to the model.
26


Reviews:
[{"review_id": "vdAdCBIHTn", "reviewer": "Reviewer_eK3j", "summary": "This paper proposes PISTOLE, a novel prediction stability-aware model merging framework designed to reconcile the trade-off between clean accuracy and adversarial robustness in vision-language models (VLMs). Instead of retraining, the method merges task vectors from naturally and adversarially fine-tuned models using gradient-informed masks and adversarial parameter trajectories. The approach leverages prediction sensitivity and curvature estimates to selectively retain stable parameters and suppress conflicting ones. Theoretical analysis and extensive experiments demonstrate improved clean-robust trade-offs and transferability to downstream tasks.", "questions": "Clarify in the introduction whether the merging operates solely on the vision encoder or also on the text encoder.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The idea of merging task vectors from conflicting objectives (natural vs. robust) is novel and well-motivated.\n\n2. The paper provides clear mathematical reasoning (Theorem 1–2) showing that the complementary masks contract cross-objective interference and PCI correlates with curvature — adding interpretability and credibility.\n\n3. Extensive experiments across datasets, backbones (ViT-B/L/H), and downstream tasks (captioning, VQA, hallucination, reasoning) demonstrate consistent improvement and generality.", "weaknesses": "1. The role of PCI versus GISM (gradient-informed stability mask) feels somewhat overlapping; clarifying their unique contributions would help.\n\n2. Although the method claims to avoid retraining, the process of computing path-integrated gradients and PCI-based weighting still requires multiple forward–backward passes. A brief complexity analysis would strengthen the practicality argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oxNp0nLSOG", "reviewer": "Reviewer_QXNC", "summary": "This paper proposes PISTOLE, a training-free way to merge a naturally fine-tuned CLIP and an adversarially fine-tuned CLIP by masking task vectors with stability cues. Instead of adding task vectors uniformly (which yields a nearly linear clean vs. robust balance), the proposed method builds complementary masks so each branch down-weights coordinates that the other loss wants to change. Furthermore, the introduced method refines those masks by accumulating gradients along adversarial weight trajectories, where each step is weighted to capture high-curvature regions. The theory proves first-order interference contraction and connects PCI to the Hessian trace. Experiments on 14 datasets plus multiple backbones and PEFT show improved performance and better downstream transfer (captioning, VQA, hallucination, and CoT).", "questions": "1. Does the text-side parameter merging benefit from the proposed merging method? The authors can briefly validate whether applying PISTOLE to the text encoder yields additional gains or harms alignment. \n2. When referencing Eq. (6) later, can you briefly remind readers what \\kappa controls (sharpening vs. sparsity)?\n3. Is mask capping (q-quantile) applied globally or per-layer? More details should be provided.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and organized. The figures are illustrative and intuitive.\n2. The proposed idea is well-motivated by the systematic analyses of the linear trade-off between clean and robust accuracy.\n3. The theoretical analysis in this paper provides additional justification of the method's effectiveness. For example, it links the prediction criticality index to the model's curvature.\n4. Component analyses and comparisons with prior approaches prove the efficacy of PISTOLE. This method can also be transferred to diverse downstream tasks in a plug-and-play way.", "weaknesses": "1. It appears that the per-layer normalization and \\kappa sharpening are important. Although the paper has provided sensitivity analyses, additional evidence that the method is stable across diverse scenarios should be provided.\n2. Most experiments focus on the vision encoder. It would be better to discuss further regarding the task vector for the text encoder to give more insights for future researchers.\n3. The discussion regarding model interpolation and soups beyond task vectors is missing. They are also closely related to this paper.\n4. (Minor) The figure readability needs to be improved. For example, I suggest increasing font sizes in figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2LKUKY8YRG", "reviewer": "Reviewer_VcKu", "summary": "This paper introduces PISTOLE (Prediction STability-aware mOdeL mErging), a training-free framework designed to reconcile the long-standing accuracy–robustness trade-off in CLIP.\nInstead of costly adversarial fine-tuning, PISTOLE merges two off-the-shelf models—one naturally fine-tuned and one adversarially fine-tuned—directly in parameter space using task vectors.\n\nTo avoid the near-linear degradation observed in naïve merging, the authors propose gradient-informed complementary masks and a prediction criticality index (PCI) that approximates curvature and identifies stable parameters for selective fusion.\nTheoretically, PISTOLE is shown to reduce the conflict between natural and robust objectives and guide the merged model toward flatter, more stable parameter regions, thereby improving both accuracy and robustness; empirically, it achieves superior clean–robust trade-offs and transfers effectively to downstream tasks such as captioning, VQA, hallucination detection, and reasoning.", "questions": "In your work, PISTOLE assumes that task vectors can be linearly combined in parameter space \n(e.g., $\\boldsymbol{\\tau}_{\\text{add}} = \\sum_i \\boldsymbol{\\tau}_i$). \nHowever, in multimodal encoders, task-specific update directions may not be linearly independent, \nand direct addition could introduce interference due to the nonlinearity of the representation space.  \nCould you clarify on that?\n\nFigure 5 shows that PISTOLE reduces loss curvature during model merging.  \nCould you clarify how this empirical result quantitatively supports Theorem1 and Theorem2?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.**Motivation is Clear:** The paper is well-motivated, shifting the focus from model retraining to direct parameter merging, which effectively leverages pretrained foundation models to achieve robustness–accuracy balance with minimal cost.\n\n2.**Method is reasonable:** The approach is conceptually sound, integrating gradient-informed and curvature-aware weighting into the model merging framework, and theoretically demonstrating that (i) complementary masks provably contract cross-objective interference, and (ii) PCI quantitatively approximates curvature through the Hessian trace.\n\n3.**Experiment proves the claim:** Experiments are comprehensive, covering zero-shot classification, captioning, VQA, hallucination, and CoT reasoning, and consistently showing improvements across both clean and robust performance metrics.", "weaknesses": "1. PISTOLE assumes access to both naturally and adversarially fine-tuned models, which are themselves costly to obtain. Thus, the “training-free” claim is conditional—effective only once such models are available—and may not reduce the overall computational cost in practice.\n\n2. The paper lacks analysis on failure cases. While quantitative results are strong, does this method works all the time with all the cases? It remains unclear under what conditions the proposed method may fail or underperform.\n\n3. The experimental section could be improved by including more comparisons and discussions with similar task-vector-based methods to better contextualize the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5x1ThN3kGj", "reviewer": "Reviewer_uKzG", "summary": "The paper investigates a training-free way to merge a naturally fine-tuned CLIP and a robustly (adversarially) fine-tuned CLIP using gradient-informed and prediction-stability masks over task vectors. Instead of uniformly adding task vectors (which they show yields an almost linear clean-vs-robust trade-off), the authors (1) estimate per-parameter stability from gradients of the *other* objective, (2) cap unstable coordinates, and (3) refine these masks along *adversarial parameter trajectories* weighted by a Prediction Criticality Index (PCI) that tracks curvature (Hessian trace). Empirically, PISTOLE bends the frontier and improves the sum of clean+robust performance across 14 datasets. Theoretically, the authors show the masks contract cross-objective first-order interference.", "questions": "1. Could the authors briefly clarify the meaning of kappa in Eq. (6)? It would help if the notation were reintroduced when referenced later in the paper.\n2. The authors introduce the term Prediction Criticality Index (PCI) in Section 3.4, but the intuition only becomes clear later. Could you add a short intuitive explanation when it’s first mentioned?\n3. Sections 3.3 and 3.4 are mathematically dense. If possible, could you add a brief ‘takeaway’ sentence at the end of each subsection summarizing the main intuition?”", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides an interesting and clear problem formulation and empirical evidence, where vanilla task vector merging fails to improve the trade-off, yet the proposed method effectively addresses this issue.\n2. The paper shows clear and logical writing along with a novel methodology.\n3. To the best of my knowledge, this is the first paper to address the accuracy-robustness trade-off without costly adversarial fine-tuning, generalizing across tasks.\n4. The theoretical justification matches intuition: masks contract first-order interference, while PCI tracks curvature.\n5. The strong ablations and analyses demonstrate the SOTA performance of the proposed method associated with improved efficiency.", "weaknesses": "1. The proposed method requires access to both natural and robust fine-tuned models. Does it mean that the quality also depends on them?\n2. The “Prediction Criticality Index” is well-motivated mathematically, but the intuition appears late (after Eq. 10). An earlier conceptual explanation could help readers.\n3. Recent VLM robustness techniques (e.g., prompt ensembling, token-level defenses) should be discussed in the Related Works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)