MergeTune: Continued Fine-Tuning of Vision-Language Models
Authors: Wenqing Wang
Da Li
Xiatian Zhu
Josef Kittler
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Continue Learning
Parameter-Efficient Fine-Tuning
Robust Fine-Tuning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Fine-tuning vision-language models (VLMs) such as CLIP often leads to catastrophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adaptation; however, forgetting often remains inevitable during this process. We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted. We propose a simple, model-agnostic CFT strategy (named MergeTune) guided by linear mode connectivity (LMC), which can be applied post hoc to existing fine-tuned models without requiring architectural changes. Given a fine-tuned model, we continue fine-tuning its trainable parameters (e.g., soft prompts or linear heads) to search for a continued model which has two low-loss paths to the zero-shot (e.g., CLIP) and the fine-tuned (e.g., CoOp) solutions. By exploiting the geometry of the loss landscape, the continued model implicitly merges the two solutions, restoring pretrained knowledge lost in the fine-tuned counterpart. A challenge is that the vanilla LMC constraint requires data replay from the pretraining task. We approximate this constraint for the zero-shot model via a second-order surrogate, eliminating the need for large-scale data replay. Experiments show that MergeTune improves the harmonic mean of CoOp by +5.6\% on base-novel generalisation without adding parameters. 
On robust fine-tuning evaluations, the LMC-merged model from MergeTune surpasses ensemble baselines with lower inference cost, achieving further gains and state-of-the-art results when ensembled with the zero-shot model.

Published as a conference paper at ICLR 2026
MERGETUNE: CONTINUED FINE-TUNING OF VISION-LANGUAGE MODELS
Wenqing Wang',, Da Liatinn
Xiatian Zhu't,
Josef Kittler
University of Surrey
2 Samsung AI Centre Cambridge
3 Queen Mary University of London
ABSTRACT
Fine-tuning vision-language models (VLMs) such as CLIP often leads to catas- trophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adaptation; however, forgetting often remains inevitable during this process. We introduce novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted. We propose simple, model-agnostic CFT strategy (named MERGE- TUNE) guided by linear mode connectivity (LMC), which can be applied post hoc to existing fine-tuned models without requiring architectural changes. Given a fine-tuned model, we continue fine-tuning its trainable parameters (e.g., soft prompts or linear heads) to search for a continued model which has two low- loss paths to the zero-shot (e.g.. CLIP) and the fine-tuned (e.g.. CoOp) solutions. By exploiting the geometry of the loss landscape, the continued model implicitly merges the two solutions, restoring pretrained knowledge lost in the fine-tuned counterpart. A challenge is that the vanilla LMC constraint requires data replay from the pretraining task. We approximate this constraint for the zero-shot model via a second-order surrogate, eliminating the need for large-scale data replay. Experiments show that MERGETUNE improves the harmonic mean of CoOp by +5.6% on base-novel generalisation without adding parameters. On robust fine- tuning evaluations. the merged model from MERGETURE TUNE surpasses ensem- ble basclines with lower inference cost, achieving further gains and state-of-the-ar results when ensembled with the zero-shot model.
INTRODUCTION
Foundational vision-language models (VLMs) such as CLIP (Radford al., 2021) have achieved strong zero-shot generalisation by pretraining web-scale image-text pairs. To further adapt these models for downstream tasks, fine-tuning is often necessary. However, a well-known drawback of fine-tuning VLMs is catastrophic forgetting of pretrained knowledge, which weakens generalisation.
range of fine-tuning strategies have been proposed to address this issue, though under different evaluation protocols. Parameter-efficient fine-tuning (PEFT) methods update only lightweight mod- ules such as prompts (Zhou et al., 2022b;a; Yao et al., 2023; et al., 2025) or adapters (Yang et al., 2024: Khattak al., 2023), achieving strong adaptation with relatively smaller amount of trainable parameters. These approaches are primarily evaluated under few-shot settings, focusing on base-to- novel generalisation and, to a lesser extent, cross-domain and cross-dataset generalisation. In par- allel, robust fine-tuning approaches (Wortsman et al., 2022b; Zhu et al., 2024) combine pretrained and finetuned models through ensembling, either in weight space (Wortsman et al., 2022a; Huang et al., 2024) or prediction space (Lakshminarayanan et al., 2017). Unlike PEFT methods, robust
"Equal contribution
1 Joint last authorship

---

Published as a conference paper at ICLR 2026
fine-tuning methods typically evaluated in many-shot settings, aligning with standard domain generalisation benchmarks (Zhou et al., 2023) while enforeing great in in-distribution performance.
While PEFT methods mitigate forgetting by restricting updates lightweight modules, they not only rely on increasingly complex modules (Yang et al., 2024) to balance adaptation with generali- sation, but also still preserve pretrained knowledge incompletely. For example, as shown in Figure 1, no single prior PEFT method consistently outperforms CLIP across all 11 evaluation datasets with- out leveraging external knowledge (Yao al., 2024). Similarly, ensembling can ease forgetting in the robust fine-tuning, but they often fail to fully reconcile pretrained knowledge with downstream adaptation (Zhu et al., 2024) and yield unstable performance (as shown in Table 4).
In this paper, we explore how to re- cover forgotten knowledge after fine- tuning is completed, which resem- bles the purpose of model ensem- bling methods in the robust fine- tuning (Wortsman et al., 2022b; Zhu et al., 2024). However, unlike ensem- bling, we take as a post hoc di- rection to the VLM adaptation tech- niques through continued fine-tuning.
20113
TGB rasss
Yhu VCLLU
LOV SPLL
We take inspiration from recent ad- vanced weight-space ensembling model merging (Yadav al., 2023; Huang et al., 2024), which demon- strates that different models can be combined to integrate complemen- tary knowledge. Building on these insights, we propose to merge a fine- tuned VLM with its zero-shot coun- terpart to restore lost knowledge and enhance performance. However, ex- isting model merging methods tuned models could lie far apart in weight space. This weight-space gap could break the mode connectivity (Garipov et al., 2018) required for model merging to be effective and may explain the unstable performance commonly observed in weight averaging approaches (Wortsman et al., 2022a; Zhu et al., 2024). To overcome this, propose learning-basee merging method that leverages linear mode connectivity (LMC) (Garipov et al., 2018) as an objective to integrate knowledge from both shot and finetuned VLMs through continued fine-tuning.
015-3P0
Figure I: Cross-dataset generalisation shows no single PEFT
method consistently outperforms CLIP across all 11 datasets, im- plying incomplete preservation of pretrained knowledge. Numbers in brackets (X/11) indicate X times method underperforms CLIP.
often ineffective in this context, where the zero-shot and fine-
Our model-agnostic MERGETUNE applies post hoc to already finetuned VLMs without requiring ar- chitectural changes. We can continue updating its trainable parameters (e.g., soft prompts, adapters, or linear heads) accordingly to search for weights that are connected to the zero-shot and the fine- tuned ones with two low-loss paths. The continued fine-tuning by searching these geometrically linear paths allows the continued model to effectively integrate pretrained knowledge into the fine- tuned model. However, the vanilla LMC constraint requires replaying the data used for training the zero-shot model (Mirzadeh et al., 2021), which is problematic. For example, the web-scale corpus used for pretraining CLIP is publicly inaccessible (Radford et al., 202 1). And even accessible for re- play, it is computationally prohibitive. We thus introduce second-orde order surrogate that approximates the constraint for the zero-shot model, eliminating the need for aarge-scale data replay.
We apply MERGETUNE to multiple PEFT and robust fine-tuning methods. Experiments demon- strate that MERGETUNE consistently improves accuracies on the benchmarks for which these mod- els were trained. On base-to-novel generalisation, MERGETUNE improves the harmonic mean of CoOp (Zhou et al., 2022b) by +5.6% without adding parameters. On robust fine-tuning tasks, the LMC-merged model from MERGETUNE achieves stronger -ut-of-distiibution (OOD) generalisa- tion than ensemble- based baselines, while maintaining comparable in-distribution (ID) accuracy. It also reduces inference cost and consistently outperforms CLIP across all evaluated cases. Moreover, simple ensembling with the zero-shot model brings further gains, setting new state-of-the -oht----rttt-tttt--- results.
2

---

Published as a conference paper at ICLR 2026
We summarise our contributions as follows: 1) While existing VLM methods seek to mitigate knowl- edge forgetting during downstream fine-tuning, we find that forgetting remains often unavoidable. In response, we propose continued fine-funing forgotten knowledge after the model has been adapted. 2) Inspired by model merging (Section 3), which integrates complementary knowledge across models, we introduce learning-based merg- ing method MERGETUNE guided by linear mode connectivity (Section 4). 3) MERGETUNE is model-agnostic and can be applied post hoc to any fine-tuned VLM without architectural changes. Extensive experiments demonstrate its effectiveness in restoring pretrained knowledge to enhance the fine-tuned model (Section 5).
new paradigm to existing methods - restore
2 RELATED WORK
VLM fine-tuning, Few-shot parameter-effiient fine-tuning (PEFT) has become a common strat- egy for adapting vision-language models (VLMs), as reduces computational costs while miti- gating catastrophic forgetting. Prompt learning methods, such as CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a), optimise continuous prompts (task-specific or image-conditioned) instead of fixed templates, with extensions including prompt distributions (Lu et al., 2022) and attribute- guided prompts (Li et al., 2025). Adapter-based methods insert lightweight trainable mod- ules, either uni-modal (Clit-Adapter (Gao et al., 2024), Tip-Adapter (Zhang et al., 2022)) multi- modal ( (Yang et al., 2024; Jiang et al., 2025; Zang et al., 2022). Most of these existing methods focus on the few-shot learning evaluation of base-to-novel generalisation, with limited attention to cross-domain and cross-dataset generalisation.
Another line of work focuses on many-shot robust fine-tuning (Zhou et al., 2023), aiming to boost in- distribution accuracy without compromising out-of-distribution generalisation (Kumar et al., 2022; Wortsman et al., 2022b; Zhu et al., 2024). One common practice weight ensembling after the model is fine-tuned, which blends zero-shot and fine-tuned model parameters to preserve both task- specific and general capabilities. Wortsman et al. (2022b) demonstrated that simple linear interpo- lation between zero-shot and fine-tuned weights can improve both II) and OOD performance. More sophisticated ensemble methods have emerged, such as VRF (Zhu et al., 2024), which conditionally combines models based on a "zero-shot failure" set, and R-Adapter (Kim et al., 2024), which inte- grates self-ensembling with lightweight adaptation. However, these complex ensembling procedures add inference overhead.
Model merging. Model merging demonstrates that different models can be effectively combined to integrate complementary knowledge, grounded in the discovery of mode connectivity (Garipov et al., 2018). Early work showed that independently trained solutions can be linked by low-loss paths in weight space (Garipov et al., 2018; Draxler al., 2018). Building on these insights, practical methods emerged: model soups combine multiple fine-tuned weights (Wortsman et al., 2022a), task arithmetic edits task vectors (Ilharco al., 2023), and approaches like TIES-Merging (Yadav al., 2023) and DARE (Yu et al., 2024) address interference during model merging.
Instead of merely mitigating forgetting during downstream fine-tuning, our goal is to restore the pretrained knowledge lost after the zero-shot model has been adapted. To this end, we introduce a continued fine-tuning strategy MERGETUNE that learns merge the zero-shot and adapted models.
3 PRELIMINARIES
We begin by introducing model merging and mode connectivity, which underpin our approach.
Model merging and mode connectivity, When two models are trained on the same task with the loss L(-) but differ in initialisation or training trajectory, they converge to different solutions wi, dh that both achieve low loss. Often, one can merge them through weight averaging.
w >a) = (1 - ai)i + aniz, a E [0, 1].
This linear interpolation can sometimes yield a model with performance comparable to its endpoints ie and ths (Izmailov et al., 2018) or yield interesting observation (Garipov et al., 2018) as follows
((7(a)) 0,
3

---

Published as a conference paper at ICLR 2026
Before MERGETUNE Training
During MERGETUNE Training
After MERGETUNE Training
spach
Casa sheee
b-0.17 73
epech J3
spoch
EH0.75
Wissi
The proposed MERGETUNE (conceptual illustration). (Left) Before MERGETUNE Training: The zero-shot model dūj and fine-tuned ún exist in separate minima with no low-loss connectivity. Linear interpo- lation between them (shown in the inset) reveals high barriers and induces performance trade-off base and novel classes. (Middle) During training, w searched to mode connected to both wi and úuz gradually inte- grating both models. (Right) After MERGETUNE Training: Our continued model Wours merging two endpoints will be used for inference of both tasks Wi and d2 where trained. The two distinct low-loss paths, Wh + Woura and Uا
Figure 2:
Wours, show smooth interpolation curves (inset) indicating stable performance.
i.e. during the interpolation, the computed loss of the resulting model consistently close
The effectiveness of model merging and the root of the observation in Eq. 1 can be explained by the phenomenon of mode connectivity (Garipov et al., 2018; Draxler et al., 2018), i.e. linear mode here. Empirical studies have shown that seemingly distinct optima discovered by independent train- ing runs can be linked by continuous low-loss paths in parameter space. This indicates that neural network solutions are not totally isolated minima but could lie in connected valleys of the loss land- scape. Mode connectivity thus provides a theoretical basis for why interpolation between models can preserve low loss.
Merging models of different tasks. Beyond the same-task setting above, model merging can be extended to integrate knowledge from different but related tasks, e.g., combining a model with another trained for different tasks (Mirzadeh al., 2021; Yadav et al., 2023; Yu al., 2024). this case, the goal is to find a merged solution w that preserves performance on both tasks. Formally, given models i and 22 trained with task losses L1 and L2, one can seek a new model whose interpolation paths to úi and 102
(1(a) wn a(w- (ow--j) w1), 72(a) =w² a(w w2) a [0,1].
(3)
satisfy
((((a)) = 0, (2(12(a)) =
(4)
That is, throughout the interpolation, model maintains two smooth, low-loss connections both endpoints, ensuring that knowledge from both tasks preserved in the single w (Mirzadeh et al., 2021; Yadav al., 2023; Yu et al., 2024).
From this perspective, model merging can be viewed as finding a solution that lies on low-loss paths connecting multiple checkpoints. This geometric view unifies heuristic averaging methods, such as training-free approaches TIES (Yadav et al., 2023) and DARE (Yu et al., 2024), with regularisation- based methods (Mirzadeh al., 2021). Enforcing mode connectivity ensures that the merged model integrates the endpoints, preserving knowledge from both. However, training-free methods (Yadav al., 2023; Yu al., 2024) lack explicit enforcement of mode connectivity, and may therefore struggle to integrate models effectively.
MERGETUNE: CONTINUED FINETUNING OF VLMS VIA MODE-CONNECTTVITY GUIDANCE
We now introduce continued fine-tuning (CFT) objective, which enforces linear mode connec- rivity between the continued model and both the zero-shot and fine-tuned solutions. This ensures that the final model preserves pretrained knowledge while retaining downstream adaptation.
4

---

Published as a conference paper at ICLR 2026
Linear mode connectivity objective. The effectiveness of model merging is grounded in the phenomenon of linear mode connectivity. We employ it directly as a learning principle, as inspired by Eq. and Eq. 4. Our hypothesis is that if model can be linearly connected to another solution through a consistently low-loss path, it can inherit and preserve the knowledge of that solution.
Concretely, given zero-shot checkpoint W1 (e.g., CLIP) and downstream fine-tuned checkpoint 2 (e.g.. CoOp), we seek a continued solution w that integrates both. We require w to remain linearly connected to both and 22 via low-loss interpolations. This leads the objective:
= arg min 10 (20 + a(w - -m1)) + C2( + +aww - w22))
(5)
where L1 and L2 are the pretraining and downstream training losses for the zero-shot and fine- tuned models, and a the interpolation coefficient uniformly sampled from [0, 1]. This formulation leverages linear mode connectivity learning objective merge knowledge from two solutions. After training, w is used for inference. An illustration of our training process is shown in Figure 2.
Challenge. A direct implementation of Eq. 5 requires the computation of both L1 and L2- How- ever, &, depends on the pretraining data (e.g., CLIP's ss web-scale corpus), which is often inaccessible propose a second-order surrogate loss to ap-
and computationally prohibitive replay. Thus, proximate the computation of Task loss term in a replay-free manner.
Second-order surrogate. We approximate the Task interpolation term using second-order Taylor expansion:
L1 (w0 + aww -1)) = Li (10 + (V) -w1)= of 2 w1) H2(w (n))
(6)
where H1 (1)) is the Hessian matrix.
We adopt assumptions: (1) VL((w))
as よे lies near local optimum for Task (2)
H1 = μl, assuming isotropic curvature for tractability following common practices. Under these assumptions, Eq. 6 simplifies
L (w + a(w - w = (1)) + ua² || w1
L1 (101 constant value, and pa² formulated as:
can be represented by The surrogate regulairiser can be 2
R = Allw - -0111
(8)
Final replay-free objective. Combining the Task 1 surrogate with the Task 2 objectives leads to our final continued fine-tuning loss:
((w) (2(w) Allw 1 Ca + a(w -b)))
(9)
This objective preserves pretrained knowledge through proximity to ú1, while enforcing low-loss connectivity to the downstream solution u. Empirically, we decoupled L2(w), where a = 1, from expectation term. In this way, the LMC terms serve as regulariser for training. Also, expectation in the third term is computationally intractable. We thus approximate it by evaluating small number of evenly spaced a values (e.g., five or ten), which yields promising results.
Application to CLIP fine-tuning methods. Our framework makes no limitation on which param- eters are trainable, allowing to be applied broadly. We optimise w under the same training con- figurations the base method: In prompt-based methods such as CoOp and KgCoOp'. w T()) corresponds to the classifier weights derived from learnable prompts and T(.) is the text encoder. In adapter-based methods such as MMA, timodal adapters In many-shot regimes, w can represent the linear classification head (linear probing) or the entire model parameters (end-to-end fine-tuning), This flexibility allows tinued fine-tuning approach to act as a general post hoc enhancement, seamlessly integrating with diverse CLIP adaptation strategies.
T(p; 0) includes fixed prompts and trainable mul-
5

---

Published as a conference paper at ICLR 2026
Table I: Fase-to-nowe generalisation experiments on 11 datasets. Our method achieves consistent average per- formance improvement over different haselines. 1: Using large language model or teacher model's 's knowledge.
Average
ImrageNier Newel
Caltech101
OxfondPats
Method
Bose
Nowel
Rase
Novel
HM
Base
Novel
HM
CLIP acv0.70
69.35 34
74.22
71.70
72.43
68 14
70.22
96.84
94,00)
95. 04
91.17
97:33 26
94.12
CoOp (iCv
82.60
63.22
71.66
76.47
67.35 67.88
71.92
98,00
89.81
93.73
93.67
95.
94.47
KgCoOp ceve
80:73
73.61
77.01
75.83
69.96
72.78
97.72
94.39
96.03
94.65
97.76
96.18
MaPLe 20)
75.14
76.66
95.43
97.96
96.58
ProcuptSRC 20
84.26
76.10
79.97
77.60
70.73
74,01
98.10
91,03
96.02
95.33
97
96.30
MMA (UV91224)
83.20
76.80
79.87
77.31
71.00
74.02
98,40
94,00
96.15
95:40
98.07
96.72
CuPPromgi (00000
77.23
77.67
55
95.67
PromptKD (CV15.20)
$6.96
80.73
83.73
80.83
74.66
77.62
98.91
96.65
97.77
96.30
98.01
97.15
CoOp ATPrommp
$2.68
68.04
74.65 (1299)
76.27
70,60
73.33
97.95
93.63
95.77 74
94.77
96. 36.59
45.67
Prompi/D ATProcppp (0cV 201
81.82
$4.35
(40.2))
80.90
74.83
77.35
96.52
97.00
96.92
98.27
97.59
CoOp + TIES (ieadts 33)
66.95
65.71
66.32 (435)
70.09
61.65
65.60
93.63
91.34
92.50
90.12
95.13
92.56
CaOp DARE nrvi N0
75.91
65.97
70.59 (100)
74.25
64.32
68.93
96.36
91.12
93. 67
93.41
96.33
94,84
CoOp MERGETUNE
80.82
73.97
77.
75.96
69.91
72.81
97.91
94,65
96.25
95.09
97.75
96.40
KgCoOp TIES nouutes
72.09
72. (4:45)
74.06
68.00
93.99
96.51
KgCaOp DARE nese. 20
78.15
72.41
75. (1144)
75.10
68.73
71.28
97.53
94,47
95. 97
92.22
97.37
94.73
KgCoOp + MERGETUNE
$1.85
74.46
77.98 (40.57)
76.49
69.35
72.75
97.87
94,91
96.37
95.32
97.76
96.53
MMA TIES pleutt 241
70.39
68.46
69.41
73.16
66.48
69.66
95.36
92.23
93.77
91.24
93.11
92.17
MMA-1
71.81 (2000)
74.17
66.76
90.52
95.53
92.96
MMA + MERGETUNE
84.27
76.94
80
77.68
70.65
74.00
98.32
94.65
96.45
95.75
98. 10
96.91
PromptiDD + TIES (NoitS
82.03
77.16
79.52 (420)
76.18
69. 32
72.56
97.61
94.9
96 24
95.23
96. 13
95.
PrompeKD' + DARE SEMIL
82.13 (1.5)
78.29
72.54
75.31
97.15
95.56
98.00
96.76
PrompiKD MERGETUNE
87.23
81.17
84.09 (40.99
80.89
74.88
77.77
98.93
96.64
97.77
96.63
98, 34
97.48
StanfordCarr
Hlwerst02
Food101
HGVCAireraft
Method
Base
Novel
HM
Buse
Novel
HM
Base
Novel
HM
Base
Novel
HM
CL.P. 20
63.37
74.89
65
72.08
77.80
74.83
90.10
91.22
90.66
27.
36.29
31.09
CoOp
78.12
68.
97.60
59.67
74.06
82.26
10.44
28.75
KgCaOp (0001
71.76
75.04
73.36
95.00
74.73
83.65
90.50
91.70
91.10
36.21
33.55
34.83
MaPl cove
72.94
74.00
73:47
95.92
72.46
82.56
90.71
92.05
91.38
37 37.44
35.61
36.50
PrompiSRC 20
78.27
76.58
98.07
76.50
91.53
42.73
40.15
MMA (2
78.50
73.10
75.70
97.77
75.93
85,48
90.13
91.30
90.71
40.57
36.33
38.33
CoPrompe
76.97
74.40
75. 26.66
97.27
76.60
85.71
90.73
92.07
91.40
40.20
39.33
39.76
PromptKD' (CVR220
82.80
83.37
83.13
99.42
82.62
90.24
92.43
93.68
93.05
49.12
41.81
45.17
CoOp ATPrompi
77.43
66:55
71.58
97.44
67.52
89.74
87.44
88.09
40.38
27.22
32.52
PromgeKD + ATPrompe 0C0V 25)
82.51
84.03
83:26
99.15
82.03
89.78
92.48
93.96
93.22
49.63
42.35
45.70
CoOp TIES (NculPS 23)
57.94
63.42
60.56
73.60
65.41
69.27
88.42
87.64
88 0Z
30.29
28.83
29. 04.54
CoOp + DARE
67.82
61.62
64.57
90.85
63.73
74.92
87.78
87.22
87.50
34.41
31.03
CaDp + MERGETUNE
71.94
75.12
73.49
95.32
74.30
83.51
90.56
91.75
91.15
36.17
34.35
35. 24
KgCoOp TIES (NailPS
65.79
74.17
69.73
74.26
72.53
73.39
90.25
91.58
90.91
30.81
28.97
29.86
KgCoOp DARE OCML 341
67.84
73.02
70.33
72.88
79.33
91.60
91.02
33.49
31.85
KgCaOp MERGETUNE
73.21
75.14
74.17
96.449
74.68
84.20
90.51
91.83
91.17
37.09
35.11
36.07
MMA + TIES (Noal25
60.72
70.12
65.08
73.16
70.74
72.07
88.79
89.83
89.31
29.27
28.06
28.65
MMA + DARE aCMIL 24
64.82
71.52
68.01
77.15
70.95
73.92
89.85
87.61
58.72
30.35
28.23
29 25
MMA + MERGETUNE
81.24
72.30
76.51
98.04
76.33
85.88
90.77
91.57
91.17
42.86
36.01
39.14
PrompoKD' TIES (NcuiPS
75.13
76.99
76.05
94.53
79.22
86.20
90.96
92.10
91.53
43.68
41.07
PrompiKD' DARE (41
80.13
81.95
81.03
97.73
80.93
88.54
91.57
93.28
92.42
46.96
40.79
43. 66.66
PromptKD + MERGETLNE
82.98
83.44
99.30
82.67
90.23
92.45
93.91
93.17
19.82
45.76
SUN397
DTT
EuroSATT
UCF101
Method
Base
Novel
HM
Novel
Basc
Novel
Base
Nowel
HM
CLJP JCML 211
69.36
75.35
72.23
53.24
59.90
56.37
56.48
64.05
60.03
70.53
77.50
73.85
CoOp (ICV
80.60
65.89
72.51
79..4
41.18
54.24
92.19
54,74
68.69
84.
56.05
67. /6
CoCoOp (2)
79.74
78.27
77.01
56.00
87.19
60.04
82.33
17.66
KgCaOp coor 220
80.29
76.53
78.36
77.55
54,99
64.35
85.64
64.34
73.48
82. 89
76.67
79. 99.66
MaPl la coVe
$0.82
78.70
79:75
80.36
59.18
68, 16
94.177
73.23
82.35
83.00
78.66
80.77
PrompiSRC (0CV22)
82.67
80.52
62.97
71.75
73.90
87.10
78.80
MMA (CV19 24
82:27
78.57
80.38
83.20
65.63
73.38
85.46
82.34
83.87
86.23
80.03
82.
CoPrompe" (XLB83))
82.63
80.03
81.30
83.13
64.73
72.79
94.60
78.57
85.84
86.90
79.57
83.07
PrompeKD' (CVR220
83.69
81 54
82.60
85.84
71.37
77.94
97.54
82.08
89.14
89.71
82.27
86. 10
CoOp ATPromptt uccv 29)
80.84
68.64
74.24
80.83
45.49
58.22
90.34
59.79
71.96
84.49
64,96
73.45
ProngeKID + PTPrompe (CCV 25)
83.87
81.35
82.59
86.92
72.34
78.96
97.05
92.07
94.49
89.29
82.44
85.73
CoOp TIES (NcultS 23)
62.41
62.99
62.70
57.33
41.87
48.40
50,17
69. 15.85
58.40 40
62,45
54. 54.68
58, 30
CoOp + DARE
70.33
66.47
72.49
57.04
65.22
57.54
66.28
CaOp + MERGETUNE
80.37
76.19
78.22
78.51
56.84
65.99
85.15
64.51
73.40
$2. 04
78.33
80.
KgCoOp + TIES (Naites
73.05
73.88
73.46
60.77
53.54
56.93
68.66
65.45
67.02
74.46
73.91
74.18
KgCoOp DARE OCM0..4
77.47
74.81
76.
72.46
54.19
62.01
84.48
63.03
72.19
81.63
78.71
KgCaOp+ MERGKTUNK
80.53
77.28
78.87
799.90
57.33
66.75
89.56
67.55
71.22
83.35
77.76
80:45
MMA + TIES (Noults
69.46
65.77
67.56
55.38
48.36
51.63
65.69
67.95
66.80
71.74
60:42
65.60
MMA DARE JUMI 20)
71.43
72.61
72.02
67.65
50.37
57.74
77.42
64.16
77..17
78.
70.78
MMA + MURGETUNE
82.46
78.48
80.44 4244
83.37
65. 34
73.26
90.09
81.91
85.81
86.33
81.02
83.61
PrompoKD (NeuiPP
79.11
77.78
65.93
71.82
88.64
81.59
84.97
82.47
77.32
79.81
PromptKD DARE (4ML224
80.34
79.66
80.00
84,22
70.04
76.48
92.48
82
86.99
86.97
80.68
83.71
PromptDD + MERGETUNE
83.92
82.69
86.77
72.10
78.76
97.81
83.82
90.28
89.99
86.24

---

Published as a conference paper at ICLR 2026
Table 2: Cross-datase generalisation results. Models are trained on ImageNet and directly evaluated on other
datasets. Avg-C average over all cross-dataset targets. *; Our reproduction. 1 Using external knowledge.
FOVC Aircraft
Stanford Cara
Callech101
Oxford Pan
Flowar 102
Faad10
SUN397
EuDSAT
UCF101
Arg-C
DTD
Method
CLIP (CMI-30)
93.30
89.10
65.70
70.70
85.90
24.90
62.60
44.30
48.30
67.60
65.24
CoOp
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41,92
46.39
66.55
63.88
-TIES (Neeets 33
93.31
88.25
65.51
67.40
85.23
23.91
62.56
44.39
42.22
65.24
63.80 64:00
DARE JCML
89.67
86.40
62.87
66,32
84.27
18.07
60.27
38.55
45.65
64.68
61.67 (2221)
MERGETUNE
93.96
89.97
65.67
70.40
86.43
23.88
66.35
46.53
46.12
68.71
65.80 (41.92)
KgCoOp ICVER 221
93.92
89.83
65.41
70.01
86.36
22.51
66.16
46.35
46.04
68.50
65.51
+TIES (NeultS 23)
91.94
87.56
64.33
66.92
85.53
21.69
63.66
41.75
46,36
65.74
63,55 6:100)
+DARE @CML. 34y
93.66
15
64.81
69.29
86.16
22.24
64.64
45.09
45.38
66.06
64.55 (4090)
*MERGETUNE
94.24
90.41
66.15
70.85
86.57
24.12
66.63
47,46
48.41
70.42
66.53
MMA (CVPR
93.80
89.65
65,41
69.86
85.79
24.47
67.31
45,26
43,70
68.82
65.41
+TIES (NauliS 259
90.54
86.21
62.46
65.35
84.21
17.98
60.33
38.35
41.69
64.13
61.13 (-4.2))
+DARE JCMI.3
91.24
87.14
63.25
66.45
84.43
19.46
62.12
40.44
42.89
65.35
62.28
-MERGETUNE
94.28
90.43
66.11
71.46
86.26
25.26
67,59
46.22
46. 16
69.23
66.30 (40,89)
PromptKD OCLR 24)
93.61
91.59
73.93
75.33
88.84
26.24
68.57
55.08
63.74
76.39
71.33
+TIES (NarlDS 25)
91.35
89.25
70.35
69.36
86.26
23.74
64.75
50.36
60.68
72.54
67.86
-DARE (CML 24)
93.26
90,36
73.26
74.26
88.36
25.14
67.47
54.26
63.03
75.19
70.46 (4.5))
-MERGETUNE
93.95
92.01
74.17
75.39
89.32
26.58
68.61
55.74
63.94
77.12
71.68 (40.35)
EXPERIMENTS
We evaluate MERGETUNE across multiple benchmarks and protocols:
Few-shot setting. Base-to-novel generalisation: We evaluate on 11 datasets, each dataset is split into base and novel classes; models are trained on base classes and evaluated on both base and novel categories, with harmonic mean (HM) reported as the main metric. Cross-dataser generalisation: Following (Zhou et al., 2022a), models are trained on ImageNet (16 shots per class) and directly evaluated on other datasets without further adaptation. Domain generalisation: To assess robustness to domain shift, it is to evaluate ImageNet-finetume models on four ImageNet variants.
Many-shot setting. ID-OOD generalisation: In the robust learning setting, models are initial- ized from pre-trained CLIP with an additional linear classifier head derived from CLIP's zero-shot text embeddings generated by encoding class-specific text prompts through the pre-trained text en- coder, then fine-tuned on ImageNet- 1K using standard supervised cross-entropy loss. The finetuned model is tested without any additional training or adaptation across both in-distribution and out- of of-distribution settings. We report performance the four domain-shifted ImageNet variants plus ObjectNet (Barbu et al., 2019). All reported results are averaged over three random seeds.
5.1 IMPLEMENTATION DETAILS
Base models. We evaluate our continued fine-tuning strategy on multiple VLM adaptation meth- ods. For sew-shot learning, we build upon CoOp (Zhou et al., 2022b). KgCoOp (Yao et al., 2023), MMA (Yang et al., 2024), and PromptKD (Li et al., 2024), while for many-shot settings we consider linear-head and full-model fine-tuning. For each baseline, we reproduce it to obtain a downstream checkpoint. We then apply MERGET T'DEEETNEEEEE to this checkpoint together with the zero-shot CLIP model, following the same training configurations as the respective bascline. All experiments use CLIP ViT- B/16 backbone. Details on datasets, training, and evaluation are in the supplement.
Competitors. We compare our MERGETUNE against two families of competing methods. First, we compare with existing training-free free model merging techniques. including TIES (Yadav al., 2023), which mitigates parameter interference via task-wise sparsification; and DARE (Yu et al., 2024), which improves cross-task transfer through reweighting strategies. Second, evaluate against rep- resentative ensembling approaches: Wise-FT (Wortsman et al., 2022h), which linearly interpolates pretrained and finetuned weights, and VRF (Zhu et al., 2024), which enhances robustness through
Also interesting to note, our MERGETUNE reduces KgCoOp when 3=0 for prompt tuning.

---

Published as a conference paper at ICLR 2026
variance-reduction ensembles. These methods provide strong baselines for assessing the effective- ness of our MURGETUNE.
5.2 FEW-SHOT MASE-TO-NOVEL GENERALISATION
Table reports base-to-novel generalisation across 11 datasets. We extend CoOp, KgCoOp, and PromptKD with training-free frree model merging methods, including TIES, DARE, and our MERGE- TUNE. Since training-free merging cannot directly handle models with structural differences (e-g., MMA, whose adapted models differ from pretrained CLIP), we instead merge only the linear heads while preserving the original MMA image encoder.
The results highlight two key observations. First, merging zero-shot and fine-tuned checkpoints TIES DARE degrades performance. For example, CoOp+TIES reduces the HM by 5.33% com- pared to vanilla CoOp, while CoOp+DARE decreases it by 1.07%. Even when applied to stronger baselines such PromptKD, these training- free approaches fail to improve performance, under- scoring the difficulty of directly interpolating between CLIP and fine-tuned checkpoints.
Second, our MERGETUNE consistently improves generalisation across all baselines. The magnitude of improvement correlates inversely with baseline performance, which is theoretically consistent with our framework. Methods that already preserve pretrained knowledge well (e.g., PromptKD at 83.13% HM) have less forgotten knowledge to recover, thus showing smaller but consistent gains (+0.36%). Conversely, methods with more catastrophic forgetting (e.g.. CoOp at 71.66% HM) ben- efit more substantially (+5.58%), as our LMC-based approach has more pretrained knowledge to restore. This demonstrates that MERGETUNE'S effectiveness scales with the degree of forgetting, making it particularly valuable for improving methods that struggle with knowledge retention.
5.3 FEW-SHOT CROSS-DATASE GENERALISATION
Table 2 presents cross-dataset generalisation results, where models trained on ImageNet are di- rectly evaluated on 10 other datasets without adaptation. Training-fres merging methods (TIES and DARE) generally degrade performance, whereas MERGETUNE delivers consistent gains across all baselines, with average HM improvements of +1.92 on CoOp, +0.86 on KgCoOp. +0.89 on MMA, and +0.35 PromptKD. The improvements are especially pronounced challenging datasets such as FGVC Aireraft, DTD and AT. Notably, by merging pretrained knowledge from the zero- shot model via our MERGETUNE, MMA is able surpass CLIP on all evaluated datasets.
5.4 FEW-SHOT DOMAIN GENERALISATION
We evaluate the models few-shot- tuned from ImageNet out-of-domain datasets shown Ta- ble 3. Again, training-free free merging approaches (TIES and DARE) gain negative performance gains across all baselines, with notable degradation on datasets featuring larger shifts, such as ImageNet-Skette and ImageNet- Adversarial.
Table 3: Domain gemeralisation results ImageNet and four distribution shifts. Avg-D D = average over domain-shifted datasets. MMA is our repraduction.
various
Method
ImageNet
.V2
-5
A
-R
Avg-D
CaOp doo 25
71.51
64.20
47.55
49.71
75.21
59.28
4TIES (N
62.84
56 26.26
41.14
44.65
70.77
53 $3.20
ADARE 00M0.30) 2q
69.02
61.75
45.87
49.10
73.85
57 57.64 61
AMBRGITTUEE
71.68
64.55
48.67
50.74
76.61
60.15 (-Iu))
KgCoOp (CVeR
70.66
61.10
48.97
50.69
76.70
60.11
+TIES (Nauts
67.77
61.47
46.36
49.86
75.91
58 58.40
+DARE OCME
69.67
62.89
47.69
50.33
76.15
59 27 (a56)
Our MERGETUNE consistently outperforms all baselines against domain shifts, achieving posi- tive gains of +0.87 for CoOp, +0.35 for KgCoOp, +0.42 for MMA, and +0.30 for PromptKD (as Avg-D). confinming enhanced robustness under distribution shift scenarios.
+MERGETUNE
71,80
64.70
49.10
51.01
77.02
60.46 (10.55)
MMA"
70.45
63.87
48.84
59.91
77.29
59.98
+TIES
64.01
$7.13
42.32
45.25
72.11
$4.20
+DARE
68.35
58.67
44.75
47.26
74.25
56.24 (3.74
+MERGE TUNE
71.11
61.41
69.26
50.43
77.51
60.40
PromptKD (CCUR
77.12
69.77
58.72
70.36
71,47
+TIES (NealEP 23)
74,63
64.22
67.47
K3.85
67
+DARE OCME
76.14
68.90
69.36
40
*MEGGEUUNE
77.14
70.21
7066
K7.22
71.77
(40306
8

---

Published as a conference paper at ICLR 2026
Table 4: ID-OOD generaisation accuracy of various methods on ImageNet and distribution shifts CLIP ViT-B/16 in the robust line-tuning evaluation. Avg-D average over domain-shifted datasets.
Distribution shifts
Method
Imagenel
Avg-I>
-V2
-A
-R
ObjectNet
Zero-shot (CLIP)
68.34
61.90
48.27
50.12
77.60
54.23
58.42
Lineur Probing (CVPR 22)
79.79
70.02
46.48
71.16
52.28
57.39
Weight (2)
79,80
70.45
48.41
47.89
73.00
53.07
58.56 (:1.17)
VRF (NeuriPS
79.84
70.36
48.67
48.08
73.87
53.36
58.87 (-1.48)
'IES (NontDS 23)
79.75
70.33
48.25
48.32
73.78
53.13
58.76 (-1.37)
DARE
70.26
47.74
73.11
(-1.00)
MERGETUNE
79.96
70.22
49.47
49.21
75.98
53.43
59.66 (40.27)
Weight ens.
79.88
70.27
50.14
50.04
76.69
54,01
60.23 (+2.86)
E2E-FT (CVPR: 22)
81.31
70.61
45.12
36.62
65.63
50,51
53,70
Weight ens, (2)
82.51
73.11
51.62
47.61
75.13
55.71
60.64
(16.94)
VRF (NeuiPS
82.32
72.12
52.93
48.41
78.72
56.41
61.72
TIES (NeurPS
82.27
72.84
51.67
47.81
74.48
54.87
60.33 (+6.33)
DARE (CMI, 24)
81.09
70.21
45.79
35.55
65.23
50.13
53.38 (40.9)
MERGETUNE
72.98
52.76
51.61
62.29 (46.9)
Weight ens.
82.18
73.21
53.10
52.68
78.68
56.84
62.90
(-0.20)
7. TTsk (pretraince) surrogate loss weight
76.42
0,68%
Averge 77.31%
77.75
(Low Senstiivit)
Harmonic Mean (%)
77.50
(77,628)
1673
77.00
77.57
77.28
77.22
77.27
77.25
17.35
Meatt
1m8
77.51
77.62
77.51
77.02
2=10
77.45
77.42
77.49
77.33
76.50
2-16
77.36
77.43
77.51
77.52
7625
1=20
77.00
77.23
77.26
77.45
76.25
7600
76.00
1e0.1
8=0.5
8=10
6=2.0
0.0
0.2
0.4
0.6
0.8
1.0
(6) P- Task2 (fine-tuned) LMC loss weight
(b) T Initiolization Parameter
Figure 3: Hyperparameter sensitivity analysis: HM averaged over 11 datasets. (a) Surrogate loss weight and Task2 LMC loss weight B. (b) Initialisation parameter of the continued model on performance (under 8.0, 0.5). where 0 means using CLIP weights for initialisation and means using the fine-tuned weights (c.g. KgCoOp here).
5.5 ID-OOD GENERALISATION
Table 4 presents the robust fine-tuning evaluation on ImageNet and five datasets under distribution shifts. We repurposed model merging methods for this evaluation. However, training-free merging methods, TIES and DARE, exhibit unstable performance and severe degradation under challenging shifts. Our MERGETUNE outperforms SOTA ensembling methods, such as VRF Notably, MER- GETUNE employs single model for inference, eliminating the need for failure set construction or per-sample distance computations, which can incur additional inference costs as VRF did.
Furthermore, when combined with weight-space ensembling - which interpolates parameters be- tween the zero-shot and our LMC-tuned checkpoint using single mixing weight our MER- GETUNE achieves the highest average OOD accuracies: 60.23 (Linear Probing) and 62.90 (E2E- FT). This demonstrates that the LMCC-tuned model a superior replacement for standard fine-tuned checkpoints in ensemble schemes. These results establish our achievement better ID-OOD trade- offs without complex design, while for the first time outperforming CLIP in all cases.
5.6 FURTHER ANALYSIS
We evaluate the sensitivity of MERGETUNE to key hyper-parameters, including surrogate loss weight X, Task2 LMC loss weight 3. and the initialisation of the continued model. We conduct this ablation averaged over 11 datasets using ure 3(a) demonstrates that our method is robust to hyperparameter choices, with harmonic mean performance varying within narrow range across different À and B combinations. Optimal perfor-
method based on KgCoOp initialisation. Fig-

---

Published as a conference paper at ICLR 2026
mance is achieved when À [8, 16] and [0.1, 0.5], resulting in balanced loss values between different terms. Figure 3(b) shows the impact of the initialisation the parameters for continued fine-tuning, Given both fine-tuned and zero-shot checkpoints,
initialise the parameters by merg- ing them as w = (1 T T) + 7i22, r [0. 1], where - = corresponds to CLIP weights and fine-tuned weights. When the initialisation combines CLIP and the fine-tuned model in balanced manner (7 E [0.3, 0.6)), performance remains optimal.
6 CONCLUSION
We introduced continued fine-tuning (CFT), a new paradigm that recovers pretrained knowledge of ldak-adapted VLM. Our method, MERGETUNE, searches for continued model that is linear-mode connected to zero-shot and fine-tuned checkpoints, thus effectively integrating their knowledge. MERGETUNE is model-agnostic and can be applied post hoc existing fine-tuned VLMs, with- out requiring architectural changes. Experimental results on base-to-novel, cross-dataset, domain generalisation, and robust fine-tuning benchmarks showed the efficacy of our MERGETUNE, establishing continued fine-tuning promising direction for advancing VLM adaptation.
7
STATEMENT
7.1
ETHICS STATEMENT
This work focuses on methodological improvements for adapting vision-language models and does involve the collection of new human subject data. All datasets used our experiments publicly available, widely adopted in prior research, and contain images and annotations curated for academic use. We follow the same usage protocols the original dataset authors and cite the relevant sources. As our method builds on pretrained models such as CLIP, potential biases or societal impacts inherited from these models may still present downstream applications. We encourage careful consideration of these risks when deploying our approach in real-world systems.
7.2 REPRODUCIBILITY STATEMENT
We ensure reproducibility by providing comprehensive details of our experimental setup. The datasets, model architectures, and baselines figurations, including hyperparameters and optimization settings, are described in the main paper and supplement. To further support replication, we include our code in the supplementary material. In addition, we perform multiple runs with different random seeds and report averaged results to ensure statistical robustness.
all publicly available. Our training con-
ACKNOWLEDGMENTS
This research was supported by the UKRI-AHRC CoSTAR National Lab for Creative Industries Research and Development (AH/Y001060/1)
REFERENCES
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund. Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- Food-101-min 1ii-iinng discriminative compo- nents with random forests. In European conference on computer vision (ECCV), pp. 446-461. Springer, 2014.
Mircea Cimpoi. Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 3606-3613, 2014.
10

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning (ICML), pp. 1309-1318. PMLR, 2018.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshop, pp. 178-178. IEEE, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision (ICV),, 132(2):581-59) 2024.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling dnns. Advances in Neural Information Pro- cessing Systems (NeurIPS), 31, 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 15262-15271, 2021b.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights Ieads to wider optima and better generalization. arXiv preprint 1rX1v:1803.05407, 2018.
Haojun Jiang, Jianke Zhang, Rui Huang, Chunjiang Ge, Zanlin Ni, Jiwen Lu, Jie Zhou, Shiji Song. and Gao Huang. Cross-modal adapter for text-video retrieval. Pattern Recognition, 2025.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19113-19122, 2023.
Sungyeon Kim, Boseung Jeong, Donghyun Kim, and Suha Kwak. Efficient and versatile robust fine- tuning of zero-shot models. European conference on computer vision (ECCV), pp. 440-458. Springer, 2024.
Jonathan Krause Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fnneggriined categorization. In International Conference on Computer Vision (ICCV) workshop, pp. 554-561, 2013.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of distribiitiiiitinn.. In International Conference on Learning Representations (ICLR), 2022.
11

---

Published as a conference paper at ICLR 2026
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promp- (kd: Unsupervised prompt distillation for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26617-26626, 2024.
Zheng Li, Yibing Song, Ming-Ming Cheng, Xiang Li, and Jian Yang. Advancing textual prompt learning with anchored attributes. In International Conference Computer Vision (ICCV), 2025.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 5206-5215, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5155 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations (ICLR), 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp- 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3498-3505 IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8026-8037, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning PmLR, 2021.
(ICML). pp.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning (ICML), pp. 5389- 5400. PMLR, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 02.0022 2012.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vasion-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint wrXiv 2025.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time. In International conference on machine learning (ICML), pp. 23965-23999 PMLR, 2022a.
12

---

Published as a conference paper at ICLR 2026
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Ro- bust fine-tuning of zero-shot models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971, 2022b.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485-3492. IEEE, 2010.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. mies-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems (NeurIPS), 36:7093-7711 --15 2023.
Lingxiao Yang, Ru- Yuan Zhang. Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23826-23837, 2024.
Hantao Yao, Rui Zhang. and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 6757-6767, 2023.
Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for visual-language model. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In International conference on machine learning (ICML), 2024.
Yuhang 7ang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision (ECCV), pp. 493-510. Springer, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Dision-language models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 2022b.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):439644415, 2023. doi: 10. 10.1109//1 100/TPPMM..02223195555
Beier Zhu, Jiequan Cui, and Hanwang Zhang. Robust fine-tuning of zero-shot models via variance reduction. Advances in Neural Information Processing Systems (NeurIPS), 37:76967-76999 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DISCLOSURE OF LLM USAGE
Large language models were used to enhance grammar and address editorial issues. All ideas, experimental design, data analysis, and conclusions are the original work of the authors.
A.2 EXPERIMENTAL SETTINGS
Our framework is implemented in PyTorch Paszke et al. (2019), and all experiments are conducted on an NVIDIA 3090 GPU.
Few-shot setting To evaluate MERGETUNE, we apply to the following base methods: the prompt learning methods CoOp (Zhou et al., 2022b) and KgCoOp (Yao et al., 2023), the adapter- based method MMA (Yang et al., 2024), and the distillation-bart based method PromptKD (Li et al., 2024). For each base method, we first reproduce the fine-tuning procedure to obtain a downstream checkpoint. Our MERGETUNE is then applied to this checkpoint in conjunction with the zero-shot CLIP model. Following the baselines, resized cropping and flipping. We use Stochastic Gradient Descent (SGD) as the optimizer. The number of prompt context tokens is set to 4, initialised "a photo of lJ". All few-shot experi- ments are trained with 16 shots.
adopt standard data augmentation scheme of random
batch size of 128 and an initial learning rate of 0.002 for the first round of downstream training over 100 epochs. Our MERGETUNE is then trained for 50 epochs using the same optimiser and training schedule as the base method.
CoOp MERGETUNE. Following the baseline,
KgCoOp+ MERGETUNE. KgCoOp follows the same training epochs, schedule, and data augmen- tation settings as CoOp. The regularisation weight is set A=8.0, as the original configuration. Our MERGETUNE reuses the same training setup as the baseline.
MMA MERGETUNE. Following the baseline, insert the multi-modal unit from transformer block 5=5 to the final block in both the language and vision branches, with a shared projection dimension of 32. In the base novel generalisation, models are trained for 5 epochs with batch size of 128 on ImageNet and 16 on the other ten datasets. For the other experimental settings, we set == and train for only one epoch. Training performed with mixed precision, and our MERGETUNE follows the same training schedule and number of epochs as the baseline for each dataset.
PromptKD MERGETUNE. We publicly available pre-trained teacher models and train stu- dent models for 20 epochs with batch size and a learning rate of 0.005. Our MERGETUNE follows the same training schedule.
Many-shot setting For robust finetuning, we apply MERGETUNE to both linear probing and end- to-end fine-tuning (E2E-FT), Same as the few-shot setting, we first fine-tune CLIP to obtain finetuned checkpoints, and then apply MERGETUNE to derive results.
Linear probing. We train only the classifier head for epochs using SGD with a learning rate of 1x x 10 batch size 512, and cosine learning-rate schedule.
End-to-end Fine-tuning. We jointly update the encoder and classifier head for 10 epochs using AdamW with learning rate of 3 x 10 - batch size 512, weight decay 0.05, and cosine learning rate schedule with warmup. After obtaining the fine-tuned checkpoint, we apply VERGeTUNN for the same number of epochs, reusing the corresponding optimiser and training schedule.
We evaluate the few-shot setting on various settings. For generalisation from base-to- novel classes and cross-dataset evaluation, we test our method on 11 diverse recognition datasets. Specifically, these include ImageNet- IK Deng et al. (2009) and Caltech-101 Fei-Fei et al. (2004) for generic object classification; OxfordPets Parkhi et al. (2012). StanfordCars Krause et al. (2013), flowers-102 Nilsback & Zisserman (2008), Food- foodd101 Bossard et al. (2014), and FGV Maji et al. (2013) for fine-gra -raiind classification; SUN-397 Xiao et al. (2010) for scene recognition; UCF. 101 Soomro et al. (2012) for action recognition; DTD Cimpoi al. (2014) for texture classification;
Datasets
14

---

Published as a conference paper at ICLR 2026
and EuroSAT Helber al. (2019) for satellite imagery recognition. For domain generalisation experiments, we use ImageNet- IK Deng et al. (2009) as the source dataset and its four variants as target datasets: ImageNet-V2 V2 Recht et al. (2019), ImageNet-Sketch Wang et al. (2019), ImageNet- Hendrycks et al. (2021b), and ImageNet-R Hendrycks et al. (2021a).
For the maot setting, report results ImageNet Deng al. (2009) and its five distribution- shift variants: (1) ImageNet-V2 (-V2) Recht et al. (2019): test images sampled a decade after the al. (2021a): contains renditions (e.g., art, cartoons, graffiti.. (3) Imagehet-Sketch (-S) Wang et al. (2019): consists of sketches instead of
original ImageNet. (2) ImageNet-R (-R) Hendrycks
natural photos. (4) ImageNet-A (-A) Hendrycks al. (2021b): real-world images that are misclas- sified by ResNet models. (5) ObjectNet Barbu et al. (2019): a test set featuring objects with diverse backgrounds, rotations, and viewpoints.
A.3 PSEUDO-CODE
Algorithm Continued finetuning MERGETUNE
I: Input: zero-shot weights 1h, fine-tuned weights W2 downstream dataset D22 downstream task loss C2(-), surrogate loss weight X, LMC loss weight 8, number of interpolation points n, initialisation coefficient т
2: Output: continued model weights
3: A 4 {1/n, 2/ .......... (n - 1)/n}
4: Initialise + (1 7 ) + Túh
is initialised by blending zero-shot and fine-tuned
weights.
5: for each training epoch do
6: for each minibatch (Ir, y) from D2 do
7:
Lrask + L2(w; y) Lurr - 1| 16 ||
Downstream task loss >> Surrogate loss for Task
8:
9:
LLMC +
10:
for each a E A do
Winterp -- 1h2 + ax (w: v22) LLMC + LIMC + L2 Wiitterp F. y)
12:
end for
LLMC LLMC/AA| 15: J f -Lusk + >L + BLLMC
14:
Average LMC loss
16:
Update with SGD step T
17:
end for
18:
end for
19:
return w
A.4 ADDITIONAL EXPERIMENTS
A.4.1 GENERALISATION TO OTHER BACKBONES
We also evaluate our MERGETUNE with other backbones to test its generalisation. We report results in Table using CLIP VIT-B/32, which was commonly used in prior robust fine-tuning works (Zhu al., 2024; Kim et al., 2024). The conclusion is consistent with the results in Table 4.
To further evaluate the scalability MERGETUNE, we conduct experiments with three additional lasion-language model backbones: CLIP-L/14, Siglip2-B/16. and Siglip2-L/16 (Tschannen et al., 2025). We compare the base CoOp method with CoOp+ the mose-l generalisation evaluation. As shown in Table 6, our method consistently improves performance with new model scales and architectures, demonstrating strong scalability and model- agnostic effectiveness that generalises well beyond specific model architectures.
across all 11 datasets for
15

---

Published as a conference paper at ICLR 2026
Table 3: ID-OOD generaisation accuracy of various methods ImageNet and distribution shifts for CLIP ViT-B/32 in the robust fine-tune iin-tuning evaluation. Avg-D average over domain-shifter datasets.
Distribution shifts
Method
Imagenet
Avg-D
-V2
-A
-R
ObjectNet
Zero-shot (CLIP)
63.34
53.94
42.33
31.45
69.26
43.46
48.49
E2E-FT (CVPR 22)
76.21
64.21
39.62
20.35
57.38
39.63
44.24
Weight ens, (CVPR 22)
77.62
66.86
45.67
28.21
66.68
44.72
50.43 (-6.19)
VRF (NeurIPS 24)
77.60
66.70
47.00
29.20
70.90
46.30
52.02 (+7.78)
+ TIES (NeurPPS 23)
77.20
65.88
46.20
28.12
69.21
44.56
50.79 (4655)
DARE (IMLL 24)
77.43
66.32
46.49
28.55
69.75
45.13
51.25 (47.01)
MERGETUNE + Weight ens.
77.81
67.02
46.83
31.80
70.67
47.01
52.67 (+8.43)
77.27
66.59
46.87
32.40
71.09
46.99
52.79 (48.55)
Table 6: Base-to-novel generalisation experiments across di lifferent vision-language models on 11 datasets. Our method achieves consistent performance improvement over CoOp baseline across all model architectures.
Average
ImageNet
Caltech101
OxfordPets
Method
Novel
Novel
CLIP-LI4
CoOp
CoOp + MERGETUNE
85.90
78.70
82, (1..0)
81.93
75.57
78.62
97.39
98.00
97.79
96.64
98.32
97.47
Siglip2-B16
CoOp
78.31
79.02
78.66
79.97
77.38
78.66
98.52
97.82
98.17
95.85
97. 97
96.90
CoOp MERGETUNE
80.26
Siglip2-L16
CoOp
81.56
83,31
82.43
84.01
98.03
97.79
96.
98.25
CoOp + MERGETUNE
83.04
84.43
83.73 vi mm
84.31
80.42
82.32
98.90
98.
98.56
97.33
98 98.41
97.87
StanfenCarr
Flowers 102
Food101
FGVCAireraf
Method
Base
Novel
HM
Base
Novel
Base
Novel
Base
Novel
CLIP-L14
CoOp
82.72
82.89
82.81
98.96
75.04
85.35
93.46
93.55
93.51
52.64
32.95
40.53
CoOp+ MERGETUNE
83.14
97.59
79.77
87.79
Siglip2-B16
CoOp
86.18
96.23
90.93
89.68
85.37
87.47
92.24
93.42
92.83
29.87
39.79
34. 12
CoOp MERGETUNE
86.24
96.26
90.98
92.40
86.33
89.27
92.49
93.51
35.25
Siglip2-L16
CoOp
90.34
97.58
93.82
92.72
87.28
89.91
94.75
95.51
95.13
35.91
54.63
43.34
CoOp+ MERGETUNE
90.66
97.50
93.96
93.42
87.34
90.28
95.11
95.53
95.32
36.03
59.23
44.80
SUN397
DTD
EurSATT
UCF101
Method
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP-LI4
CoOp
84.67
72.33
78.02
82.49
59.50
69.
93.40
64,94
76.61
87.63
74.16
80,33
CoOp MERGETUNE
73.31
79.10
80.82
62.45
70.46
92.33
Sig
CoOp
83.27
70.81
76.54
77.35
69.81
73.39
56.75
58.17
57.45
71.72
82,40
76.69
CoOp MERGETUNE
78.09
79.98
72.06
75.81
Siglip2-1.16
CoOp
84.97
74.11
79.17
78.93
72.62
75.65
61.26
70.10
65.38
78.16
87.56
82.59
CoOp MERGETUNE
85.87
75.16
80.16
81.10
78.26
79.65
70.56
70.59
70.58
80. 17
88.05
83.92
A.4.2 HYPERPARAMETER EVALUATION
We evaluate the sensitivity of MERGETUNE to key hyper-paramene parameters,, including surrogate loss weight X. Task2 LMC loss weight 8, and the iitialiisation parameter of the continued model. Here is the detailed experimental results. Detailed experimental results are provided in Table 7 and Table 8.
16

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter sensitivity analysis: reported accuracies are averaged over datasets, across surrogate loss weight À and Task2 LMC loss weight B.
Parameters
Performance (%)
À
B
Avg Basc Ace
Avg New Acc
Avg Harmonic Mcan
0.1
82.86
70.94
76.44
1
0.5
82.74
71.66
76.80
82.89
70.22
76.03
1
2
82.75
70.67
76.23
82.38
72.90
77.35
4
0.5
82.47
72.15
76.97
4
1
82.64
71.94
76.92
4
2
82.70
71.63
76.77
6
0.1
82.31
73.34
77.57
6
0.5
82.41
72.76
77.28
82.56
72.53
77.22
2
82.80
72.43
77.27
82.04
73.45
77.51
8
0.5
82.11
73.60
77.62
8
1
82.39
73.18
77.51
2
82.64
72.11
77.02
10
0.1
81.68
73.64
77.45
10
0.5
82.02
73.31
77.42
10
82.20
73.29
77.49
10
2
82.58
72.72
77.33
16
80.96
74.06
77.36
16
0.5
81.42
73.82
77.43
16
1
81,65
73.76
77.51
16
2
82.10
73.43
77.52
20
0.1
79.93
74.27
77.00
20
0.5
80.54
74.18
77.23
20
80.88
73.95
77.26
20
2
81.78
73.55
77.45
Table Initialisation analysis continued model: performance comparison across datasets with
and 3 = 0.5. Here, = 0 corresponds to initialising the continued model with the CLIP weights, while 7 = corresponds to the finetuned KgCoOp weights.
T
Base Ace
New Acc
Harmonic Mean
0.0
81.24
73.07
76.94
0.1
81.47
73.01
77.01
0.2
81.84
73.33
77.35
0.3
82.11
73.60
77.62
0.4
81.91
73.41
77.43
82.01
73.34
77.43
0.6
82.10
73,42
77.51
0.7
82.07
72.90
77.21
0.8
82.15
73.13
77.37
0.9
82.03
73.07
77.29
82.16
72.87
77.24
17

---

Published as a conference paper at ICLR 2026
A.4.3 SENSITIVITY TO INTERPOLATION NUMBER Na
To better understand our MERGETUNE, conducted additional analysis. In Table 9, we varied the number of interpolation points No. which is used to approximate the LMC expectation term. Perfor- mance improves consistently as Na increases from 10, after which it saturates. Meanwhile, the training time gradually increases with scaling Na up: when using KgCoOp+MERGET about 3x that of KgCoOp at N, = 5, increases to Thus, although N₂ = 10 provides slight further improvement in accuracy, the associated overhead more noticeable, making Na a more balanced choice.
the cost N₂, 10, and reaches N, =
Table Effect of the number of interpolation points (a) in the LMC approximation. Results averaged datasets using MERGETUNE with KgCoOp weights. Performance improves steadily as increases from to 5, beyond which accuracy saturates and further increases in a only raise fine-tuning cost.
Number Interpolation Points (N,)
Base Acc
New Acc
Harmonic Mean
81.75
73.12
77.19
3
82.01
73.51
77.53
5
82.11
73.60
77.62
10
82.13
73.71
77.69
15
82.13
73.72
77.70
A.4.4 PERFORMANCE OF INTERPOLATED MODELS
We also evaluate the interpolated models between our continued model and the original zero-shot CLIP, as well as between and the initially fine-tuned KgCoOp model the base-to-novel setting. As shown in Figure 4, the results demonstrate smooth, low-loss (high-accuracy illustrated) paths between our continued model and both endpoints. Interpolation with the fine-tuned KgCoOp (blue line) shows stable, gradually improving base-class performance, while interpolation with zero-shot CLIP (orange line) maintains consistent, slightly improving novel-class performance throughout. The two smooth trajectories without performance fluctuations indicate that our continued model re- sides in connected region of the loss landscape that spans both endpoints, confirming that MERGE- TUNE successfully establishes linear mode connectivity and effectively integrates knowledge from both the zero-shot and fine-tuned solutions.
R2
Avg Harmocic Mem
70-
fine-tuned continue, base performance
78
zoro-shot clip - consinued, novel class performance
27-
75-
74
0.0
1.0
Interpolation Weight
Figure 4: Linear mode connectivity analysis on base-to-novel generalisation. We interpolate between tinued model and both the zero-shot CLIP model (orange line, novel-class performance) and the fine-tuned KgCoOp model (blue line, base-class performance). The smooth paths confirm that MERGETUNE success- fully establishes linear mode connectivity with both endpoints, maintaining strong performance throughout interpolation. Results averaged over 11 datasets.
A.4.5 WILL CONTINUED FINETUNING INTRODUCE OVER-MERGING?
To evaluate potential over-merging from our continued finetuning, we analysed MERGETUNE's performance across all 11 datasets, ranging from 10 to 100 epochs using CoOp MERGETUNE
18

---

Published as a conference paper at ICLR 2026
100
DM
Harmalic Mean (%)
Beosl
70
Peediot
Inuaree
Daftad Rlwer
Oafied Pes
Stafeed Cas
Ste277
Littn
40-
Continued Finetuning Epochs
Figure 5: MERGETUNE exhibits over-merging across extended training (ranging from 10 to 100 epochs). Performance trajectories across all 11 datasets show either stable performance continuous improvement without degradation. No performance deeline validates its robustness against over-merging through our dual- anchored linear mode connectivity objective.
on Siglip2-B/16 (as in Figure 5). Results exhibit two primary patterns: stable performance (on Caltech101, Food101, Oxford Pets, and ImageNet) and continuous improvement (on DTD, Ox- ford Flowers, Stanford Cars, UCF101, EuroSAT, and FGVC Aircraft). Across all datasets, no performance degradation is observed during continued finetuning, confirming that our linear-mode connectivity-guus guiddd model merging does lead over-merging, thanks the effective regulari- sation applied to both zero-shot and fine-tuned endpoints.
19


Reviews:
[{"review_id": "7bZ9f6jJBB", "reviewer": "Reviewer_wfKc", "summary": "The paper proposed a novel model merging method named Continue fine-tuning, which utilizes the low-loss path phenomenon in the model merging to effectively combine knowledge from the pre-training model and the fine-tuned version, achieving high performance across various tasks, models, and fits well with different tuning methods.", "questions": "- Does the low-loss path phenomenon always exist, no matter how complex the models are? Since the more recent VLM architecture has become increasingly complex, handling complex real-world tasks that generalize across various tasks, the loss space will become much more complicated. I am curious whether such LMCs discovered at an early age still affect?\n- How is Figure 2 constructed? Is it just an illustration or sampled from the training process? Could the author provide more detail on the illustration part.\n\nI will raise my score once my concerns are solved.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework shows a strong performance gain compared to the simple model merging baseline\n- Achieving competitive performance with low inference cost\n- Theoretical analysis on the low-loss path phenomenon and its relation with easing catastrophic forgetting is interesting.", "weaknesses": "- While overall performance gain looks good, the improvement on the different tuning methods is unstable, sometimes hurts the base performance while largely improves the novel class, and sometimes degrades the novel performance while the base performance becomes better. This indicates the instability of the proposed framework\n- Figure 1 is really unfriendly for the reader. I strongly suggest that the author update the scale for different datasets to emphasize the performance difference between models instead of packing them together...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l066dJiyuz", "reviewer": "Reviewer_wRZr", "summary": "This paper proposes MERGETUNE, a continued fine-tuning framework for vision-language models such as CLIP. Instead of preventing catastrophic forgetting during adaptation, the authors attempt to recover pre-trained knowledge after fine-tuning has already taken place. The key idea is to exploit linear mode connectivity (LMC) to find new parameters with two low-loss connections to the zero-shot and fine-tuned checkpoints, effectively merging their knowledge. A second-order surrogate term approximates the pretraining loss to avoid data replay from the pretraining task. The method is model-agnostic and post-hoc, requiring no architectural modifications. Experiments on base-to-novel, cross-dataset, domain generalization, and robust fine-tuning benchmarks show consistent improvements across CoOp, KgCoOp, MMA, and PromptKD, achieving state-of-the-art results with minimal overhead. Nevertheless, I am not an expert in VLMs, so the following comments and questions could be based on potentially inaccurate understandings or biased perspectives.", "questions": "1. How sensitive is MERGETUNE to violations of the isotropic curvature assumption ($H_1 \\approx \\mu I$)?\n\n2. What is the additional training or computational overhead compared to standard fine-tuning?\n\n3. Could continued fine-tuning lead to “over-merging,” where downstream adaptation performance is harmed after too many epochs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n\n1. Introduces a novel post-hoc fine-tuning paradigm focusing on knowledge restoration instead of prevention, offering a new conceptual direction.\n\n2. The approach is simple, elegant, and general, requiring no model modifications and applying to various adaptation methods.\n\n3. Using linear mode connectivity as an explicit optimization objective provides solid geometric intuition and interpretability.\n\n4. Experimental evaluation is extensive and convincing, demonstrating consistent gains across diverse datasets and settings.", "weaknesses": "1. The theoretical justification is limited, and it is unclear why continued fine-tuning converges to a mode-connected region.\n\n2. The final objective closely resembles standard $L_2$-regularized fine-tuning, so the novelty might be overstated.\n\n3. The isotropic Hessian assumption in the surrogate loss is strong and unvalidated; its practical effect remains unclear.\n\n4. The paper does not quantify the extra training cost or test scalability on larger or multimodal models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3lNFF37zkt", "reviewer": "Reviewer_uAR8", "summary": "The paper proposes MERGETUNE, a novel *continued fine-tuning* (CFT) framework that recovers pretrained knowledge in vision-language models (VLMs) after standard fine-tuning has already occurred. Unlike prior work that attempts to mitigate forgetting during adaptation, MERGETUNE operates post hoc by leveraging linear mode connectivity (LMC) to find a model that maintains low-loss interpolation paths to both the original zero-shot model (e.g., CLIP) and the fine-tuned model (e.g., CoOp). To avoid the infeasible requirement of replaying the massive pretraining data, the authors introduce a second-order surrogate loss that approximates the LMC constraint for the zero-shot model using a simple ℓ² regularizer toward the pretrained weights. The method is model-agnostic, requires no architectural changes, and consistently improves performance across multiple VLM adaptation strategies (prompt-based, adapter-based, linear probing, full fine-tuning) and evaluation protocols (base-to-novel, cross-dataset, domain generalization, robust fine-tuning). Notably, MERGETUNE achieves a +5.6% harmonic mean gain over CoOp without adding parameters and outperforms CLIP in all evaluated settings, with further gains possible via ensembling.", "questions": "Please weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) Introduces a conceptually fresh and practical paradigm—continued fine-tuning—that decouples adaptation from knowledge recovery, enabling post hoc enhancement of any existing fine-tuned VLM.  \n(2) Proposes a theoretically grounded yet simple method (MERGETUNE) based on linear mode connectivity, with a clever second-order surrogate that eliminates the need for pretraining data replay—a major practical bottleneck.  \n(3) Demonstrates consistent and significant improvements across diverse adaptation methods (CoOp, MMA, PromptKD, etc.) and evaluation settings (few-shot, many-shot, OOD robustness), validating the generality of the approach.  \n(4) Provides comprehensive empirical analysis, including ablation studies on hyperparameters, comparisons with training-free merging (TIES, DARE) and ensembling baselines (VRF, Wise-FT), and shows MERGETUNE reduces inference cost while improving performance.", "weaknesses": "(1) The surrogate loss assumes the Hessian of the pretraining loss at the zero-shot checkpoint is isotropic (H₁ ≈ μI) and that the gradient is near zero. While common, this may not hold for large-scale VLMs like CLIP trained on noisy web data. Could the authors provide empirical validation of these assumptions (e.g., via Hessian spectrum estimation on a subset) or discuss how violations might affect MERGETUNE’s performance?  \n(2) In Table 1, MERGETUNE improves CoOp by +5.58% HM but only +0.36% on PromptKD. The paper attributes this to PromptKD already preserving more knowledge. However, is it possible that MERGETUNE’s ℓ² regularizer toward CLIP inadvertently conflicts with PromptKD’s distillation objective, limiting gains? An analysis of the weight trajectory or feature similarity between CLIP, PromptKD, and MERGETUNE-enhanced PromptKD would help clarify this.  \n(3) The method samples a small number of α values (e.g., 5–10) to approximate the expectation in the LMC loss. How sensitive are results to the number and spacing of these α points? Did the authors experiment with adaptive sampling or importance weighting along the interpolation path to better capture high-loss regions?  \n(4) For adapter-based methods like MMA, which modify model architecture, MERGETUNE is applied only to the linear head (per Section 5.2). This seems inconsistent with the claim of being “model-agnostic.” Can MERGETUNE be applied to the full adapter parameters? If not, what architectural constraints limit its applicability, and how does partial application affect knowledge recovery?  \n(5) The continued model is initialized as a convex combination of w̃₁ and w̃₂ (τ ∈ [0,1]). Figure 3(b) shows optimal τ ∈ [0.3, 0.6], but this requires tuning. In real-world deployment, the ideal τ may be unknown. Does MERGETUNE include a practical strategy for selecting τ without validation data (e.g., based on fine-tuning loss magnitude or forgetting estimates)?  \n(6) All experiments use CLIP ViT-B/16 (and briefly ViT-B/32). How does MERGETUNE scale to larger backbones (e.g., ViT-L/14) or non-CLIP VLMs (e.g., ALIGN, BLIP)? Given that mode connectivity properties can vary with model size and training data, it would strengthen the paper to show results beyond the standard CLIP setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zvrxXZmafx", "reviewer": "Reviewer_Poyv", "summary": "This paper addresses the problem of catastrophic forgetting in fine-tuned vision-language models (VLMs). The authors propose a new paradigm called Continued Fine-Tuning (CFT), which aims to recover pretrained knowledge after a model has already been adapted, rather than just mitigating forgetting during adaptation. Extensive experiments on CLIP are conducted to verify the effectiveness.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n\n2. The experiments are comprehensive.", "weaknesses": "1. Some assumptions are overly strong and insufficiently discussed. The derivation of the second-order surrogate is not entirely convincing. For example, for smaller models such as ViT-B/16, it is questionable whether the gradients of the trained model are close to zero, as the authors assume. Additionally, the assumption \n𝐻1≈μI appears too strong.\n\n2. Comparing a training-free method with training-based methods seems unfair, as they operate under fundamentally different settings.\n\n3. The performance improvement is limited. The proposed method requires significantly more training time to train a new model, yet the gains on downstream tasks are modest. In the base-to-new setting, it even underperforms compared to ATPrompt.\n\n4. There is no discussion of training time or computational cost. It would be helpful to include an analysis of the training cost of the proposed method.\n\n5. The paper claims that the proposed method is designed for vision-language models, yet only CLIP is evaluated. A broader experimental validation would strengthen this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)