DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation
Authors: Changdae Oh
Yixuan Li
Kyungwoo Song
Sangdoo Yun
Dongyoon Han
Venue: ICLR 2025 Poster (2025)
Keywords: robustness
distribution shift
robust fine-tuning
weight interpolation
model merging
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Adapting a pre-trained foundation model on downstream tasks should ensure robustness against distribution shifts without the need to retrain the whole model. Although existing weight interpolation methods are simple yet effective, we argue their static nature limits downstream performance while achieving efficiency. In this work, we propose DaWin, a training-free dynamic weight interpolation method that leverages the entropy of individual models over each unlabeled test sample to assess model expertise, and compute per-sample interpolation coefficients dynamically. Unlike previous works that typically rely on additional training to learn such coefficients, our approach requires no training. Then, we propose a mixture modeling approach that greatly reduces inference overhead raised by dynamic interpolation. We validate DaWin on the large-scale visual recognition benchmarks, spanning 14 tasks across robust fine-tuning -- ImageNet and derived five distribution shift benchmarks -- and multi-task learning with eight classification tasks. Results demonstrate that DaWin achieves significant performance gain in considered settings, with minimal computational overhead. We further discuss DaWin's analytic behavior to explain its empirical success.

Published as a conference paper at ICLR 2025
DAWIN: TRAINING-FREE DYNAMIC WEIGHT INTER- POLATION FOR ROBUST ADAPTATION
Changdae Oh" Yixuan Li', Kyungwoo Song"1, Sangdoo Yun Dongyoon Han University of Wisconsin-Madisonn Yonsei University SNAVER Lab
(changdae, sharonliecs wisc.edu) kyungwoo.. wttp:/oossooogoosei....oo ac.kr {sangdoo. yun, dongyoon www.:eewwwawwo.c.ccoooooo
ABSTRACT
Adapting pre-trained foundation model on downstream tasks should ensure ro- bustness against distribution shifts without the need to retrain the whole model. Although existing weight interpolation methods are simple yet effective, we ar- gue their static nature limits downstream performance while achieving efficiency. In this work, we propose DaWin, a training- free dynamic weight interpolation method that leverages the entropy of individual models over each unlabeled test sample to assess model expertise, and compute per-sample interpolation coeffi- cients dynamically. Unlike previous works that typically rely on additional train- ing to learn such coefficients, our approach requires no training. Then, we propose a mixture modeling approach that greatly reduces inference overhead raised by dynamic interpolation. We validate Dawin on the large-scale visual recognition benchmarks, spanning 14 tasks across robust fine-tuning ImageNet and derived five distribution shift benchmarks - and multi-task learning with eight classifica- tion tasks. Results demonstrate that DaWin achieves significant performance gain in considered settings, with minimal computational overhead. We further discuss DaWin's 's analytic behavior to explain its empirical success. Here our code.
INTRODUCTION
The emergence of foundation models (Bommasani al., 2021; Radford al., 2021; Brown al., 2020) has significantly lowered the barrier to deploying artificial intelligence solutions across a wide range of real-world problems. Leveraging the strong general knowledge acquired through large-scale pre-training, foundation models can be efficiently adapted for numerous tasks. However, recent studies have shown that while fine-tuning improves performance on specific downstream tasks, it may often undermine the model's generalizability and robustness (Wortsman et al., 2022b). For example, a model tne-ttned on ImageNet has better accuracy on in-distribution (ID) data yet may underperform in
(OOD) data such ImageNet-A (Hendrycks et al., 2021b).
To address this issue, robust fine-tuning methods (Wortsman et al., 2022b) have been recently devel- oped to adapt models to ID while maintaining strong OOD generalization. Some approaches incor- porate regularization into the learning objective (Ju et al., 2022; Tian et al., 2023a; Oh et al., 2024), while others focus on preserving the knowledge of the pre-trained model by modifying fine-tuning procedure (Kumar et al., 2022; Lee et al., 2023; Goyal et al., 2023). Notably, weight interpolation approaches allow for the integration of knowledge from multiple models via simple interpolation or averaging and have proven effective in both robust fine-tuning (Wortsman et al., 2022b;a; Jang et al., 2024) and multi-task. learning (Ilharco al., 2022; Yadav al., 2023; Yu et al., 2024) settings.
The weight interpolation approaches are particularly appealing because they are easily applied to any fine-tuned model as post-hoc plug-in method, delivering competitive performance. Most existing works (Wortsman et al., 2022b;a) focus on creating a single merged model using static global interpolation coefficient for all test samples: (1 2)0g + A0,, where Og and 0, are the weights of two individual models. While these methods efficiently achieve strong performance, we argue that the optimal coefficients vary across input data samples, leaving notable potential for improving the performance of the interpolation baase approaches. Recent studies on dynamic merging (Cheng et al., 2024; Lu et al., 2024; Tang et al., 2024) explore sample- wise interpolation with (1 (1-A(x))0p)
WWok done during an internship NAVER AT Lab. Equal correspondence.

---

Published as a conference paper at ICLR 2025
X(x))1, which introduce extra learnable modules to replace the global coefficient with sample- sample-wise coefficient A(x) from a given sample æ. Yet, these methods require additional training and careful design of the router modules to determine A(r), which brings non-trivia complexity.
In this work, we propose a dynamic weight interpolation framework, DaWin, that performs sample- wise interpolation without any additional training. To begin with, we conduct a pilot study in- vestigate the upper-bound performance of dynamic interpolation methods. Specifically, we simulate an oracle sample-wise weight interpolation by leveraging ground truth test labels for sample-wise model expertise estimation via the cross-entropy (X-entropy) loss. Given a test sample, a model yielding a smaller X-entropy incurs larger importance for that sample during weight interpolation, reflecting the expertise of the corresponding model. Here, we observed that fine- grained dynamic interpolation (e.g., at a sample level) indeed significantly outperforms static interpolation.
Our pilot study informatively guides our methodology design by approximating the upper-bound performance in practical scenarios when the ground truth labels of test samples are not available. Specifically, we design entropy ratio-based score function that can act reliable alternative to the X-entropy ratio to robustly determine the sample-wist interpolation coefficients across different samples from diverse domains. The rationale behind using entropy the observation that the entropy of a sample's 's predicted probability distribution strongly correlates with X-entropy. Lastly. to resolve the computation overhead induced by sample-wise interpolation operation during inference time, we further devise a mixture modeling-hased (Ma & Leijon, 201 1) coefficient clustering method that dramatically reduces the computation. Compared with the most competitive baseline, DnWin improves the performance by 4.5% and 1.8% in terms of the OOD ac- curacy and multi-task learning average accuracy for CLIP (Radford et al., 2021) image classification, even though DaWin requires far less computational cost during the training or inference.
surrogate is grounded in
Our contributions can be summarized in three key points:
i. We present an intuitive numerical analysis of oracle dynamic interpolation methods and show that the X-entropy ratio is a reliable metric to compute per-sample interpolation coefficients.
i. We propose a practical implementation, DaWin, that approximates the oracle dynamic interpo- lation by leveraging the prediction entropy ratio of individual models on unlabeled test samples to determine sample-wise interpolation coefficients automatically.
iii. Extensive validation shows that DaWin consistently improves classification accuracy distri- bution shift and multi-task learning setups while not remarkably increasing the inference time, and we provide theoretical analysis to explain the empirical success of DaWin.
PRELIMINARY
2.1 BACKGROUND
Classification under distribution shift. We consider a classification problem over a domain X' with input x and a label space y= = {I,..., C}, where the goal is to approximate the true labeling function h : X' + y with a parametric model f X + A C-1 This model maps x to C dimensional simplex, which aims to minimize error ) y R inputs drawn from any potential target distribution Py. In the robust fine-tuning scenario (Wortsman et al., 2022b; Kumar al., 2022), the target distribution Pr, where a pre-trained model f being adapted to, is typically assumed to be covariate-shifted version (OOD) of the ID source distribution Ps. Both distributions share the same class label space V, and have the same conditional distribution over target labels, i.e., Ps (s(ylr) (v)) but have different marginal distributions over input Ps(x) A (r))..
Model merging. Let f(::0) and f(; 01) denote models that are individually trained on the same or different datasets but have identical architecture. For example, f(; (g) could represent a pre-trained model such as CLIP (Radford et al., 2021), while f(: 01) is the fine-tuned counterpart on a particular downstream task. Model merging approach (Wortsman et al., 2022b;a) constructs merged model f(:00) which achieves a better trade-off between ID and OOD performance than the individual models by interpolating in the weight space 0₂ (1 A)0o + 1O, We will use the terms inter- polation and merging interchangeably. Throughout the paper, we use the term static interpolation to denote methods that induce single merged model corresponding to single interpolation coef- ficient applied for all test samples. In contrast, dynamic interpolation refers to methods that yield multiple merged models, with interpolation coefficients depending the sample domain X.
2

---

Published as a conference paper at ICLR 2025
Table 1: Pilot experiments for upper-bound analysis. We evaluate zero-shot (ZS), line-tuned (FT), and several interpolation methods with CLIP ViT- on ImageNet (IN) and its five distribution shifts: ImageNet (IN)-V/R//AS and ObjectNet (ObjNet). Given test domain X. domain-wise coefficients X (2t) are found by grid search over cach test set, and the sample-wise coefficients (x) for cach test sample x are determined by the X-entropy loss ratio of individual models. f denotes oracles that utilize ground truth test labels.
Method
Model Weight
Acc. Under Distribution Shifts
IN
IN-V2
IN-R IN-A
IN-S
Obj
Avg
7S (Radford al., 2021)
0o 01
63.4
55.9
69.3
31.4
42.3
43.5
48.5
FT (Wortsman et al., 2022b)
78.4
67.2
59.3
24.7
42.2
42.0
47.9
WiSE-FT (Wortsman al., 2022b)
(1 X16 + A01
79.1
68.4
65.4
29.4
46.0
45.9
51.0
Dynamic Interpolationt (domain)
(1 (0 + X (x
79.1
68.5
72.9
36.3
48.5
48.9
55.0
Dynamic Interpolationt (sample)
(1 - X2 (x))) + X (x)01
83.4
74.4
77.9
42.9
53.4
54.6
60.6
IN
IN-V2
IN-R
IN-A
IN-S
ObjNet
Confirest
74
Confiione
Distribution of interpolation coefficients. Histograms of sasple-wise interpolation coefficients induced by the X-entropy ratio between two models (ZS and FT). Oracle coefficient estimates vary significantly within domain (IN) and across domains (IN v.s. IN-V2/R/AIS/OHjNee regarding symmetry and skewness.
Figure 1:
2.2
PILOT STUDY
Our goal. We begin by conducting a pilot study to understand the benefits of dynamic interpolation, which adapts interpolation coefficients at a finer granularity (such as sample level, X(x)), as opposed to using global coefficient À for all samples (Wortsman al., 2022b;a). To explore the upper limits of these methods, we experiment with ground truth labels as an oracle' to estimate upper-bound performance and understand the maximum potential of model interpolation methods. This study will further guide our method design in Sec. 3 by approximating the oracle performance.
Setup and hypothesis. We assess the top-1 classification accuracy of several approaches on ImageNet-IK (IN) (Russakovsky et al., 2015) and distribution-shifted benchmarks IN- V2/R/A/S/ObjNet (Recht et al., 2019; Hendrycks al., 2021a;b; Wang et al., 2019; Barbu et al., 2019) by fine-tuning the CLIP ViT-B/32 (Radford et al., 2021) IN. Besides the individ- ual models (zero-shot; ZS and fine-tuned; FT), we include a representative static interpolation method, WiSE-FT (Wortsman et al., 2022b), which determines the interpolation coefficient À via grid search on the ID validation set. Then, we implement two oracle interpolation methods: Dy- namic Interpolationf per domain and per sample. For the domain wise dynamic interpolation, acle coefficients (X') are determined by grid search over all test samples within each domain X, such as art or sketch. In contrast, for sample-wise oracle interpolation coefficients (*(x), we use negative X-entropy to measure models' expertise on specific input which results in (z)x) = exp(-I(f(e))) 01), ())/(exp(-((((a)) fo). y)) + exp(-i(f(x: 01), y))). Note that we can regard the model [z; 0)) as an estimator of a conditional probability density function, e.g., Po ()). Then, given the equality between the X-entropy over the one-hot label and negative log-likelihood, posterior of Bernoulli distribution in the noise-contrastive estimation (Gutmann & Hyvärinen, 2010), i.e., X*xx) = pe, (y (ylz))((pe) (w|z)) + pe, (y(z)), which models the probability whether a data (I, v) comes from the distribution po, (y/lr). See Appendix A. for more.
(z) can be interpreted
In this experiment, we hypothesize that (1) Fine-grain interpolation, which adapts coefficients at finer level (such as the sample level), can lead to substantial improvements compared to static interpolation, such as WiSE-FT. (2) Per-sample interpolation coefficients can effectively estimated with X-entropy-baate based model expertise measurement.
WWe use the term oracle to denote scenario where we utilize the ground truth test label, and it should not he confused with optimal. Optimality of X" (xx) regarding weight interpolation is further discussed in Tab C. We employed the pre-trained model available https: / / thub. com/ opena /CLIP.
3

---

Published as a conference paper at ICLR 2025
TrueTrue
Truefalse
p-04
6-821
p- 0.086
p 0.56
0.9
RVF 1i 2) Rireee
P-05
M0 R001
Figure 2: Correlation between entropy ratio and X-entropy ratio. We split each test set into four subsets (TrueTrue, FrueFalse, FalseTrue, FalseFalss aeee based on the correctness of two models' predictions (ZS and FT). On each split and the entire data, show scatter plots of the entropy ratio (x-axis) and x-entropy ratio (y-axis) computed by models with corresponding Pearson correlations. The top and bottom rows denote results from ImageNet and ImageNet-R. Results imply that the entropy ratio strongly correlates with X-entropy ratio overall, except FalseFall al case, wherein we could not expect satisfactory performance for individual models and interpolation methods as inherent limitation given fixed individual models.
Results and interpretations. We verify our hypothesis in Table Here, the Nomain-wiss. and sample-wise dynamic interpolation methods perform far better than individual models or static in- terpolation. Moreover, compared with the domain- wise interpolation method, sample-wise interpo- lations show much higher performance in all cases. This supports interpolation leads to better downstream accuracy, and the negative X-entropy can be used as a reliable estimator of per-sample model expertise to induce the interpolation coefficients.
hypotheses that fine- -re-graii
In Figure 1, the interpolation coefficients X (x), computed using the X-entropy ratio (corresponding to the last row of Table 1), vary significantly within and across domains. On the one hand, the timated coefficients are widely distributed from zero to one on all evaluation datasets. On the other hand, the distributions of coefficients also vary depending on the domain, e.g., IN and IN-A hibit right-skewed and left-skewed distributions, respectively. This confirms that optimal coefficient estimation varies by sample, motivating the exploration of the sample-wisg wise dynamic interpolation method. In summary, we conclude that determining proper interpolation coefficients on a sample- wise basis dramatically elevates the achievable performance of model merging- based approaches.
3 METHOD
3.1 REVISITING ENTROPY AS A MEASURE OF MODEL EXPERTISE
While the pilot study in the previous section provides encouraging results, those oracle methods cannot work in practice. This is because do not have access the ground truth labels for incoming test samples. This leads us to the question: how can we reliably estimate the interpolation coefficient solely based on the test input x? There extensive literature which adopts entropy as proxy of X-entropy (Grandvalet & Bengio, 2004; Chapelle & Zien, 2005; Shu et al., 2018; Wang al., 2021; Prabhu et al., 2021). The rationale behind using entropy as a surrogate is grounded in the observation that the entropy of a sample's prediction distribution strongly correlates with X- entropy, even under distribution shifts (Wang et al., 2021) those models have not explicitly trained on. This work presents the first attempt to leverage the sample-wise entropy to measure each model's expertise to determine the interpolation coefficients given each test sample. It is worth noting that this approach differs from AdaMerging (Yang et al., 2024b), which seeks to learn global coefficient inducing a single merged model to minimize the expected entropy over the entire test set.
In this section, we will show that entropy is well-correlatdd with X-entropy and can thus be used to estimate model expertise. Specifically, we divide the test set into four splits, [TrueTrue, TrueFalse, Fa FalseTrue, F FalseFalss slse), based the correctness of predictions made by two models (zero-shot and fine-tuned CLIP). For example, TrueTrue represents the set ((x,, I 0o) yi and ((x;01) 3: We then compute the Pearson correlation coeffi-
4

---

Published as a conference paper at ICLR 2025
(01))
1- 1x)(x)
A(x)
ll
0.4
0205)
0.4
+
0.9
8,
f(x B2(x,)
x1
(()
111 0.6
0.4 0.9
0.4 + 0.9
LEE 0.8
B2(8))
0.6
8b
0.3
8
f(x2) O2cxa)
+ 92
X2
exp(-M (()
0.6 + 0.3
0.6 16++0.. 0.3
0.9
((:8,)
0.3
(Atra)
0.8
0o 0.8
0.2
81
f(xs: B20mn)
P3
1E - 0.2
0.8 + 0.2
0.2
test samples
expertise estimation
dynamic interpolation
Figure 3: Framework overview. DaWin estimates per sample model expertise (exponentiated negative en- tropy) and then produces coefficients based on the relative expertise of models for dynamic interpolation.
cients between entropy and X-entropy within each split. Figure 2 shows scatter plots with fitted regression lines of the entropy and X-entropy ratios between two models on IN (top) and IN-R (bot- ((((x:o)la)
H[[(w:0o)) tom). Specifically, we plot +[ and [U((=:))),))++(([1ee)))) for entire test samples and compute Pearson correlation coefficient between them, where H denotes the entropy.
Here, aside from the FalseFalss case where both models failed to produce reliable predictions?, we observe strong correlations (Schober et al., 2018) across the remaining splits and the entire dataset. This implies that entropy is reasonable proxy to the X-entropy for estimating per-sample model expertise and, ultimately, computing sample-wise interpolation coefficients.
3.2 DYNAMIC INTERPOLATION VIA ENTROPY RATIO
After confirming strong correlation between the entropy and X-entropy, we propose a new dynamic weight interpolation, DaWin, by defining sample-wise interpolation coefficients A(x) as below:
exp( (<p(-H(((=000))) X(x) exp(-H(f((;; (o))) + exp(=H(f(=;01)
(1)
Here, H represents the entropy over the output probability distribution of model ((: given input x. The value of X(x) approaches I when model f(::0) 0,) exhibits lower entropy (greater expertise), whereas it approaches O if the entropy of model (()=0)) becomes higher. Figure 3 illustrates the overall framework. Intuitively, this approach is analogous to classifier selection methods (Giacinto & Roli, 2001; Ko al., 2008), which aims to dynamically select suitable classifier(s) given a test sample. However, DaWin differs in terms of its motivation, expertise metric, and goal as we pursue finding the interpolation coefficients that may induce better performance compared with the best model selection method (See Table 5). We further discuss the connection between DaWin and the selection- basod method in Sec. 6. Note that our sample-wise expertise estimations do not require any hyperparameters for computing interpolation coefficients. This eliminates the need for inten- sive hyperparameter tuning often required in prior works (Wortsman et al., 2022b; Ilharco et al., 2023) to achieve the best result. Meanwhile, if we assume access to all test samples simultancously, we can refine some unstable coefficient estimations by introducing per-domain expertise offset. We use this offset adjustment by default (Sec. A. Besides, to bypass using overconfident output (Guo et al., 2017) during expertise estimation, we conduct the temperature scaling on ID validation set.
Sample-wise entropy valley hypothesis. One of the most popular hypotheses that explain the success of weight interpolations is linear mode connectivity (LMC; Frankle et al. (2020)), which ensures that interpolated models are also laid on good solution space as well as low-loss converged individual models. Although entropy is a concave function for the probability density, we believe that a similar statement can be claimed, which we refer to sample-wise entropy valley as follows:
i) There exists À such that E. (f X0 + (1 x)01)] < m (x; 0u)) E. [H 0x))))
Given À obeyed also exists A(x) such that E,, (f A(z)00 + (1 A(=))0 < (f] AO. + (1 x)01)]-
and evidence of sample-wise linear feature connectivity (Zhou et al., 2023), there
As empirical validation of this hypothesis, Figure 6 and Figure H show that the interpolated model actually induces smaller entropy than individuals (i) average, and DaWin achieves lower tropy than static interpolation (ii). Given the strong correlation between entropy and X-entropy, this hypothesis advocates the benefit of DaWin than static interpolation methods.
However, even individuals fail, interpolated model can yield correct predictions (Yong et al., 2024).
5

---

Published as a conference paper at ICLR 2025
3.3 EFFICIENT DYNAMIC INTERPOLATION BY MIXTURE MODELING
Although our primary goal is to achieve better downstream task performance by way of dynamic in- terpolation, performing interpolation every sample inevitably increases the inference-time com- putation, with the cost scaling linearly with the number of model parameters (Lu et al., 2024). To ensure practicality while pursuing state-of-the-att downstream task performance, we devise an effi- cient dynamic interpolation method by leveraging mixture modeling.
Let { denote the N test samples and {M(z)) the corresponding interpolation coefficients computed by DaWin framework. Note that X(z;) E [o, 1] for all i ,,..., .N, and they can be regarded as sampled observations from some Beta distributions. Then, we accurately model the probability density function of interpolation coefficients via Beta mixture model' as follows:
A₂ ~ k Th (etBaaa(,;);)))oa ag, by),
where A, indicates random variables of (zz)) and { are the prior probabilities for each Beta distribution Beta(-;a), a,, bi), Beta(-; ax, bx). We initialize the parameters (ax, ba) randomly for each Beta distribution and estimate the initial membership of each sample via K KC-means cluster- ing. Then, the expectation expectation-mxximizze maximization (EM) algorithm is adopted to iteratively refine the param- eters of the Beta mixture and the membership inference quality. This process only takes around 30 seconds for 50K samples. Finally, given test sample Ty, infer the membership of that sample via maximum likelihood k- = arg max, (at;()(a;); ): ag. by), and use the mean of the corresponding Beta distribution Oge tb.- This approach significantly reduces the computational burden of sample- wise dynamic interpolation from N (number of test samples) K (number of pre-defined clusters), where K < N. We outline the detailed procedures of DaWin in Algorithm (Appendix A.
4 EXPERIMENTS
4.1 SETUP
Tasks and datasets. We validate DaWin. on two scenarios: (I) robust fine-tuning (Wortsman al., 2022b) and (2) multi-task learning (Iiharco et al., 2022) with focusing on the top-1 classifi- cation accuracy (Acc). Following robust fine-tuning literature (Wortsman et al., 2022b; Kumar et al., 2022), we use ImageNet-1k 1K and its five OOD variants, ImageNet- V2, ImageNet-R, ImageNet-A, InageNet-Sketch, and ObjectNet, to evaluate robustness under distribution shifts. For multi-task learning, we follow the standard evaluation protocol (Ilharco et al., 2022: Yang et al., 2024b) us- ing eight benchmark datasets: SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD. Please see the Appendix A.2 for the omitted references and description for each dataset.
Models and baselines. For robust fine-tuning, we adopt CLIP VIT- B/16, L/14] (Radford et al., 2021) as zero-shot (ZS) models to ensure a fair comparison with (Wortsman et al., 2022b;a; Ilharco et al., 2022; Yang et al., 2024b), and fine-tuned (FT) checkpoints for each backbone from Jang et al. (2024) (see details in Appendix A.2). For the weight interpolation baseline methods, we include WiSE-FT (Wortsman et al., 2022b), Model Soup (Wortsman al., 2022a), and Model Stock (Jang et al., 2024), along with the traditional output ensemble method. In addition, we consider several ofate-of-the-art robust fine-tuning methods such as LP-FT (Kumar et al., 2022), CAR-FT (Mao et al., 2024). FLYP (Goyal et al., 2023), Lipsum-FT (Nam et al., 2024), and CaRot (Oh et al., 2024). For multi-task learning, we use CLIP ViT-B/32 as our backbone and consider the model merging baselines as follows: weight averaging. Fisher Merging (Matena & Raffel, 2022), RegMean (Jin et al., 2023), Task Arithmetic (Ilharco et al., 2023), Ties-Merging (Yaday et al., 2023), AdaMerging (Yang et al., 2024b), and Pareto Merging (Chen & Kwok, 2024). For a fair comparison, we do not experiment with other craiiin-intensing dynamic methods that introduce lots additional parameters (Tang al., 2024), the post-hoe methods (Yang et al., 2024a) and test-time prompt tuning (Shu et al., 2022), which are orthogonal to baseline and our methods. For DaWin, we set K to 3, 5, 2 for ViT- (B/32, B/16, l/14] in the robust fine-tuning and K in the multi-task setups.
4.2 RESULTS ON ROBUST FINE-TUNING
Better performance trade-off between ID and OOD. In Figure 4, we present the accuracy for ID (x-axis) and OOD average (y-axis) across three backbone models, along with baseline performance.
We advocate Beta mixture more than parametriic method, e.g., K-means, for better performance due the distributional structure. For cases of more than two models, it can be easily extended to Dirichlet mixture.

---

Published as a conference paper at ICLR 2025
Ave, Accuncy on 00D
prg sccuree on 000
Avg. Accuraty on 000
25
Oraeed Craemble
CAR-FT
Ron
Ougut Graeroh
Gredy Soup
Upcere
Cafer Hodd
Made Mack
05.
125
125
gegeetetteeeoo Accuracy
geNet Accuracy
ImageNet Accuracy
Figure 4: ID and OOD trade-off analysis. Trade-off's in terms of ID and OOD classification accuracy Ima- geNet distribution shift benchmarks with CLIP ViT-(B/32, B/16, L/14) from left to right. The hyperparameter grid of WiSE-FT is set to o.,, given two models, ZS and FT (See Figure A in Appendix A.3 for more results).
DaWin achieves performance beyond the Parcto-optinal points
Achieving superior ID/OOD trade-offs beyond or even comparable to the WiSE-FT hyperparame- ter sweep is challenging. Nevertheless, DaWi provides remarkably better performance trade-off compared to baseline methods. It also worth noting that DaWin: does not require any hyperparam- eter tuning. Instead, it dynamically generates interpolation coefficients solely based on the trained models' output. This distinguishes DaWin from other hyperparameter-ineense
fine-tuning such as CaRot (Oh et al., 2024) and Lipsum-FT (Nam al., 2024) or existing interpolation methods (Wortsman et al., 2022b). Table 2 provides detailed train/inference costs and ID/OOD accuracies. Compared with Model Soups (Uniform and Greedy Soups) and Model Stock, which require more than two models to ensure diversity among interpolation candidate models (Wortsman et al., 2022a) or periodic interpolations during training (Jang et al., 2024), DaWin operates with only a single fine- tuned model while achieving significantly better performance trade-off. These trends hold across other backbones (see Tables and 4), verifying DaWin's generality in terms of modeling scale. Although DaWin compromises the inference-time efficiency for accuracy, given that existing works typically require hyperparameter tuning in practice (Wortsman et al., 2022b), relative computation overhead is minor given that we (K + M) S H in all our experiments (See Figure 8).
Table Accuracy on ImageNet (ID) and its OOD for CLIP ViT-B/32. Cost denote number of training runs required to build the final model and the number of evaluations during inference, respectively, Given M models, N denotes the number of test samples, and H and K denote the size of the hyperparameter grid and the number of mixture components, respectively. Note that (K M) & H in all our experiments. Coefficients for Output ensemble and WiSE-FT were picked by grid search on ID validations
Method
Cost (T)
Cost (I)
ImageNet Acc (ID) Avg. Acc on OOD
ZS
O(1)
63.35
48.48
((1)
78.35
47,08
Output ensemble
O(M)
78.97
51.76
WiSE-FT (Wortsman et al., 2022h)
O(H)
79.14
$1.02
Uniform Soup (Wortsman et al., 2022a)
48
O(1)
79.76
52.08
Greedy Soup (Wortsman et al., 2022a)
48
O(1)
80.42
50.83
Model Stock (Jang al., 2024)
2+a
O(1)
79.89
50.99
DaWin w/o mixture modeling
O(N + M)
78.71
54.41
DaWin
O(K + M)
78.70
54.36
Ablation study. We explore alternative metrics beyond entropy for estimating sample-wise model expertise, as shown in Figure 5. Specifically, consider four different pseudo label (PL) ap- proaches (Lee et al., 2013) to replace the entropy, (ū.ū} x {soft, hard], where ú f((;; 0g) + fx; 01) and ÿ (z; + (1). The set- tings (soft, hard} indicate whether the argmax operation applied to PL or not. We observe that some PLs slightly outperform entropy in terms of ID accuracy but bring no gains OOD accuracy. To maintain simplicity in line with Occam's 's razor (Good, 1977), we adopt entropy as the expertise metric on unlabeled test samples.
Ho: Sum
HUc:
ImageWer Accuracy
Figure 5: Effectiveness of our expertise met- ric. We evaluate five different expertise metrics on ID (left) and OOD (right). Entropy (H) he- haves like loss (7) with the pseudo labels, where its performance nearly matches or surpasses the best- performing though simplicity.
7

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy on ImageNet (ID) and distribution shifts (OOD) for CLIP VIT-B/16.
Table 4: Accuracy on ImageNet (ID) and distri- bution shifts (OOD) for CLIP ViT-L/14.
Method
ID
OOD Avg.
Method
ID
OOD Avg.
ZS
68.3
57.7
FT
82.8
56.3
ZS
75.5
70.9
LP-FT (Kumar et al., 2022)
82.5
61.3
FT
87.0
70.6
CAR-FT (Mao et al., 2024)
83.2
59.4
FLYP
86.2
71.4
FLYP (Goyal et al., 2023)
82.7
60.6
CaRot
87.0
74.1
Lipsum-FT FT (Nam al., 2024)
83.3
62.7
Ouiput ensemble
87.3
73.3
CaRot (Oh al., 2024)
83.1
64.0
WiSE-FT
87.3
73.2
Output ensemble
83.4
63.6
Model Stock
87.7
73.5
WiSE-FT (Wortsman al.. 2022b)
83.5
64.2
DaWin
86.9
76.0
Model Stock (Jang et al., 2024)
84.1
62.4
DaWin
83.4
66.9
Extre
Tuehee
Mattec
Mathec
Mothedd
Figure 6: Entropy comparison on ImageNet-A for CLIP VIT-B/32. We visualize the average entropy from each method on the entire datasct and four different splits based on the correctness zero-shot (ZS) and fine- tuned (FT) models. Compared with individual models, weight averaging (WA) induces lower entropy overall, and our DaWin achieves the lowest entropy across all splits (See Figure in Appendix for full results).
Entropy analysis. Our DaWin method is built the correlation between entropy and X-entropy. While the analyses from Sec. 2.2 support using entropy as a proxy of X-entropy, it does not answer how the entropy of interpolated final models behaves. To explore this, we analyze the average entropy of interpolated models generated by DaWin in Figure 6. We see that the simple weight averaging (WA) produces lower entropy compared to individual models, and our DaWin achieves the lower entropy in all cases. This indicates that DaWing ''s expertise-baate based interpolation successfully weighs individual experts for accurate prediction and decreases per-sample entropy accordingly. In Sec. 6, we further provide a discussion on the analytic behavior of DaWin's weighting strategy.
dynamic selection and dynamic output ensemble. Given that DaWin is motivated by the concept of "entropy as a measure of model expertise," we can extend this beyond weight interpolation to two other approaches, dynamic classifier selection (DCS; Giacinto & Roli (2001); Britto et al. (2014)) and dynamic output ensemble (DOE; Alam et al. (2020); Li et al. (2023)).
Applications:
To be specific, DCS aims to select the most suitable classifier for each test sample based on the competence (referred as expertise in our terminology) among multiple classifiers. We provide the results of DCS by adopting entropy as competence measurement for classifier selection, i.e., selecting lower entropy model per sample. Besides. we also present a dy- namic output ensemble (DOE) by our A(x) directly on the output probability space rather than weight space. In Table 5, see that entropy-basee bring large performance gains compared with single fine-tuned model (FT), while the DaWin achieves the largest gain. This implies that users can flexibly build their dynamic system using test sample entropy accord- ing to their budget and performance criteria.
Table 5: Results of per-sample dynamic clas- sifier selection (DCS), dynamic output ensem- ble (DOE), and dynamic weight interpolation (DaWin n) ImageNet (ID) and under distribu- tion shifts (OOD) for different CLIP VIT back- bone models.
Method
Model
FT
DCS
DOE
DaWin
DCS and DOE
B/32
78.35
78.59
78.71
78.71
B/16
82.80
82.15
83.24
83,38
L/14
87.07
86.53
87.07
86.88
B/32
47.08
52.87
52.71
54.41
OOD
B/16
56.25
64.90
64.85
66.85
L/14
70.59
74.71
75.14
76.01
4.3 RESULTS ON MULTI-TASK LEARNING
We now evaluate existing merging approaches and our DaWin on multi-task learning benchmarks with CLIP ViT-B/32, Following the standard evaluation protocol (Yang al., 2024b), we have M
8

---

Published as a conference paper at ICLR 2025
Table 6: Multi-task learning performance. We CLIP ViT-B/32 for evaluation across eight benchmark tasks. f denotes using the ground truth domain indicator to select expert models for each domain. We report the layer-wise method for AdaMerging (Yang et al., 2024b) here, which surpasses those of the task-wise method. Note that AdaMerging and Pareto Merging both require non-trivial amount of training.
Method
SUN397
Cars RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Pre-trained
63.2
59.6
60.2
45.2
31.6
48.3
44,4
48.1
Jointly fine-tuned
73.9
74.4
93.9
98.2
95.8
98.9
77.9
88.9
Individualst
75.3
77.7
96.1
99.8
97.5
98.7
99.7
79.4
90.5
Weight Average (Ilharco ct al., 2022)
65.3
63.3
71.4
72.6
64.2
$2.8
87.5
50.1
65.9
Fisher Merging (Matena & Rattel, 2022)
68.6
69.2
70.7
66.4
72.9
$1.1
87.9
59.9
68.3
RegMean (Jin et al., 2023)
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
Task Arithmetic (Ilharco al., 2023)
55.3
54.9
66.7
75.9
80.2
69.7
97.3
50.1
68.8
Ties-Merging ( (Yadav et al., 2023)
65.0
64.3
74.7
75.7
81.3
69.4
96.5
54.3
72.6
AdaMerging (Yang et al., 2024b)
64.2
68.0
79.2
93.0
87.0
92.0
97.5
58.8
80.0
AdaMerging++ ++ (Yang ct al., 2024b)
65.8
68.4
82.0
93.6
89.6
89.0
98.3
60.2
80.9
Parelo Merging (Chen & Kwok, 2024)
71.4
74.9
87.0
97.1
92.0
96.8
98.2
61.1
84.8
DaWin
66.2
66.7
91.3
99.2
94.7
98.1
99.5
74.6
86.3
tasks and models individually fine-tuned on each task (where M 8). We do not know where each test sample arises from during evaluation and cannot choose true experts per domain. While both AdaMerging and DaWin use unlabeled testset, DaWin produces dynamic merged models given tasks or samples, whereas AdaMerging induces a single merged model for all tasks by default. Here, we modify the interpolation formula in Sec. 2.1 into task arithmetic formulation, i.e., given weights of pre- trained model f and fine-tuned models {0, (f)])=1. dynamic interpolation is defined as (x(=) 0u + Ao ) (z) where 7j 0, 0o- (1)2,, and Ao denote the task vector (Ilharco et al., 2023), weight for j-th task vector, and scal- ing term (set 0.3 following Ilharco et al. (2023)), respec- tively. In Table 6, DaWin greatly outperforms advanced weight averaging methods (Jin al., 2023) and adaptive merging methods (Chen & Kwok, 2024) that require tough training, whereby approaching ground truth expert selec- tion method, e.g., Individualsf. This verifies the versatility of DaWin, n, which is beneficial for adapting the model on multiple tasks well as single-task setup. Figure 7 shows the average of estimated sample-wise coefficients (y-axis) per dataset (x-axis). DaWin assigns the highest weights true experts (diagonal) per task and leverages relevant experts by reflecting task-wise similarity, e.g.. SVHN MNIST for digit recognition and EuroSAT RESISC45 for scenery classification.
03
Treek
414
011
D
Figure 7: Merging coefficient visualiza- tion. We visualize DaWin's coefficients across 8 datasets (columns) with 8 fine- tuned models (rows).
RELATED WORK
Robust fine-tuning aims to adapt a model on a target task while preserving the generalization capa- bility learned during pre-training. A straightforward approach injects regularization into the learning objective. For example, Ju et al. (2022) proposed a regularization motivated by Hessian analysis, Tian al. (2023a;b) devised a trainable projection method constrain the parameter space, CAR- FT (Mao et al., 2024) devised the introduced a regularization based on singular values. Another line of works modifies the training procedure to keep the pre-trained knowledge by decoupling the tuning of linear head from the entire model (Kumar et al., 2022), employing data-dependent tunable module (Lee et al., 2023), utilizing bi-level optimization (Choi al., 2024), mimicking the pre-training procedure (Goyal et al., 2023). In contrast, weight interpolation approaches emerged as an effective yet efficient solu- tion that conducts simple interpolation of individual model weights (Izmailov et al., 2018; Wortsman et al., 2022a;b; Jang et al., 2024). Unlike existing works inducing a single interpolated model, we propose dynamic interpolation method that produces per-sample models for better adaptation.
regularization, and CaRot (Oh et al., 2024)
Model merging studies mainly focus on integrating multiple models trained on different tasks into single model to create versatile, general-purpose multi-task model. After some seminal works

---

Published as a conference paper at ICLR 2025
in the era of foumdation models (Ilharco et al., 2022; 2023; Jin et al., 2023), numerous advances have been made those aim at reducing conflict between merged parameters (Yadav et al., 2023; Yu et al., 2024; Marczak et al., 2024), merging with weight disentanglement (Wang et al., 2024; Ortiz-Jimenez et al., 2024; Jin et al., 2024), optimizing interpolation coefficients on unlabeled test samples (Yang al., 2024b; Chen & Kwok, 2024) or learning additional modules generating per- sample/domain coefficients dynamically (Cheng et al., 2024; Lu et al., 2024; Tang et al., 2024; Yang al., 2024a). While those methods provide huge performance gains compared with static methods such as (Wortsman et al., 2022a), they all bring extra learnable modules and training. We focus on methods that do not induce extra complex training and devise a training-fres free dynamic merging method, DaWin, which seeks good trade-off between efficiency and downstream performance.
6 DISCUSSION
54.5
Accuracy and runtime trade-off. While focus
540
24
on improving accuracy through model merging ap- proaches, it is crucial to ensure the extra computa- tion demands of DaWin during inference are man- ageable. Fig. 8 presents trade-offs between the accu- racy and wall-clock time for various merging meth- ods. For methods requiring hyperparameter tun- ing, e.g., WiSE-FT and Task Arithmetic, the wall- block time (logarithm of second) reflects a cumu- lative time for evaluations across all hyperparame- ters. For methods like DaWin n or AdaMerging, we include the time required for additional workloads. In both settings, DaWin shows favorable trade-offs offs that outperform the most efficient one in terms of accuracy while its runtime is far less than computation-heavis methods such as per sample inter- polation without mixture modeling (Sample-wise Interp.) or AdaMerging. Thus, DaWi n provides a geghpperforiin merging solution that can be flexibly adapted considering a given cost budget.
533
Taik drthonds::
530
Atahhrgin
W22 F1
Atshurggg*+
Unfirs Soiaa wez
530
si Vi Will-lock time (log-scale)
Well-lork time llog-scale
Figure 8: Accuracy and wall-clock inference time with CLIP ViT-B/32. We compare search- based methods, test-time method (AdaMerging), and ours regarding the accuracy and inference time on robust fine-tuning (left) and multi-task learning evaluation on the SVHN dataset (right).
Theoretical analysis. To understand the empirical success of DaWin, we present an analytic behav- ior of entropy-based dynamic weight interpolation in contrast to input-independent uniform weight interpolation, which yields a uniform interpolation coefficient (Wortsman et al., 2022a).
Lemma 6.1 (Expert-biased weighting behavior of DaWin; Proof is deferred to Appendix A.4). Suppose we have M different models (f(:;))),, parameterized by (0, 10 with a homogeneous architecture defined by [()). Lel ((x) = coefficient vector given v, and [f 8) denotes the probability mass for class c. Then, we have
be the sample-wise interpolation
)jer(x) 2 M if H(f(z;0jez)) S ((f((;; Vj and k.
where J == f argmax 0.)] y)
Lemma 6.1 implies that, under the -orop---y assumption, DaWin always produces per- sample expert-biased coefficient vectors, which result in the interpolated models being biased to- wards true experts, i.e., models that produce correct prediction given æ. This desirable behavior is aligned with the motivation of dynamic classifier selection discussed in Sec. 4.2, whereas DaWin conducts interpolation rather than selection. Meanwhile, as the number of models participating in interpolation increased, samples that at least one model correctly classifies also increased. There- fore, the coverage of DaWin for weighing the correct experts expands accordingly. This analysis endows a potential clue for remarkable gains observed in the multi-task setting, which conducts merging beyond two models (See Sec. 4.3).
Conclusion. This work has presented a novel training-free dynamic weight interpolation method, DaWin, that estimates the sample-wise model expertise with output entropy to produce reliable per- sample interpolation coefficients. We further proposed a mixture modeling approach, which greatly enhances computational efficiency. We then extensively evaluated DaWin with three different back- bone models on two application scenarios, robust fine-tuning and multi-task learning, spanning 14 benchmark classification tasks. Regarding ID/OOD trade-offs in robust fine-tuning setup and overall accuracy in multi-task setup. DaWin consistently outperforms existing methods, implying its gen- erality. This empirical success was further explained by discussing an analytic behavior of Dawin n.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the researchers and interns at NAVER Lab Byeongho Heo, Taekyung Kim, Sanghyuk Chun, Sehyun Kwon, Yong-Hyun Park, and Jaeyoo Park for their invaluable feedback. Most of the experiments are based on the NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017). We also appreciate on the constructive comments from Max Khanov, Hyeong Kyu Choi, and Seongheon Park at University of Wisconsin-Madison This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024- 00457216, NRF-RSS-022400466666 FY2025).
REFERENCES
Kazi Md Rokibul Alam, Nazmul Siddique, and Hojjat Adeli. A dynamic ensemble learning algo- rithm for neural networks. Neural Computing and Applications, 32(12):8675-866 2020.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Diffusion soup: Model merging for text-to- image diffusion models. arXiv preprint wrXiv:2406.08411 2024.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On opportu- nities and risks of foundation models. arXiv preprint (XXi:2200.00755: 2021.
Alceu S. Britto, Robert Sabourin, and Luiz E.S. Oliveira. Dynamic selection of classifiers comprehensive review. Patter Recognition, 47(11):3665-3688 2014. ISSN 0031-3203. doi: https://doi.org/10..000 http:://www......oooooooo//..... sci sccienceeartticleep
URL https: www. sciencedirect. com/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In International workshop on artificial intelligence and statistics, pp. 57-64. PMLR, 2005.
Weiyu Chen and James Kwok. You only merge once: Learning the pareto set of preference-avee aware model merging. arXiv preprint 2rXiv: 12105, 2024.
Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Bertasius. Dam: Dynamic adapter merging for continual video qa learning. arXiv preprint arX1v:2403.08755 2024.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, (0((10)1885-1888) 2017.
Caroline Choi, Yoonho Lee, Annie Chen, Allan Zhou, Aditi Raghunathan, and Chelsea Finn. Autoft: Robust fine-tuning by optimizing hyperparameters on ood data. arXiv preprint arXiv:2401.10220 2024.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023, pp.
2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, 2014.

---

Published as a conference paper at ICLR 2025
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner. Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference Machine Learning, pp. 7480-7512. PMLR, 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. International Conference on Machine Learning, pp- 3259-3269 PMLR, 2020.
Giorgio Giacinto and Fabio Roli. Dynamic classifier selection based multiple classifier be- haviour. Pattern Recognition, 34(9):187-1188, 2001. ISSN 0031-3203, doi: https://doi.oog org//0. 1006/500811920000000015000 URL https: /www. sci encedi rect. www.teeetttttteeeeeeeeeeeeeeeeee /cc scnce//
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.132557 2024.
Good, Explicativity: mathematical theory of explanation with statistical applications. Proceed- ings of the Royal Society of London. A. Mathematical and Physical Sciences, 354(16799 1977.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19338-19347, 2023.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv
preprint arXiv:2312.00752 2023.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2021b.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzbak Gadre, Shuran Song. Hannanch Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. Advances in Neural Information Processing Systems, 35:29262-29277 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2025
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All need is just few fine-tuned models. European Conference on Computer Vision (ECCV), 2024.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. Fine-tuning linear layers only is simple yet effective way for task arithmetic. arXiv preprint arXiv:2407. 00000999, 2024.
Xisen Jin, Xiang Ren, Daniel Freotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023.
Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In International Conference on Machine Learning, pp- 10431-10461. PMLR, 2022.
Albert H.R. Ko, Robert Sabourin, and Alceu Souza Britto, Jr. From dynamic classifier selection to dynamic ensemble selection. Pattern Recognition, (1(5)1778--131, 2008. ISSN 0031-3203. doi: http:/iooorrr001.0111ij.oaaoooo 10.005. URL https: /tww///www////////////////// www.scediiieeeeeeeoeooecoec rect. com/ www.encceaiiiiiiiceeeeeeees https//neeeaarrccceea/////////0/ /ttp:///e/eeeee////0000000001000
Jonathan Krause, Michael Stark, Jía Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pp. 554-561, 2013.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Con- ference on Learning Representations, 2022.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML. Atlanta, 2013.
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference Learning Representations, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, 2021.
Ziyue Li, Kan Ren, XINYANG JIANG, Yifei Shen, Haipeng Zhang, and Dongsheng Li. SIMPLE: Specialized model-sample matching for domain generalization. The Eleventh International Conference on Learning Representations, 2023. httpst http::////p/eeeeeeeeeeeeee net forum? id=BqrPe2_e52
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
Zhuang Liu, Hanzi Mao, Chao-Yuu Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986 2022.
Ziming Liu. Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljačíć, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov- networks. arXiv preprint arXiv:2404. 19756, 2024.
13

---

Published as a conference paper at ICLR 2025
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. arXiv preprint arXiv:2406. 15479 2024.
Zhanyu Ma and Are Leijon. Bayesian estimation of beta mixture models with variational inference.
IEEE Transactions on Pultern Analysis and Machine Intelligence 33(1 (1:11:066-2213. 2011.
Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. International Journal of Computer Vision, 13265))1655-170M 2024.
Daniel Marezak, Bartlomiej Twardowski, Tomasz Trzciński,, and Sebastian Cygent Magmax: Lever- aging model merging for seamless continual learning. European Conference on Computer Vision (ECCV), 2024.
Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703 2022.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprini (rXX::2501119393, 2025.
Nithin Gopalakrishnan Nair, Jeya Maria Jose Valanarasu, and Vishal M Patel. Maxfu- sion: Plug&play multi-modal generation text-to-image diffusion models. arXiv preprint 0rXiv:2404.09977,, 2024.
Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In The Twelfth International Conference on Learning Representations, 2024.
Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jacgul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision- language models. Advances Neural Information Processing Systems, 2024.
Changdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, and Yixuan Li. Understanding multimodal Ilms under distribution shifts: An information--heooreiic approach. arXiv preprint arXiv:2502.00577, 2025.
OpenAL. Gpt-4v(ision) system card, 2024.
/ttts:///penaa..com/ttt
gpt 4v- sysstem-card..
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent
space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, 36, 2024.
Karl Pearson. Method of moments and method of maximum likelihood. Biometrika, 28(1/2):3459, 1936. ISSN 00063444, 14643510. URL http: : / www. jitor wttp://ttteeee//..1111.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8538-8567, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Interna- tional Conference on Machine Learning, pp. 28656-220000 28679. PMLR, 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean- SaantBaptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by in- terpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2023.
14

---

Published as a conference paper at ICLR 2025
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJC,, (15(3):211-2555) 2015.
Patrick Schober, Christa Boer, and Lothar A Schwarte. Correlation coefficients: Appropriate use and interpretation. Anesth. Analg., (665):1763-1700 May 2018.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35, 2022.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representations, 2018.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033 2024.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, pp. 1453-1460, IEEE, 2011.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngk- wan Kim, Gayoung Lee, Donghyun Kwak, Jung-Wor Woo Ha. al. Nsml: A machine learning platform that enables you to focus on your models. arXiv preprint arXiv:1712.05902 2017.
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weighl-ensembling mixture of experts. In Forty-first International Conference on Machine Learning. 2024.
Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yar Ma, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7836-7845, 2023a.
Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. Advances in Neural Information Processing Systems, 36, 2023b.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations, 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-limenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. ICML, 2024.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time. In International conference machine learning, pp. 23965-23999 PMLR, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022b.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring large collection of scene categories. IJCV, 119:3-22, 2016.
15

---

Published as a conference paper at ICLR 2025
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging. Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36, 2023.
Enneng Yang, Li Shen, Zhenyi Wang. Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. JCML, 2024a.
Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024b.
LIN Yong, Lu Tan, Yifan HAO, Ho Nam Wong. Hanze Dong, WEIZHONG ZHANG, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. In The Twelfth International Conference Learning Representations, 2024. URL https: // /ppeneeview
7RE.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Ab- sorbing abilities from homologous models as free lunch. In Forty-firtt International Conference on Machine Learning. 2024.
Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In NIPS Work- shop on Deep Learming and Unsupervised Feature Learning, 2011.
Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang. Shikai Guan, Ji- azheng Xu, and Peng Cui. On the out-o--distribution generalization of multimodal large language models. arXiv preprint (tti:22022.0599, 2024.
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Advances Neural Information Processing Systems, 36:60853-60877, 2023.
16

---

// Per-sample interp. coefficient Published as a conference paper at ICLR 2025
A APPENDIX
We provide the following items in this Appendix:
(A.1) Algorithm and additional details on DaWin
(A.2) Additional details on the experiment setup
(A.3) Additional empirical results
(A.4) Missing proof
(A.5) Limitation and future work
A.1 ALGORITHM AND ADDITIONAL DETAILS DAWIN
Algorithm 1: Procedure for training- free dynamic weight interpolation (DaWin)
Input: Test samples X = { E RNND, models f(=:0)) and (((:0)) of the same architecture, entropy function H-).. and the number of mixture component K
for i = I....,N do (H H,,1) + (H(((x,;;o)), (HH (((((;(;;) ([) if offset adjustment then
// Per-sample model expertise
X(x:) expl Hopl-H Ho. H,,1)
expl H (1)))0(()//((
else
exp( H X(x:) & expl Hep(_H. H..)
end
end
fitt bmm f- BetaMixture ((((:)) .fittti ()
{m, N ;i=1 +- hmm. predict (X) // Membership inference on test samples
for k = 0, ..... K do
X1 + X X[(m;;; = :]
X(X²) bmm [k] mean ()
(x(x.) + (1
// Cluster-wise dynamic interp.
end
Incorporating domain-wise expertise. While Eq. yields sample-wise interpolation coefficients solely based on the per-sample expertise estimation, if we can access the whole test samples si- multancously, we can refine the coefficient computation by introducing domain-wise offset terms. These terms are automatically estimated by the per-domain entropy of each model. Given the signif- icant performance improvements with domain- dynamic interpolation (c.f. Table 1), expect domain-wisn wise expertise per model can complimentary benefit to compute the interpolation coef- ficient. Therefore, we modify the sample-wise coefficient term in Eq. as below:
exp(-H(f(x; A(x) exp( 1app--H(0 -H(((((((;; + exp(-H((((=;0))) + O(X)
((x)/T(()
(td(H(((X(0o)) mean(F[(f(Xx00.))
(F((H(
T(X) = H(((X;00)) + H( H(F(X;00))
O(X) = 2 meann([(f(X;00))
where O(X) and T(X) denote the per-domain offset (formulated as an average coefficient of vari- ation) and relative domain expertise terms, respectively. By incorporating domain-wise expertise, DaWIN offers more stable dynamic interpolation coefficients, correcting some of the inaccuracies in sample-wist estimation (see Table A). It is worth noting that these sample-wise and expertise estimations do not require any hyperparameters for computing interpolation coeffi- cients. This eliminates the need for intensive hyperparameter tuning that often required in prior works (Wortsman et al., 2022b; Ilharco et al., 2023) to achieve the best performance.
X-entropy ratio and likelihood ratio. Let fa; 0) = (y(() be a classifier parameterized with 0 that models the ground truth conditional probability distribution p(y(z). Given that observe one-hot encoded target labels so that l(f(x; (x; 8), y) = ())))))))) then we can rewrite the oracle sasele-wise coefficient in Sec 2.2 follow, \ po, (ulx)/ (po, (v/x:) po, (ylx))
17

---

Published as a conference paper at ICLR 2025
Details on Beta Mixture Model (BMM) with Expectation Maximization algorithm. To hance the inference- efficiency of dynamic interpolation, we propose a mixture modeling ap- proach over the estimated per-sample interpolation coefficients to reduce the number of interpolation operations from N (entire test sample) to K (pre-defined number of mixture components). Here, we elaborate the detailed procedure of Beta Mixture Modeling'.
We have coefficient estimates over test samples via Eq. or Eq. as (A(x;))== Our goal is to model those coefficients with mixture of K Beta distributions as below:
I(ax + 6x), X(xx)) (xrr))==((111) {(x;))
Beta ((xr)); ak, by)
(3)
((ax)
p(((a:))) = Σ Ty (estaa((sss) bb)
(4)
where OA > O and by 0 are the shape parameters of component k, r(.) and denote the Gamma function and mixing prior probabilities for each Beta component satisfying Σ TR and my 2 O. We first initialize the responsibilities (rax by applying K-Means clustering to (>(x;))
assign the initial membership per each observation one of K components. We also initialize
the parameter estimates of BMM as below:
Mixing (Tio m
(0)
N
k
where N=(0)
7
(0)
(5)
1=1
Shape Parameters (ax, 6R)
(0)
= (0)
X X + t,
(6)
0R
(0)
(0)
X (1 - 1(00
+
1 N
where 1
Vik ((xx),
(8)
No
i=1
2.(0)
1
(0)
(A(z;) - -x 10)
$6
NO
Vik
(9)
t=1
(L
XQ) 1,
(10)
S% 2.(0)
where is a small positive constant to ensure numerical stability. Here, the initial shape parameters are estimated by method-of loaddoff-ooot momonts (Pearson, 1936). Then, we conduct the expectation step (E- step) and the maximization step (M-step) alternatively until convergence to refine the parameter estimate as follows:
E-step:
-Compute log responsibilities Iny
-Update responsibilities 7/k = exp(In (t)
M-step:
Update mixing priors xi ((+1)
-Update shape parameters a (1+1) ((+1) using methotsof-moments estimation.
Convergence Check:
- -Compute log-likelihood L(++1), where (()) = E = Inp(A(z:))
If [(t-)) _(0) < tolerence, stop the iterations.
Then, we get the estimated parameter @ {Tk, Ok by b of BMM to infer per-sample weight interpolation coefficients.
'The same procedure is adopted for the Dirichlet Mixture Model in multi-task learning scenario by modi- fying the probability density function from Beta distribution to Dirichlet distribution.
18

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL DETAILS ON THE EXPERIMENT SETUP
In this section, we provide extended details for task definition, baseline methods, and implementa- tion details. Some contents might be duplicated from the main paper.
A.2.1 TASKS AND DATASETS
We validate DaWin n two scenarios: (1) robust fine-tuning (Wortsman et al., 2022b) and (2) multi-task learning (Ihhacoo et al., 2022) with focusing on the top-1 classification accuracy (Acc). Following robust fine-tuning literature (Wortsman
al., 2022b; Kumar al., 2022), we use ImageNet-IK K (Russakovsky et al., 2015) and its five variants, ImageNet-V2 (Recht et al., 2019), a post-decade reproduced version of the original ImageNet test set by following the dataset generating process of ImageNet, ImageNet-R (Hendrycks et al., 2021a), rendition-spesnt specific collection of 200 ImageNet classes, ImageNet-A (Hendrycks et al., 2021b), an actual examples from ImageNet test set misclassified by a ResNet-50 model over 200 ImageNet classes, ImageNet-Sketch (Wang et al., 2019), a sketch-specific collection of 1000 ImageNet classes, and ObjectNet (Barbu et al., 2019) for evaluating robustness under distribution shifts. For multi-task learning, follow the standard evaluation protocol (Ilharco et al., 2022; Yang et al., 2024b) using eight benchmark datasets from the optical character images, traffic signs, scenery satellite imagery, and fine-grain categorization over cars and texture: SUN397 (Xiao et al., 2016), Cars (Krause et al., 2013), RESISC45 (Cheng al., 2017), ET (Helber et al., 2019), SVHN (Yuval, 1), GTSRB (Stallkamp al., 201 I), MNIST (LeCun, 1998), and DTD (Cimpoi et al., 2014).
A.2.2 MODELS AND BASELINES
For robust fine-tuning, we adopt CLIP ViT- VIT-(B/32, B/16, L/14) (Radford et al., 2021) as zero-shot (ZS) models to ensure fair comparison with (Wortsman al., 2022b;a; Ilharco et al., 2022; Yang et al., 2024b), and fine-tuned (FT) checkpoints for each CLIP ViT backbone from Jang et al. (2024). For the weight interpolation baseline methods, we include WiSE-FT (Wortsman al., 2022b), which conducts weight interpolation between pre-trained model and a fine-tuned model given a single pre-defined interpolation coefficient, Model Soup (Wortsman et al., 2022a), which conducts aver- aging of all models' weights (Uniform Soup) or greedily selected models" weights (Greedy Soup) trained with different training hyperparameter configurations, and Model Stock (Jang et al., 2024), iteratively interpolate a pre-trained model with few fine-tuning model based on the cosine distance between pre-trained model weight and the tinettuning model weights, along with the traditional out- put ensemble method. In addition, we consider several ofateoof-the-ae art robust fine-tuning methods such as LP-FT (Kumar et al., 2022), a two-stage method to avoid pre-trained feature distortion, CAR-FT (Mao al., 2024), regularized fine-tuning method leveraging context-aware prompt, FLYP (Goyal et al., 2023), a contrastive learning based fine-tuning method, Lipsum-FT (Nam et al., 2024), regularized fine-tuning method motivated by energy score gap between zero-shot and fine- tuned models, and CaRot (Oh et al., 2024), a theory-inspired singular value regularization method.
For multi-task learning, baselines as follows: simple weight averaging (Ilharco et al., 2022), Fisher Merging (Matena & Raffel, 2022), a Fisher information betric-based weighted averaging method, RegMean (Jin al., 2023), an averaging method that minimizes L2 distance between the averaged weight and individual weights, Task Arithmetic (Ilharco et al., 2023), a method perform arithmetic across task vectors rather the original weight vector itself which are produced by the subtractions between a pre-trained model weight and individual fine-tuned model weights, Ties-Merging (Yadav et al., 2023), a post- hoc weight refinement method mitigating conflicts between task vectors, AdaMerging (Yang al., 2024b) and Pareto Merging (Chen & Kwok, 2024) methods those driving additional optimization procedure to a global interpolation coefficient and the conditional coefficient generation models that trained to minimize entropy over entire test sample.
CLIP ViT-B/32 as backbone and consider the model merging
A.2.3 IMPLEMENTATION DETAILS
For fine-tuning CLIPs on ImageNet, Wortsman et al. (2022a) and its successor (Jang et al., 2024) conducted multiple training with different training configurations such data augmentation, learn- ing rate, weight decay, and random initialization seeds given fixed epochs (16) and batch size (512). Here, we use the best model weight provided by the authors of Jang et al. (2024) per each backbone.
19

---

Published as a conference paper at ICLR 2025
For fine-tuning weights of CLIP multi-task learning setup, adopt the official checkpoints from Ilharco et al. (2023) on the eight datasets.
On DaWin's evaluation, first get the entropy of batch test samples from the interpolation didate models (wherein temperature scaling (Guo et al., 2017) applied in the robust fine-tuning setup with ID validation set), then we compute interpolation coefficients by building softmax-like model expertise ratio term with exponentiated negative entropy of each model. We further perform the mixture modeling over the batch test samples and finally conduct dynamic model merging with interpolation coefficients corresponding to estimated membership from the fitted mixture model to obtain the prediction per sample. About the Beta (on robust fine-tuning setup) and Dirichlet (on multi-tasks learning setup) mixture modeling on interpolation coefficients for DaWin, we set K to in the multi-task setups.
3, 5, for VBT- B/16, L/14] in the robust fine-tuning and Unless otherwise mentioned, we adopt the offset adjustment term (Eq. 2) by default, assuming that the entire test samples per task are available and fit the Beta (and Dirichlet) mixture model on the entire coefficients per task, likewise assumption of Yang et al. (2024b;a); Chen & Kwok (2024).
A.3 ADDITIONAL EMPIRICAL RESULTS
Table A: Ablation study on offset adjustment. validate the effect of using the offset adjustment term on ImageNet ID and OOD accuracy.
Model
Offset Adjustment
ID
Avg. Acc OOD
78.3
54.1
ViT-B/32
78.7
54.4
83.1
66.6
ViT-B/16
/
83.4
66.9
86.7
75.9
ViT-L/14
86.9
76.0
We ablate the offset adjustment and expertise metric to investigate the effectiveness of the design choices of each component. Firstly, the ID and OOD accuracy across all cases, which supports the use of domain-wise relative expertise (average entropy over all test samples) to enhance sample-wise expertise estimation. To secure
see in Table A, offset adjustment consistently boosts
Table B: Sensitivity analysis on sample size. We report DaWin's ImageNet and its OOD variants' accuracy of CLIP VIT-B/32 under varying sample wise for fitting Beta Mixture Model (where the number of mixture components K=3), DaWin shows robustness against varying sample sizes.
Sample size
ImageNet Accuracy
Avg. Ace OOD
32
78.55
54.30
64
78.69
54.27
128
78.71
54.27
256
78.70
54.28
512
78.70
54.25
1024
78.71
54.24
2048
78.67
54.26
N
78.70
54.36
inference time efficiency, we adopt the Beta mixture modeling approach on the batch-wise DaWin's coefficients. The fitness of the Beta mixture model may be improved as the sample size is increased, whereas a smaller sample size enables more granular interpolations evaluate the performance of DaWin n under varying sample sizes. Table B presents the IDD and OOD performance of DaWin on the ImageNet distribution shift benchmark for the CLIP ViT-B/32 back- bone model. DaWin shows strong robustness against varying sample sizes.
be applied. Therefore, we
'Candidate models constituted with the pre-trained and fine-tuned models for robust fine-tuning setting, the task-specific eight fine-tuned models for multi-task learning setting.
20

---

Published as a conference paper at ICLR 2025
Nenl ACCur aucuraacyoo on 0.0
Dupue Crearro
Dog Aecureg
Beds Accures
Wegge triepokitie
Weigh Mitisiationn
Oeran WISETT
Our WEW6TT
irageict Acturacy
Imogelict Acouracy
Progelet-Accunne
Figure ID and OOD trade-off analysis. We visualize performance trade-offss regarding ID and OOD classification accuracy on the ImageNet distribution shift benchmarks with CLIP ViT--B/32,, B/16, L/14) from left to right. Interpolation coefficients are swept over 0.2,,0.2 0.9.)
In Figure A, present the results of DaWin. with WiSE-FT plug-in augmentation the robust fine-tuning setup. We multiply our estimated coefficients from Beta mixture by a scalar coefficient a, e.g., (a)) X(x)
Although it brings slight benefits in the case of CLIP ViT-B/32, the WiSE-FT interpolation trace becomes almost a line on the ViT-B/16 and ViT-L/14 cases. This implies that DaWin already achieves performance beyond the Pareto-optimal trade-ofTs and cannot be further improved by WiSE-FT, given models f(; 0o) and f(:; Meanwhile, Figure reveals that DaWin produces larger X(x) for samples that are hard recogmize the target object due to overwhelming background semantics, which the pre-trained model may wrongly pay attention to.
5 (x)
Top-5 1(x)
Fis Years
WORHT
nutte
Thann reant
side Franecc
SKUNK
paner snake
patter
Arstcanreda
expesient
sabite
Indd
Sengn Lore Mide
Fige
Figure B: DaWIN's bottom-5 and top-5 estimated coefficient analysis. We visualize the images with labels corresponding to bottom-5 and top-5 sample-we wise interpolation coefficients estimated by DaWIN of pre-trained and ImageNet fine-tuned tuned CLIP VIT-B/32. Each denotes the actual test samples from ImageNet. ImageNet- ImageNet-R. R, ImageNet-A. ImageNet-Skee
and ObjectNet. We see that images corresponding to coefficients lean towards the fine-tuned model, which typically contains multiple semantics, corresponding to the ground truth label is overwhelmed by other semanties.
object
21

---

Published as a conference paper at ICLR 2025
Entire
TrueTrue
TrueFalse
FalseTrue
FalseFalse
Ps,
P.
Ps 0.433
Ph, 0,896
Ps 0.089
Ps - 0.513 0.417
Po, = 0.932 0,903
Ps, 0.907
0.24
As - 0.052
0.055
0.921
0.000
Ps. - 0.664 0.589
Ps, = 0.943
As. 0.94
0.372
Pa. - 0.149
0.949
0.063
0.54
0.095
= 0267 0.064
0.94 0.938
= 0.92
0.115
0.003
-01162
0:929
0223
0.579
0.93
0.917
0.36
0.3
0.05
0.483
0.945
0.528
4.059
((f(x;;),))
(xf(x;))
Figure C: Correlation between entropy and X-entropy. We split each testset into four subsets (TrueTrue, TrueFalse, FaiseTrue, FaiseFalse) based on the correctness of the models' predictions. On each data split and the entire dataset, we visualize scatter plots entropy (x-axis) and cross-entropy (y-axis) computed by two models (zero-shot and fine-tuned) with corresponding Pearson correlation coefficients per model. Each row from top to bottom shows results from ImageNet, ImageNetV2, ImageNetR, ImageNetA, ImageNetSketch, and ObjectNet.
DaWin adopts entropy as a proxy of X-entropy to estimate the model expertise without access to the true target label. Figure C presents the correlation between entropy and X-entropy of the pre- trained CLIP ViT-B/32 (:::0) and ImageNet fine-tuned counterpart f(; 01). Results indicate that the model producing correct predictions holds strong correlation between entropy and X-entropy while the model failing to correctly predict test samples shows bad or no correlation. However, if least one model success in making correct prediction in a given sample œ, and the entropy of the correct predictor may be smaller than that of another model, thereby DaWin would be likely to produce A(r) biased towards the correct predictor's weight (See Lemma 6.
22

---

Published as a conference paper at ICLR 2025
Entire
pm
p=07
p-009
p. 0.300
P
10/0x: (( 10/ (op
H(((#:8)
H(f (()+
Figure D: Correlation between entropy ratio and X-Entropy ratio. We split each testset into four subsets (TrueTrue. TrueFalses www.see.lleeeeeeeeeeeeaeeeeeeeee SallsTrue,,Fassee se) based on the correctness the two models' predic- tions. each split and entire dataset, we visualize seatter plots entropy ratio (x-axis) and cross-entropy ratio (y-axis) computed by two models (zero-shot and fine-tuned) with corresponding Pearson correlation co- efficients. Each row from top bottom shows results from ImageNet, ImageNetV2, ImageNetR, ImageNetA, ImageNetSketch, and ObjectNet.
In Figure D, we provide the extended results of Figure 2, showing the correlations between entropy ratio and X-entropy ratio, on the whole evaluation datasets of robust fine-tuning setup. The entropy ratio approximates the X-entropy ratio overall across datasets and sub-populations of each dataset, even though for the most challenging OOD, i.e., ImageNet-A, which constructed with natural ad- versarial examples, there is tond---e--nn-trivieee correlation (Schober et al., 2018) between entropy and X-entropy. Moreover, we note that weight interpolation (or output ensemble) has an advantage that elicits the correct prediction by modifying the relative feature importance (Yong et al., 2024) even though two individual models fail to produce correct predictions (i.c., in the F Falsel selseFad alse case), thereby expected outperform the model selection method (See Table 5).
23

---

Published as a conference paper at ICLR 2025
Entire
Vuelnue
Tuefalte
FaseTrue
Rasefalse
&rence
Jud Caero Jud Srrnge
Aud drmapy
at Dtanyy
Por Canner
Coett deit
Entire
TreThee
Thyefaice
FaiteTrue
Paicefatee
Add in Orreey
for on bonny
Jud en Dronp
Huhv Oringy
Craffic
Conticiee
Contfcine
Hot of toveer
frrnnce
Aut Autweteeee
Coaft
Conficirtt
24 14
Conticeer
Confiiient
Trafiru
TacheTru
Faluehidee
Art in son
Asd et thinu
151 with
Cotto
Coeficent
Entire
Tretuue
halse
Falsehulse
AUd in sup
Hut ot cansup
Aus eananip
Cooft dert
Confictntt
Entire
Tecfrue
Trucfise
FobeeTus
Fabechisz
Aut snrre
Oxc CI
Cadlukri
CodRukii
Figure E: Estimated coefficient density analysis on ImageNet distribution shift benchmarks with CLIP VTT-B/32. We visualize histograms of estimated sample-wise interpolation coefficients by the expertise ratio, which is compuied using X-entropy (Oracle: we can not access it in reality) and entropy expertise met- rics. Each column denotes the splits (Entire, True, TrueFalse FalseTrue, FalseF Fasee] of the test set, those are categorized by the correctness zero-shot CLIP, and ImageNet fine-tuned CLIP's pre- dictions. Each low denotes the result of the test set: from ImageNet, ImageNet- V2, ImageNet-R, ImageNet-A. ImageNet-Skette and ObjectNet. Entropy-based coefficient estimation shows remarkably good fitness in the TrueTrue, FalseTruc splits, and produces left-skewed distribution in the case of TrucFalse which desired to construct the interpolated model biased towards zero-shot model weight. Overall, except the Fa 1 seFa a alse se case, the entropy-based coefficient estimation provides reasonable alternatives X-entropy- hased oracle coefficients.
We visualize the histogram of interpolation coefficients computed by entropy and X-entropy ratio in Figure E. While the entropy- -base coefficients quite diverge from the oracle X-entropy- -bsed coef- ficients in some cases (e.g., FrueFalss and Fal seFalse of ImageNet-Sketth and ObjectNet), the overall distributions of entropy-based coefficients show good fitness to X-entropy-baase based coeffi- cients across datasets. This result supports using entropy proxy of X-entropy to estimate model expertise given unlabeled test-time input to determine the per-sample interpolation coefficient.
24

---

Published as a conference paper at ICLR 2025
Endie
Tustinue
D1000
Entrrey
Mudfood
Musiiod
Mettod
Muttool
Method
Trashalee
tabotube
20
010000
Entrage
Nethod
Endine
Tuethe
Thurhalee
Delehatte
70
Methoo
040
Tustiue
Truehalse
Natuel ue
Passmubee
50000
Eriepey
Method
Hestrod
Methed
Method
Nerrod
25
Endie
Tuclne
Kasohate
170000
Entree
Mattoo
Mustro
Mestod
Neshod
Figure F: Entropy comparison on ImageNet distribution L with CLIP VIT-B/32. Across IN, IN-V2, IN-R, IN-A, IN-S, and ObjNet from top to bottom, visualize the final output entropy from each model on the entire dataset and four different splits based on the correctness of zero-shot (ZS) and fine-tuned (FT) models. Compared with individual models, weight averaging (WA) induces lower entropy overall, and our DaWin achieves the lowest entropy across all splits.
In Figure F, provide the extended results of Figure 6, showing the average entropy over test samples, on the whole evaluation datasets of robust fine-tuning setup. In almost all of cases, weight interpolation achieves smaller entropy than individual models, and DaWin achieves the smallest entropy. This supports our sample-wise entropy valley hypothesis in Section 3.2 and helps us to understand DaWini n's great performance gains.
25

---

Published as a conference paper at ICLR 2025
MMD between datasets
FID between datasets
SUN397
SUN397
RESISC45
EUrOSAT
EUrOSAT
SVHN
SVHN
GTSRB
GTSRB
MNIST
MNIST
DTD-
SUN391
RESISCAS
EuroSAT
SWNN-
WNIT
DTD.
Cars
RESISDAS
EuroSAT
SVAN
GTSRB
WNIST
DID-
Figure Distance between datasets. visualize mean maximum discrepancy (MMD) Frechet
inception distance across eight datasets used multi-task learning benchmarks in the left and right panels, respectively. To compute MMD and FID, we adopt the pre-trained OpenAl CLIP VIT-B/32 and ImageNet pre- trained Inception V3 models feature extractors respectively.
ImageNet
ImageNetv2
ImageNett
14
14
Aurrage Entrony
Avereqea Entraay
ImageNetA
ImagerNetSeetchh
ObjectNet
2.00
1.75
Average Fatrapy
2,00
Figure Empirical validation on the sample-wise entropy valley hypothesis. visualize the average entropy for each dataset across various methods including single model evaluations (ZS and FT), static weight interpolation (WiSE-FT 0.05, 0.95), some dynamic weight interpolation methods (O, 1), (0.5,, 0.2*)), N(0.8, 0.1%), Max logit ratio, Confidence ratio, and Confidence difference ratio), and Dawin method.
In Figure G, we visualize the distance between eight datasets used in the multi-task learning se- tups. The computed cross-dataset distances are matched with the average coefficients derived from DaWin in Figure 7 to some extent.
Similar to the linear mode connectivity phenomenon that prevails between pre-trained and fine-tuned checkpoints of foundation models (Wortsman al., 2022b; Ramé et al., 2023) which indicates interpolations between two checkpoints do not increase the expected loss (e.g., X-entropy) on down- stream, in Section 3.2, we hypothesized the expected entropy of model output may also show similar convexity trend over interpolations between pre-trained and fine-tuned model checkpoints. In Fig- ure H. we see that this claim empirically holds for the six datasets including ID (ImageNet) and OOD (its five variances), i.e., there exists interpolated model that achieves smaller entropy than individual models (ZS and FT). Besides, we further observed that there exists a per-sample dy- namic interpolation method that achieves much smaller entropy compared the static interpolation method. Therefore, our two statements in the sample-war wise entropy valley hypothesis show empirical evidence.
26

---

Published as a conference paper at ICLR 2025
Table C: Optimality check for (x) We provide mean squared (MSE) between the optimally timated per-sample interpolation coefficients and our X-entropy ratio-based estimation in Section 2.2. estimate the optimal coefficient per sample by conducting a finer-gras grained grid search for the coefficients, e.g.. (0.05, 0.10, ..., 0.90, 0.95} per sample to minimize the X-entropy loss of interpolated model given a sample x. As baseline coefficients, (10 1) and WiSE-FT constant (0.8) denote per sample random uniform variable and constant scalar 0.8 for all samples respectively.
ImugeNet
ImageNetV2
ImageNetR
ImageNetA
ImageNetšketch
ObjectNet
Std. of per-sampll optimal coefficients (x)
0,2827
0.3068
0.3237
0.3287
0.3413
0.3438
MSE between (x) and (0, 1)
0.1797
0.1823
0.1869
0.1976
0,1980
0.2013
between (1)) WiSE-FT constant
0.1390
0.2081
0.2427
MSE between X (ax) and X eniropy ratio (ours)
0.0382
0.1559
0.0479
0.0364
0,0400
0.0440
In Section 2.2, we adopted the negative X-entropy as an oracle model expertise metric and used the exponentiated negative X-entropy ratio as our desired per-sample interpolation coefficients. Al- though the exponentiated negative X-entropy ratio A (x(z) has a nice interpretation connected the density estimation (Sec A.I), it is not guaranteed that the (ar)'ss optimality in the context of weight interpolation. To investigate how close our X (xx) are to the true optimal per-sample interpolation coefficients that minimize the X-entropy losses of interpolation model given samples, we measure mean squared error (MSE) between ours and optimally estimated per-sample coefficients (we esti- mate these by conducting grid search for interpolation coefficient over {0.05, 0.10, ..., 0.90, 0.95) per sample). Table presents the results indicating that our X (x) are generally much closer to the optimally estimated coefficient compared with random uniform variable tuned WiSE-FT constant coefficient. Given that the standard deviation of oracle coefficients are about from 0.28 to 0.34, the closeness between optimal and X-entropy ratio-based coefficients is remarkable.
Table D: Alternative per-sample interpolation coefficients. We provide results from some different design choices for per-sample dynamic interpolations (Beta Mixture Modeling with K=3 applied for efficient infer- ence) ImageNet distribution shift benchmarks. Although other data-independent data-dependent methods struggle strike good balance between ID and OOD performances, DaWin i n shows outstanding performance among considered candidates demonstrating its effectiveness well-grounded with unique motivation.
Category
Coefficient per sample
ID) Acc
OOD Acc Avg.
No interpolation (FT)
78.4
47.1
Static interpolation
0.8 (WiSE-FT)
79.1
51.0
Data-independent
UO, 1) (0.55 0.2²) NO..8, 0.1²)
75.9
52.9
Data-independent
77.5
54.1
Data-independent
79.1
51.1
Data-dependent
Max Logit Ratio
76.6
54.3
Data-dependens
Confidence Diff. Ratio
77.5
54.2
Data-dependent Data-dependent
Confidence Ratio Entropy Ratio (ours)
77.5
54.2
78.7
54.4
In Table D, we provide some alternative design choices for functions that estimate per-sample inter- polation coefficients from two categories, data-independent and data-dependent. We see that DaWin outperforms other baseline methods indicating our careful design choice of DaWin motivated by pilot study has non-trivial benefits rather than other heuristic alternatives.
Table E: Performance of DaWin w/ and w/o confidence calibration on ImageNet (ID) and its variants (OOD). Here, we adopt temperature scaling on validation set to calibrate individual models' outputs.
Method
ID Acc
OOD Acc
WiSE-FT
79.1 1
51.0
DaWin (default)
78.7
54.4
Da w/ bad calibration
76.6
54.0
Meanwhile, DaWin adopts the entropy of output probability distributions of individual models. As mentioned in Section A.2, apply temperature scaling individual merging candidate models
27

---

Published as a conference paper at ICLR 2025
to get calibrated outputs from them. The natural question is "how does the uncertainty calibration of individual models affect the final performance DaWin?". We adopt temperature scaling on the ID validation set by default to calibrate individual models in the robust fine-tuning setup, now ablate the temperature scaling in Table E. We can see that DaWin n without temperature scaling (denoted as bad calibration) somewhat compromises IDD performance. However, it still significantly outperforms the static merging method on OOD performance which demonstrates the effectiveness of our method even without delicate tuning.
Table F: Runtime, FLOPs, and Peak memory allocation comparison between static (WiSE-FT) and dy- namic (DaWin) weight interpolation for evaluation on ImageNet variants. Here, we use ViT-B/32 back- bone model on NVIDIA A A100 GPU(s). DaWin (parallel) conducts the expertise computation process in parallel (i..., forward evaluations for pe-trained and fine-tuned models are conducted simultaneously) with multiple GPUs for faster inference.
Method
Total Runtime (sec)
FLOPs per sample
Peak memory (MB) per sample
OOD Avg. Ace
WiSE-FT DaWin
1040
4,4139
454
51.0
1802
13.2452
1338
54.4
DaWin (parallel)
1341
8.8313
1338
54.4
Along with Figure 8, to further investigate the trade-offs between DaWin and simpler method, we provide evaluation results with additional metrics (FLOPs and peak memory allocation) and variants of DaWin in Table F. Here, DaWin (parallel) denotes the setting where we parallelize the evaluations of individual models during the expertise estimation phase before conducting the interpolation'. DaWin shows three times FLOPs than WiSE-FT, but the actual runtime is less than twice that of WiSE-FT because DaWin does not require intensive hyperparameter tuning, unlike WiSE-FT. Besides, the computational complexity can be further reduced by adopting the parallel inference pipeline for the model expertise estimation phase of DaWin.
Table G: Fine-grained analysis interpolation candidates. Model Pool denotes the candidate models (fine- tuned on those datasets) use to build the interpolated model. We simulate some scenarios where we use relevant expert models only or mixed candidates of experts and non-experts to get the interpolated model. Relevant datasets are determined based on task and semantic similarity between datasets.
Scenario
Model Pool
Evaluation data
Task Arithmetic
AdaMerging
DaWin
Experts only
SVIIN, MNIST
SVIIN SVHN SVIIN
87.5
94.1
90.5
Mixed
SVHN, MNIST, EuroSAT
84.4
93.7
93.8
Mixed
SVIIN, MNIST, EuroSAT, RESISCAS
81.5
93.4
94.8
Experts only
FuroSAT, RESISCAS
FureS AT
96.4
98.1
98.6
Mixod
SVHN, MNIST, EuroSAT
BureS
86.9
97.7
99.6
Mixed
SVHN, MNIIST, EuroSAL, RESISC45
BuroSAT
89.6
97.7
99.7
Including our method, the success of model merging methods (also traditional ensemble methods) depends on the relation between the training datasets of merging candidate models and the evaluation dataset. In Table G, we systematically analyze the effect of the relation between train and evaluation datasets on the performance of the merged method by constructing a model pool for merging based on the distance between train datasets (where the candidate models are trained) and test datasets. Please refer to Figure G and Figure 7 to check the quantities of similarity between datasets.
We simulate scenarios where all models are relevant to solve the downstream task, or some models are relevant but others are not to solve the downstream task. Although the baseline methods perform well when we have expert models trained on datasets that are relevant to the evaluation dataset, they suffer from performance degeneration as the non-expert models are included in the merging pool. This indicates that there are some interferences between models (and train corresponding datasets) that hurt post-merging performance if we inappropriately determine the interpolation coefficient. Meanwhile, DaWin takes benefits from even non-expert models by leveraging the entropy ratio- based weighting strategy, and the performance is improved as more models participate in merging.
This treatment is orthogonal to the commonly used distributed data-parallel inference mode and is only applicable to DaWin.
28

---

Published as a conference paper at ICLR 2025
A.4 MISSING PROOF
In Sec. 6, we provided an analytic behavior of DaWin, which generated true-expert biased interpo- lation weight vectors below:
Lemma A.1 (Restatement of expert-biased weighting behavior of DaWin). Suppose we have M different models {[(:0;))} M parameterized by {0,}}=1 with homogeneous architecture defined by {()). Let X(x)) (Z),..... (............ (x)x)) be the sample-wise interpolation coefficient vector given 2, and (1; o) denotes the probability mass for class c. Then, we have
)gEr(a)
M if H(f(2;0Jez)) & H(f(x; Bxer)) Vj and l.
where fill arg f 0.)] y)
Proof. The proof is very straightforward, given the assumption and definitions of problem setup,
H(f(=: & ((((x;0kes))
(By assumption)
exp( (((=(((xx())))) 2 (x;
{xEr(x) 2 {nes(z) (z)
(By definition of A(x))
{xE 2 M
(Given that Σ/ >y(x) 1)
A.5 LIMITATION AND FUTURE WORK
By following previous works (Wortsman al., 2022b; Ilharco et al., 2023; Yang et al., 2024b), we limited our validation scope to an image classification of fully fine-tuned visual foundation models with restricted scale and architecture, e.g., CLIP ViT-(B/32, B/16, L/14]. Exploring DaWin on diverse model architecture (Liu et al., 2022; Gu & Dao, 2023; Liu et al., 2024), large-scale modeling setup (Dehghani et al., 2023), large language models (Yu et al., 2024; Goddard al., 2024; Rame et al., 2023), multimodal generative models (Liu al., 2023; Biggs al., 2024; Nair et al., 2024), continual adaptation scenario (Marezak et al., 2024), and parameter-efficient tuning regime (Li & Liang, 2021; Hu al., 2021; Chronopoulou et al., 2023) can be exciting future work directions.
Meanwhile, the increased amount of computation during inference time is another limitation of DaWin. However, note that scaling test-time computation can be cost-effective solution for foundation models to address challenging tasks and data in the wild (Snell et al., 2024; Jaech et al., 2024; Muennighoff al., 2025). Given the observations that modern multimodal foundation models still struggle with out-of-distribute improving the robustness of foundation models by allocating more budgets for test-time computation through dynamic merging, such as DaWin, is worth to be investigated further.
query (OpenAI, 2024; Zhang et al., 2024; Oh et al., 2025),
29


Reviews:
[{"review_id": "wdeUDdfU82", "reviewer": "Reviewer_sGAj", "summary": "A new method for robust fine-tuning, called \"dynamic weight interpolation\" (DaWin) is proposed. It is directly related to the well known model averaging method WiSE-FT. However, while WiSE-FT averages model parameters, using a fixed coefficient $\\lambda$ that does not depend on the input data, DaWin proposes to make the coefficient data dependent, i.e., to use $\\lambda(x)$. The optimal $\\lambda(x)$ is computed, using a heuristic that is based on the entropy of the model output. DaWin is tested against many existing robust fine-tuning methods, yielding slightly better OOD performance compared to WiSE-FT.", "questions": "* Could you please comment on the memory requirement of DaWin and how it compares to other methods, for both with and without mixture modeling?\n* I am also not so sure about your provided costs for inference in Table 2. Could you point me where $H$ is defined and elaborate a bit the provided terms? Am I correct, that DaWin (without mixture modeling) requires 3 model evaluations (2 to compute the entropy for the individual models for a given $x$ and 1 for the averaged model)?\n* In the experiments, DaWin sacrifices ID performance for better OOD performance compared to WiSE-FT. Could you elaborate in which application scenario I would want to choose DaWin over WiSE-FT. I would think in practice good ID performance still wins over good OOD performance if you cannot have both.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In general, the paper written in a clear and understandable way. It provides novel experimental evidence, that averaging the weights of pre-trained and fine-tuned models to keep OOD performance high might not be fine grained enough. The proposed heuristic to select sample based averaging coefficients intuitively makes sense and is backed up by well designed experiments.\nThe performance and computational complexity of DaWin is compared to a sensible selection of existing methods.", "weaknesses": "I see two problems:\n\n1) In section 2.2, the a pilot study is provided to show the necessity of sample-wise averaging coefficients $\\lambda(x)$ for good out of distribution performance after fine-tuning. As I understood, the goal is to find the best possible OOD performance that we can achieve with an oracle $\\lambda^*(x)$ for the sample-wise coefficients that actually knows the true labels for a given $x$.\n\nThe paper states, that such an oracle would be $\\lambda^*(x) = p_{\\theta_1}(y|x) / (p_{\\theta_0}(y|x) + p_{\\theta_1}(y|x))$, i.e., the probability whether data $(x,y)$ comes from the distribution $p_{\\theta_1}(y|x)$. However, while this would make sense if we merge the output of the models (analogous to classifier selection), I cannot imagine that it is the seeked best oracle for $\\lambda^*(x)$ if we perform weight averaging. To obtain this oracle, we would have to test for each $x$, if there exists a $\\lambda^*(x)$, such that the model with parameters $\\theta_{\\lambda^*}$ predicts the correct $y$.\n\nIn the following sections, the paper makes some approximations to compute $\\lambda^*(x)$ based on the entropy of the model output, which does not require the knowledge of $y$. The reader is lured into the impression, that the only performance gap to the oracle is caused by these approximations. However, I think this is not true, since the oracle itself might not be optimal. However, this issue is about presentation and could be fixed.\n\n2) I think performance gains compared to WiSE-FT and other methods are often minimal. Although DaWin does not require fine-tuning, there is still a considerable memory overhead. To be able to perform per-sample averaging, we need to store both sets of weights. For large models, this might be a severe problem that is not worth the effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fr3NqMDM1x", "reviewer": "Reviewer_Rb5n", "summary": "This paper proposes a dynamic weight interpolation method DaWin. It merges pre-trained and fine-tuned models based on per-sample entropy without training. It further uses Beta mixture models to cluster interpolation coefficients to keep computation efficient. DaWin achieves considerable performance, offering a scalable solution for adaptable model merging.", "questions": "1. From my point of view, the dynamic weighting strategy per sample is another form of confidence-based selection, where the preference is determined solely by model confidence. Given this, what if we fuse the outputs instead of the model weights? Do the tasks (e.g., ImageNet - ImageNet-A) reveal any trends in the weighting of these models?\n\n2. How can we determine whether the performance is correlated with the **distribution distances** between datasets?\n\n3. Could the proposed method result in negative weight mixtures for some datasets? If so, why?\n\n4. Is it possible that some checkpoints could **decrease** the final performance on specific datasets? Furthermore, do the semantic distances and relationships between classes from different datasets (and their corresponding checkpoints) affect the performance of one another?\n\n5. For the evaluation of wall-clock time, is it tested per dataset? or per sample?\n\n6. Please also review the drawbacks section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. DaWin requires no additional training, making it highly practical for real-world deployment.\n2. The usage of Beta mixture models ensures adaptation scalability, reducing the inference-time overhead of dynamic interpolation.\n3. DaWin shows its performance on both robust fine-tuning and multi-task learning problems.\n4. DaWin does not require extensive hyperparameter tuning.", "weaknesses": "1. The model utilizes Beta Mixture Models for efficiency, but only complexity and wall-clock time results are provided. There are no reported MACs or FLOPs per sample, limiting insights into computational costs.  \n\n2. Since the model performs per-sample interpolation, it requires three inferences per sample, increasing computational demands.  \n\n3. The dynamic interpolation adjustment can be viewed as an incremental improvement over existing interpolation methods, rather than a novel breakthrough. The level of novelty remains unclear.  \n\n4. While the paper presents a comprehensive experimental study, the explanations for the results are shallow. There is limited discussion on why DaWin performs well or poorly on specific datasets or distributions (e.g., SUN397 and Cars). Additionally, relationships among datasets are critical but remain under-explored.  \n\n5. The entropy-based expert assumption is built on simple object concepts, implying that the datasets are sufficiently learnable through fine-tuning a CLIP model, where entropy should ideally align with correct predictions. However, in real applications, entropy can be misleading, resulting in overconfident but incorrect predictions. The model design does not address how to correct these wrong predictions. While the TrueTrue/TrueFalse breakdown in Figure 2 is insightful, it is not directly leveraged in the experiments, which limits the depth of the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QRfyZMjPIv", "reviewer": "Reviewer_uT7T", "summary": "This paper describes a method of doing sample based parameter merging in order to improve performance on OOD test samples. The main idea is to estimate the entropy of each model and then to use this to perform the merging step. As this is quite slow, they also describe a faster mixture-modelling technique. They first demonstrate with 'GT' data that if you had an oracle for the entropy you would do well. They then show that this oracle / estimated entropy is correlated and finally that they achieve superior results over similar methods.", "questions": "Please refer to questions in weaknesses but here are additional questions that are not weaknesses but I'm curious about:\n\n1. Did they see any trends in where they obtained similar interpolation coefficients for samples? Maybe certain clusters of samples are always better explained by a given model or not. Are there conclusions one could draw about the test set from this ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is quite interesting and presents their main idea clearly. I summarise the strengths below:\n1. Clarity: the paper is clear and gets across the main message (can we use an estimate of entropy at the sample level in order to improve OOD performance).  I like the 'GT' set up as it clearly shows the potential improvements of such an approach and gets across the intuition of why this is useful.\n\n2. Experiments: they have a range of results backing up their claims and showing that their method improves over the baselines. They further generalise their approach to other settings (e.g. dynamic classifier selection or ensemble selection)\n\n3. The consider compute and performance and show how they are trading off compute (though maybe not as badly as others) for improved performance.", "weaknesses": "1. Clarity: There are a few parts of the paper that are lacking details or are confusing:\n1a. in the intro a foundational model != ImageNet. People refer to foundational models primarily for large scale models trained on vast amounts of data (GPT,Gemini,Flamingo)\n1b. It is unclear how precisely H(-) is computed. I appreciate this is probably obvious to the authors but it would be helpful to briefly mention.\n1c. I am unconvinced by their linear mode connectivity explanation. My understanding is the point of the linear mode connectivity is that *along that line between model weights*, the error is bounded to some height. But this is *not* what's being checked -- it's being done for a single interpolated example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hx2H0du1JU", "reviewer": "Reviewer_yGMX", "summary": "The paper titled \"DaWin: Training-Free Dynamic Weight Interpolation for Robust Adaptation\" introduces DaWin, a novel method for dynamically merging model weights without additional training. Traditional weight interpolation methods, such as WiSE-FT, use static coefficients for merging models, limiting their effectiveness when models exhibit different levels of expertise across samples. DaWin addresses this limitation by proposing sample-wise dynamic weight interpolation, utilizing entropy-based measures to adjust interpolation coefficients in real-time. DaWin achieves state-of-the-art performance on both robust fine-tuning tasks (ImageNet benchmarks) and multi-task learning (e.g., EuroSAT, MNIST) with minimal computational overhead.", "questions": "1. How well does DaWin generalize to tasks beyond classification benchmarks (e.g., NLP or reinforcement learning)?\n2. Can the reliance on entropy as an expertise proxy limit DaWin's performance in certain scenarios, such as multi-modal data?\n3. How does DaWin compare in terms of inference speed and energy consumption to AdaMerging and WiSE-FT?\n4.What scenarios, if any, might lead to simpler methods outperforming DaWin, and how can this be systematically explored?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Training-free approach: DaWin eliminates the need for additional model training, providing a plug-and-play solution.\n2. Entropy-based dynamic interpolation: This novel use of entropy helps improve sample-wise performance across diverse tasks.\n3. State-of-the-art results: DaWin outperforms WiSE-FT and other interpolation methods on ImageNet and multi-task benchmarks with minimal computational overhead.\n4. Computational efficiency: The use of mixture modeling reduces the inference cost, making DaWin scalable to large datasets.", "weaknesses": "1. **Incremental novelty**: While the method provides refinements over static interpolation, it shares similarities with existing methods (e.g., AdaMerging), which reduces its overall originality.\n2. Over-reliance on entropy: The assumption that entropy reliably measures model expertise across all scenarios is not thoroughly validated, and alternative metrics could be explored further.\n3. Trade-offs not fully explored: The paper does not sufficiently address the performance trade-offs between per-sample interpolation and simpler methods like weight averaging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)