DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging

## Abstract

Abstract Model merging has emerged as an efficient and flexible paradigm for multi-task learning, with numerous methods being proposed in recent years.
However, these state-of-the-art techniques are typically evaluated on benchmark suites that are highly favorable to model merging, and their robustness in more realistic settings remains largely unexplored.
In this work, we first investigate the vulnerabilities of model-merging methods and pinpoint the source-model characteristics that critically underlie them.
Specifically, we identify two factors that are particularly harmful to the merging process: (1) disparities in task vector norms, and (2) the low confidence of the source models. To address this issue, we propose DisTaC ( Dis tillation for Ta sk vector C onditioning), a novel method that pre-conditions these problematic task vectors before the merge. DisTaC leverages knowledge distillation to adjust a task vector’s norm and increase source-model confidence while preserving its essential task-specific knowledge. Our extensive experiments demonstrate that by pre-conditioning task vectors with DisTaC, state-of-the-art merging techniques can successfully integrate models exhibiting the harmful traits—where they would otherwise fail—achieving significant performance gains.
The source code is available at https://github.com/katoro8989/DisTaC

## 1 Introduction

The recent wave of open-sourcing both large pretrained models (Devlin et al., 2019; Rombach et al., 2022; Achiam et al., 2023; Grattafiori et al., 2024) and their fine-tuned downstream variants (Wolf et al., 2019; Taori et al., 2023) has put an unprecedented variety of neural networks within easy reach of anyone. This democratization has, in turn, accelerated research on model merging (Wortsman et al., 2022b; a; Ilharco et al., 2023; Yadav et al., 2023; Akiba et al., 2025)—techniques that create new, customized models by integrating existing fine-tuned models without the need for additional large-scale training.
In particular, a flurry of recent methods aims to build multi-task models by merging networks that have been fine-tuned independently for each task, rather than retraining a single shared model from scratch (Ilharco et al., 2023; Yadav et al., 2023; Ortiz-Jimenez et al., 2023; Wang et al., 2024; Yoshida et al., 2025; Gargiulo et al., 2025). Many of these techniques require only minimal extra training—or none at all. Compared with conventional multi-task learning (MTL), they offer two key advantages: (i) they eliminate the need to aggregate all task-specific labeled data in one location, sidestepping data-access constraints, and (ii) they make it easy to add or edit the model’s skill on a particular task after deployment (Yang et al., 2024a).

On established benchmarks, these approaches have shown promising gains, in some cases approaching the performance of traditional MTL (Gargiulo et al., 2025). Yet those benchmarks are built under conditions that are highly idealized for model merging; how robust current merging methods remain in more practical, pessimistic settings is still largely unknown. Bridging this gap is a prerequisite for real-world application.

To that end, we first pinpointed where generic multi-task model merging pipelines break down. Our analysis reveals two especially harmful factors: (1) differences in task vector norms and (2) low prediction confidence of source models. Figure [1(a)](https://arxiv.org/html/2508.01148v1#S1.F1.sf1) illustrates the vulnerability of recent merging methods to these factors using CLIP (Radford et al., 2021) with a ViT-B-32 (Dosovitskiy et al., 2021) backbone on the eight vision tasks defined in Section [5.1](https://arxiv.org/html/2508.01148v1#S5.SS1): blue bars show the effect of training models with different learning rates, thereby altering task vector norms (see Figure [1(b)](https://arxiv.org/html/2508.01148v1#S1.F1.sf2)), while yellow bars show the effect of label smoothing (Müller et al., 2019) (LS), which reduces model confidence (see Figure [1(c)](https://arxiv.org/html/2508.01148v1#S1.F1.sf3)). In the plot, the horizontal axis lists the merging methods, and the vertical axis reports the average normalized accuracy (Norm. ACC) across the eight tasks, defined as the post-merge accuracy relative to the pre-merge accuracy obtained by individual models for each task. In both cases, every method’s performance degrades substantially compared to the standard baseline, represented by the gray bars (a uniform learning rate of $10^{-5}$ with hard labels), with a maximum 24% drop in Norm. ACC.

Figure: (a)
Refer to caption: x1.png

These failure modes often arise in real-world deployments. For instance, differences in task vector norms can stem from varied learning rates, fine-tuning steps, or weight decay used during the individual fine-tuning of each task (Devlin et al., 2019; Wightman et al., 2021). Low confidence often results from techniques such as LS, Mixup (Zhang et al., 2017), and focal loss (Lin et al., 2017). We therefore contend that models should be pre-conditioned before merging to remove their latent harmfulness.
To this end, we propose DisTaC—Distillation for Task-vector Conditioning—a lightweight knowledge distillation (KD) procedure that tackles both issues using only unlabeled data:
To correct task vector–norm disparities, DisTaC first rescales each vector to a chosen target norm and then restores any performance lost through this scaling by distilling knowledge from the original model.
To address low source-model confidence, it trains the student with a higher temperature than the teacher ($T_{\text{stu}}>T_{\text{tcr}}$), so the student ultimately produces lower-entropy—that is, more confident—predictions.
Algorithm [1](https://arxiv.org/html/2508.01148v1#alg1) combines these two conditioning steps, allowing them to be carried out in a single pass.
Because DisTaC leverages the already-trained task vectors as the initialization for KD and relies solely on unlabeled data, it incurs minimal computational overhead and imposes only modest practical requirements, yet markedly improves the robustness of existing model merging techniques in challenging scenarios.

Empirically, on eight vision tasks with ViT-B-32/L-14 backbones, DisTaC increased post-merge accuracy by up to 20.8 percentage points and restored the best-performing TSVM merge’s normalized accuracy from 68% to 92% under low-confidence conditions, thereby matching the conventional “ideal” benchmark performance (i.e., merging high-confidence models with uniform task vector norms)—all with minimal computational cost.

Our contributions are as follows:

- •
We identify two failure modes in model merging: (i) the task vector norms of the source models differ (Section [3.1](https://arxiv.org/html/2508.01148v1#S3.SS1)), and (ii) the source models’ outputs are low-confidence or even well-calibrated (i.e., their predicted probabilities match the true frequency of correctness) (Section [3.2](https://arxiv.org/html/2508.01148v1#S3.SS2)). We provide theoretical explanations and empirical results for each of these phenomena.
- •
We propose DisTaC, a distillation method of source model’s weights under appropriate conditions (Section [4](https://arxiv.org/html/2508.01148v1#S4)), and demonstrate that it mitigates aforementioned failure modes (Section [5.2.1](https://arxiv.org/html/2508.01148v1#S5.SS2.SSS1)). Our DisTaC is a computationally efficient method, as it requires only a small number of training steps and relies solely on unlabeled data (Section [5.2.2](https://arxiv.org/html/2508.01148v1#S5.SS2.SSS2)).
- •
From our analysis, we present two practical guidelines for practical model merging: (i) when the task vector norms differ, it is better to shrink the larger vector rather than stretch the smaller one (Section [6.1](https://arxiv.org/html/2508.01148v1#S6.SS1)); and (ii) when the source models have low confidence, it is more effective to make them overconfident before merging, and then apply a calibration method to the merged model (Section [6.2](https://arxiv.org/html/2508.01148v1#S6.SS2)).

## 2 Preliminaries

##### Notation.

For any parameter vector $\bm{\theta}\in\mathbb{R}^{d}$,
let $f(\cdot\,;\bm{\theta}):\mathcal{X}\to\mathbb{R}^{C}$
denote the neural network that maps an input $x\!\in\!\mathcal{X}$ to a $C$-dimensional logit vector, where $C$ is the number of classes.
We target a multi-task scenario comprising $T$ supervised tasks.
Let $\bm{\theta}_{\text{pre}}\in\mathbb{R}^{d}$ be the parameters of an open-source pretrained backbone.
For each task $t\in\{1,\dots,T\}$, we obtain a model that has already been fine-tuned on the corresponding labelled dataset
$\mathcal{D}_{t}=\bigl{\{}(x_{i}^{(t)},y_{i}^{(t)})\bigr{\}}_{i=1}^{n_{t}},$
yielding task-specific weights $\bm{\theta}_{t}\in\mathbb{R}^{d}$.

### 2.1 Model Merging for Multi-Task Learning

Recent model merging techniques operate on the task vectors (Ilharco et al., 2023) $\bm{\tau}_{t}\;:=\;\bm{\theta}_{t}-\bm{\theta}_{\text{pre}}$ and obtain a single multi–task model by linearly combining them:

$$ $\bm{\theta}_{\text{mtl}}\;=\;\bm{\theta}_{\text{pre}}\;+\;\sum_{t=1}^{T}P_{t}\,\bm{\tau}_{t},$ (1) $$

where each $P_{t}\in\mathbb{R}^{d\times d}$ is a method-specific matrix that mitigates inter-task interference.

In the following, we explain the $P_{t}$ used in each merging method.

Uniform averaging: $P_{t}=\tfrac{1}{T}\,I_{d}$.

Task arithmetic (Ilharco et al., 2023): $P_{t}=\lambda_{t}\,I_{d},\quad\text{where}\ \lambda_{t}\in\mathbb{R}$.

Ties-Merging (Yadav et al., 2023): $P_{t}=\lambda_{t}\,\bm{m}_{\text{Ties},t}\,I_{d},\quad\text{where}\ \lambda_{t}\in\mathbb{R},\ \bm{m}_{\text{Ties},t}\in\{0,1\}^{d}$.
$\bm{m}_{\text{Ties},t}$ is determined by the norm of each weight parameter to mitigate inter-task conflicts.

Consensus Merging (Wang et al., 2024): $P_{t}=\lambda_{t}\,\bm{m}_{\text{Cons},t}\,I_{d},\quad\text{where}\ \lambda_{t}\in\mathbb{R},\ \bm{m}_{\text{Cons},t}\in\{0,1\}^{d}$.
The framework is the same as Ties-Merging, but the binary mask $\bm{m}_{\text{Cons},t}$ is determined in the following steps.
First, create the TALL mask $\bm{m}_{\text{TALL},t}$, which is a binary mask of weights where each element is set to 1 if the norm of $\bm{\tau}_{t}$ is larger than the weighted distance between $\bm{\tau}_{t}$ and $\sum_{t=1}^{T}\bm{\tau}_{t}$. Then, create $\bm{m}_{\text{Cons},t}$, where each element is set to 1 if the corresponding element of $\bm{m}_{\text{TALL},t}$ is 1 in at least $k$ tasks, reflecting agreement among the source models regarding the importance.

TSVM (Gargiulo et al., 2025) cannot be expressed within the framework of Eq. (1). Instead, it suppresses task interference by whitening the matrices $\mathbf{U}_{t}$ and $\mathbf{V}_{t}$ obtained from the singular value decomposition of the task vectors $\bm{\tau}_{t}=\mathbf{U}_{t}\mathbf{\Sigma}_{t}\mathbf{V}_{t}^{\top}$.

### 2.2 Knowledge Distillation

Knowledge distillation (KD) is a model compression and transfer paradigm in which a compact student network is trained to replicate the behavior of a larger, well-performing teacher network (Hinton et al., 2015). By minimizing a joint loss that combines ground-truth supervision with a soft-target signal derived from the teacher’s output distribution, the student acquires the teacher’s dark knowledge—fine-grained inter-class relationships encoded in the soft logits—while retaining a substantially smaller parameter footprint.
Formally, for a given input $\bm{x}$, let $\bm{z}_{\text{tcr}}:=f(\bm{x}\,;\bm{\theta}_{\text{tcr}})\in\mathbb{R}^{C}$ and $\bm{z}_{\text{stu}}:=f(\bm{x}\,;\bm{\theta}_{\text{stu}})\in\mathbb{R}^{C}$ be the output logits from the teacher and student models, parameterized by $\bm{\theta}_{\text{tcr}}\in\mathbb{R}^{d}$ and $\bm{\theta}_{\text{stu}}\in\mathbb{R}^{d}$, respectively.
The KD objective then augments the conventional cross-entropy loss $\mathcal{L}_{\text{CE}}$ with a softened Kullback-Leibler (KL) divergence term:

$$ $\displaystyle\mathcal{L}_{\text{KD}}=(1-\zeta)\,\mathcal{L}_{\text{CE}}\bigl{(}\bm{z}_{\text{stu}},\,y\bigr{)}\;+\;\zeta\ T_{\text{tcr}}T_{\text{stu}}\,\mathrm{KL}\Bigl{(}\sigma(\bm{z}_{\text{tcr}}/T_{\text{tcr}})\,\bigl{\|}\,\sigma(\bm{z}_{\text{stu}}/T_{\text{stu}})\Bigr{)},$ (2) $$

where $\sigma$ denotes the softmax, $T_{\text{tcr}},T_{\text{stu}}\geq 1$ is the distillation temperature, and $\zeta\in[0,1]$ balances hard versus soft supervision.

## 3 Failure Modes in Model Merging

### 3.1 Task Vector Norm Disparity

We begin by demonstrating that differences in task vector norms can severely impair model merging.
In practical fine-tuning, practitioners select diverse hyperparameters, including learning rate, number of training steps, weight decay, and optimizer, each of which influences the distance between the final weights and their initialization, i.e. the norm of the task vector.

To quantify this effect, we fine-tuned CLIP models with Vision Transformers (ViTs) backbones, specifically ViT-B-32, on eight vision tasks as introduced in Section [5.1](https://arxiv.org/html/2508.01148v1#S5.SS1) with two learning rates, $10^{-5}$ (gray) and $10^{-4}$ (blue), and plotted the resulting task vector norms in Figure [1(b)](https://arxiv.org/html/2508.01148v1#S1.F1.sf2). Across all tasks, we observe a $5$-$7\times$ gap between the two settings. Crucially, the difference is not confined to any particular layer: parameter scales diverge consistently throughout the network, as demonstrated in Section [C.1](https://arxiv.org/html/2508.01148v1#A3.SS1).

Figure [1(a)](https://arxiv.org/html/2508.01148v1#S1.F1.sf1) reports the corresponding merge performance.
The gray bars denote the baseline where all eight tasks are fine-tuned with $10^{-5}$, while the blue bars show the average over eight experiments in each of which one task is replaced with a higher learning rate of $10^{-4}$ and the other seven remain unchanged. We measure performance using normalized accuracy. Injecting a single high-norm task vector degrades every merging method, with losses of up to $14\%$. These results confirm that norm discrepancies pose a fundamental obstacle to robust task vector merging.

The detrimental effect of norm disparity on model merging can be explained with a straightforward theoretical analysis. Let us consider two task vectors, $\bm{\tau}_{1}$ and $\bm{\tau}_{2}$, derived from different tasks, and assume their norms have a large discrepancy, i.e., $||\bm{\tau}_{2}||\gg||\bm{\tau}_{1}||$. In a simple addition merge, the resulting task vector is $\bm{\tau}_{\text{merge}}=\bm{\tau}_{1}+\bm{\tau}_{2}$. The directional alignment of this merged vector can be analyzed using cosine similarity.

The similarity between $\bm{\tau}_{\text{merge}}$ and the high-norm vector $\bm{\tau}_{2}$ approaches one. By expanding the cosine similarity definition, we have $\cos(\bm{\tau}_{\text{merge}},\bm{\tau}_{2})=(\bm{\tau}_{1}\cdot\bm{\tau}_{2}+||\bm{\tau}_{2}||^{2})/(||\bm{\tau}_{1}+\bm{\tau}_{2}||\cdot||\bm{\tau}_{2}||)$.
Since $||\bm{\tau}_{2}||\gg||\bm{\tau}_{1}||$, the numerator is dominated by the $||\bm{\tau}_{2}||^{2}$ term, while in the denominator, $||\bm{\tau}_{1}+\bm{\tau}_{2}||\approx||\bm{\tau}_{2}||$. This leads to the approximation

$$ $\cos(\bm{\tau}_{\text{merge}},\bm{\tau}_{2})\approx||\bm{\tau}_{2}||^{2}/(||\bm{\tau}_{2}||\cdot||\bm{\tau}_{2}||)=1.$ $$

Conversely, its alignment with the low-norm vector $\bm{\tau}_{1}$ becomes negligible. The corresponding similarity is $\cos(\bm{\tau}_{\text{merge}},\bm{\tau}_{1})=(||\bm{\tau}_{1}||^{2}+\bm{\tau}_{1}\cdot\bm{\tau}_{2})/(||\bm{\tau}_{1}+\bm{\tau}_{2}||\cdot||\bm{\tau}_{1}||)$. Assuming the tasks are orthogonal ($\bm{\tau}_{1}\cdot\bm{\tau}_{2}=0$), as observed by Ilharco et al. (2023), and using the same approximation for the denominator, the expression simplifies to

$$ $\cos(\bm{\tau}_{\text{merge}},\bm{\tau}_{1})\approx||\bm{\tau}_{1}||^{2}/(||\bm{\tau}_{2}||\cdot||\bm{\tau}_{1}||)=||\bm{\tau}_{1}||/||\bm{\tau}_{2}||\approx 0.$ $$

This demonstrates that the merged solution almost entirely inherits the directional characteristics of the high-norm task, leading to a catastrophic drop in performance for the task associated with the low-norm vector and thus degrading the overall utility of the merged model.

### 3.2 Low-Confidence Source Models

We now show that low confidence constitutes a second, equally damaging
failure mode. Paradoxically, models that are well calibrated can be
fragile from the perspective of model merging; conversely, we argue that
the more over-confident a source model is, the more robust it becomes
to merging.

A model’s decisiveness can be quantified by the entropy of its predictive
distribution. Using the same experimental configuration as in
Section [3.1](https://arxiv.org/html/2508.01148v1#S3.SS1), we replaced the learning-rate manipulation
with a single change: turning label smoothing on or off. Figure
[1(c)](https://arxiv.org/html/2508.01148v1#S1.F1.sf3) plots the resulting prediction entropies: the gray bars
correspond to training without label smoothing, while the yellow bars use
$\alpha=0.1$. The vertical axis is logarithmic; with label smoothing the
entropy increases by up to three orders of magnitude.

Figure [1(a)](https://arxiv.org/html/2508.01148v1#S1.F1.sf1) (yellow bars) shows how this reduced
confidence affects merging. Across all algorithms, the normalized accuracy
drops markedly—by as much as 24% compared to the no-smoothing
baseline. This degradation exceeds that caused by norm discrepancies in the
previous section, underscoring how harmful low-confidence source models can be.
In short, routine training choices that alter confidence (e.g. label
smoothing, Mixup, focal loss) can induce large swings in post-merge performance.
These phenomena can also be supported from a theoretical perspective. (Appendix [A](https://arxiv.org/html/2508.01148v1#A1))

## 4 Knowledge Distillation for Task vector
Conditioning

Figure: Algorithm 1 DisTaC

Here, we propose DisTaC—Distillation for Task vector Conditioning—a KD–based pre-conditioning method that eliminates the harmful effects of individual task vectors during model merging, as identified in Section [3](https://arxiv.org/html/2508.01148v1#S3).

### 4.1 Task Vector Norm Conditioning

First, to correct task vector norm disparity, DisTaC harmonizes the norms while preserving single-task accuracy.
A naive countermeasure is to adjust the norm by scaling the task vector, i.e. replacing $\bm{\tau}_{t}$ with $\kappa_{t}\bm{\tau}_{t}$ using a scalar scaling factor $\kappa_{t}$.
Unfortunately, this constant rescaling offers no guarantee of performance retention and can severely degrade accuracy relative to the pre-merge model.

We therefore propose to recover the lost performance through KD: starting from
$\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}_{t}$, we treat the pre-merge model as the teacher and distill its predictions into the rescaled student using only unlabeled data from the same task as the one underlying $\bm{\tau}_{t}$.
Since DisTaC relies solely on unlabeled data, it uses soft-target distillation only, i.e., we fix $\zeta=1$ in Eq. [2](https://arxiv.org/html/2508.01148v1#S2.E2), omitting the cross-entropy loss entirely.

Although one might instead fine-tune $\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}_{t}$ with labeled examples, obtaining a sufficiently large supervised corpus at merge time is typically impractical.
By contrast, access to unlabeled data is commonly assumed during model merging (Yang et al., 2024b; Yan et al., 2025; Yoshida et al., 2025), and KD imposes only mild additional requirements.

To prevent the task vector norm from drifting far from $\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}_{t}$ during KD, we include an $\ell_{2}$ regularizer on their difference, as shown in Algorithm [1](https://arxiv.org/html/2508.01148v1#alg1).

### 4.2 Source Model Confidence Conditioning

To mitigate low-confidence issues, DisTaC aims to increase each source model’s confidence before merging, thereby rendering the model more robust to the merge.
Here the student and the teacher are identical at initialization, i.e. $\bm{\theta}_{t}=\bm{\theta}_{\text{pre}}+\bm{\tau}_{t}$.
We set the student temperature $T_{\text{stu}}$ higher than the teacher temperature $T_{\text{tcr}}$ so that the student, trained on a higher-entropy distribution, is pushed toward a lower-entropy (more confident) output when the temperature is later reset to 1.
Consequently, the distilled student becomes more confident than its teacher.

One may worry that the over-confidence harms model reliability in practice.
However, standard post-hoc calibration methods (e.g. temperature scaling) can mitigate over-confidence, whereas merging with an under-confident model leads to large performance drops that make the merged model impractical.
A detailed discussion appears in Section [6.2](https://arxiv.org/html/2508.01148v1#S6.SS2).

##### Unified algorithm.

The two conditioning strategies above are unified by Algorithm [1](https://arxiv.org/html/2508.01148v1#alg1).
When both norm disparity and low-confidence coexist, they can be mitigated simultaneously by choosing an appropriate scaling factor $\kappa_{t}$ and temperature pair $(T_{\text{tcr}},T_{\text{stu}})$.

## 5 Experiment

**Table 1: Comparison of post-merge accuracy across fine-tuning configurations and the effect of DisTaC. Absolute accuracy is displayed in a large font size, whereas normalized accuracy appears in parentheses in a smaller font. “Individual” denotes the average performance of the source models on their respective tasks, and “MTL” represents the performance of conventional MTL. When the task vector norms diverge (Norm Mismatch) or the source models exhibit low confidence (Low Confidence), performance consistently degrades relative to the standard benchmark setting (Original). Under these conditions, DisTaC effectively pre-conditions the source models, achieving performance comparable to Original even in both stringent settings.**
| Method |  | Original | Norm Mismatch | Low Confidence |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ViT-B-32 | ViT-L-14 | ViT-B-32 | ViT-L-14 | ViT-B-32 | ViT-L-14 |
| Pre-trained |  | 47.3 | 65.1 | 47.3 | 65.1 | 47.3 | 65.1 |
| Individual |  | 89.9 | 93.7 | 89.3 | 93.3 | 89.8 | 94.0 |
| MTL |  | 87.8 | 92.6 | - | - | - | - |
| Task arithmetic |  | 70.4 (78.0) | 84.0 (89.3) | 63.6 (71.8) | 78.6 (84.2) | 51.0 (58.3) | 66.9 (71.5) |
| Task arithmetic | + DisTaC | - | - | 70.0 (78.2) | 83.9 (89.6) | 63.6 (72.2) | 77.6 (83.3) |
| TIES |  | 74.0 (82.0) | 85.0 (91.9) | 59.1 (66.4) | 74.0 (79.5) | 54.5 (62.0) | 68.3 (73.0) |
| TIES | + DisTaC | - | - | 73.1 (81.0) | 84.4 (90.2) | 68.7 (77.9) | 79.4 (85.4) |
| Consensus TA |  | 74.8 (82.8) | 85.3 (90.7) | 68.8 (77.0) | 82.0 (87.6) | 54.6 (62.0) | 68.6 (73.2) |
| Consensus TA | + DisTaC | - | - | 73.7 (82.2) | 84.9 (90.7) | 67.7 (76.5) | 80.0 (85.8) |
| TSVM |  | 83.3 (92.4) | 90.5 (96.3) | 72.2 (80.2) | 84.8 (90.7) | 60.7 (68.4) | 71.6 (76.4) |
| TSVM | + DisTaC | - | - | 82.9 (91.8) | 90.3 (96.6) | 81.5 (91.8) | 89.7 (96.2) |

### 5.1 Setup

We conducted experiments in a multitask setting following Ilharco et al. (2023). Specifically, we adopted eight vision tasks—Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp et al., 2011), MNIST (LeCun, 1998), RESISC45 (Cheng et al., 2017), SUN397 (Xiao et al., 2016), and SVHN (Netzer et al., 2011). Our models applied ViT-B-32 and ViT-L-14 to CLIP.
We evaluated post-merge performance using absolute accuracy and normalized accuracy under the two aforementioned failure modes: the case with diverged task vector norms (Norm Mismatch) and the case with low-confidence source models (Low Confidence). The detailed settings for each scenario followed those described in Section [3](https://arxiv.org/html/2508.01148v1#S3).
We adopt four merging methods as baselines: task arithmetic (Ilharco et al., 2023), Ties-Merging (TIES) (Yadav et al., 2023), Consensus Merging (Consensus TA)(Wang et al., 2024), and TSVM (Gargiulo et al., 2025).
For DisTaC, knowledge distillation was run for $K=500$ steps.
In the Norm Mismatch regime we assign a task–specific scaling coefficient $\kappa_{t}$ individually for each of the eight norm–disparity configurations: the task vector with the largest $\ell_{2}$-norm is rescaled so that, after scaling, its norm equals the mean norm of the remaining seven task vectors. A neutral temperature pair is then used, $(T_{\text{tcr}},T_{\text{stu}})=(10,10)$.
In the Low Confidence regime we instead fix $\kappa_{t}=1$ and sharpen the student by adopting a more asymmetric temperature pair, $(T_{\text{tcr}},T_{\text{stu}})=(1,10)$.
More detailed settings can be found in Appendix [B](https://arxiv.org/html/2508.01148v1#A2).

Figure: (a) Norm Mismatch
Refer to caption: x4.png

### 5.2 Results

#### 5.2.1 Merging Performance

Table [1](https://arxiv.org/html/2508.01148v1#S5.T1) summarizes the results.
Absolute accuracy is displayed in a larger font, whereas normalized accuracy appears in parentheses in a smaller font.
As noted in Section [3](https://arxiv.org/html/2508.01148v1#S3), all methods exhibit a substantial and consistent performance decline relative to the conventional configuration (Original) under both failure modes, revealing a clear vulnerability (white rows).
The rows highlighted in gray show the performance obtained by first applying DisTaC for pre-conditioning and then merging. DisTaC consistently enhances merge performance, yielding gains of up to 20.8% absolute accuracy for ViT-B-32 and 18.1% for ViT-L-14.
Moreover, for TSVM, which achieves the highest merge performance, DisTaC raises the accuracy under both failure modes to a level comparable with the Original configuration, indicating that the intended merge performance is robustly maintained even in challenging scenarios.

#### 5.2.2 Efficiency of DisTaC

Here, we present how the single-task performance on each task, the task vector norm, and the prediction entropy change during the KD process of DisTaC, as well as the computational cost required for sufficiently thorough training.

Figure [2](https://arxiv.org/html/2508.01148v1#S5.F2) shows the average over eight vision tasks of the training history when KD by DisTaC is applied to ViT-B-32. The blue curve denotes the test accuracy relative to the teacher’s test accuracy, the green curve the task vector norm relative to its value at the initialization point, and the orange curve the test prediction entropy.

First, Figure [2(a)](https://arxiv.org/html/2508.01148v1#S5.F2.sf1) depicts the training history under the Norm Mismatch setting in Table [1](https://arxiv.org/html/2508.01148v1#S5.T1). It achieves performance comparable to, or even surpassing, the teacher model’s test performance within 500 steps, while the $\ell_{2}$ regularizer of DisTaC keeps the task vector norm to roughly $1.1\times$ that of the initialization point, $\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}$, at the end of the 500 steps.

Of particular interest is that DisTaC occasionally surpasses the teacher model’s test performance. We identify two factors underlying this phenomenon. The first is the scale given by $\kappa_{t}$. In particular, we observed that reducing $\kappa_{t}$ can sometimes improve generalization performance. That is, the DisTaC initialization point already outperforms the teacher model, and we observed this in every instance in which the teacher model was exceeded. This phenomenon of the student outperforming the teacher is confirmed in (Furlanello et al., 2018), where it has been shown that a student can surpass the teacher by repeating KD between identical architectures. Furthermore, in this case, since KD is performed while keeping the student’s norm smaller than the teacher’s, it is plausible that a regularization effect similar to weight decay is being exhibited.

Next, Figure [2(b)](https://arxiv.org/html/2508.01148v1#S5.F2.sf2) presents the training history under the Low Confidence setting in Table [1](https://arxiv.org/html/2508.01148v1#S5.T1). Within 500 steps—particularly during the first 100 steps—it achieves a substantial reduction in prediction entropy while maintaining test accuracy at a level nearly equivalent to that of the teacher model.

## 6 Discussion

### 6.1 Stretching vs. Shrinking Task Vectors

Figure: Figure 3: Effect of scaling task vectors on test accuracy. For each of the eight vision tasks (ViT-B-32), we evaluate the model $\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}$ as the scaling factor $\kappa_{t}$ varies from $0.0$ to $3.0$. Model performance is more robust to shrinking the task vector than to stretching it, suggesting that when harmonizing task vector norms, longer vectors should be shrunk to match shorter ones.
Refer to caption: x6.png

When task vectors differ significantly in norm, a natural question arises:
Should shorter vectors be stretched to match longer ones, or should longer vectors be shrunk to match the shorter ones?
Our findings support the latter—we advocate shrinking the longer vectors.

There are several reasons for this.
First, it is conceivable that model performance is more robust to scaling down a task vector than scaling it up.
Figure [3](https://arxiv.org/html/2508.01148v1#S6.F3) shows how test accuracy varies across vision tasks when applying different scaling factors $\kappa_{t}$ to the task vector, i.e., evaluating $\bm{\theta}_{\text{pre}}+\kappa_{t}\bm{\tau}$ for $\kappa_{t}\in[0.0,3.0]$.
Shrinking the task vector ($\kappa_{t}<1.0$) retains performance comparable to or even better than the original fine-tuned model across a broad range.
In contrast, stretching beyond $\kappa_{t}=1.0$ degrades accuracy, and by $\kappa_{t}=3.0$, the model underperforms even the zero-shot baseline across all tasks.
A similar trend was also observed for ViT-L-14 (see Section [C.3](https://arxiv.org/html/2508.01148v1#A3.SS3)).

As shown earlier in Figure [1(b)](https://arxiv.org/html/2508.01148v1#S1.F1.sf2), real-world fine-tuning pipelines often result in over $5\times$ variation in task vector norm due to differing learning rates or training durations.
In such cases, stretching small-norm vectors to match larger ones risks disrupting the pretrained model’s useful representations and is therefore undesirable.

Furthermore, Ilharco et al. (2023) observed that merging task vectors with smaller norms tends to yield better performance.
A likely explanation is that smaller displacements remain within the local linear regime around $\bm{\theta}_{\text{pre}}$, where first-order approximations hold more accurately.
This also aligns with the NTK perspective discussed in Ortiz-Jimenez et al. (2023); Yoshida et al. (2025), under which merging remains valid and weight disentanglement is preserved near the pretrained initialization.

Taken together, these observations strongly suggest that when normalizing task vectors for merging, it is preferable to shrink the longer ones rather than stretch the shorter ones.

### 6.2 Confidence Reliability in Model Merging

Figure: (a) Reliability diagram over different label smoothing strengths
Refer to caption: x7.png

As noted in Section [3.2](https://arxiv.org/html/2508.01148v1#S3.SS2), successful model merging often conflicts with maintaining reliable confidence estimates in both the source and merged models.
Figure [4](https://arxiv.org/html/2508.01148v1#S6.F4) illustrates this trade-off by sweeping the label-smoothing strength $\alpha$ used during fine-tuning of the source models.

First, the calibration curves in Figure [4(a)](https://arxiv.org/html/2508.01148v1#S6.F4.sf1) show that a model trained without label smoothing (dark-purple line) is strongly over-confident—consistent with the well-known tendency of modern deep networks (Guo et al., 2017).
As $\alpha$ increases from 0.01 to 0.1 (red $\to$ yellow), the models become well-calibrated and eventually under-confident, matching the observations of Müller et al. (2019).

Figure [4(b)](https://arxiv.org/html/2508.01148v1#S6.F4.sf2) then reports the normalized accuracy obtained when these source models are merged.
Accuracy decreases monotonically with larger $\alpha$, revealing an inverse correlation between label-smoothing strength and merge performance.

In short, current merging methods perform best when the source models are deliberately over-confident.
To retain reliable confidence after merging, we therefore advocate applying post-hoc calibration—such as temperature scaling (Guo et al., 2017)—to the merged model rather than trying to calibrate the sources beforehand.

## 7 Related Work

### 7.1 Model Merging and Task Arithmetic

Research on integrating multiple neural network models by performing operations on their parameters has been widely conducted, starting with Utans (1996).
These techniques enable a model to learn diverse tasks with less time and computational resources, and have become increasingly important in recent years as the number of model parameters has grown dramatically.
For instance, in early approaches to model merging, models with the same architecture were fine-tuned and then merged by averaging their parameters (Wortsman et al., 2022a; Choshen et al., 2022).
More sophisticated methods have since been proposed, such as Fisher Merging (Matena & Raffel, 2022), which is based on maximizing the posterior probability of the model, and RegMean (Jin et al., 2023), which minimizes the distance between output activations before and after merging.
In contrast, task arithmetic (Ilharco et al., 2023) focuses on the task vector, defined as the difference in parameters between a fine-tuned model and a pre-trained model, and performs addition and subtraction of task vectors in parameter space.
This approach offers the advantage of allowing flexible, localized modifications to the model and has found applications across diverse tasks (Huang et al., 2024; Su et al., 2024; Yoshikawa et al., 2025; Naganuma et al., 2025).

Recent research on task arithmetic has theoretically analyzed the simple addition of task vectors and proposed multiple methods to address its shortcomings.
Approaches aimed at improving the properties of task vectors focus on the linearity in fine-tuning (Ortiz-Jimenez et al., 2023; Yoshida et al., 2025).
These methods, based on the Neural Tangent Kernel (NTK) (Jacot et al., 2018), treat the model’s output as linear during fine-tuning in order to reflect vector operations in parameter space onto the model’s inputs and outputs.
Meanwhile, several studies have been conducted from the perspective of mitigating interference between task vectors.
TIES-Merging (Yadav et al., 2023) emphasizes the removal of redundant elements and the consideration of sign in each vector dimension.
AdaMerging (Yang et al., 2024b), on the other hand, automatically adjusts merging coefficients per task and per layer to reduce task interference and enhance robustness through test-time adaptation.
Wang et al. (2024) introduced a framework for pinpointing the parameters that carry information shared across tasks and, on that basis, proposed Consensus Merging, which builds task-wise masks that align more closely with inter-task consensus than the masks used in TIES-Merging.
More recently, TSVM (Gargiulo et al., 2025) interprets task interference as non-orthogonality among the layer-wise singular vectors of the task vectors; by whitening those singular directions, TSVM further improves merge quality.

Despite these advances, Ilharco et al. (2023) and nearly all follow-up studies on multi-task model merging benchmark their methods under highly idealized settings, leaving real-world failure modes largely unexplored.
In this work, we show that (i) discrepancies in task vector norms and (ii) low source-model confidence are key sources of interference. We introduce DisTaC as a simple pre-conditioning step that mitigates both problems before merging.

### 7.2 Knowledge Distillation

DisTaC addresses the limitations of existing task arithmetic methods by incorporating knowledge distillation.
Knowledge distillation is a technique proposed for transferring knowledge from a teacher model to a smaller student model (Hinton et al., 2015). Although initially intended for model compression (Hinton et al., 2015; Kim et al., 2018; Sanh et al., 2020), it has also been applied in contexts such as self-distillation, where repeated distillation between models of the same architecture leads to performance improvement (Furlanello et al., 2018; Zhang et al., 2019; Zhang & Sabuncu, 2020).
Among these applications, several studies have explored generating models that can handle multiple tasks by distilling knowledge from multiple teacher models (Luo et al., 2019; Hao et al., 2023).
These approaches achieve distillation by mapping the parameters of multiple teacher models into a shared space for the student model.
Conversely, it is also possible to distill models with different architectures individually to obtain task vectors, which can then be merged using task arithmetic (Merugu et al., 2025).
DisTaC adopts the latter approach and resolves the issue of variability in the norms of task vectors by obtaining them through distillation.

Applying distillation to task arithmetic requires addressing the impact of soft targets.
Numerous studies have analyzed the effects of label smoothing in the context of knowledge distillation (Müller et al., 2019; Shen et al., 2021; Chandrasegaran et al., 2022; Zheng & YANG, 2024).
In this study, we demonstrate that fine-tuning with soft targets significantly affects the models obtained through mode merging, and propose a method to mitigate this effect by increasing the confidence of the student model.

## 8 Conclusion

We presented DisTaC, a lightweight and practical pre-conditioning method for task vectors that improves the robustness of model merging in multi-task learning. Our analysis identified two major failure modes—norm disparity and low source-model confidence—that frequently occur in real-world merging scenarios. DisTaC addresses both issues simultaneously via KD on unlabeled data, requiring only minimal computational cost and no access to task labels. Through extensive experiments, we demonstrated that DisTaC not only recovers performance degraded by task vector scaling, but also enhances confidence in the source models without sacrificing generalization. Furthermore, we showed that DisTaC enables state-of-the-art merging methods to succeed in challenging cases where they would otherwise fail. Our findings highlight the importance of task vector conditioning, and we believe that DisTaC provides a simple yet powerful tool to make model merging more reliable and broadly applicable.

## Acknowledgments

This work was (partly) achieved through the use of large-scale computer systems at D3 Center, The University of Osaka.
Hiroki Naganuma acknowledges funding
support from the ANRI Fellowship for this work.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Akiba et al. (2025)
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha.
Evolutionary optimization of model merging recipes.
*Nature Machine Intelligence*, pp.  1–10, 2025.
- Chandrasegaran et al. (2022)
Keshigeyan Chandrasegaran, Ngoc-Trung Tran, Yunqing Zhao, and Ngai-Man Cheung.
Revisiting label smoothing and knowledge distillation compatibility: What was missing?
In *International conference on machine learning*, pp.  2890–2916. PMLR, 2022.
- Cheng et al. (2017)
Gong Cheng, Junwei Han, and Xiaoqiang Lu.
Remote sensing image scene classification: Benchmark and state of the art.
*Proceedings of the IEEE*, 105(10):1865–1883, 2017.
- Choshen et al. (2022)
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz.
Fusing finetuned models for better pretraining.
*arXiv preprint arXiv:2204.03044*, 2022.
- Cimpoi et al. (2014)
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.
Describing textures in the wild.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp.  3606–3613, 2014.
- Devlin et al. (2019)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
In *Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)*, pp.  4171–4186, 2019.
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=YicbFdNTTy](https://openreview.net/forum?id=YicbFdNTTy).
- Furlanello et al. (2018)
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.
Born again neural networks.
In Jennifer Dy and Andreas Krause (eds.), *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*, pp.  1607–1616. PMLR, 10–15 Jul 2018.
URL [https://proceedings.mlr.press/v80/furlanello18a.html](https://proceedings.mlr.press/v80/furlanello18a.html).
- Gargiulo et al. (2025)
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola.
Task singular vectors: Reducing task interference in model merging.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp.  18695–18705, 2025.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guo et al. (2017)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.
On calibration of modern neural networks.
In *International conference on machine learning*, pp.  1321–1330. PMLR, 2017.
- Hao et al. (2023)
Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu.
One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp.  79570–79582. Curran Associates, Inc., 2023.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/file/fb8e5f198c7a5dcd48860354e38c0edc-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/fb8e5f198c7a5dcd48860354e38c0edc-Paper-Conference.pdf).
- Helber et al. (2019)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7):2217–2226, 2019.
- Hinton et al. (2015)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
*arXiv preprint arXiv:1503.02531*, 2015.
- Horn & Johnson (2012)
Roger A Horn and Charles R Johnson.
*Matrix analysis*.
Cambridge university press, 2012.
- Huang et al. (2024)
Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee.
Chat vector: A simple approach to equip LLMs with instruction following and model alignment in new languages.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp.  10943–10959, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.590.
URL [https://aclanthology.org/2024.acl-long.590/](https://aclanthology.org/2024.acl-long.590/).
- Ilharco et al. (2023)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.
Editing models with task arithmetic.
*arXiv preprint arXiv:2212.04089*, 2023.
- Jacot et al. (2018)
Arthur Jacot, Franck Gabriel, and Clément Hongler.
Neural tangent kernel: Convergence and generalization in neural networks.
*Advances in Neural Information Processing Systems*, 31, 2018.
- Jin et al. (2023)
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.
Dataless knowledge fusion by merging weights of language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=FCnohuR6AnM](https://openreview.net/forum?id=FCnohuR6AnM).
- Kim et al. (2018)
Jangho Kim, Seonguk Park, and Nojun Kwak.
Paraphrasing complex network: Network compression via factor transfer.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018.
URL [https://proceedings.neurips.cc/paper_files/paper/2018/file/6d9cb7de5e8ac30bd5e8734bc96a35c1-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2018/file/6d9cb7de5e8ac30bd5e8734bc96a35c1-Paper.pdf).
- Kimura & Naganuma (2025)
Masanari Kimura and Hiroki Naganuma.
Geometric insights into focal loss: Reducing curvature for enhanced model calibration.
*Pattern Recognition Letters*, 189:195–200, 2025.
- Krause et al. (2013)
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization.
In *Proceedings of the IEEE international conference on computer vision workshops*, pp.  554–561, 2013.
- LeCun (1998)
Yann LeCun.
The mnist database of handwritten digits.
*http://yann. lecun. com/exdb/mnist/*, 1998.
- Lin et al. (2017)
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In *Proceedings of the IEEE international conference on computer vision*, pp.  2980–2988, 2017.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations*, 2019.
URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7).
- Luo et al. (2019)
Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song.
Knowledge amalgamation from heterogeneous networks by common feature learning.
In *Proceedings of the 28th International Joint Conference on Artificial Intelligence*, IJCAI’19, pp.  3087–3093. AAAI Press, 2019.
ISBN 9780999241141.
- Matena & Raffel (2022)
Michael S Matena and Colin A Raffel.
Merging models with fisher-weighted averaging.
*Advances in Neural Information Processing Systems*, 35:17703–17716, 2022.
- Merugu et al. (2025)
Ranjith Merugu, Bryan Bo Cao, and Shubham Jain.
Statsmerging: Statistics-guided model merging via task-specific teacher distillation, 2025.
URL [https://arxiv.org/abs/2506.04567](https://arxiv.org/abs/2506.04567).
- Müller et al. (2019)
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.
When does label smoothing help?
*Advances in neural information processing systems*, 32, 2019.
- Naganuma et al. (2025)
Hiroki Naganuma, Kotaro Yoshida, Laura Gomezjurado Gonzalez, Takafumi Horie, Yuji Naraki, and Ryotaro Shimizu.
On fairness of task arithmetic: The role of task vectors.
*arXiv preprint arXiv:2505.24262*, 2025.
- Netzer et al. (2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.
Reading digits in natural images with unsupervised feature learning.
In *NIPS workshop on deep learning and unsupervised feature learning*. Granada, 2011.
- Ortiz-Jimenez et al. (2023)
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard.
Task arithmetic in the tangent space: Improved editing of pre-trained models.
*Advances in Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=0A9f2jZDGW](https://openreview.net/forum?id=0A9f2jZDGW).
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp.  8748–8763. PMLR, 2021.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  10684–10695, 2022.
- Sanh et al. (2020)
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
URL [https://arxiv.org/abs/1910.01108](https://arxiv.org/abs/1910.01108).
- Shen et al. (2021)
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides.
Is label smoothing truly incompatible with knowledge distillation: An empirical study.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=PObuuGVrGaZ](https://openreview.net/forum?id=PObuuGVrGaZ).
- Stallkamp et al. (2011)
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.
The german traffic sign recognition benchmark: a multi-class classification competition.
In *The 2011 international joint conference on neural networks*, pp.  1453–1460. IEEE, 2011.
- Su et al. (2024)
Hsuan Su, Hua Farn, Fan-Yun Sun, Shang-Tse Chen, and Hung-yi Lee.
Task arithmetic can mitigate synthetic-to-real gap in automatic speech recognition.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp.  8905–8915, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.503.
URL [https://aclanthology.org/2024.emnlp-main.503/](https://aclanthology.org/2024.emnlp-main.503/).
- Taori et al. (2023)
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.
Alpaca: A strong, replicable instruction-following model.
*Stanford Center for Research on Foundation Models*, 2023.
- Utans (1996)
Joachim Utans.
Weight averaging for neural networks and local resampling schemes.
In *Proceedings of the AAAI Workshop on Integrating Multiple Learned Models*, pp.  133–138, 1996.
- Wang et al. (2024)
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard.
Localizing task information for improved model merging and compression.
In *International Conference on Machine Learning*, 2024.
- Wightman et al. (2021)
Ross Wightman, Hugo Touvron, and Hervé Jégou.
Resnet strikes back: An improved training procedure in timm.
*arXiv preprint arXiv:2110.00476*, 2021.
- Wolf et al. (2019)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.
Huggingface’s transformers: State-of-the-art natural language processing.
*arXiv preprint arXiv:1910.03771*, 2019.
- Wortsman et al. (2022a)
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
In *Proceedings of the International Conference on Machine Learning*, volume 162, pp.  23965–23998. PMLR, 2022a.
URL [https://proceedings.mlr.press/v162/wortsman22a.html](https://proceedings.mlr.press/v162/wortsman22a.html).
- Wortsman et al. (2022b)
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.
Robust fine-tuning of zero-shot models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp.  7959–7971, 2022b.
- Xiao et al. (2016)
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.
Sun database: Exploring a large collection of scene categories.
*International Journal of Computer Vision*, 119:3–22, 2016.
- Yadav et al. (2023)
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.
TIES-merging: Resolving interference when merging models.
*Advances in Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=xtaX3WyCj1](https://openreview.net/forum?id=xtaX3WyCj1).
- Yan et al. (2025)
Kunda Yan, Min Zhang, Sen Cui, Qu Zikun, Bo Jiang, Feng Liu, and Changshui Zhang.
CALM: Consensus-aware localized merging for multi-task learning.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=OgfvSDn73E](https://openreview.net/forum?id=OgfvSDn73E).
- Yang et al. (2024a)
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao.
Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.
*arXiv preprint arXiv:2408.07666*, 2024a.
- Yang et al. (2024b)
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.
Adamerging: Adaptive model merging for multi-task learning.
In *The Twelfth International Conference on Learning Representations*, 2024b.
URL [https://openreview.net/forum?id=nZP6NgD3QY](https://openreview.net/forum?id=nZP6NgD3QY).
- Yoshida et al. (2025)
Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryosuke Yamaki, Ryotaro Shimizu, Yuki Saito, Julian McAuley, and Hiroki Naganuma.
Mastering task arithmetic: $\tau$jp as a key indicator for weight disentanglement.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=1VwWi6zbxs](https://openreview.net/forum?id=1VwWi6zbxs).
- Yoshikawa et al. (2025)
Yuya Yoshikawa, Ryotaro Shimizu, Takahiro Kawashima, and Yuki Saito.
Transferring visual explainability of self-explaining models through task arithmetic.
*arXiv preprint arXiv:2507.04380*, 2025.
- Zhang et al. (2017)
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.
mixup: Beyond empirical risk minimization.
*arXiv preprint arXiv:1710.09412*, 2017.
- Zhang et al. (2019)
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.
Be your own teacher: Improve the performance of convolutional neural networks via self distillation.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, October 2019.
- Zhang & Sabuncu (2020)
Zhilu Zhang and Mert Sabuncu.
Self-distillation as instance-specific label smoothing.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp.  2184–2195. Curran Associates, Inc., 2020.
URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/1731592aca5fb4d789c4119c65c10b4b-Paper.pdf).
- Zheng & YANG (2024)
Kaixiang Zheng and EN-HUI YANG.
Knowledge distillation based on transformed teacher matching.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=MJ3K7uDGGl](https://openreview.net/forum?id=MJ3K7uDGGl).

## Appendix A Theoretical Insights into Task Vector Merging for Models Optimized with Distinct Objectives

This appendix provides a step-by-step derivation of the theoretical results concerning the effect of calibration penalties on the arithmetic merging of task vectors. We demonstrate how calibration can introduce a first-order degradation in cross-entropy (CE) performance upon merging, an effect not observed when merging standard CE-trained task vectors.

### A.1 Notation and Assumptions

We begin by formalizing the notation and assumptions used throughout this analysis.

##### Notation

- •
Base parameter vector: $\bm{\theta}_{0}\in\mathbb{R}^{d}$. This is the shared starting point for fine-tuning, e.g., a pre-trained model’s parameters.
- •
Tasks and datasets: Two distinct downstream tasks, $T_{1}$ and $T_{2}$, with corresponding datasets $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$.
- •
Cross-Entropy (CE) loss: The standard objective for task $i$ is defined as
$J_{i}^{\mathrm{CE}}(\bm{\theta}):=-\,\mathbb{E}_{(x,y)\sim\mathcal{D}_{i}}\bigl{[}\log p_{\bm{\theta}}(y\mid x)\bigr{]}.$
- •
Calibrated objective: A modified objective function for task $i$ that includes a generic penalty term $\mathcal{C}_{i}(\bm{\theta})$ (^1^11For example, a detailed description of evaluating focal loss can be found in Kimura & Naganuma (2025)), weighted by $\lambda_{i}>0$.
$J_{i}^{\mathrm{CAL}}(\bm{\theta}):=J_{i}^{\mathrm{CE}}(\bm{\theta})+\lambda_{i}\,\mathcal{C}_{i}(\bm{\theta}).$
- •
Task-Specific optima: The parameter vector that minimizes a given objective for task $i$. We denote this by
$\bm{\theta}_{i}^{\star}:=\arg\min_{\bm{\theta}}J_{i}^{\star}(\bm{\theta}),\quad\text{where }\star\in\{\mathrm{CE,CAL}\}.$
- •
Task vectors: The displacement from the base parameters $\bm{\theta}_{0}$ to the task-specific optimum $\bm{\theta}_{i}^{\star}$.
$\bm{\tau}_{i}^{\star}:=\bm{\theta}_{i}^{\star}-\bm{\theta}_{0}.$

##### Assumptions

- 1.
The objective functions $J_{i}^{\mathrm{CE}}$ and $J_{i}^{\mathrm{CAL}}$ are twice continuously differentiable ($C^{2}$) in a neighborhood of the base parameter vector $\bm{\theta}_{0}$.
- 2.
The Hessian matrix of the CE loss evaluated at $\bm{\theta}_{0}$, denoted $H_{i}:=\nabla^{2}J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})$, is positive-definite. This ensures that $\bm{\theta}_{0}$ is located in a convex region of the CE loss landscape.
- 3.
We define the following gradients at the base point $\bm{\theta}_{0}$ for notational convenience:
$\textbf{g}_{i}:=\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{0}),\qquad\textbf{b}_{i}:=\nabla\mathcal{C}_{i}(\bm{\theta}_{0}).$

### A.2 Derivation of the Standard Task Vector

The optimal parameter vector $\bm{\theta}_{i}^{\mathrm{CE}}$ for the standard cross-entropy loss satisfies the first-order optimality condition, which states that the gradient at this point is zero.

$$ $\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{i}^{\mathrm{CE}})=0.$ (3) $$

Using the definition of the task vector, we can write $\bm{\theta}_{i}^{\mathrm{CE}}=\bm{\theta}_{0}+\bm{\tau}_{i}^{\mathrm{CE}}$. Substituting this into the optimality condition yields:

$$ $\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{0}+\bm{\tau}_{i}^{\mathrm{CE}})=0.$ (4) $$

We now perform a first-order Taylor series expansion of the gradient function $\nabla J_{i}^{\mathrm{CE}}(\cdot)$ around the point $\bm{\theta}_{0}$.

$$ $\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{0}+\bm{\tau}_{i}^{\mathrm{CE}})=\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})+\nabla^{2}J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})\bm{\tau}_{i}^{\mathrm{CE}}+\mathcal{O}(\|\bm{\tau}_{i}^{\mathrm{CE}}\|^{2}).$ (5) $$

Using our established notation for the gradient ($\textbf{g}_{i}$) and the Hessian ($H_{i}$) at $\bm{\theta}_{0}$, this becomes:

$$ $\textbf{g}_{i}+H_{i}\bm{\tau}_{i}^{\mathrm{CE}}+\mathcal{O}(\|\bm{\tau}_{i}^{\mathrm{CE}}\|^{2})=0.$ (6) $$

For fine-tuning scenarios where the task-specific solution $\bm{\theta}_{i}^{\mathrm{CE}}$ is close to the pre-trained model $\bm{\theta}_{0}$, the norm of the task vector $\|\bm{\tau}_{i}^{\mathrm{CE}}\|$ is small. We can therefore neglect the higher-order terms.

$$ $\textbf{g}_{i}+H_{i}\bm{\tau}_{i}^{\mathrm{CE}}\approx 0.$ (7) $$

Since $H_{i}$ is assumed to be positive-definite, it is invertible. We can solve for the task vector $\bm{\tau}_{i}^{\mathrm{CE}}$:

$$ $H_{i}\bm{\tau}_{i}^{\mathrm{CE}}=-\bm{\tau}_{i},$ (8) $$

which gives the well-known result from a single Newton-Raphson step:

$$ $\bm{\tau}_{i}^{\mathrm{CE}}=-H_{i}^{-1}\textbf{g}_{i}.$ (9) $$

### A.3 Derivation of the Calibrated Task Vector

We now apply the same procedure to the calibrated objective function $J_{i}^{\mathrm{CAL}}(\bm{\theta})$.

#### A.3.1 Gradient and Hessian at the Base Point

First, we compute the gradient and Hessian of $J_{i}^{\mathrm{CAL}}(\bm{\theta})$ at the base point $\bm{\theta}_{0}$.
The gradient is:

$$ $\displaystyle\nabla J_{i}^{\mathrm{CAL}}(\bm{\theta}_{0})$ $\displaystyle=\nabla\left(J_{i}^{\mathrm{CE}}(\bm{\theta})+\lambda_{i}\mathcal{C}_{i}(\bm{\theta})\right)\Big{|}_{\bm{\theta}=\bm{\theta}_{0}}$ (10) $\displaystyle=\nabla J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})+\lambda_{i}\nabla\mathcal{C}_{i}(\bm{\theta}_{0})$ (11) $\displaystyle=\textbf{g}_{i}+\lambda_{i}\textbf{b}_{i}.$ (12) $$

Let $A_{i}:=\nabla^{2}\mathcal{C}_{i}(\bm{\theta}_{0})$ be the Hessian of the calibration term. The Hessian of the calibrated objective, which we denote by $\tilde{H}_{i}$, is:

$$ $\displaystyle\tilde{H}_{i}:=\nabla^{2}J_{i}^{\mathrm{CAL}}(\bm{\theta}_{0})$ $\displaystyle=\nabla^{2}\left(J_{i}^{\mathrm{CE}}(\bm{\theta})+\lambda_{i}\mathcal{C}_{i}(\bm{\theta})\right)\Big{|}_{\bm{\theta}=\bm{\theta}_{0}}$ (13) $\displaystyle=\nabla^{2}J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})+\lambda_{i}\nabla^{2}\mathcal{C}_{i}(\bm{\theta}_{0})$ (14) $\displaystyle=H_{i}+\lambda_{i}A_{i}.$ (15) $$

#### A.3.2 Neumann Series Expansion of H ~ i − 1 \tilde{H}_{i}^{-1} over~ start_ARG italic_H end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT

To solve for the calibrated task vector $\bm{\tau}_{i}^{\mathrm{CAL}}$, we need the inverse of the calibrated Hessian, $\tilde{H}_{i}^{-1}$. For a small penalty weight $\lambda_{i}$, we can approximate this inverse. We begin by factoring out $H_{i}$:

$$ $\tilde{H}_{i}=H_{i}+\lambda_{i}A_{i}=H_{i}\left(I+H_{i}^{-1}(\lambda_{i}A_{i})\right)=H_{i}\left(I+\lambda_{i}H_{i}^{-1}A_{i}\right).$ (16) $$

The inverse is then given by $\tilde{H}_{i}^{-1}=(I+\lambda_{i}H_{i}^{-1}A_{i})^{-1}H_{i}^{-1}$. We can expand the term $(I+\lambda_{i}H_{i}^{-1}A_{i})^{-1}$ using a Neumann series (Horn & Johnson, 2012), $(I+X)^{-1}=\sum_{k=0}^{\infty}(-X)^{k}$, which converges if the spectral radius of $X$ is less than 1. Assuming $\lambda_{i}$ is sufficiently small such that $\|\lambda_{i}H_{i}^{-1}A_{i}\|<1$, we have:

$$ $\displaystyle(I+\lambda_{i}H_{i}^{-1}A_{i})^{-1}$ $\displaystyle=I-\lambda_{i}H_{i}^{-1}A_{i}+(\lambda_{i}H_{i}^{-1}A_{i})^{2}-\dots$ (17) $\displaystyle=I-\lambda_{i}H_{i}^{-1}A_{i}+\mathcal{O}(\lambda_{i}^{2}).$ (18) $$

Substituting this back into the expression for $\tilde{H}_{i}^{-1}$:

$$ $\displaystyle\tilde{H}_{i}^{-1}$ $\displaystyle=(I-\lambda_{i}H_{i}^{-1}A_{i}+\mathcal{O}(\lambda_{i}^{2}))H_{i}^{-1}$ (19) $\displaystyle=H_{i}^{-1}-\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}+\mathcal{O}(\lambda_{i}^{2}).$ (20) $$

#### A.3.3 Solving for τ i CAL \tau_{i}^{\mathrm{CAL}} italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_CAL end_POSTSUPERSCRIPT

The calibrated task vector $\bm{\tau}_{i}^{\mathrm{CAL}}$ is found by applying the first-order optimality condition to $J_{i}^{\mathrm{CAL}}$ and linearizing around $\bm{\theta}_{0}$:

$$ $\nabla J_{i}^{\mathrm{CAL}}(\bm{\theta}_{i}^{\mathrm{CAL}})=\nabla J_{i}^{\mathrm{CAL}}(\bm{\theta}_{0})+\nabla^{2}J_{i}^{\mathrm{CAL}}(\bm{\theta}_{0})\bm{\tau}_{i}^{\mathrm{CAL}}+\mathcal{O}(\|\bm{\tau}_{i}^{\mathrm{CAL}}\|^{2})=0.$ (21) $$

Using the expressions from B1 and ignoring higher-order terms:

$$ $(\textbf{g}_{i}+\lambda_{i}\textbf{b}_{i})+\tilde{H}_{i}\bm{\tau}_{i}^{\mathrm{CAL}}\approx 0.$ (22) $$

Solving for $\bm{\tau}_{i}^{\mathrm{CAL}}$ gives:

$$ $\bm{\tau}_{i}^{\mathrm{CAL}}\approx-\tilde{H}_{i}^{-1}(\textbf{g}_{i}+\lambda_{i}\textbf{b}_{i}).$ (23) $$

Now, we substitute the approximation for $\tilde{H}_{i}^{-1}$ from equation [20](https://arxiv.org/html/2508.01148v1#A1.E20):

$$ $\displaystyle\bm{\tau}_{i}^{\mathrm{CAL}}$ $\displaystyle\approx-\left(H_{i}^{-1}-\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}+\mathcal{O}(\lambda_{i}^{2})\right)(\textbf{g}_{i}+\lambda_{i}\textbf{b}_{i})$ (24) $\displaystyle=-\left(H_{i}^{-1}\textbf{g}_{i}+\lambda_{i}H_{i}^{-1}\textbf{b}_{i}-\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}\textbf{g}_{i}-\lambda_{i}^{2}H_{i}^{-1}A_{i}H_{i}^{-1}\textbf{b}_{i}\right)+\mathcal{O}(\lambda_{i}^{2})$ (25) $\displaystyle=-H_{i}^{-1}\textbf{g}_{i}-\lambda_{i}H_{i}^{-1}\textbf{b}_{i}+\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}\textbf{g}_{i}+\mathcal{O}(\lambda_{i}^{2}).$ (26) $$

We recognize the first term as the standard task vector, $\bm{\tau}_{i}^{\mathrm{CE}}=-H_{i}^{-1}\textbf{g}_{i}$. The expression becomes:

$$ $\bm{\tau}_{i}^{\mathrm{CAL}}=\bm{\tau}_{i}^{\mathrm{CE}}-\lambda_{i}H_{i}^{-1}\textbf{b}_{i}+\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}\textbf{g}_{i}+\mathcal{O}(\lambda_{i}^{2}).$ (27) $$

In many practical scenarios, especially after extensive pre-training, the initial gradient norm $\|\textbf{g}_{i}\|$ is small. Consequently, the term $\lambda_{i}H_{i}^{-1}A_{i}H_{i}^{-1}\textbf{g}_{i}$, which is of order $\mathcal{O}(\lambda_{i}\|\textbf{g}_{i}\|)$, is often negligible compared to the term $-\lambda_{i}H_{i}^{-1}\textbf{b}_{i}$, which is $\mathcal{O}(\lambda_{i})$. Under this simplifying assumption, we can define the first-order correction due to calibration as:

$$ $\bm{\delta}_{i}:=-\lambda_{i}H_{i}^{-1}\textbf{b}_{i}.$ (28) $$

This allows us to express the calibrated task vector as a simple perturbation of the standard task vector:

$$ $\bm{\tau}_{i}^{\mathrm{CAL}}=\bm{\tau}_{i}^{\mathrm{CE}}+\bm{\delta}_{i}+\mathcal{O}(\lambda_{i}^{2},\lambda_{i}\|\textbf{g}_{i}\|).$ (29) $$

### A.4 Task Vector Merging

We consider merging two task vectors using a simple linear combination with positive coefficients $\alpha,\beta>0$. We define two types of merged parameters:

$$ $\displaystyle\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}}$ $\displaystyle:=\bm{\theta}_{0}+\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}},$ (30) $\displaystyle\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}}$ $\displaystyle:=\bm{\theta}_{0}+\alpha\bm{\tau}_{1}^{\mathrm{CAL}}+\beta\bm{\tau}_{2}^{\mathrm{CAL}}.$ (31) $$

### A.5 Taylor Expansion of the CE Loss for Merged Vectors

Our goal is to evaluate the CE loss $J_{i}^{\mathrm{CE}}$ not at its own optimum, but at the merged parameter points. We use a second-order Taylor expansion of $J_{i}^{\mathrm{CE}}(\bm{\theta})$ around $\bm{\theta}_{0}$:

$$ $J_{i}^{\mathrm{CE}}(\bm{\theta})-J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})=\textbf{g}_{i}^{\top}(\bm{\theta}-\bm{\theta}_{0})+\frac{1}{2}(\bm{\theta}-\bm{\theta}_{0})^{\top}H_{i}(\bm{\theta}-\bm{\theta}_{0})+\mathcal{O}(\|\bm{\theta}-\bm{\theta}_{0}\|^{3}).$ (32) $$

#### A.5.1 Merging of CE Vectors

Let $\Delta\bm{\theta}^{\mathrm{CE}}=\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}}-\bm{\theta}_{0}=\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}}$. The change in CE loss for task $i$ is:

$$ $J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}})-J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})=\textbf{g}_{i}^{\top}(\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}})+\mathcal{O}(\|\bm{\tau}\|^{2}).$ (33) $$

Let’s analyze the first-order term in the expansion. Using $\textbf{g}_{i}=-H_{i}\bm{\tau}_{i}^{\mathrm{CE}}$ from equation [7](https://arxiv.org/html/2508.01148v1#A1.E7):

$$ $\displaystyle\textbf{g}_{i}^{\top}(\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}})$ $\displaystyle=\alpha\textbf{g}_{i}^{\top}\bm{\tau}_{1}^{\mathrm{CE}}+\beta\textbf{g}_{i}^{\top}\bm{\tau}_{2}^{\mathrm{CE}}$ (34) $\displaystyle=\alpha(-H_{i}\bm{\tau}_{i}^{\mathrm{CE}})^{\top}\bm{\tau}_{1}^{\mathrm{CE}}+\beta(-H_{i}\bm{\tau}_{i}^{\mathrm{CE}})^{\top}\bm{\tau}_{2}^{\mathrm{CE}}$ (35) $\displaystyle=-\alpha(\bm{\tau}_{i}^{\mathrm{CE}})^{\top}H_{i}\bm{\tau}_{1}^{\mathrm{CE}}-\beta(\bm{\tau}_{i}^{\mathrm{CE}})^{\top}H_{i}\bm{\tau}_{2}^{\mathrm{CE}}.$ (36) $$

The term for task $i$ itself ($i=1$ and analyzing $\bm{\tau}_{1}^{\mathrm{CE}}$, or $i=2$ and analyzing $\bm{\tau}_{2}^{\mathrm{CE}}$) is $-\alpha(\bm{\tau}_{i}^{\mathrm{CE}})^{\top}H_{i}\bm{\tau}_{i}^{\mathrm{CE}}=-\alpha\|\bm{\tau}_{i}^{\mathrm{CE}}\|_{H_{i}}^{2}$. Since $H_{i}$ is positive-definite, this self-term is strictly negative. The cross-term’s sign is indefinite. However, the dominant contribution to the loss change is typically negative and of order $\mathcal{O}(\|\bm{\tau}\|^{2})$, indicating that merging CE vectors does not increase the loss at first order.

#### A.5.2 Merging of Calibrated Vectors

Let $\Delta\bm{\theta}^{\mathrm{CAL}}=\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}}-\bm{\theta}_{0}=\alpha\bm{\tau}_{1}^{\mathrm{CAL}}+\beta\bm{\tau}_{2}^{\mathrm{CAL}}$. The change in loss is:

$$ $J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}})-J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})=\textbf{g}_{i}^{\top}(\alpha\bm{\tau}_{1}^{\mathrm{CAL}}+\beta\bm{\tau}_{2}^{\mathrm{CAL}})+\mathcal{O}(\|\bm{\tau}\|^{2},\lambda^{2}).$ (37) $$

We substitute $\bm{\tau}_{j}^{\mathrm{CAL}}\approx\bm{\tau}_{j}^{\mathrm{CE}}+\bm{\delta}_{j}$:

$$ $\displaystyle\textbf{g}_{i}^{\top}(\alpha\bm{\tau}_{1}^{\mathrm{CAL}}+\beta\bm{\tau}_{2}^{\mathrm{CAL}})$ $\displaystyle\approx\textbf{g}_{i}^{\top}\left(\alpha(\bm{\tau}_{1}^{\mathrm{CE}}+\bm{\delta}_{1})+\beta(\bm{\tau}_{2}^{\mathrm{CE}}+\bm{\delta}_{2})\right)$ (38) $\displaystyle=\underbrace{\textbf{g}_{i}^{\top}(\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}})}_{\text{Original term, }\mathcal{O}(\|\bm{\tau}\|^{2})}+\underbrace{\alpha(\textbf{g}_{i}^{\top}\bm{\delta}_{1})+\beta(\textbf{g}_{i}^{\top}\bm{\delta}_{2})}_{\text{Additional term, }\mathcal{O}(\lambda\|\bm{\tau}\|)}.$ (39) $$

Let’s analyze the additional term introduced by calibration. Using the definitions of $\textbf{g}_{i}$ and $\bm{\delta}_{j}$:

$$ $\textbf{g}_{i}^{\top}\bm{\delta}_{j}=(-H_{i}\bm{\tau}_{i}^{\mathrm{CE}})^{\top}(-\lambda_{j}H_{j}^{-1}\textbf{b}_{j})=\lambda_{j}(\bm{\tau}_{i}^{\mathrm{CE}})^{\top}H_{i}H_{j}^{-1}\textbf{b}_{j}.$ (40) $$

This term is first-order in $\lambda_{j}$ and its sign is not guaranteed to be negative. If this term is positive, it can cause an increase in the CE loss. Since its magnitude is $\mathcal{O}(\lambda\|\bm{\tau}\|)$, it can dominate the $\mathcal{O}(\|\bm{\tau}\|^{2})$ terms when $\|\bm{\tau}\|$ is small, leading to a net increase in the CE loss.

### A.6 Main Result and Proof

###### Theorem 1 .

Under the assumptions stated, if the vectors $\{\textbf{g}_{i}^{\top}\bm{\delta}_{j}\}_{j=1,2}$ are not pathologically structured (e.g., they are not both zero or strictly negative), then there exist merge coefficients $\alpha,\beta>0$ such that for at least one task $i\in\{1,2\}$,

$$ $J_{i}^{\mathrm{CE}}\bigl{(}\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}}\bigr{)}>J_{i}^{\mathrm{CE}}\bigl{(}\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}}\bigr{)}.$ $$

This difference is of first order in the calibration weights $\lambda_{1},\lambda_{2}$.

###### Proof.

We analyze the difference in the CE loss for task $i$ between the two merging strategies. Let $\Delta\bm{\theta}^{\mathrm{CE}}=\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}}-\bm{\theta}_{0}$ and $\Delta\bm{\theta}^{\mathrm{CAL}}=\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}}-\bm{\theta}_{0}$.

$$ $\displaystyle J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}})-J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}})$ $\displaystyle=\left(J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})+\textbf{g}_{i}^{\top}\Delta\bm{\theta}^{\mathrm{CAL}}+\mathcal{O}(\|\Delta\bm{\theta}^{\mathrm{CAL}}\|^{2})\right)-\left(J_{i}^{\mathrm{CE}}(\bm{\theta}_{0})+\textbf{g}_{i}^{\top}\Delta\bm{\theta}^{\mathrm{CE}}+\mathcal{O}(\|\Delta\bm{\theta}^{\mathrm{CE}}\|^{2})\right)$ $\displaystyle=\textbf{g}_{i}^{\top}(\Delta\bm{\theta}^{\mathrm{CAL}}-\Delta\bm{\theta}^{\mathrm{CE}})+\mathcal{O}(\|\bm{\tau}\|^{2},\lambda^{2}).$ (41) $$

The difference between the merged displacement vectors is:

$$ $\displaystyle\Delta\bm{\theta}^{\mathrm{CAL}}-\Delta\bm{\theta}^{\mathrm{CE}}$ $\displaystyle=\left(\alpha\bm{\tau}_{1}^{\mathrm{CAL}}+\beta\bm{\tau}_{2}^{\mathrm{CAL}}\right)-\left(\alpha\bm{\tau}_{1}^{\mathrm{CE}}+\beta\bm{\tau}_{2}^{\mathrm{CE}}\right)$ $\displaystyle=\alpha(\bm{\tau}_{1}^{\mathrm{CAL}}-\bm{\tau}_{1}^{\mathrm{CE}})+\beta(\bm{\tau}_{2}^{\mathrm{CAL}}-\bm{\tau}_{2}^{\mathrm{CE}})$ $\displaystyle=\alpha(\bm{\delta}_{1}+\mathcal{O}(\lambda_{1}^{2}))+\beta(\bm{\delta}_{2}+\mathcal{O}(\lambda_{2}^{2}))$ $\displaystyle=\alpha\bm{\delta}_{1}+\beta\bm{\delta}_{2}+\mathcal{O}(\lambda^{2}).$ (42) $$

Substituting this back, the leading term of the loss difference is:

$$ $J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CAL}})-J_{i}^{\mathrm{CE}}(\bm{\theta}_{\mathrm{merge}}^{\mathrm{CE}})\approx\alpha(\textbf{g}_{i}^{\top}\bm{\delta}_{1})+\beta(\textbf{g}_{i}^{\top}\bm{\delta}_{2}).$ (43) $$

The terms $\textbf{g}_{i}^{\top}\bm{\delta}_{1}$ and $\textbf{g}_{i}^{\top}\bm{\delta}_{2}$ are scalars of order $\mathcal{O}(\lambda\|\bm{\tau}\|)$. Their signs depend on the geometry of the loss landscapes. Unless both scalars are non-positive for both tasks $i=1,2$, we can choose positive coefficients $\alpha,\beta$ that result in a positive sum for at least one task. For instance, if $\textbf{g}_{i}^{\top}\bm{\delta}_{1}>0$ for a given $i$, we can select a small enough $\beta>0$ relative to $\alpha>0$ such that the total sum $\alpha(\textbf{g}_{i}^{\top}\bm{\delta}_{1})+\beta(\textbf{g}_{i}^{\top}\bm{\delta}_{2})$ is positive.

This positive term is of order $\mathcal{O}(\lambda\|\bm{\tau}\|)$. It dominates the other terms of order $\mathcal{O}(\|\bm{\tau}\|^{2})$ and $\mathcal{O}(\lambda^{2})$ when $\|\bm{\tau}\|$ and $\lambda$ are sufficiently small, leading to a net increase in the CE loss for calibrated merging compared to standard merging.
∎

###### Theorem 1 .

###### Proof.

##### Interpretation

This result provides a theoretical basis for the observation that merging task vectors trained with certain penalties can be detrimental. The calibration penalty introduces a linear perturbation term $\bm{\delta}_{i}$ to the task vector. This term is not necessarily aligned with the descent direction of the cross-entropy loss $J_{i}^{\mathrm{CE}}$. When multiple such vectors are added, these misaligned perturbations can combine constructively to push the merged parameter vector into a region of higher CE loss. This increase is of first order in $\lambda$ and can therefore be significant. In contrast, merging pure CE vectors does not introduce such a first-order degradation term.

## Appendix B Experiment Details

All experiments were run on NVIDIA A100 GPUs (40 GB memory each). Fine-tuning jobs used four GPUs in parallel, whereas all evaluations were performed on a single GPU.

##### Fine-tuning Details.

Our training protocol closely mirrors the public code of Ilharco et al. (2023).
For each task, we fine-tuned CLIP backbones (ViT-B-32 and ViT-L-14) for 2000 updates using AdamW (Loshchilov & Hutter, 2019) with a weight-decay factor of 0.1. We adopted a cosine-annealed learning-rate schedule preceded by 200 warm-up steps and used a mini-batch size of 128; ViT-L-14 training employed gradient accumulation to match this effective batch size. Following the findings of Ilharco et al. (2023), we kept CLIP’s text encoder frozen and treated the logits obtained from class-specific prompts (e.g., “a photo of a {classname}”) as a fixed classification head, updating only the image encoder during fine-tuning.
Regarding the learning rate, we used $10^{-4}$ only when training task vectors with large norms in the Norm Mismatch setting, and $10^{-5}$ for all other cases. In the Low Confidence setting, the label smoothing strength was set to $\alpha=0.1$.

##### Merging Details.

For all four merging methods adopted in this study, it is necessary to tune the task vector coefficient $\lambda_{t}$. Following Ilharco et al. (2023), we imposed a unified constraint on all $\lambda_{t}$ and searched the range from 0.0 to 1.0 (in increments such as 0.05) based on validation accuracy.

##### Distillation Details.

The distillation procedure generally followed the fine-tuning settings described above, except that the number of steps was set to 500 and the learning rate was fixed at $10^{-5}$ for all cases.
The $\ell_{2}$ regularizer weight $\beta$ was set to 0.5.

### B.1 Normalized Accuracy

The normalized accuracy for a task $t$ on its dataset $\tilde{\mathcal{D}}_{t}$ is defined as the ratio of the post-merge model’s accuracy to the single-task model’s accuracy:

$$ $\text{normalized accuracy}_{t}=\frac{\text{accuracy}(\bm{\theta}_{\text{mtl}},\tilde{\mathcal{D}}_{t})}{\text{accuracy}(\bm{\theta}_{t},\tilde{\mathcal{D}}_{t})},$ $$

where the function $\text{accuracy}(\bm{\theta},\mathcal{D})$ denotes the accuracy of the model $f(\cdot;\bm{\theta})$ on a dataset $\mathcal{D}$.

## Appendix C Additional Results

### C.1 Norm Comparison across Layers

Figure [5](https://arxiv.org/html/2508.01148v1#A3.F5) (weights) and Figure [6](https://arxiv.org/html/2508.01148v1#A3.F6) (biases) visualize how the
parameter norm of each ViT-B-32 layer changes when the learning rate is raised from $10^{-5}$ (gray
bars) to $10^{-4}$ (blue bars). The scale shift is not confined to a few layers; rather, every
block exhibits a consistent multiplicative increase. In other words, tuning with a larger learning
rate stretches the entire task vector almost uniformly, across both weight matrices and bias terms.
This layer-wise coherence implies that any merge-time correction must adjust the global
scale of the model, not merely a subset of layers.

Figure: (a) MLP
Refer to caption: x9.png

Figure: (a) MLP
Refer to caption: x12.png

### C.2 Other Confidence Calibration Method and Merging Performance

We assessed two additional confidence–calibration techniques—Mixup and focal loss—alongside label smoothing.
For each of the eight vision tasks we fine-tuned ViT-B-32 with Mixup or focal loss and then merged the resulting task vectors.
For Mixup, the interpolation coefficient was sampled independently at each iteration from the uniform distribution $\mathcal{U}(0,1)$.
For focal loss, we set the focusing parameter to $\gamma=10$.
Table [2](https://arxiv.org/html/2508.01148v1#A3.T2) reports the outcomes.
Like label smoothing, both Mixup and focal loss markedly reduced merge accuracy relative to the Original configuration, confirming that they also raise prediction entropy and thus interfere with model merging.
In every case, however, applying DisTaC restored accuracy to a level on par with Original, demonstrating that DisTaC reliably conditions confidence even when the source models were calibrated with Mixup or focal loss.

**Table 2: Impact of confidence–calibration fine-tuning on merge accuracy. Source models (ViT-B-32) are fine-tuned with three popular calibration techniques—label smoothing (LS), Mixup, and focal loss—before merging. In every case the resulting merge accuracy drops far below the Original benchmark, showing that low-confidence sources hamper model merging. When the same models are first processed with DisTaC, accuracy is restored to a level on par with Original, confirming that DisTaC’s confidence conditioning is effective across all three calibration schemes.**
| Method |  | Original | LS | Mixup | Focal Loss |
| --- | --- | --- | --- | --- | --- |
| Task arithmetic |  | 70.4 (78.0) | 51.0 (58.3) | 52.3 (60.5) | 55.5 (63.9) |
| Task arithmetic | + DisTac | - | 63.6 (72.2) | 66.8 (75.2) | 67.2 (76.9) |
| TIES |  | 74.0 (82.0) | 54.5 (62.0) | 55.5 (63.9) | 59.4 (68.8) |
| TIES | + DisTac | - | 68.7 (77.9) | 69.5 (78.7) | 72.1 (82.4) |
| Consensus TA |  | 74.8 (82.8) | 54.6 (62.0) | 54.8 (63.0) | 58.9 (68.2) |
| Consensus TA | + DisTac | - | 67.7 (76.5) | 69.4 (77.8) | 71.7 (81.7) |
| TSVM |  | 83.3 (92.4) | 60.7 (68.4) | 60.9 (69.6) | 69.3 (79.5) |
| TSVM | + DisTac | - | 81.5 (91.8) | 80.1 (90.0) | 81.8 (93.0) |

### C.3 Impact of Task Vector Scaling on ViT-L-14

We carried out the same scaling experiment (see Figure [3](https://arxiv.org/html/2508.01148v1#S6.F3)) on the larger ViT-L-14 backbone.
As shown in Figure [7](https://arxiv.org/html/2508.01148v1#A3.F7), the trend matches that of Figure [3](https://arxiv.org/html/2508.01148v1#S6.F3): shrinking the task vector ($\lambda<1$) leaves single-task accuracy largely unchanged—often even slightly higher—whereas stretching it ($\lambda>1$) rapidly erodes performance.
These results further support the recommendation that, when task-vector norms are mismatched, one should shrink the longer vectors rather than stretch the shorter ones for robust model merging.

Figure: Figure 7: Effect of scaling task vectors on test accuracy. For each of the eight vision tasks (ViT-L-14), we evaluate the model $\bm{\theta}_{\text{pre}}+\lambda\bm{\tau}$ as the scaling factor $\lambda$ varies from $0.0$ to $3.0$. Shrinking the task vector ($\lambda<1.0$) often preserves or even improves accuracy relative to the fine-tuned model ($\lambda=1.0$), while stretching the vector ($\lambda>1.0$) leads to sharp degradation. At $\lambda=3.0$, performance falls below that of the zero-shot model on all tasks. These results support shrinking long task vectors to match shorter ones when resolving norm disparities.
Refer to caption: x15.png

### C.4 Scaling Alone Is Insufficient to Overcome Norm Mismatch

Figure: Figure 8: Normalized merge accuracy for ViT-B-32 on the eight-task benchmark under three conditions. Gray: Original. Light-orange: Norm Mismatch after rescaling the longest task vector to the mean norm of the others. Red: same rescaled vectors followed by DisTaC. Simple scaling narrows the gap only slightly, whereas DisTaC fully restores accuracy to the Original level. “Avg.” denotes the average across all tasks.
Refer to caption: x16.png

To test whether simple rescaling is sufficient, we revisited the
Norm Mismatch scenario and aligned the longest task vector to
the mean norm of the remaining vectors before merging. Figure
[8](https://arxiv.org/html/2508.01148v1#A3.F8) reports the resulting normalized accuracy
for ViT B-32 on the eight vision tasks: Original (gray),
Norm Mismatch after *only* scaling (light orange), and
Norm Mismatch followed by DisTaC (red). The $x$-axis lists
the task whose vector was lengthened; “Avg.” is the mean over all tasks.

Scaling alone lifts accuracy slightly but still leaves a sizeable gap
to Original. In contrast, applying DisTaC after scaling
recovers the lost performance and matches the baseline across every
task. As explained in Section [6.1](https://arxiv.org/html/2508.01148v1#S6.SS1), even
*shrinking* a task vector inevitably hurts its single-task
accuracy; DisTaC is therefore essential for restoring that accuracy
before merging.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Please see the weaknesses.
soundness: 4
strengths: 1. The paper addresses a novel and highly relevant problem. While most research focuses on improving performance on idealized benchmarks, this work astutely investigates the robustness of merging methods in more practical, pessimistic settings, which is a critical step for real-world applicability.

2. The problem formulation is exceptionally clear and well-motivated. The failure modes are demonstrated with convincing empirical evidence (Figure 1), making the motivation for the proposed solution immediately apparent and easy to follow.

3. The paper provides valuable and actionable insights for practitioners. The discussions around "shrinking vs. stretching" task vectors and the trade-off between source model calibration and merge performance offer practical guidelines that extend beyond the method itself.
summary: This paper investigates the robustness of multi-task model merging methods under realistic, non-idealized conditions. The authors identify and empirically validate two key factors that degrade the performance of existing merging techniques: (1) disparities in the norms of task vectors, often arising from different fine-tuning hyperparameters, and (2) low prediction confidence of the source models, which can result from calibration techniques like label smoothing. To address these issues, the paper proposes DisTaC, a pre-conditioning step that uses knowledge distillation on unlabeled data. DisTaC adjusts task vector norms and increases model confidence before the merge, thereby making state-of-the-art merging algorithms more robust. Experiments on vision tasks show that DisTaC significantly improves performance in these challenging scenarios, often recovering it to the level of an ideal merge.
weaknesses: 1. The empirical evaluation is limited in scope. All experiments are conducted on CLIP/ViT models for vision classification tasks. Given the current prominence of model merging in the context of Large Language Models (LLMs), the absence of any experiments on language tasks is a significant limitation and leaves the generalizability of the findings in question.

2. The investigation of failure modes, while insightful, feels somewhat narrow. The paper focuses on norm disparity and low confidence but does not explore other plausible sources of incompatibility, such as differences in optimizer states, dataset quality/size, or the specific architecture of parameter-efficient fine-tuning (e.g., LoRA rank).

3. The method introduces a dependency on unlabeled data from the task distribution. While the authors argue this is a reasonable requirement, it makes the method less self-contained than truly data-free merging techniques and may be a practical constraint in some scenarios.

4. Proposition 1 formalizes a basic geometric intuition about vector addition but offers little deep insight into the functional consequences within a neural network. The choice of cosine similarity as the primary metric is also debatable, as its connection to actual task performance is not explicitly established.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How does DisTaC's independent conditioning of each task vector affect the final merge performance when tasks are fundamentally antagonistic or highly complementary?
soundness: 2
strengths: 1. The paper provides a sharp empirical and theoretical analysis of robustness shortcomings in existing model merging approaches, uncovering how norm disparity and low confidence can produce significant accuracy degradation after merging. 
2. DisTaC is a clear, well-motivated conditioning method, leveraging knowledge distillation with unlabeled data for practical pre-conditioning. By integrating both norm scaling and confidence sharpening, it directly addresses the two key failure modes.
summary: The paper investigates the robustness limits of state-of-the-art multi-task model merging methods, identifying two critical failure modes: disparities in task vector norms and low confidence in source models. To address these, the authors propose DisTaC—a pre-conditioning framework utilizing knowledge distillation to harmonize task vector norms and boost model confidence before merging. Extensive experiments on eight vision tasks with CLIP backbones demonstrate that DisTaC consistently restores or enhances merging performance under challenging, realistic scenarios where conventional merging fails.
weaknesses: 1. Since the pre-conditioning step is applied independently to each source model, the total computational cost scales linearly with the number of models. For large-scale ensembles, this cumulative overhead can become prohibitive, undermining the method's primary efficiency advantage over full-scale retraining. The distillation cost also scales with model parameter count, becoming computationally prohibitive for large-scale foundation models (e.g., LLMs). 
2. The study’s empirical validation is confined to vision tasks, leaving its applicability to other modalities, particularly Large Language Models (LLMs), unverified. Given that task vector dynamics in LLMs may be substantially more complex, the method's effectiveness for merging models with diverse reasoning and generative capabilities is not guaranteed.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Why not provide the results of the Original version when combined with DisTaC in Table 1? It is helpful for us to understand that DisTaC may be harmful for performance in some cases.
soundness: 3
strengths: The experiments are extensive, including ablation studies, visual analyses (e.g., layer-wise norm shifts in Figs. 5–6), and additional tests with Mixup and focal loss to demonstrate generality.

The authors go beyond idealized benchmarks and pinpoint practical causes of model-merging failures, which is interesting and meaningful.
summary: The paper investigates the robustness of model merging methods. The authors identify two critical failure modes in existing model-merging pipelines:

+ Task vector norm disparities — differences in magnitudes caused by heterogeneous fine-tuning hyperparameters (e.g., learning rates, steps).

+ Low pretrained-model confidence — often resulting from training techniques such as label smoothing, Mixup, or focal loss.

To mitigate these issues, the authors propose DisTaC, a lightweight rescaling task vectors to a target norm and increasing model confidence by training students with higher temperatures than teachers. Empirical evaluations on eight vision tasks demonstrate that DisTaC restores and often improves post-merge accuracy.
weaknesses: The experiments are restricted to vision-only tasks using CLIP ViT backbones. No tests are conducted on NLP, speech, or multimodal architectures.

While the paper argues that unlabeled data are “readily available,” this assumption may not hold in all domains, such as the security issue, or we can not find a comprehensive set for LLMs.

While DisTaC generally improves results, the paper lacks a sensitivity analysis of unlabeled data quality that would clarify when it might fail or overfit.

The authors assert that DisTaC is computationally lightweight (“500 steps, unlabeled data only”), but do not quantitatively compare its runtime or cost to baseline merges, nor provide results on large-scale LLMs such as Qwen.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Figure 3 and the discussion in Section 6.1 suggest that shrinking a task vector can sometimes lead to better performance than the original model. This is a very interesting observation. Could you elaborate on the potential reasons for this? Does this imply a regularization effect, suggesting that the initial fine-tuning may have overshot a more optimal, shorter task vector?

2. Regarding the influence of learning rate and training steps on the effectiveness of model merging, you may refer to the error analysis in Theorem 3.1 of [1], where the impact of the norm of the task vector was also observed. In [2], analysis of the properties of task vectors was conducted.

*[1] Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging. arXiv preprint arXiv:2505.19892*  
*[2] MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation. arXiv preprint arXiv:2406.07529*
soundness: 3
strengths: 1. The paper's primary contribution is the identification and diagnosis of two practical failure modes in model merging. While the component techniques (knowledge distillation) are not new, their application to pre-condition task vectors to solve these specific robustness issues is novel.

2. The claims are substantiated by a thorough and well-designed set of experiments. The inclusion of theoretical motivation (Proposition 1 and the analysis in Appendix C) adds rigor and provides a deeper understanding of the observed phenomena. The ablation studies, such as the analysis of shrinking versus stretching vectors in Figure 3, are particularly convincing and support the practical guidelines offered.

3. By addressing the gap between idealized academic benchmarks, the paper provides a crucial step towards making model merging a more practical and reliable tool. DisTaC is a simple, effective, and computationally cheap solution that can be readily adopted by practitioners.
summary: The authors identify two critical failure modes that are common in practical scenarios but overlooked in idealized benchmarks: (1) Task Vector Norm Disparity, where task vectors have vastly different magnitudes; and (2) Low-Confidence Source Models, where models produce high-entropy predictions, often as a result of calibration techniques like label smoothing.

The paper proposes a lightweight pre-conditioning method. DisTaC uses KD on unlabeled data to prepare models for merging. To correct norm disparities, it first rescales a task vector to a target norm and then uses KD to distill knowledge from the original, high-performing model, thereby recovering the performance lost during scaling. To correct for low confidence, it distills knowledge using a higher temperature for the student model than the teacher, which encourages the student to produce lower-entropy, more confident predictions.

The authors demonstrate empirically on eight vision tasks with ViT-B-32 and ViT-L-14 backbones that these failure modes significantly degrade the performance of state-of-the-art merging methods. They then show that applying DisTaC before merging consistently restores performance to the level of the idealized benchmark, with minimal computational overhead. Finally, the paper offers practical guidelines for model merging.
weaknesses: * The experiments are confined exclusively to CLIP-based models on vision tasks. The general applicability of DisTaC would be strengthened by demonstrating its effectiveness in other modalities, such as merging fine-tuned LLMs, or on MLLMs.

* The method's reliance on unlabeled data from the task distribution is a potential limitation. The paper would benefit from an analysis of the method's sensitivity to the quantity of this unlabeled data. For instance, how much data is needed for DisTaC to be effective, and how does performance degrade if the unlabeled data is from a slightly shifted distribution?


Decision: Accept (Poster)