ATM: Improving Model Merging by Alternating Tuning and Merging

## Abstract

Abstract Model merging has recently emerged as a cost-efficient paradigm for multi-task learning. Among current approaches, task arithmetic (Ilharco et al., 2022 ) stands out for its simplicity and effectiveness.
In this paper, we motivate the effectiveness of task vectors by linking them to multi-task gradients. We show that in a single-epoch scenario, task vectors are mathematically equivalent to the gradients obtained via gradient descent in a multi-task setting, and still approximate these gradients in subsequent epochs. Furthermore, we show that task vectors perform optimally when equality is maintained, and their effectiveness is largely driven by the first epoch’s gradient.
Building on this insight, we propose viewing model merging as a single step in an iterative process that A lternates between T uning and M erging (ATM). This method acts as a bridge between model merging and multi-task gradient descent, achieving state-of-the-art results with the same data and computational requirements. We extensively evaluate ATM across diverse settings, achieving up to 20% higher accuracy in computer vision and NLP tasks, compared to the best baselines.
Finally, we provide both empirical and theoretical support for its effectiveness, demonstrating increased orthogonality between task vectors and proving that ATM minimizes an upper bound on the loss obtained by jointly finetuning all tasks.

## 1 Introduction

The pretrain-and-finetune paradigm has become the standard for many deep learning tasks, where a model pretrained on large-scale, unlabeled data is adapted to a specific downstream task with minimal tuning. However, when working with multiple tasks, a major drawback is the need to store separate finetuned models for each task. Model merging addresses this challenge by combining task-specific models into a single model capable of handling all tasks. This significantly reduces storage costs, as the unified model’s size remains comparable to that of a single-task model, regardless of the number of tasks. Among numerous model merging methods, task arithmetic (Ilharco et al., ) stands out for its simplicity and effectiveness. Given a pretrained model $\theta_{0}$ and a model $\theta_{i}$ finetuned on task $t_{i}$, the task vector $\tau_{i}=\theta_{i}-\theta_{0}$ is defined as the difference between the finetuned and pretrained weights. For multi-task learning with $n$ tasks, task arithmetic sums the $n$ task vectors, scales the sum with a coefficient $\alpha$, and adds the resulting vector back to the pretrained model.

Figure: Figure 1: The ATM method, illustrated up to iteration $h$ with two tasks (A and B). In each iteration, the Tune step finetunes the pretrained model $\theta_{\text{base}}^{(0)}$ separately on both tasks, and the Merge step aggregates the task vectors and applies the resulting multi-task vector to the base model. This process repeats, with each iteration using the updated model as the new base, continuing until a stopping condition is met. We observe increased task vector orthogonality as ATM iterations progress.
Refer to caption: extracted/5981356/images/atm-diagram.jpg

In this paper, we explain the effectiveness of task arithmetic by linking task vectors to the gradients of the average loss across all tasks. We start from a simple observation: when a model is finetuned for a single epoch using gradient descent (GD), the corresponding task vector is the additive inverse of the loss gradient, scaled by the learning rate. Similarly, the multi-task vector, obtained by summing individual task vectors, is equivalent to the additive inverse of the average loss gradient. Thus, task addition is analogous to performing a GD step on the sum of the average task losses.

When finetuning spans multiple epochs, this equality becomes an approximation, with an error dependent on the learning rate.
Despite the single-epoch assumption not always holding in practice, the gradient analogy still sheds light on why task vectors are effective. In fact, we reveal that the first epoch often contributes the most to the overall gradient norm during finetuning. Even when this is not the case, subsequent gradients tend to align with the first, confirming that the initial direction predominantly dictates the task vector’s effectiveness.

In this view, aggregation and merging in task arithmetic correspond to a noisy GD step when finetuning on the combined tasks, using the sum of the average losses as the objective. In practice, this implies that the one-step nature of these techniques likely leads to overshooting the multi-task minimum, as they effectively perform GD on a multi-task dataset with a single, noisy step. The scaling factor optimized over the validation set essentially acts as the learning rate in this process.

##### Contribution

Building on these insights, we address the limitations of one-step task vector methods with Alternating Tuning and Merging (ATM) – a novel framework that iteratively alternates between finetuning and merging. This approach generalizes task arithmetic, allowing for a more gradual and refined integration of task-specific knowledge.

Given a compute budget of $b$ epochs per task, traditional methods allocate $b$ epochs to a single finetuning pass. In contrast, ATM distributes this budget across $k$ iterations, performing $\frac{b}{k}$ epochs of finetuning in each iteration, followed by task vector aggregation. The unified model from each iteration serves as the starting point for the next, and after $k$ iterations, the final unified model is deployed and evaluated. Notably, ATM is agnostic to the merging framework, allowing the integration of any interference-resolution techniques during the merge step to enhance performance. Overall, ATM significantly reduces time overhead compared to current baselines; extensive experiments in vision and NLP demonstrate its state-of-the-art results without hyperparameter tuning.

All code, experiment configurations, and checkpoints are available for reproducibility (^1^11https://github.com/LuckerZOfficiaL/Alternating-Tuning-and-Merging).
To summarize, our contributions are four-fold:

- •
We show that, under specific conditions, task vectors are equivalent to or approximate the gradients of the corresponding task losses.
- •
We point out that existing one-shot merging frameworks often overshoot the multi-task optimum, especially when task vectors have large norms.
- •
We introduce Alternating Tuning and Merging (ATM), a novel model merging framework that generalizes task arithmetic. Its flexibility allows it to incorporate any interference-resolution method with no additional overhead.
- •
We empirically demonstrate that ATM increases task vector orthogonality compared to standard methods, and mathematically prove that it reduces the loss of a jointly finetuned multi-task model.

## 2 Related work

##### Mode connectivity and model merging

Mode connectivity studies how weights characterize local minima in the loss landscape. Frankle et al. () explored linear mode connectivity in models trained from the same initialization, while Entezari et al. () suggested that all models converge to a shared basin once neuron permutations are resolved. Building on this, permutation-based model merging combines diverse models into a single one, inheriting their capabilities without ensembling overhead. Singh & Jaggi () introduced an optimal-transport weight-matching method, while Git Re-Basin (Ainsworth et al., ) proposed three variants acting both on weights and activations, and REPAIR (Jordan et al., ) demonstrated significant barrier reduction through activation renormalization. Most recently, Navon et al. () suggested merging models in the embedding space of deep neural networks, while Crisostomi et al. () proposed a cycle-consistent matching procedure for improved merging.
When models share the same pretrained initialization, Wortsman et al. () proposed fusing them via a simple average. Jolicoeur-Martineau et al. () proposed to merge models by pushing them towards the population mean to ensure stability. RegMean (Jin et al., ) and Fisher-weighted averaging (Matena & Raffel, ) fall under the regime of weighted averaging, where the weights are optimized according to some criteria. Daheim et al. () shed light on the positive relation between post-averaging multi-task performance and the gradient mismatch between the constituent models. Finally, Choshen et al. () proposed model merging as a replacement for pretraining; they argue that pretrained checkpoints are not always the optimal starting point for further finetuning, and a model obtained by merging finetuned models can be a better starting point than any of its constituents.

##### Task vectors

Task vector-based merging (Ilharco et al., ) finetunes a pretrained model on different tasks to obtain task vectors (differences between finetuned and original checkpoints). Arithmetic operations on these vectors enable forgetting, analogy learning, and multi-task learning. Several works aim to improve task vector merging by reducing task interference (Deep et al., ; Wang et al., ; Huang et al., ). Some methods include sparsifying task vectors or finetuning only lottery tickets (Panda et al., ). TIES-merging (Yadav et al., ) merges vectors by pruning, selecting a unified sign vector, and merging disjointly, while Model Breadcrumbs (Davari & Belilovsky, ) prunes both small and large-magnitude weights. DARE Merging (Yu et al., ) randomly masks out a portion of weights and scales up the rest. AdaMerging (Yang et al., ) optimizes aggregation coefficients, while Yang et al. () proposed task-specific modules for test-time adaptation. Ortiz-Jimenez et al. () introduced the concept of weight disentanglement and recommended finetuning in the tangent space. In contrast to these one-shot methods, we present an iterative model merging approach that progressively refines a base model to achieve improved multi-task performance.

## 3 Task Vectors as Gradients

In this section, we show that task vectors are tightly related to the loss gradients over the union of the tasks.

###### Theorem 3.1 .

Let $\left\{\theta^{(k)}_{t}\right\}_{t=1}^{|T|}$ be a set of models obtained by finetuning the base model $\theta_{\text{base}}$ for $k$ epochs on tasks $T$ using GD with a learning rate $\eta$, where finetuning task $t\in T$ minimizes the loss $\overline{L}_{t}(\theta)=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\ell(x_{i},y_{i},\theta)$.
Additionally, let $\left\{\tau_{t}^{(k)}\right\}_{t=1}^{|T|}$ denote the corresponding set of task vectors, with each $\tau^{(k)}_{t}=\theta^{(k)}_{t}-\theta_{\text{base}}$. Let $\tau^{(k)}_{\text{MT}}$ be the multi-task vector $\tau^{(k)}_{\text{MT}}=\sum_{t\in T}\tau_{t}^{(k)}$.
Finally, let $\theta_{\text{MT}}^{(k)}$ represent the model obtained by minimizing the combined loss $\sum_{i=1}^{|T|}\overline{L}_{i}$ for $k$ epochs using GD with a learning rate of $\alpha\eta$. It holds that

$$ $\displaystyle\tau^{(1)}_{\text{MT}}=-\eta\nabla\sum_{t\in T}\overline{L}_{t}( \theta_{\text{base}})$ (1) $\displaystyle\tau^{(k)}_{\text{MT}}=-\eta\sum_{t\in T}\sum_{j=0}^{k-1}\nabla \overline{L}_{i}(\theta_{\text{MT}}^{(j)})+\frac{\eta^{2}}{2}C(\{\theta_{\text {MT}}^{(j)}\}_{j=1}^{k-2})+O(\eta^{3})$ (2) $$

with

$$ $C(\{\theta_{\text{MT}}^{(j)}\}_{j=1}^{h})=\sum_{t\in T}\sum_{\ell=0}^{h}\nabla ^{2}\overline{L}_{t}(\theta^{(\ell)}_{\text{MT}})\sum_{m=0}^{\ell}\left[\alpha \sum_{t^{\prime}\neq t,t^{\prime}\in T}\nabla\overline{L}_{t}^{\prime}(\theta^ {(m)}_{\text{MT}})+(\alpha-1)\nabla\overline{L}_{t}(\theta^{(m)}_{\text{MT}})\right]$ (3) $$

We provide the proof in Appendix A.1. To better appreciate the relationship between a task vector and the gradient computed on the corresponding task dataset, consider the single-task case, where the task vector is exactly the additive inverse of the gradient, scaled by the learning rate $\eta$.

Figure: Figure 2: Accuracy of a model merged using task vectors over the task-specific models finetuned for $1$ epoch and to convergence.
Refer to caption: extracted/5981356/images/ta-1epoch-vs-ta-convergence.png

###### Remark 3.1 .

From Theorem 3.1, it follows that, for a single task $t$, and after a single finetuning epoch,

$$ $\tau_{t}=-\eta\nabla\overline{L}_{t}(\theta_{\text{base}})$ (4) $$

where $\eta$ is the learning rate.

This implies that, under the theorem’s assumptions, adding the task vector to the pretrained model approximates the effect of finetuning the model; the same intuition applies to the multi-task case.

In fact, in Fig. 2 we report evidence that the multi-task model obtained by merging models finetuned for a single epoch outperforms the one obtained by merging models finetuned to convergence. In other words, *the optimal merged model is achieved when task arithmetic is exactly a GD step.*

We formalize this through the following corollary.

###### Corollary 3.1.1 .

Let $\theta_{\text{TA}}^{(k)}=\theta_{\text{base}}+{\alpha}\sum_{t=1}^{T}\tau_{t}^{
(k)}$ be the model obtained using vanilla task arithmetics.
Using the same notation of Theorem 3.1, it holds that

$$ $\displaystyle\theta_{\text{TA}}^{(1)}=\theta_{\text{MT}}^{(1)}$ (5) $\displaystyle\theta_{\text{TA}}^{(k)}=\theta_{\text{MT}}^{(k)}+\frac{\eta^{2}} {2}C(\{\theta_{\text{MT}}^{(j)}\}_{j=1}^{k-2})+O(\eta^{3})\;\;\;\text{for}\ k>1$ (6) $$

The corollary indicates that for $k>1$, the multi-task vector still approximates the gradient of a model finetuned for the same number of epochs via GD, with an error on the order of $o(\eta^{2})$. Despite the lack of exact equivalence, the effectiveness of task vectors still holds and can be explained by the fact that most of the model’s finetuning trajectory is driven by the gradient from the first epoch.

In Fig. 3(a) we plot the epoch-wise normalized gradient norm $\frac{\|\nabla_{\theta}^{(k)}L\|}{\sum_{k^{\prime}=1}^{K}\|\nabla_{\theta}^{(k
^{\prime})}L\|}$. We observe that the first epoch contributes the most, accounting for up to $70\%$ of the total gradient norms across all epochs. In cases where this is not true, such as with the RESISC45 dataset, we speculate that the direction is still largely determined by the first epoch. As depicted in Fig. 3(b), the gradients from the first $5$ epochs maintain a high cosine similarity $(>0.8)$ with the first epoch’s gradient.

It is worth recalling that this analysis does not apply exactly when using SGD instead of GD. However, treating SGD as an approximation to GD preserves the intuition. From this view, reducing task interference corresponds to minimizing conflicting gradients in multi-task learning.

Figure: (a) Normalized gradient norms after $5$ epochs of finetuning.
Refer to caption: x1.png

###### Theorem 3.1 .

###### Remark 3.1 .

###### Corollary 3.1.1 .

## 4 ATM: Alternating Tuning and Merging

Building upon the insights of Section 3, we argue that task arithmetic is an approximation to a single GD step over the union of all the tasks. Following this parallel, we advocate taking further update steps iteratively.

The overall framework of ATM is depicted in Fig. 1. Specifically, starting from a pretrained checkpoint as the base model $\theta_{\text{base}}^{(0)}$, we finetune it separately on each task to obtain the first-iteration task vectors $\tau_{1}^{(1)},\dots,\tau_{|T|}^{(1)}$. These are then aggregated and added to the base model to form the next-iteration unified model $\theta_{\text{base}}^{(1)}$.
The procedure is iterated according to the following equation:

$$ $\theta_{\text{base}}^{(k+1)}=\theta_{\text{base}}^{(k)}+\frac{\alpha}{|T|}\sum _{t\in T}\tau_{t}^{(k)}\quad\forall k=0,\dots,{K-1}\,.$ (7) $$

The $k$-th iteration task vector for task $t$ is obtained as $\tau^{(k)}_{t}=\theta_{t}^{(k)}-\theta_{\text{base}}^{(k)}$, where $\theta_{t}^{(k)}$ is a model obtained finetuning the $k$-th iteration base model $\theta_{\text{base}}^{(k)}$ on task $t$. The total number of iterations $K$ can be predefined or based on a stop condition.

In practice, each iteration of ATM involves finetuning the current base model on all $|T|$ tasks of interest, thereby obtaining $|T|$ task vectors. These task vectors determine the task-specific directions the current base model should follow in order to attain enhanced performance on the corresponding tasks.

The merging step of ATM at a given iteration simply consists of summing the mean current-iteration task vectors to the current base model, although any interference-resolution method in the task vector literature can be integrated. This step is intended to pull the base model closer to the multi-task basin on the loss landscape. Taking the average over $T$ ensures the magnitude of the update remains insensitive to the number of tasks.

Note that after each iteration, the task vectors of the previous iterations can be safely discarded. Therefore, at any stage of ATM, we only store the current base model and one task vector for each of the $|T|$ tasks, incurring no additional memory requirements.

## 5 Upper bounding the multi-task loss

In this section, we explore the relationship between the ATM loss, defined as the mean of average losses over all tasks, and the loss of a model trained jointly on all the datasets. Analogously to Section 3, we conduct this analysis under the assumption that GD, rather than SGD, is used for optimizing the model parameters. This simplifying assumption removes the stochasticity introduced by random sampling, enabling a more straightforward analysis while still providing valuable insights into the underlying dynamics of the optimization process.
With a slight abuse of notation, we denote with $t$ both the task and its corresponding dataset with cardinality $n_{t}$. The total number of samples for all tasks is given by $N=\sum_{t\in T}n_{t}$.

Inspired by Daheim et al. (), we define the *target loss* for model merging as $L_{\text{target}}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell(x_{i},y_{i},\theta)$, which is the loss of a model trained jointly on all the datasets.
By Theorem 3.1, when merging occurs after one step of finetuning on each dataset, the ATM update is given by:

$$ $\theta_{\text{base}}^{(k+1)}=\theta_{\text{base}}^{(k)}+\frac{\alpha}{|T|}\sum _{t\in T}\tau_{t}^{(k)}=\theta_{\text{base}}^{(k)}-\alpha\eta\nabla\left(\frac {1}{|T|}\sum_{t\in T}\overline{L_{t}}\right)\,,$ (8) $$

which corresponds to performing a GD step over the loss $L_{\text{ATM}}=\frac{1}{|T|}\sum_{t\in T}\overline{L_{t}}$.

Having established that one step of ATM in GD minimizes $L_{\text{ATM}}$, a crucial question arises: under what conditions does minimizing $L_{\text{ATM}}$ imply the minimization of $L_{\text{target}}$? In other words, when can we be certain that optimizing the ATM loss will also minimize the loss associated with training jointly on all datasets?

To answer this question, we first note that $L_{\text{ATM}}$ is an unweighted average of the individual dataset losses, while the target loss is a weighted average:

$$ $L_{\text{target}}(\theta)=\frac{\sum_{t\in T}n_{t}\overline{L}_{t}(\theta)}{ \sum_{t\in T}n_{t}}\,.$ (9) $$

We now analyze the parameter update from $\theta^{(k)}$ to $\theta^{(k+1)}$. For both ATM and target methods, we denote the change in loss, $L_{\text{method}}$, as
$\Delta L_{\text{method}}=L_{\text{method}}(\theta^{(k)})-L_{\text{method}}(
\theta^{(k+1)}).$
In the following theorem, we prove that if the drop in ATM loss exceeds a threshold $\delta$, the target loss will also decrease. The value of $\delta$ depends on the size of the largest dataset with a decreasing loss and the smallest dataset with an increasing loss. In particular, if the former dataset is larger than the latter, a reduction in ATM loss reduces the target loss.
In practice, this is ensured when the loss is reduced on the largest dataset.

###### Theorem 5.1 .

Let $D=\{t\mid\Delta\overline{L}_{t}>0\}$ be the set of datasets where the loss decreases after a parameter update, and $I=\{t\mid\Delta\overline{L}_{t}\leq 0\}$ be the set of datasets where the loss increases or remains unchanged.
If the reduction in the ATM loss satisfies $\Delta L_{\text{ATM}}>\delta$, where

$$ $\delta=\frac{1}{|T|}\left(1-\frac{\min_{t\in I}n_{t}}{\max_{t\in D}n_{t}} \right)\sum_{t\in I}|\Delta\overline{L}_{t}|\,,$ $$

then the target loss $L_{\text{target}}$ will also decrease, i.e., $\Delta L_{\text{target}}>0$.

We redirect the reader to Appendix A.2 for the formal proof.

###### Remark 5.1 .

If we choose the target loss to be the maximum of the average loss across all datasets $L_{\text{target}}=\max_{t\in T}\overline{L}_{t}$, by leveraging the equivalence between the $L_{1}$-norm and the max norm, we obtain the bound $L_{\text{target}}\leq T\cdot L_{\text{ATM}}$.

###### Theorem 5.1 .

###### Remark 5.1 .

## 6 Experiments

In this section, we compare ATM against several recent baselines across a number of classification tasks in computer vision and NLP.

Figure: Figure 4: Comparisons as computational budget $K$ varies for ViT-B-16.
Refer to caption: extracted/5981356/images/vitRadars.png

### 6.1 Experimental setting

##### Datasets and Models

For computer vision tasks, we test ATM with a ViT-B-16 backbone (Dosovitskiy et al., ) and evaluate it on a diverse set of datasets: CIFAR100 (Krizhevsky et al., ), DTD (Cimpoi et al., ), EuroSAT (Helber et al., ), GTSRB (Houben et al., ), MNIST (Lecun et al., ), RESISC45 (Cheng et al., ), and SVHN (Netzer et al., ).
For NLP tasks, we employ RoBERTa-base (Liu, ) and BERT-base-uncased (Devlin et al., ), evaluating them on the GLUE benchmark (Wang et al., ).

##### Baselines and Metrics

To gauge the performance of ATM, we consider several model merging baselines, including task arithmetic (TA) (Ilharco et al., ), TIES-merging (Yadav et al., ), and model breadcrumbs (Davari & Belilovsky, ) for both computer vision and NLP tasks, and DARE merging (Yu et al., ) for NLP tasks only. We adhere to the author-recommended hyperparameters, whenever needed or available, in order to ensure fair comparisons across experiments. Specifically, for TIES-merging, we retain the top $15\%$ of weights based on magnitude ranking. For model breadcrumbs, we set $\beta=0.85$ and $\gamma=0.993$. For DARE merging, we use a drop rate of $0.9$. In all settings, we adopt mean aggregation of task vectors and use a scaling factor of $1$ when applying them to the base model.

### 6.2 Impact of Epoch Distribution on Performance

Figure: Figure 5: Multi-task accuracy for different budget distributions (ViT-B-16).
Refer to caption: extracted/5981356/images/HOTA_distribution.png

In this experiment, we first establish a fixed compute budget of $10$ finetuning epochs for each task. Then, we seek the optimal distribution of epochs among different numbers of ATM iterations. To exemplify, if $10$ epochs are distributed among $5$ iterations, then in each iteration a task is finetuned for $2$ epochs.

As depicted in Fig. 5, with a fixed compute budget, maximizing iterations while minimizing epochs per iteration yields the best results for ATM. This indicates that more gradual updates to the base model are preferable to abrupt ones. Splitting $10$ epochs across $10$ iterations achieves the highest average accuracy of $89\%$, outperforming the $1$ iteration of $10$ epochs setting (analogous to task arithmetic) by $\mathbf{21\%}$. Based on this, we use the $1$ epoch, $10$ iterations setting for most subsequent experiments.

### 6.3 Effect of Compute Budget

Figure: Figure 6: Average multi-task accuracy as budget varies.
Refer to caption: extracted/5981356/images/decreasingAccs.png

We extend the comparison of ATM against baseline methods across different compute budgets. Specifically, we vary the per-task finetuning epochs ($K$) within$t$ and $te$ and compare the average test accuracy across tasks. As shown in Figs. 4 and 7, ATM consistently outperforms the baselines, with its advantage growing as the budget increases. ATM surpasses the best baseline by$7\%$, $11\%$, and $21\%$ for budgets of $2$, $4$, and $10$ epochs, respectively. Detailed results for vision and NLP benchmarks can be found in Appendix 11.

Interestingly, while ATM’s accuracy improves with more finetuning epochs, the baseline methods exhibit the opposite trend; see Fig. 6. In other words, *the more specialized the task-specific models, the lower the performance of the unified multi-task model*. ATM avoids this issue by gradually specializing the intermediate multi-task models.

Figure: Figure 7: Comparisons as computational budget $K$ varies for RoBERTa-base.
Refer to caption: extracted/5981356/images/robertaRadars.png

### 6.4 Comparisons in Original Settings

Given the available training data, we compare three ATM settings against various baselines under their original settings:
(i) ATMfinetuned on validation data (valFT ATM) for $10$ iterations of $1$ epoch,
(ii) ATMfinetuned on training data for $10$ iterations of $1$ epoch, and
(iii) ATMfinetuned on training data to convergence for $30$ iterations of $1$ epoch.
As illustrated in Table 1, all ATM variants significantly outperform the baselines. With a 10-epoch budget and available training data, ATM achieves an average accuracy of $89\%$, leading by $\mathbf{17\%}$ over the best-performing baseline. Without compute restrictions, ATM converges after 30 iterations, achieving a remarkable accuracy of $91\%$.

### 6.5 Training-data Free Setting

A common realistic constraint is the lack of per-task training data, as finetuned models are often sourced from online repositories. In this section, we assume only the availability of validation data, typically used by baselines for hyperparameter tuning. Unlike the baselines, ATM uses this validation data for finetuning the tasks, leaving hyperparameters untuned.

We define this variant as valFT ATM and compare its performance against standard ATM and the baselines, using author-recommended hyperparameters for fair comparisons. As presented in Table 1, valFT ATM outperforms the best baseline by $10\%$. While the $7\%$ gap between valFT ATM and ATM is due to the limited finetuning data, valFT ATM performs comparably to ATM on all tasks except EuroSAT. A similar trend is observed on the NLP benchmark.

**Table 1: Accuracy comparison under original baseline settings (ViT-B-16).**
|  | CIFAR100 | DTD | EuroSAT | GTSRB | MNIST | RESISC45 | SVHN | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Task Arithmetic | 0.59 | 0.46 | 0.78 | 0.61 | 0.96 | 0.63 | 0.77 | 0.69 |
| TIES | 0.70 | 0.51 | 0.76 | 0.60 | 0.94 | 0.73 | 0.78 | 0.72 |
| Breadcrumbs | 0.60 | 0.47 | 0.77 | 0.60 | 0.95 | 0.63 | 0.75 | 0.68 |
| ValFT ATM: 10 Orders | 0.78 | 0.61 | 0.53 | 0.97 | 0.99 | 0.88 | 0.95 | 0.82 |
| ATM: 10 Orders | 0.79 | 0.61 | 0.98 | 0.97 | 0.99 | 0.89 | 0.96 | 0.89 |
| ATM: 30 Orders | 0.83 | 0.68 | 0.99 | 0.99 | 1.00 | 0.94 | 0.97 | 0.91 |

## 7 Discussion

### 7.1 Orthogonality

Orthogonality between task vectors has been recommended as a desirable property for multi-task merging (Ilharco et al., ). Davari & Belilovsky () adopt pairwise cosine similarity between task vectors as a proxy for task interference. Following this line, we again back the validity of this observation by identifying a positive correlation between ATM performance and task vector orthogonality. Fig. 10 highlights that as the number of ATM iterations increases, the magnitude of cosine similarity between task vectors tends to shrink, suggesting greater orthogonality as performance improves. Furthermore, as depicted in Fig. 10, we find that ATM task vectors exhibit lower-magnitude average cosine similarity compared to the baseline methods.

### 7.2 Task Proficiency is not Mergeability

As supported by figures 6 and 2, task-specific expertise does not imply multi-task performance. We observe that better-performing task-specific models result in worse multi-task models when adopting baseline methods, hinting that *downstream performance is not a predictor of post-merging performance*. We speculate that specialized models end up in highly dispersed locations in the parameter space, and merging them abruptly in a one-shot fashion generates a suboptimal multi-task model; this can be observed in Fig. 8, where baseline methods end up all in the same (suboptimal) loss basin. From the perspective of gradient approximation, this degradation can be explained by the fact that vanilla task vectors approximate increasingly noisy multi-task gradients when finetuning occurs for multiple epochs. On the contrary, a lower degree of specialization ensures the task-specific models remain closer to the pretrained checkpoint in the parameter space, as evinced by Ortiz-Jimenez et al. () from the perspective of weight disentanglement in the tangent space, thus leading to less aggressive updates when merging. The above insights translate to less aggressive updates (shorter-norm task vectors) being more amenable to merging. This phenomenon is also consistent with the findings of Lu et al. (), explaining this phenomenon by the vanquishing common knowledge due to task-specific finetuning. Again, in view of gradient approximation, fewer epochs of finetuning yields a better approximation of the multi-task gradient, with a single epoch achieving the best approximation. Hence, our theoretical explanation generalizes the previous observations on this phenomenon. Capitalizing on this, ATM gradually aggregates task-specific models and updates the base model accordingly, merging less aggressively but over multiple iterations. At each iteration, the ATM task vectors represent the best nudges for the current base model without referring back to the initial pretrained model as all baseline methods do.

### 7.3 Educated Trajectory

Figure: Figure 8: 2D PCA projection of various approaches.
Refer to caption: extracted/5981356/images/pca6.png

Task arithmetic performs the aggregation step abruptly in a one-shot fashion over the initial pretrained checkpoint, likely overshooting the multi-task optimum. In ATM, however, the loss landscape is traversed iteration by iteration as the base model updates, leading to more informed nudges toward the multi-task optimum. Fig. 8 depicts the 2D projection of various checkpoints via PCA. Notably, TIES and breadcrumbs, being post-hoc enhancements of task arithmetic, end up around the same basin, whereas ATM takes gradual steps toward a different and better basin, signaling the effectiveness of our novel iterative merging paradigm.

Figure: Figure 9: Average magnitudes of pairwise cosine similarity between ATM task vectors.
Refer to caption: extracted/5981356/images/avgCosSims.png

### 7.4 Time and Memory Complexities

**Table 2: Comparison of methods in terms of time and memory complexity.**
| Method | Time | Memory |
| --- | --- | --- |
| Task Arithmetic | $O(n*d)$ | $O(d)$ |
| TIES | $O(n*d*log(d))$ | $O(d)$ |
| Breadcrumbs | $O(n*d*log(d))$ | $O(d)$ |
| ATM | $O(k*n*d)$ | $O(d)$ |

We compare ATM to the baselines in terms of time and memory consumption, assuming the backbone has $d$ parameters. Memory-wise, ATM has no additional requirements compared to task arithmetic, as task vectors can be deleted after each merge step. The time complexity of iteration-$k$ ATM is equivalent to performing task arithmetic $k$ times, as it involves $k$ rounds of finetuning and merging. This time complexity is generally negligible compared to TIES and breadcrumbs, whose dominant overhead originates from pruning task vectors post-hoc according to the magnitude ranking of all weights, with a time complexity of $O(d*\log(d))$. As outlined in Table 2, ATM is faster than TIES and breadcrumbs as long as $\log(d)$ is asymptotically greater than $k$, which is typically the case since $k$ is usually a small constant (e.g., $10$).

### 7.5 Limitations of Iterative Merging

While we have demonstrated the benefits of ATM in various settings, its iterative nature has some drawbacks. Specifically, iterative merging does not immediately provide a task representation for new tasks, as it produces a series of $K$ vectors per task. However, it remains the best approach for obtaining a single model that performs well across all tasks. We ensured a fair comparison by using the same compute budget and data requirements, showing that a small validation set suffices for effective use of the framework – this is standard in the literature, typically used for hyperparameter optimization.
Lastly, although ATM resembles gradient descent on the union of tasks, it retains a key advantage of task arithmetic: by generating task vectors independently, it avoids centralizing data, making it suitable for federated settings where data privacy is critical.

## 8 Conclusions

In conclusion, this paper identifies a key limitation of task arithmetic – overshooting due to abrupt model merging – and establishes its connection to gradient descent, forming the basis for our proposed model merging framework. We present Alternating Tuning and Merging (ATM), an iterative framework that addresses the shortcomings of one-shot merging techniques. By alternating between finetuning and merging, ATM effectively prevents overshooting and enhances multi-task performance. Extensive experiments on computer vision and NLP benchmarks demonstrate that ATM achieves state-of-the-art accuracy while maintaining computational efficiency comparable to existing baselines. We explain the effectiveness of ATM from the perspective of multi-task gradient approximation. Additionally, our theoretical analysis reveals that ATM optimizes the upper bound of the loss over the union of all the tasks and improves task vector orthogonality. The flexibility of ATM opens numerous future research directions, including the integration of interference-mitigation techniques and further refinements through advancements from the gradient descent literature.

## Ethics Statement

This research was conducted with a strong commitment to ethical standards in both data usage and experimental methodology. All datasets utilized in this study are publicly available. No personally identifiable information was accessed or used during the course of this research. Additionally, the experiments were designed to ensure fair comparisons across methods. We encourage future work that adheres to these same principles and addresses the broader societal impacts of machine learning.

## Reproducibility Statement

We are committed to ensuring the reproducibility of our results and have taken steps to facilitate this for the broader research community. The code, datasets, and configurations used for the experiments in this paper are made available via a public repository. We are open to providing further instructions on the usage of our code. The hyperparameters, frameworks, and evaluation metrics have been thoroughly documented, and we provide clear descriptions of our experimental setup to allow for straightforward replication of our findings. We encourage the community to utilize these resources and provide feedback for further improvements.

#### Acknowledgments

This work is supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAIR.

## Appendix A Proofs

### A.1 Proofs of Theorem 3.1 and Corollary 3.1.1

In this section, we provide proofs for Theorem 3.1 and Corollary 3.1.1.
For clarity, we restate both the theorem and corollary.

###### Theorem .

Let $\left\{\theta^{(k)}_{t}\right\}_{t=1}^{|T|}$ be a set of models obtained by finetuning the base model $\theta_{\text{base}}$ for $k$ epochs on tasks $T$ using GD with a learning rate $\eta$, where finetuning task $t\in T$ minimizes the loss $\overline{L}_{t}(\theta)=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\ell(x_{i},y_{i},\theta)$.
Additionally, let $\left\{\tau_{t}^{(k)}\right\}_{t=1}^{|T|}$ denote the corresponding set of task vectors, with each $\tau^{(k)}_{t}=\theta^{(k)}_{t}-\theta_{\text{base}}$. Let $\tau^{(k)}_{\text{MT}}$ be the multi-task vector $\tau^{(k)}_{\text{MT}}=\sum_{t\in T}\tau_{t}^{(k)}$.
Finally, let $\theta_{\text{MT}}^{(k)}$ represent the model obtained by minimizing the combined loss $\sum_{i=1}^{|T|}\overline{L}_{i}$ for $k$ epochs using GD with a learning rate of $\alpha\eta$. It holds that

$$ $\displaystyle\tau^{(1)}_{\text{MT}}=-\eta\nabla\sum_{t\in T}\overline{L}_{t}( \theta_{\text{base}})$ (10) $\displaystyle\tau^{(k)}_{\text{MT}}=-\eta\sum_{t\in T}\sum_{j=0}^{k-1}\nabla \overline{L}_{i}(\theta_{\text{MT}}^{(j)})+\frac{\eta^{2}}{2}C(\{\theta_{\text {MT}}^{(j)}\}_{j=1}^{k-2})+O(\eta^{3})$ (11) $$

with

$$ $C(\{\theta_{\text{MT}}^{(j)}\}_{j=1}^{h})=\sum_{t\in T}\sum_{\ell=0}^{h}\nabla ^{2}\overline{L}_{t}(\theta^{(\ell)}_{\text{MT}})\sum_{m=0}^{\ell}\left[\alpha \sum_{t^{\prime}\neq t,t^{\prime}\in T}\nabla\overline{L}_{t}^{\prime}(\theta^ {(m)}_{\text{MT}})+(\alpha-1)\nabla\overline{L}_{t}(\theta^{(m)}_{\text{MT}})\right]$ (12) $$

###### Corollary .

Let $\theta_{\text{TA}}^{(k)}=\theta_{\text{base}}+{\alpha}\sum_{t=1}^{T}\tau_{t}^{
(k)}$ be the model obtained using vanilla task arithmetics.
Using the same notation of Theorem 3.1, it holds that

$$ $\displaystyle\theta_{\text{TA}}^{(1)}=\theta_{\text{MT}}^{(1)}$ (13) $\displaystyle\theta_{\text{TA}}^{(k)}=\theta_{\text{MT}}^{(k)}+\frac{\eta^{2}} {2}C(\{\theta_{\text{MT}}^{(j)}\}_{j=1}^{k-2})+O(\eta^{3})\;\;\;\text{for}\ k>1$ (14) $$

We recall that $\theta^{(k)}_{i}$ is the model obtained by finetuning on task $i$ for $k$ epochs, and that both the finetuing on different tasks and the training on the average loss start from a pretrained model $\theta_{\text{base}}$.

To prove the statement of the theorem and of the corollary we need a intermediate result.
We introduce the following notation:

$$ $\displaystyle r_{i}(\theta)=\alpha\sum_{j\neq i}\nabla\overline{L}_{j}(\theta) +(\alpha-1)\nabla\overline{L}_{i}(\theta)=\alpha\sum_{j=1}^{|T|}\nabla \overline{L}_{j}(\theta)-\nabla\overline{L}_{i}(\theta_{\text{base}})$ (15) $\displaystyle p_{i}^{k}(\theta_{\text{base}},\theta^{(1)}_{\text{MT}},\dots, \theta^{(k)}_{\text{MT}})=\sum_{j=0}^{k}r_{i}(\theta^{(k)}_{\text{MT}})$ (16) $\displaystyle s^{k}_{i}(\theta_{\text{base}},\dots,\theta^{(k)}_{\text{MT}})= \sum_{j=0}^{k}\nabla^{2}\overline{L}_{i}(\theta^{(j)}_{\text{MT}})[p^{j}_{i}( \theta_{\text{base}},\dots,\theta^{(j-1)}_{\text{MT}})].$ (17) $$

###### Lemma A.1 .

Using the notation introduced in Theorem 3.1, it holds that

$$ $\theta^{(1)}_{i}=\theta^{(1)}_{\text{MT}}+\eta p_{i}^{0}(\theta_{\text{base}})$ (18) $$

and for $m\geq 2$

$$ $\theta^{(m+1)}_{i}=\theta^{(m+1)}_{\text{MT}}+\eta p_{i}^{m}(\theta_{\text{ base}},\dots,\theta^{(m)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-1}_{i}(\theta_{ \text{base}},\dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ (19) $$

###### Proof.

We first show that the statement is true for $m=1$, and then prove the results for $m\geq 2$ by induction.
In this case, the base case is given for $m=2$. In the induction step, instead, we prove that if the statement holds for any given case
$m$ then it must also hold for the next case $m+1$.

###### Theorem .

###### Corollary .

###### Lemma A.1 .

###### Proof.

##### m = 1 𝑚 1 m=1 italic_m = 1 . First epoch

For each task $i=1,\dots,|T|$

$$ $\theta^{(1)}_{i}=\theta_{\text{base}}-\eta\nabla\overline{L}_{i}(\theta_{\text {base}})\text{ while }\theta^{(1)}_{\text{MT}}=\theta_{\text{base}}-\alpha\eta \sum_{i\in T}\nabla\overline{L}_{i}(\theta_{\text{base}}).$ $$

Consequently, it holds that

$$ $\displaystyle\theta^{1}_{i}$ $\displaystyle=\theta^{(1)}_{\text{MT}}+\eta\left[\alpha\sum_{j\neq i}\nabla \overline{L}_{j}(\theta_{\text{base}})+(\alpha-1)\nabla\overline{L}_{i}(\theta _{\text{base}})\right]$ $\displaystyle=\theta^{(1)}_{\text{MT}}+\eta r_{i}(\theta_{\text{base}})=\theta ^{(1)}_{\text{MT}}+\eta p_{i}^{0}(\theta_{\text{base}}).$ $$

##### m = 2 𝑚 2 m=2 italic_m = 2 . Second epoch

$$ $\displaystyle\theta^{(2)}_{i}$ $\displaystyle=\theta^{(1)}_{i}-\eta\nabla\overline{L}_{i}(\theta^{(1)}_{i})$ $\displaystyle=\theta^{(1)}_{\text{MT}}+{\eta}r_{i}(\theta_{\text{base}})-\eta \nabla\overline{L}_{i}\left(\theta^{(1)}_{\text{MT}}+{\eta}r_{i}(\theta_{\text {base}})\right)$ $\displaystyle\overset{Taylor}{\approx}\theta^{(1)}_{\text{MT}}+{\eta}r_{i}( \theta_{\text{base}})-\eta\nabla\overline{L}_{i}(\theta^{(1)}_{\text{MT}})- \frac{\eta^{2}}{2}\nabla^{2}\overline{L}_{i}(\theta_{\text{MT}}^{(1)})r_{i}( \theta_{\text{base}})+O(\eta^{3})$ $\displaystyle=\theta^{(1)}_{\text{MT}}-\eta\nabla\overline{L}_{i}(\theta^{(1)} _{\text{MT}})+\eta r_{i}(\theta_{\text{base}})-\frac{\eta^{2}}{2}\nabla^{2} \overline{L}_{i}(\theta_{\text{MT}}^{(1)})r_{i}(\theta_{\text{base}})+O(\eta^{ 3})$ $\displaystyle=\theta^{(1)}_{\text{MT}}-\eta\nabla\overline{L}_{i}(\theta^{(1)} _{\text{MT}})+\eta\alpha\sum_{t\in T}\nabla\overline{L}_{i}(\theta^{(1)}_{ \text{MT}})-\eta\alpha\sum_{t\in T}\nabla\overline{L}_{i}(\theta^{(1)}_{\text{ MT}})+\eta r_{i}(\theta_{\text{base}})$ $\displaystyle-\frac{\eta^{2}}{2}\nabla^{2}\overline{L}_{i}(\theta_{\text{MT}}^ {(1)})r_{i}(\theta_{\text{base}})+O(\eta^{3})$ $\displaystyle=\theta^{(1)}_{\text{MT}}+\eta r_{i}(\theta^{(1)}_{\text{MT}})+ \eta r_{i}(\theta_{\text{base}})-\frac{\eta^{2}}{2}\nabla^{2}\overline{L}_{i}( \theta_{\text{MT}}^{(1)})r_{i}(\theta_{\text{base}})+O(\eta^{3})$ $\displaystyle=\theta^{1}_{\text{MT}}+\eta p_{i}^{1}(\theta_{\text{base}},\dots ,\theta^{(1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{0}_{i}(\theta_{\text{base}})+O (\eta^{3})$ $$

##### Inductive step

Let us assume that

$$ $\displaystyle\theta_{i}^{(m)}$ $\displaystyle=\theta^{(m)}_{\text{MT}}+\eta p_{i}^{m-1}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-2}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-2)}_{\text{MT}})+O(\eta^{3})$ $$

We can derive that

$$ $\displaystyle\theta_{i}^{(m+1)}$ $\displaystyle=\theta^{(m)}_{i}-\eta\nabla\overline{L}_{i}(\theta^{(m)}_{i})$ $\displaystyle=\theta^{(m)}_{\text{MT}}+\eta p_{i}^{m-1}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-2}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-2)}_{\text{MT}})-\eta\nabla\overline{L}_{i}(\theta^{m} _{i})+O(\eta^{3})$ $\displaystyle=\theta^{(m)}_{\text{MT}}+\eta p_{i}^{m-1}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-2}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-2)}_{\text{MT}})$ $\displaystyle-\eta\nabla\overline{L}_{i}\left(\theta^{(m)}_{\text{MT}}+\eta p_ {i}^{m-1}(\theta_{\text{base}},\dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2 }}{2}s^{m-2}_{i}(\theta_{\text{base}},\dots,\theta^{(m-2)}_{\text{MT}})\right) +O(\eta^{3})$ $\displaystyle=\theta^{(m)}_{\text{MT}}+\eta p_{i}^{m-1}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-2}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-2)}_{\text{MT}})$ $\displaystyle-\eta\nabla\overline{L}_{i}(\theta^{(m)}_{\text{MT}})-\frac{\eta} {2}\nabla^{2}\overline{L}_{i}(\theta^{(m)}_{\text{MT}})\left(\eta p_{i}^{m-1}( \theta_{\text{base}},\dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m- 2}_{i}(\theta_{\text{base}},\dots,\theta^{(m-2)}_{\text{MT}})\right)+O(\eta^{3})$ $\displaystyle=\theta^{(m)}_{\text{MT}}+\eta p_{i}^{m-1}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-2}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-2)}_{\text{MT}})$ $\displaystyle-\eta\nabla\overline{L}_{i}(\theta^{(m)}_{\text{MT}})-\frac{\eta^ {2}}{2}\nabla^{2}\overline{L}_{i}(\theta^{(m)}_{\text{MT}})p_{i}^{m-1}(\theta_ {\text{base}},\dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ $\displaystyle=\theta^{(m+1)}_{\text{MT}}+\eta p_{i}^{m}(\theta_{\text{base}}, \dots,\theta^{(m)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-1}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ $$

∎

###### Proof Theorem and Corollary.

For the first epoch

$$ $\theta^{(1)}_{\text{TA}}=\theta_{\text{base}}+\alpha\sum_{i\in T}\tau^{(1)}_{i }=\theta_{\text{base}}-\eta\alpha\sum_{i\in T}\nabla\overline{L}_{i}(\theta_{ \text{base}})$ $$

while, choosing $\alpha\eta$ as learning rate for the loss $\sum_{i\in T}\overline{L}_{i}$ :

$$ $\theta^{(1)}_{\text{MT}}=\theta_{\text{base}}-\alpha\eta\sum_{i\in T}\nabla \overline{L}_{i}(\theta_{\text{base}}).$ $$

So $\theta^{(1)}_{\text{MT}}=\theta^{(1)}_{\text{TA}}$.

For $k\geq 2$, notice that

$$ $\theta^{(k)}_{\text{MT}}=\theta_{\text{base}}-\alpha\eta\sum_{j=0}^{k-1}\nabla \sum_{t\in T}\overline{L}_{i}(\theta^{(j)}_{\text{MT}}).$ (20) $$

Now, using Lemma A.1, we get:

$$ $\displaystyle-\alpha\eta$ $\displaystyle\sum_{j=0}^{k-1}\nabla\sum_{t\in T}\overline{L}_{t}(\theta^{(j)}_ {\text{MT}})+\eta p_{i}^{k-1}(\eta^{0},\dots,\eta^{k-1}_{\text{MT}})$ $\displaystyle=-\alpha\eta\sum_{j=0}^{k-1}\nabla\sum_{t\in T}\overline{L}_{t}( \theta^{(j)}_{\text{MT}})+\sum_{j=0}^{k-1}r_{i}(\theta^{k}_{\text{MT}})$ $\displaystyle=-\alpha\eta\sum_{j=0}^{k-1}\nabla\sum_{t\in T}\overline{L}_{i}( \theta^{(j)}_{\text{MT}})+\sum_{j=0}^{k-1}\alpha\sum_{j\in T}\nabla\overline{L }_{j}(\theta^{(j)}_{\text{MT}})-\nabla\overline{L}_{i}(\theta^{(j)}_{\text{MT}})$ $\displaystyle=-\eta\sum_{j=0}^{k-1}\nabla\overline{L}_{i}(\theta^{(j)}_{\text{ MT}})\,.$ $$

Namely:

$$ $\displaystyle\theta_{i}^{(m+1)}$ $\displaystyle=\theta^{(m+1)}_{\text{MT}}+\eta p_{i}^{m}(\theta_{\text{base}}, \dots,\theta^{(m)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-1}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ $\displaystyle=\theta_{\text{base}}-\alpha\eta\sum_{j=0}^{m}\nabla\sum_{t\in T} \overline{L}_{i}(\theta^{(j)}_{\text{MT}})+\eta p_{i}^{m}(\theta_{\text{base}} ,\dots,\theta^{(m)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-1}_{i}(\theta_{\text{ base}},\dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ $\displaystyle=\theta_{\text{base}}-\eta\sum_{j=0}^{m}\nabla\overline{L}_{i}( \theta^{(j)}_{\text{MT}})-\frac{\eta^{2}}{2}s^{m-1}_{i}(\theta_{\text{base}}, \dots,\theta^{(m-1)}_{\text{MT}})+O(\eta^{3})$ $$

we can rewrite the tasks vectors as

$$ $\displaystyle\tau_{i}^{(k)}$ $\displaystyle=\theta^{(k)}_{i}-\theta_{\text{base}}$ (21) $\displaystyle=-\eta\sum_{j=0}^{k-1}\nabla\overline{L}_{i}(\theta^{(j)}_{\text{ MT}})-\frac{\eta^{2}}{2}s^{k-2}_{i}(\theta_{\text{base}},\dots,\theta^{(k-2)}_ {\text{MT}})+O(\eta^{3})$ (22) $$

Consequently the model obtained with TA is

$$ $\displaystyle\theta^{(k)}_{\text{TA}}$ $\displaystyle=\theta_{\text{base}}+\alpha\sum_{i\in T}\tau^{(k)}_{i}$ $\displaystyle=\theta_{\text{base}}-\eta\alpha\sum_{j=0}^{k-1}\sum_{i\in T} \nabla\overline{L}_{i}(\theta^{(j)}_{\text{MT}})-\alpha\sum_{i\in T}\frac{\eta ^{2}}{2}s^{k-2}_{i}(\theta_{\text{base}},\dots,\theta^{(k-2)}_{\text{MT}})+O( \eta^{3})$ $\displaystyle=\theta^{(k)}_{\text{MT}}-\alpha\sum_{i\in T}\frac{\eta^{2}}{2}s^ {k-2}_{i}(\theta_{\text{base}},\dots,\theta^{(k-2)}_{\text{MT}})+O(\eta^{3})\,.$ $$

∎

###### Proof Theorem and Corollary.

### A.2 Proof of Theorem 5.1

In this section, we provide a rigorous proof of Theorem 5.1, establishing the conditions under which ATM implicitly optimizes the loss of the model trained on the union of datasets.

###### Proof.

Suppose that when transitioning from parameters $\theta^{(i)}$ to parameters $\theta^{(i+1)}$, the change in the average loss for each dataset $D_{k}$ is given by $\Delta\overline{L_{k}}=\overline{L_{k}}(\theta^{(i)})-\overline{L_{k}}(\theta^
{(i+1)})$.
We denote by $P$ the set of tasks for which the delta of the loss is positive and by $P$ the set for which it is negative, namely
$P=\{k\in$T$\text{ s.t. }\Delta\overline{L_{k}}>0\}$ and $N=\{k\in$T$\text{ s.t. }\Delta\overline{L_{k}}\leq 0\}$.
The set $\{1,\dots,n\}=P\cup N$.
In the following formulas we will use the symbol $|$ for different purposes. For sets, like $D_{i}$, $|D_{i}|$ denotes the cardinality of the set, while for scalars, such as $\Delta\overline{L_{k}}$, $|\Delta\overline{L_{k}}|$ denotes their absolute value. Since the tasks in $N$ have negative $\Delta\overline{L_{k}}$, it holds that

$$ $\sum_{j\in N}|D_{j}|\Delta\overline{L_{j}}=-\sum_{j\in N}|D_{j}||\Delta \overline{L_{j}}|\,.$ $$

Under the hypothesis that $\Delta L_{\text{ATM}}>\delta$, this implies $\sum_{j=1}^{n}\Delta_{j}>n\delta$. We have that
$\sum_{j\in P}\Delta\overline{L_{j}}+\sum_{j\in N}\Delta\overline{L_{j}}=\sum_{
j\in P}\Delta\overline{L_{j}}-\sum_{j\in N}|\Delta\overline{L_{j}}|>n\delta$, namely $\sum_{j\in P}\Delta\overline{L_{j}}>n\delta+\sum_{j\in N}|\Delta\overline{L_{j
}}|$.

We want to prove that $\Delta L_{\text{target}}>0$.
Let us now consider $\Delta L_{\text{target}}>0$ iff $\sum_{j=1}^{n}|D_{j}|\Delta_{j}>0$.

$$ $\displaystyle\sum_{j=1}^{n}|D_{j}|\Delta_{j}$ $\displaystyle=\sum_{j\in P}|D_{j}|\Delta\overline{L_{j}}+\sum_{j\in N}|D_{j}| \Delta\overline{L_{j}}$ $\displaystyle=\sum_{j\in P}|D_{j}|\Delta\overline{L_{j}}-\sum_{j\in N}|D_{j}|| \Delta\overline{L_{j}}|$ $\displaystyle>\min_{j\in P}|D_{j}|\sum_{j\in P}\Delta\overline{L_{j}}-\max_{j \in N}|D_{j}|\sum_{j\in N}|\Delta\overline{L_{j}}|$ $\displaystyle>\min_{j\in P}|D_{j}|\left[n\delta+\sum_{j\in N}|\Delta\overline{ L_{j}}|\right]-\max_{j\in N}|D_{j}|\sum_{j\in N}|\Delta\overline{L_{j}}|$ $\displaystyle=n\min_{j\in P}|D_{j}|\delta+(\min_{j\in P}|D_{j}|-\max_{j\in N}| D_{j}|\sum_{j\in N})|\Delta\overline{L_{j}}|\,.$ $$

The last line of the previous equation is positive by hypothesis, since we assumed
$\delta>\frac{1}{n}(1-\frac{\min_{j\in N}|D_{j}|}{\max_{j\in P}|D_{j}|})\sum_{j
\in N}|\Delta\overline{L_{j}}|$.
∎

###### Proof.

## Appendix B Additional Results

### B.1 Full results over varying computational budget

In the main paper, we pictorially illustrated the multi-task accuracy of the baselines and ATM variants, in the form of radar plots. For deeper analysis, here we report the full results of ATM compared to the baselines, as the computational budget varies for 2, 4, 7, and 10 epochs. See Tables 3, 4, and 5 respectively.

**Table 3: ATM vs. Baselines as budget varies (ViT-B-16)**
|  |  | CIFAR100 | DTD | EuroSAT | GTSRB | MNIST | RESISC45 | SVHN | average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $K=2$ | Task Arithmetic | 0.58 | 0.39 | 0.78 | 0.59 | 0.98 | 0.58 | 0.83 | 0.68 |
| TIES | 0.76 | 0.46 | 0.71 | 0.55 | 0.96 | 0.71 | 0.81 | 0.71 |  |
| Breadcrumbs | 0.72 | 0.48 | 0.80 | 0.61 | 0.97 | 0.70 | 0.81 | 0.72 |  |
| ValFT ATM | 0.70 | 0.53 | 0.84 | 0.84 | 0.98 | 0.78 | 0.90 | 0.80 |  |
| ATM | 0.74 | 0.51 | 0.89 | 0.76 | 0.98 | 0.74 | 0.89 | 0.79 |  |
| $K=4$ | Task Arithmetic | 0.71 | 0.48 | 0.80 | 0.62 | 0.97 | 0.71 | 0.82 | 0.73 |
| TIES | 0.75 | 0.45 | 0.69 | 0.56 | 0.97 | 0.69 | 0.84 | 0.70 |  |
| Breadcrumbs | 0.68 | 0.43 | 0.79 | 0.61 | 0.97 | 0.67 | 0.81 | 0.71 |  |
| ValFT ATM | 0.66 | 0.55 | 0.73 | 0.85 | 0.98 | 0.83 | 0.94 | 0.79 |  |
| ATM | 0.75 | 0.55 | 0.95 | 0.90 | 0.99 | 0.82 | 0.94 | 0.84 |  |
| $K=7$ | Task Arithmetic | 0.67 | 0.44 | 0.79 | 0.62 | 0.97 | 0.67 | 0.83 | 0.71 |
| TIES | 0.72 | 0.43 | 0.66 | 0.56 | 0.97 | 0.66 | 0.85 | 0.69 |  |
| Breadcrumbs | 0.65 | 0.41 | 0.76 | 0.62 | 0.97 | 0.62 | 0.81 | 0.69 |  |
| ValFT ATM | 0.75 | 0.62 | 0.66 | 0.96 | 0.99 | 0.85 | 0.96 | 0.83 |  |
| ATM | 0.77 | 0.58 | 0.98 | 0.96 | 0.99 | 0.87 | 0.95 | 0.87 |  |
| $K=10$ | Task Arithmetic | 0.63 | 0.41 | 0.76 | 0.63 | 0.97 | 0.62 | 0.83 | 0.69 |
| TIES | 0.68 | 0.41 | 0.70 | 0.56 | 0.98 | 0.62 | 0.86 | 0.69 |  |
| Breadcrumbs | 0.60 | 0.40 | 0.78 | 0.58 | 0.98 | 0.59 | 0.81 | 0.68 |  |
| ValFT ATM | 0.78 | 0.61 | 0.53 | 0.97 | 0.99 | 0.88 | 0.95 | 0.82 |  |
| ATM | 0.79 | 0.61 | 0.98 | 0.97 | 0.99 | 0.89 | 0.96 | 0.89 |  |

**Table 4: ATM vs Baselines as budget varies (RoBERTa-base)**
|  |  | CoLA | SST2 | MRPC | QQP | MNLI | QNLI | RTE | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $K=2$ | Task Arithmetic | 0.70 | 0.56 | 0.66 | 0.37 | 0.47 | 0.54 | 0.51 | 0.54 |
| TIES | 0.69 | 0.51 | 0.68 | 0.37 | 0.37 | 0.51 | 0.47 | 0.51 |  |
| Breadcrumbs | 0.69 | 0.53 | 0.66 | 0.37 | 0.45 | 0.52 | 0.48 | 0.53 |  |
| DARE | 0.84 | 0.53 | 0.68 | 0.41 | 0.32 | 0.50 | 0.47 | 0.54 |  |
| valFT ATM | 0.71 | 0.78 | 0.68 | 0.46 | 0.64 | 0.68 | 0.67 | 0.66 |  |
| ATM | 0.72 | 0.78 | 0.69 | 0.44 | 0.65 | 0.68 | 0.62 | 0.66 |  |
| $K=4$ | Task Arithmetic | 0.71 | 0.60 | 0.66 | 0.37 | 0.51 | 0.54 | 0.51 | 0.56 |
| TIES | 0.69 | 0.51 | 0.68 | 0.37 | 0.42 | 0.51 | 0.47 | 0.52 |  |
| Breadcrumbs | 0.70 | 0.57 | 0.66 | 0.37 | 0.49 | 0.53 | 0.51 | 0.55 |  |
| DARE | 0.83 | 0.50 | 0.65 | 0.41 | 0.33 | 0.50 | 0.46 | 0.53 |  |
| valFT ATM | 0.70 | 0.83 | 0.70 | 0.68 | 0.67 | 0.73 | 0.62 | 0.70 |  |
| ATM | 0.73 | 0.81 | 0.70 | 0.58 | 0.70 | 0.80 | 0.67 | 0.71 |  |
| $K=7$ | Task Arithmetic | 0.71 | 0.65 | 0.65 | 0.38 | 0.52 | 0.56 | 0.55 | 0.57 |
| TIES | 0.69 | 0.51 | 0.68 | 0.37 | 0.42 | 0.51 | 0.47 | 0.52 |  |
| Breadcrumbs | 0.71 | 0.61 | 0.65 | 0.37 | 0.50 | 0.54 | 0.52 | 0.56 |  |
| DARE | 0.84 | 0.52 | 0.65 | 0.43 | 0.33 | 0.49 | 0.44 | 0.52 |  |
| valFT ATM | 0.69 | 0.85 | 0.69 | 0.75 | 0.68 | 0.72 | 0.60 | 0.71 |  |
| ATM | 0.72 | 0.79 | 0.70 | 0.73 | 0.73 | 0.82 | 0.67 | 0.74 |  |
| $K=10$ | Task Arithmetic | 0.71 | 0.66 | 0.60 | 0.38 | 0.56 | 0.59 | 0.60 | 0.59 |
| TIES | 0.69 | 0.51 | 0.68 | 0.37 | 0.43 | 0.51 | 0.47 | 0.52 |  |
| Breadcrumbs | 0.71 | 0.61 | 0.62 | 0.38 | 0.54 | 0.57 | 0.57 | 0.57 |  |
| DARE | 0.85 | 0.51 | 0.64 | 0.42 | 0.33 | 0.49 | 0.44 | 0.52 |  |
| valFT ATM | 0.68 | 0.86 | 0.66 | 0.76 | 0.70 | 0.73 | 0.63 | 0.72 |  |
| ATM | 0.72 | 0.83 | 0.69 | 0.76 | 0.74 | 0.81 | 0.66 | 0.74 |  |

**Table 5: ATM vs. Baselines as budget varies (BERT-base-uncased)**
|  |  | CoLA | SST2 | MRPC | QQP | MNLI | QNLI | RTE | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $K=2$ | Task Arithmetic | 0.61 | 0.70 | 0.39 | 0.65 | 0.41 | 0.57 | 0.54 | 0.55 |
| TIES | 0.32 | 0.50 | 0.32 | 0.63 | 0.37 | 0.50 | 0.54 | 0.45 |  |
| Breadcrumbs | 0.60 | 0.68 | 0.39 | 0.65 | 0.40 | 0.57 | 0.54 | 0.55 |  |
| DARE | 0.82 | 0.54 | 0.69 | 0.40 | 0.33 | 0.50 | 0.46 | 0.53 |  |
| ValFT ATM | 0.66 | 0.71 | 0.47 | 0.65 | 0.44 | 0.57 | 0.56 | 0.58 |  |
| ATM | 0.68 | 0.68 | 0.39 | 0.64 | 0.46 | 0.56 | 0.61 | 0.58 |  |
| $K=4$ | Task Arithmetic | 0.59 | 0.66 | 0.39 | 0.67 | 0.41 | 0.55 | 0.57 | 0.55 |
| TIES | 0.32 | 0.50 | 0.32 | 0.63 | 0.38 | 0.51 | 0.53 | 0.46 |  |
| Breadcrumbs | 0.58 | 0.65 | 0.40 | 0.67 | 0.41 | 0.55 | 0.56 | 0.54 |  |
| DARE | 0.82 | 0.53 | 0.66 | 0.43 | 0.33 | 0.50 | 0.47 | 0.54 |  |
| ValFT ATM | 0.67 | 0.75 | 0.47 | 0.70 | 0.54 | 0.64 | 0.62 | 0.62 |  |
| ATM | 0.68 | 0.76 | 0.44 | 0.74 | 0.59 | 0.66 | 0.67 | 0.65 |  |
| $K=7$ | Task Arithmetic | 0.58 | 0.65 | 0.38 | 0.66 | 0.42 | 0.55 | 0.58 | 0.55 |
| TIES | 0.33 | 0.50 | 0.33 | 0.63 | 0.38 | 0.51 | 0.53 | 0.46 |  |
| Breadcrumbs | 0.58 | 0.65 | 0.38 | 0.66 | 0.41 | 0.55 | 0.59 | 0.55 |  |
| DARE | 0.83 | 0.52 | 0.66 | 0.43 | 0.33 | 0.51 | 0.46 | 0.53 |  |
| ValFT ATM | 0.66 | 0.75 | 0.47 | 0.70 | 0.59 | 0.67 | 0.64 | 0.64 |  |
| ATM | 0.69 | 0.81 | 0.51 | 0.78 | 0.63 | 0.70 | 0.66 | 0.68 |  |
| $K=10$ | Task Arithmetic | 0.59 | 0.66 | 0.39 | 0.64 | 0.42 | 0.55 | 0.59 | 0.55 |
| TIES | 0.35 | 0.51 | 0.34 | 0.64 | 0.38 | 0.51 | 0.53 | 0.47 |  |
| Breadcrumbs | 0.59 | 0.65 | 0.39 | 0.64 | 0.42 | 0.55 | 0.60 | 0.55 |  |
| DARE | 0.82 | 0.51 | 0.69 | 0.42 | 0.33 | 0.51 | 0.47 | 0.54 |  |
| ValFT ATM | 0.66 | 0.73 | 0.50 | 0.71 | 0.59 | 0.69 | 0.63 | 0.64 |  |
| ATM | 0.68 | 0.81 | 0.53 | 0.78 | 0.65 | 0.72 | 0.66 | 0.69 |  |

### B.2 Cosine similarity of epoch-wise gradients

We report in Fig. 11 the cosine similarity of gradients for the first $10$ epochs over DTD and EuroSAT, as these do not have a marked difference in gradient norms between the first epoch and the remaining ones in Fig. 3(a). We see that the alignment of subsequent gradients observed in RESISC45 still holds in DTD, even if with less marked similarities. On the other hand, this does not seem to hold for EuroSAT.

Figure: (a) Dataset EuroSAT
Refer to caption: extracted/5981356/images/cosine_similarity_heatmap_EuroSAT_latex_font.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: The proposed method (ATM) is fairly simple, but is shown to have significantly higher performance over “one-shot” baselines and methods, where models are fine-tuned once and then merged. This demonstrates the value of merging models intermittently, rather than training models fully in a task-specific manner and then merging them.

The paper also has some interesting analysis regarding the gradients of individual tasks over the course of training, which can help explain the efficacy of ATM and help us understand why merging earlier as opposed to later can be beneficial.
summary: This paper proposes a novel model “merging” algorithm, where the goal is to merge separately tuned, task-specific models into a joint multi-task model. The paper begins by making several observations regarding the paradigm of task-specific fine-tuning followed by model merging. First, they demonstrate the relationship between fine-tuning then model merging and standard multi-task learning (i.e. following the average task gradient at each step). Additionally, they show that merging the model after a single epoch results in stronger generalization than merging after the task-specific models converge on their respective tasks, and present some empirical analysis to help explain this finding.

Motivated by these findings, the authors propose Alternating Tuning and Merging (ATM), wherein training alternates between making task-specific updates to task-specific models and making a “global” update by merging the task-specific models together. The authors demonstrate that ATM significantly outperforms other model merging baselines, which all consist of a single merge step. Additionally, the authors provide analysis of performance regarding the frequency at which models are merged vs. tuned under an epoch budget, showing that merging more frequently improves performance.
weaknesses: My understanding is that model merging is primarily a technique for taking advantage of models which are fine-tuned separately from one another, e.g. combining separate open-source models trained by different individuals. In other words, model merging is a technique to construct multi-task models when multi-task data is not available, i.e. when a single joint model cannot be trained. The proposed method, however, assumes that we have access to the entire multi-task data at one time, in order to alternate between tuning and merging.

In this case, it is not clear to me why we would not simply do multi-task learning, and indeed this paper seems to suggest that we should be doing precisely that: performance goes up as we merge more often during training, and as we merge more often during training we more closely approximate multi-task learning. However, pure MTL (i.e. no task-specific updates) is not considered here as a baseline. Thus, it is not clear to me how this new method (ATM) fits into the current paradigm of model merging or transfer learning, outside of an analysis on the impact of the rate at which models are merged versus tuned.

It is also worth mentioning that the proposed method is similar to heterogeneous federated averaging, where learning algorithms often alternate between making local “task-specific” updates to task-specific models and making global “multi-task” updates using changes to the task-specific models. In this literature, it has been established that “global SGD”, which makes no local updates, has better convergence guarantees but that “local SGD” which makes some task-specific updates before merging can have some generalization benefits [1,2]. I feel strongly that this literature needs to be acknowledged and discussed in this paper, given their similarities.

A smaller comment: in line 420 it is claimed that the hyperparameters are untuned in the data-free setting, and then later mentioned that the author-suggested hyperparameters are used. I am not sure that both can be true however, as it is likely the case that the previous authors used the validation data for tuning the hyperparameters, and therefore that the hyperparameters actually have seen the data being used for training in this case.

The floating figures need to be fixed, there are 3 pages where the figure hangs off the bottom edge of the page and messes up the formatting of the next page as well.

[1] Woodworth et al., 2022; Minibatch vs Local SGD for Heterogeneous Distributed Learning
[2] Bao et al., 2024; Provable Benefits of Local Steps in Heterogeneous Federated Learning for Neural Networks: A Feature Learning Perspective

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: 1. How is section 5 connected to the method specifically? It seems to be restating multi-task learning findings
soundness: 1
strengths: The proposed method has a GPU memory advantage compared to gradient-balancing multi-task learning approaches, since the fine-tuning for each task can be performed independently.
summary: The paper proposes a method that replaces the finetune-and-then-merge paradigm with alternating fine-tuning and merging. The paper also connects task vectors with multi-task gradients and evaluates on standard vision and NLP benchmarks.
weaknesses: The paper has many flaws, from poor placement in the literature, misconfiguring baselines and ignoring the most relevant ones, overstating the results and representing known results as novel. Specifically:


1. The method is wrongly presented as model merging, while it is more closely related to joint fine-tuning. Imo, it cannot be seen as model merging, since the task combination is known a priori. The method is inflexible and defeats the purpose of model merging
2. Moreover, there is not a single reference for multi-task learning, nor is it considered as a baseline. The method mostly resembles gradient-based MTL solutions [1-3, inter alia], where instead of gradients being computed at every step, task vectors are computed and merged after some steps. This can actually be implied by lines 88-91. The paper would benefit from looking into this perspective by first motivating the need of gradient-balancing solutions (vs loss-balancing [4])in the fine-tuning paradigm (which might actually not be the case) and then formulating the method as a way to combat the memory issues associated with gradient-balancing
3. Task vectors vs gradients: this is an obvious connections. For one step of (S)GD, the definitions are the same.
4. Lack of proper citations: no mention of multi-task literature as well as other places. For instance: “under this unifying lens, reducing task interference is analogous to reducing conflicting gradients in multi-task learning” misses [5,6] for interference and [3] for conflicting gradients.
5. Section 6 is poorly structured, starting with ablation studies culminating into the main results deep into the section
6. Lack of scientific rigor: “In all setting, we adopt mean aggregation of task vectors and use a scaling factor of 1 when applying them to the base model”. This shows that the baselines are misconfigured, making the comparison unfair.
7. Overstating the results: Given the lack of proper baseline selection and correct setup of used baselines, the stated improvements of “up to 19% in computer vision and 20% in NLP over the best baselines” are far-fetched. 
8. “greatest contribution is given by the first epoch”: this is presented as a novel insight but it is not. This occurs naturally during training since the model adapts to the new distribution and, the same applies to Fine-tuning
9. Increased orthogonality: the comparison is unfair, since the authors choose to compare the task vectors of the last iteration with the “general” ones form standard fine-tuning. They will be more orthogonal since the joint features are already embedded in the multi-task vector via the proposed joint fine-tuning and the last fine-tuning only focuses on task-specific features. 

Given the above, I disagree with all four of the stated contributions of this work.

[1] Navon A, Shamsian A, Achituve I, Maron H, Kawaguchi K, Chechik G, Fetaya E. Multi-task learning as a bargaining game. ICML 2022.

[2] Chen Z, Badrinarayanan V, Lee CY, Rabinovich A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational conference on machine learning 2018 Jul 3 (pp. 794-803). PMLR.

[3] Yu T, Kumar S, Gupta A, Levine S, Hausman K, Finn C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems. 2020;33:5824-36.

[4] Kendall, Alex, Yarin Gal, and Roberto Cipolla. "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[5] Yadav, P., Tam, D., Choshen, L., Raffel, C.A. and Bansal, M., 2024. Ties-merging: Resolving interference when merging models. *Advances in Neural Information Processing Systems*, *36*.

[6] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F. and Frossard, P., 2024. Localizing Task Information for Improved Model Merging and Compression. *ICML 2024*.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: None.
soundness: 1
strengths: 1. The paper addresses model merging, an emerging area of high relevance for reducing fine-tuning costs in multi-task settings.

2. The initial theoretical insights linking task vectors with gradients of the loss for the corresponding tasks after the first gradient descent step are novel.
summary: The paper proposes *Alternating Tuning and Merging* (ATM), a novel approach for merging single-task models into a multi-task model. ATM alternates between fine-tuning each task separately and merging through weight interpolation techniques, such as task arithmetic or TIES. Experimental results show ATM’s advantages over standard model merging baselines for small ViT models on computer vision tasks and BERT-like models on an NLP benchmark.
weaknesses: 1. **Misalignment with Model Merging Goals**. The paper appears to overlook the key aims of model merging. The motivation for merging models fine-tuned on different tasks is not only saving memory – as the introduction of the paper claims – but instead bypassing expensive joint fine-tuning by enabling the combination of models fine-tuned *independently* on separate tasks. Indeed, model merging methods often rely on available checkpoints from online hubs. Contrary to this goal, ATM’s merging process mixes models already at the fine-tuning stage. This loses all the modularity properties and undermines the very benefits of model merging. Consequently, the baseline most relevant for this study is standard multi-task learning, i.e., joint fine-tuning, which is missing (and expected to obtain superior performance over ATM). Table 2 comparing computational costs is therefore misleading since ATM requires joint fine-tuning. For example, adding one task with Task Arithmetic or TIES is straightforward. Adding one task with ATM requires full retraining of the model on all tasks.

2. **Baseline Comparison and Parameter Choices**. The baselines used in the experiments rely on suboptimal hyperparameters (see, e.g., line 350 "in all settings, we adopt mean aggregation of task vectors and use a scaling factor of 1 when applying them to the base model"). This decision severely weakens the validity of all results of the paper and prevents a fair comparison with past approaches. Moreover, the paper cites but omits evaluation against state-of-the-art merging methods such as AdaMerging, which, despite its significant computational overhead, is far less resource-intensive than ATM.

3. **Experimental Scope**. The experiments are limited to small and/or outdated models. For instance, in the case of computer vision, only the ViT B16 model is used. Standard benchmark architectures such as ViT large, T5, or LLaMA models are not included. Arguably, this is due to the excessive computational overhead of ATM compared to present methods, which hinders the scalability of ATM, further reinforcing the concerns above.

4. **Presentation of Results**. The paper opts for qualitative radar charts over numerical tables, complicating performance comparisons. Single-task accuracies after standard fine-tuning are never reported for the baselines, normalized accuracies are also missing. Together with the use of suboptimal hyperparameters and a choice of datasets that are not the standard in the field, this makes it difficult to compare the results against previous work. 

5. **Theoretical Analysis and Testing**. The theoretical analysis lacks soundness. Firstly, the fact that task vectors are precisely equal to single-task gradient updates just for the first step of gradient descent does not imply that the best merging performance is obtained when merging after the first step. In fact, the accuracy of the model also improves in the next steps, so even if errors accumulate, this does not exclude the improvement of performance after merging. Secondly, the paper does not test this hypothesis directly with gradient descent; instead, it uses stochastic gradient descent, for which the analysis does not hold. On the one hand, the natural approximation is to approximate one step of GD with one step of SGD (with large batch size) in the early phases of training and not with a full epoch consisting of hundreds if not thousands of steps. On the other hand, testing it with gradient descent is straightforward using gradient accumulation.  If done, results would likely show that merging performance post first step is not optimal, as the model would still closely reflect zero-shot capabilities at this stage.

6. **Presentation Quality**. The paper suffers from several presentation issues. Figures are of low quality, extend beyond page margins, and are occasionally truncated. Some captions are missing. Table 2 is presented before Table 1, ignoring the logical flow. Additionally, the paper’s notation is inconsistent, e.g., symbols such as $\theta_0, \theta_{base}, \theta_{init}$ appear to refer to the same quantity.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. When observing Figure 6 and Table 1 together, is the baseline result reported in Table 1 corresponding to the result at epoch 10? Although the authors' method achieves the best results at 10 or even 30 epochs, is 10 epochs too many for these single-task fine-tuning tasks?
2. In Figure 6, the authors illustrate, through the declining curve of baselines like Task Arithmetic, the expert capability increased with the decreased performance of the final merged model. However, could the performance decline of the merged model be due to over-training of individual expert models? It is suggested that the authors report the performance changes of a single expert model across epochs to further support their claim.
3. Regarding valFT ATM, reaching such astonishing results with only the validation set is noteworthy. Could the authors provide more information on this setting?
4. In my understanding, Theorem 3.1 and Corollary 3.1.1 conclude that models obtained by GD then merged and the model obtained by MTL loss (defined as \bar{L}_t(\theta) in Theorem 3.1) are somewhat equivalent only for 1 epoch. When exceeding 1 epoch, additional errors are introduced (not sure if I understood correctly?). I strongly suggest the authors report the results of the standard MTL (simultaneously fine-tuning multiple tasks) for further theoretical validation.

Typos:
Formatting errors in various figures and tables, including Figures 6 and 8 and Table 2.
The caption of Figure 3 is missing.
soundness: 3
strengths: 1. **Theoretical Support**: The authors use proven theories in some discussions to support their viewpoints.
2. **Comprehensive Ablation Study**: There is discussion and experimental verification of various aspects of the proposed method.
summary: In this paper, the authors first discover through theory and experiment that merging models of fine-tuning until convergence is not the optimal choice. Consequently, they introduce a technique that involves independently training each model for one epoch and then using model merging to create a new model to serve as the starting point for the next training round. This process is repeated to obtain the final merged model. The authors conducted experiments on models such as ViT-B-16 and achieved remarkable results.
weaknesses: 1. **Scope of the Method and Application Scenario**: The proposed method requires original training data (at least validation data) and training, whereas the baselines (including Task Arithmetic, TIES, DARE, etc.) used for comparison are in fact data-free and training-free merging methods, allowing them to directly merge existing models. The requirements for data and training are obligatory for the authors’ method, thus making the application scenarios quite different. The authors should compare their method with others that require similar amounts of data and training (at least with the most standard MTL results). Alternatively, authors could define the approach as a new training paradigm that utilizes merging method, rather than simply as a model merge method.
2. **Inappropriate Cost Comparison**: For the same reason, the comparison of computational and storage costs in Section 6.6 is also unreasonable. The merging stage of the authors’ method remains Task Arithmetic, but it combines with training to change the timing of merging. Thus, it is unreasonable to only consider the computational and storage costs of the merging stage; training costs are unavoidable.