Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models
Authors: Filippo Rinaldi
Aniello Panariello
Giacomo Salici
Fengyuan Liu
Marco Ciccone
Angelo Porrello
Simone Calderara
Venue: ICLR 2026 Poster (2026)
Keywords: model rebasin
fine-tuning transfer
transfer learning
model editing
model patching
compositionality
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Gradient-sign masking transfers task vectors across foundation models with few samples, outperforming naive transfer and few-shot fine-tuning.

Abstract: When a new release of a foundation model is published, practitioners typically need to repeat fine-tuning, even if the same task was already tackled in the previous version. A promising alternative is to reuse the parameter changes (i.e., task vectors) that capture how a model adapts to a specific task. However, these vectors often fail to transfer across different pre-trained models because their parameter spaces are misaligned. In this work, we show that successful transfer depends strongly on the gradient-sign structure of the new model. Based on this insight, we propose GradFix, which approximates the ideal sign structure and leverages it to transfer knowledge using only a handful of labeled samples. Notably, this requires no additional fine-tuning: we only compute a few target-model gradients without parameter updates and mask the source task vector accordingly. This yields an update that is locally aligned with the target loss landscape, effectively rebasing the task vector onto the new pre-training. We provide a theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant performance gains on vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning. We further show that transporting task vectors improves multi-task and multi-source model merging. Code is available at https://github.com/fillo-rinaldi/GradFix.

Published as a conference paper at ICLR 2026
GRADIENT-SIGN MASKING FOR TASK VECTOR TRANSPORT ACROSS PRE-TRAINED MODELS
Filippo Rinaldi', Aniello Panariello's Giacomo Salici', Fengyuan Liu
Marco Ciccone", Angelo Porrello', Simone Calderara'
University of Modena and Reggio Emilia, Italy
Vector Institute, Canada *University of Toronto, Canada
name aaxname@unniore name www.ssnnnnveeetttiinnitttttttue.
ABSTRACT
When new release of foundation model published, practitioners typically need to repeat fine-tuning, even if the same task was already tackled in the previ- ous version. A promising alternative to reuse the parameter changes (i.e., task vectors) that capture how model adapts to specific task. However, these vectors often fail to transfer across different pre-trained models because their parameter spaces are misaligned. In this work, we show that successful transfer depends strongly on the gradient-sign structure of the new model. Based on this insight, we propose GradFix, which approximates the ideal sign structure and leverages it transfer knowledge using only a handful of labeled samples. Notably, this requires additional fine-tuning: we only compute few target-model gradients without parameter updates and mask the source task vector accordingly. This yields an up- date that is locally aligned with the target loss landscape. effectively rebasing the task vector onto the new pre-training. We provide theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant per- formance gains vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning. We further show that trans- porting task vectors improves multi-task and multi-source model merging. Code is available at athttpss://githiim com/ fillo rinal di di/GradFix.
INTRODUCTION
Over the past few years, the paradigm in deep learning has shifted from training models from scratch to fine-tuning large pre-trained models. Adapting these large models to downstream tasks is advan- tageous, as it leads to stronger performance at fraction of the cost. Such a shift has been evident in natural language processing and computer vision, where pre- treitriined models such BERT (Dev le ct al. 2019). CLIP (Radford et al.] 2021), and their successors (OpenAl et al. 2023 Liu et al. 2023) have become the standard starting point for developing
applications.
Since companies and researchers often update checkpoints using more data or improved training pipelines, practitioners frequently need to repeat fine-tuning the same downstream tasks. This creates redundancy: the work invested in adapting one release is not directly reusable on the next. To address this issue, several lines of research have investigated how to systematically relate transfer knowledge ledge across parameter spaces. The model rebasin literature (Ainsworth et al. 2023) Rinaldi al. 2025) studies how align and merge independently trained models by exploiting permutation symmetries in their parameters. In parallel, Task Arithmetic (Ilharco et al.) [203)
al. 2023; Yadav et al. 2023 Panariello et al., 2025b; Porrello et al.] (2026) has shown that task vectors (i.e., the difference subtracted, and merged across models to induce new capabilities. On a similar note, the literature on mode connectivity (Garipov et al. 2018 Frankle et al.) 2020) demonstrates that different fine- tuned solutions can be linked by low- loss paths, highlighting that model parameters encode highly structured and transferable representations. Together, these advances suggest that parameters encode rich and transferable structure that can be systematically manipulated to obtain the desired behavior reduced cost.
-/"- 8" between base and fine-tuned parameters) can be added,

---

Published as a conference paper at ICLR 2026
In particular, Rinaldi al. (2025) formalizes this setting and proposes technique to transport task vectors across transformer- basedd architectures. Yet a large gap remains between the transported fine-tune and an actual fine-tumed model, which highlights key challenge: while task vectors are informative about adaptation, their direct transfer across different pre-trained models is not guaran- teed to align with the loss geometry of the target model. In fact, naive transfer may introduce harmful directions in parameter space, i.e., components of the task vector that are misaligned with the de- scent directions of the target loss, thus increasing the loss and limiting its effectiveness. Addressing this problem is crucial both for reducing the cost of adapting rapidly evolving foundation models and for enabling their use in low-data regimes, where re- re-running full fine-tuning infeasible.
In this work, we introduce a framework for transporting task- knowledge across pre- pre-trained models using gradient-sign masking. Our key insight is that, although a fine-tuning trajectory encodes valuable task information, its effectiveness on a new pre-trained model depends on the local loss geometry of the target. Inspired by findings from the optimization and distributed training literature (Bernstein et al. 2018 Alistarr et al. 2017), we exploit the observation that the sign of the gradient provides a robust surrogate for the descent direction. Leveraging this insight, we introduce simple yet effective method to transport a task vector from source model to target pre- tre-trained model: we mask the source task vector using the gradient signs of the target, keeping only the components aligned with the target's local loss landscape. We further provide formal guarantee that, to first order, this transported update reduces the target loss, ensuring a principled safeguard against harmful or misaligned transfer.
Empirically, we show that this method enables highly effective transfer of fine-tuning knowledge from an outdated pre-trained model to newer one, even in the low-data regime where gradients can only be estimated from a handful of samples, partially closing the gap between naive transfer and full tne-ttnnng on the target model. We also evaluate GradFix in model-merging pipelines, showing gains in both multi- task and multi-source transport settings. Our contributions are:
We establish a theoretical connection between the oracle task vector, the ideal fine-tuning update on the target model, and quantities we can actually compute, namely the source task vector and the gradient at the zero-shot target model. We show that the sign of the zero-shot gradient provides a reliable proxy for the descent directions encoded in the target model.
Building on this insight, we propose GradFix, simple mechanism that filters the source task vector using the target model's local loss geometry. We formally prove that, to first order, the transported update reduces the target loss.
We empirically show that our method enables effective transport of fine-tuning knowledge across pre-trained models in both vision and text domains, even in the low-data regime where gradients must be estimated from only a handful of samples. We further validate that GradFix improves model-merging performance in both multi-task and multi-soure settings, showing that the transported updates remain useful beyond single-task transfer.
2 RELATED WORK
Model merging, Prior work studies how to merge fine-tuned checkpoints from the same pre-trained model. Model soups show that simple weight averaging can improve generalization (Wortsman eet al., (022). Task Arithmetic interprets fine-tuning deltas as task vectors that can be combined through linear operations (lhhacco et al. 2023), and TTES--Merging addresses conflicts by enforcing sign consistency (Yadav et al.) 2023). Recent extensions include curvature-awar composition for improved adaptation (Porrello et al. (225)) and modular embedding recomposition for more flexible transfer across diverse tasks (Panariello et al. 2025a).
Model rebasin. A different family of methods focuses on explicit rehasing, aligning independently pre-trained models into shared parameterization so that task vectors can be transported across different pre-trainings. Git Re- Re-Basin introduced permutation matching to map two networks into common basin (Ainsworth et al. 2023). For transformers, [mfeld et al. (2024) applies Optimal Transport to softly align components, while Rinaldi et al. (2025) proposes permutation- and spectral- based procedures that enable task vector transfer across distinet pre-trained models. Extensions such as permutation least-squares (Nasery et al.) (025) and supernet formulations (Stoica et al.) 2024) address more heterogeneous or large-scale settings.
2

---

Published as a conference paper at ICLR 2026
Gradient information. A complementary line of work studies the utility of gradient signs and compressed gradient information. SignSGD (and its majority-vote variant) shows that one-bit sign information suffice for convergence in distributed settings (Bernstein et al., 2018), while quan- tization and error-feedback analyses formalize guarantees for compressed updates (Alistarh et al.) 2017 Karimireddy et al., 2019). More recent methods leverage gradient magnitudes or sign statis- tics for efficient adaptation: Gradient-Mask Tuning masks low-importance parameters during LLM fine-tuning (Li et al., 2025), and sign-based federated variants weight client updates to address het- erogeneity (Park et al. 2024).
3 PRELIMINARIES
Let Aa and On denote the parameters of the same architecture, pre-trained on different datasets (or with different hyperparameters), The fine-tuned A on a downstream task is denoted by 0
Model rebasin. The goal of rebasin (Ainsworth al. 2023 is to align two independently trained models by mapping the parameters of one into the loss basin of the other, so that they become functionally compatible. This setting concerns only the pre- pre-trainet traitre weights, without involving fine- tuning updates.
Task Arithmetic. A complementary perspective to model rebasin is offered by Task Arithmetic (II- harco et al. [023] Yadav et al., 2023), which studies linear operations on task-specific parameter updates. Given pre-trained model 8 and fine-tuned counterpart 0', the difference vector in pa- rameter space 7 := 0 0" is called task vector. Task vectors describe how a base model adapts to the task and can be added, subtracted, or merged to induce new behaviors. This setting usually assumes that all models share the same initialization 60, which ensures comparability across tasks.
Our setting. In contrast, our goal is to apply task-vector transfer when the target and source models do not share the same initialization: we want to transfer 7Λ = 0 A from a source pre-trained model Oa to a different pre-trained model OB. This connects rebasin in that the bases differ, but unlike traditional rebasin approaches, we do not seek to explicitly find a permutation to align parameters. Instead, we ask:
Which components of TA are truly transferable, and which would instead harm pn?
This question motivates our method, which leverages the local gradients of Og to selectively filter тA into a compatible and transferable update, effectively performing direct task vector transportation.
4
METHOD
Here we introduce GradFix framework for transferring task vectors across different pre-trained models by filtering them with gradient information from the target model. As conceptual starting point, first consider oracle setting where the target task vector obtained from full fine-tuning defines the ideal transferable directions (Sec. 4.I). We then approximate this oracle with a single gradient step the target model, using anti-gradient signs estimate the direction of the full fine- tuning trajectory. This yields a gradient-sign mask that selectively filters the source task vector into compatible update (Sec. (.2). Finally, we extend the approach to the limited-data regime, where gradients are estimated from only a handful of labeled samples (Sec. 4.3
4.1 GRADFIX (GRADIENT-SIGN MASKING)
We begin by considering an ideal scenario where the true fine-tuned task vector тB f/ AB of model B and the full target dataset D are available. This vector represents the optimal parameter change to adapt B to D. In this setting, we can construct mask that retains only the components of a candidate update (e.g., TA) that are aligned with TB, ensuring that every retained coordinate contributes to decreasing the loss. In other words, TB (or its sign structure) defines the gold standard for locally beneficial directions. Formally, we define the oracle mask m E (0. 1)d, where d is the total number of model parameters and i E (1,...,d d} indexes a parameter coordinate:
m 1 (sign sign(Tn.))-
3

---

Published as a conference paper at ICLR 2026
Gradient-Sign Mask
Source Task Vector
Target Update
+
x
x
x
+
x
+
+
+
x
+
+
+
+
x
+
+
+
+
X
+
X
m
TA
8-
Figure 1: Illustration of our masking procedure. The gradient mask suppresses harmful directions in the task vector TA while preserving those aligned with the target model.
As shown in Fig. applying this mask to TA produces the oracle-masked update 8*, which preserves only the components consistent with TB:
8 : m T
where @ denotes element-wise multiplication. This vector 8* represents reliable transfer of Ta onto 0g. since it filters out all components of TA that are misaligned with the true adaptation directions of B. In practice, however, TB (and thus 8*) unavailable because requires access the fine- tuned target model 0f which defeats the purpose of transporting the solution from A to B. To approximate this ideal mask, we use the gradient of the zero-shot target model surrogate for TB. since it captures locally beneficial directions:
g voL(OB). L(0) = : ((a.))-D(((fo)) v)],
where l is the training objective (e.g.. cross-entropy) and (x, y) is a labeled example from D. Based on this gradient, we define the gradient-sigh sign mask m, which retains only the components of тA whose sign matches that of the corresponding anti-gradient coordinate:
m. == 1 ((sign(TA.)) sign(-9:)}.
(4)
Intuitively, -g acts as signal for local alignment with the loss geometry of B. Notably, in the idealized setting where B is fine-tuned using full-batch gradient descent for single epoch, the resulting task vector TB is proportional to -9. so the gradient-sign mask coincides with the oracle. This observation justifies using the gradient-sign mask as approximation of the ideal update, even when only a few labeled examples are available. The mask retains only components of тд whose sign matches the anti mati-eradient of L(0x). pruning coordinates that would increase the loss for target model B. In this way, the gradient-sign mask serves as a practical surrogate for the trajectory- informed directions encoded in the unavailable TB.
4.2 TRANSPORTING THE UPDATE
Given the gradient-sign mask m from Eq. 4 we define the updated target parameters by directly applying the masked task vector with a scaling factor a >> 0:
B
Ag 5 8 = (m O TA)-
(5)
We explicitly use the convention g VoC(BB) (ascent direction), so useful transport directions should align with -g. It is important to note that points in a descent direction for model A, whereas the gradient g of the target model points in the ascent direction of its loss. By selecting only coordinates aligned with 9 and then adding 84, each retained component moves along a descent- aligned direction for B. contrast, 8* is oracle-aligned with the target task direction because its mask is constructed from TB-
This construction induces a coordinate wise filtering effect: we do not alter the direction of retained entries, but only suppress coordinates that aign-nnomppatble with the target anti- gradient. As result, the transported update preserves task information from TA while reducing the risk of injecting locally harmful directions into Oz. We formalize this intuition with first-order loss analysis.
4

---

Published as a conference paper at ICLR 2026
Descent guarantee. To understand why this gradient masking provides effective transfer, analyze its effect on the loss of the target model B. Consider the transported update from Eq. (5). By expanding the target loss L around p via first-order Taylor approximation, we obtain:
L(0B 5²)= L(0s) + TT where g VoL(fg).
The sign of the inner product g T 8 determines whether the update increases or decreases the loss first order. By construction, the gradient-sign mask retains only components of 7л that are aligned with g. Concretely, for each coordinate i, we have:
f- ITA if sign(TA.) = (g:)( otherwise,
9: (M,TA.)) = 00
which is always nonpositive. Coordinates with = 0 or TA. 0 are naturally covered by this expression and contribute zero. Therefore, the overall inner product satisfies the following:
g g -0 Σ malge |TA,e S 0.
(8)
Thus, for sufficiently small a, the update 5 is guaranteed to be a descent direction for L. Practically, the mask removes all sign-mismatched components TA. so that every retained entry contributes to reducing the loss. Without masking, could contain harmful directions that increase the loss for B; with masking, the transported update is locally aligned with the descent geometry of the target model.
4.3 LIMITED DATA REGIME
In Sec. 4.1 we have assumed access to the full target dataset D to compute the gradient g at the zero-shot target model fg. In practice, one of the main motivations for task vector transport the few-shot or limited data regime. If we had access to the entire dataset, we could directly fine-tune 03 to obtain 0 making task vector transfer unnecessary in that setting.
When only a small number of samples is available, we estimate the anti-gradient signs using a subset of labeled examples. Let D. C DD denote a small subset of N samples. For each parameter coordinate i, we compute the sign of the anti mati-gradient via majority voting across these samples:
8 sign
(Dn.NN)ED, sign (Vol (fog (In), (1n)) vn)))
Lemma (Concentration of Majority Vote Sign Estimator). Let p. Pr[sign( V Fel(fes (z), a)) (ggn( denote the probability that a single-sampi gradient sign matches the true gradient sign coordinate i. Then, under mild independence assumptions and for ps > 1/2, the majority-vote estimator satisfies:
2 -xpp (-2N(p: (- -1/))².
Pr
(10)
which shows that the estimated sign concentrates around the true anti- anfi-gradient direction the number of samples N grows.
We provide proof of this lemma Appendix A which uses Hoeffding's inequality (Hoeffding 1963]. In practice, even few samples provide robust estimate of the true descent direction. Each gradient acts vote for the correct sign, and majority voting filters out noisy conflicting di- rections. This implies that, with high probability. the masked task vector 5 points in a descent direction, preserving the first-order loss reduction behavior of the full-data update. As shown in Sec.5.3 this approach is robust to small sample sizes, making it particularly attractive when direct fine-tuning of 0B is expensive or prone to overfitting. Compared to mean-based aggregation, ma- jority voting is less sensitive to outlier magnitudes because it depends only sign frequency and provides a more stable transfer (Sec. 5.4).
Algorithm D outlines the gradient-sign masked task vector transport procedure, showing how the source task vector is selectively applied to the target model using only a small subset of labeled data.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Gradient-Sign Masked Transport
Require: Source model 0A, 0ff. target model On, target data subset D., scaling a. Ensure: Transported model Og
1: Compute source task vector: TA 0 A
2: for (In: Vn.) E D3 do
3: g(z)
4: Compute anti-gradien signs & by majority voting
Eq. (
5: Build gradient-sign mask: m, <- 1 (sign((T,,) = 8;}
6: Compute transported update: 8 a (m ① TA)
7: return B t 8B + 5d
> Updated target model
EXPERIMENTAL RESULTS
Implementation details. For the vision settings, we consider CLIP ViT-B/16 and ViT-L/14 Vision Transformers (Radlord et al.] 202I), implemented in Open-CLIP (Cherti et al. pre-trained weights are denoted A and the target pre- trained weights
2023). The source
denoted 8g. For ViT-B/16, A was pre-trained on Datacomp XL (s13b, b90k) and 8g on LAION-28 -B (s 34b, b8 8k). For VIT- L/14, A was pre-trained Datacomp XL (s13b, b90k) and On on LAION-2B (s32b, b82k). For the language settings, we use TS-base variants (Raffel et al. 2020 As AA we use T5v1. .1 pre-trained on C4 (Raffel al. 2020) without supervised training. As fg. we use FLAN-IS T5 (Chung et al. 2024), pre-trained and instruction-tunet on several datasets, including GSM8K (Cobbe et al., 2021 AQUA-RAT (Ling et 2017). and LAMBADA (Paperno al., 2016). Task vectors were obtained following the fine-tuning protocol of Ilharco et al. (2023): 2000 iterations, batch size 128, learning rate 1 10-5, cosine annealing with 200 warm-up steps, Adam W optimizer (Loshchilov & Hutter 2019). weight decay 0.1. The text encoder backbone was kept frozen following Cherti et al. (2023). For fairness, compared transport methods use the same source/target checkpoints and the same supervision budget D., and results aggregated across random seeds. The is chosen through a validation set following standard model merging pipelines, more details in Appendix
Baselines. We evaluate our method against several baselines. As lower bound, we consider the zero-shot target model (0p zero-shor), i.e., the base model without fine-tuning. As an upper bound, we report 0B + 5*, obtained by adding the source task vector тA masked with the signs of the true task vector TB the target model. We also include the performance of the fully fine-tuner target model (0g fine-tune) and the naive Task Arithmetic transport (0g + TA). In addition, we compare against TransFusion (Rinaldi al., (0255), which transports task vectors across transformer- models via permutation alignment. Finally, we report the performance of a target model fine-tuned with the same number of randomly sampled examples per class [D]] used by approach.
Supervision budget D., In all experiments, the subset D, is drawn from the full downstream fine- tuning dataset 'D and constitutes only a fraction its size. Throughout the tables, [De] indicates the number of examples per class used estimate anti-gradient signs for the target model 0g. The corresponding proportions of D used to form D, are provided in Appendix D Additionally, include ablation study in Appendix C.2 analyzing the impact of different subset selection heuristics on anti- mati-gradient sign estimation.
5.1 TRANSPORT EXPERIMENTS
Transport in the vision setting. summarins the results task vector transport across CLIP VIT-B/16 and CLIP ViT-L/14 architectures, averaged over multiple random seeds that determine the composition of the sampled D. (standard deviations are reported in Appendix I). Our GradFix, denoted by 0B + 84, yields a consistent improvement over naive task vector addition (0 TA) even when using a single sample per class to approximate true anti-gr gradient signs. Notably, the naive addition performs nearly at the level of zero-shot initialization and fails to transfer meaningful task knowledge. This confirms that GradFix effectively suppresses misaligned components of T preventing negative transfer due pre-training mismatch.

---

Published as a conference paper at ICLR 2026
Table 1: Cross-dataset performance comparison for ViT-B/16 and ViT-L/14 models.
EUROSAT
SVHN
GTSRB
RESISC45
DTD
Model
[D[]
B/16
I/14
B/16
T/14
B/16
I/14
B/16
T/14
B/16
I//14
On zero-shot
49.41
62.80
50.58
37.28
48.29
56.12
67.98
73.12
55.96
63.35
AB fine-tume
98.70
98.95
97.45
97.80
98.65
99.16
95.66
97.06
83.19
83.56
Op + +TA
49.58
62.77
50.84
39.09
49.31
56.03
67.87
73.49
56.27
63.56
B +
95.06
96.75
92.04
92.60
82.92
88.65
87.06
90.30
71.44
72.66
TransFusion
50.12
63.21
53.26
37.38
50.24
56.78
67.99
73.36
56.70
64.10
B
56.61
64.65
61.32
62.51
56.08
63.97
69.25
74.54
56.21
63.76
Op 8A
61.91
69.67
71.07
70.15
60.88
66.82
70.05
76.45
58.32
65.50
59.49
70.76
62.01
45.23
61.70
69.91
71.20
76.62
57.00
64.97
Bg + 8
65.07
74.10
70.19
54.31
64.33
71.55
71.42
76.97
58.51
66.10
07
61.99
69.75
67.03
67.11
63.08
73.25
73.01
75.41
59.65
66.72
0n + 8
66.05
75.59
73.59
74.41
66.61
73.14
71.57
76.82
60.02
66.95
Table 2: Cross-dataset performance of T5 models on different NLP tasks.
Model
SNLI
MNLI
RTE
QNLI
SCITAIL
AVG
Bg zero-shot
34.24
35.21
47.20
50.54
50.38
43.51
BB fine-tuee
88.20
86.30
84.40
92.79
95.32
89.40
On
31.61
30.75
47.36
50.52
50.46
42.12
On
58.69
69.97
72.93
65.32
62.38
65.86
0ost
50
35.09
26.05
47.29
51.45
51.78
42.33
BB + 8
50
68.06
49.68
54.25
60.50
59.89
58.48
To further evaluate our approach, compare it against few-shot fine-tuning of 0p. denoted as gopt, using the same limited target samples and number of training steps (see Appendix F for de- tailed computational cost analysis). This comparison isolates the effect of update construction: both methods observe the same supervision budget, but 0 relies on iterative parameter optimization, whereas GradFix applies single masked transport step. GradFix achieves better performance both V and ViT-L/14, while exhibiting smaller variance across seeds with respect to few-shot fine-tuning (Tabs. 5 and 6). Moreover, the D, size increases, our method continues provide stable gains, whereas 0hrh shows larger fluctuations due subset composition. These trends suggest that GradFix is not only stronger on average. but also more reliable under realistic data-san mata-sampling variability. practical low-shot settings, this matters because adaptation quality should remain sta- ble even when the selected subset is not carefully curated. Overall, gradient-sign masking yields more data-efficient and predictable adaptation mechanism, where robustness achieved with single howward-backwann pass to obtain the mask m.
Transport in the language setting. Table reports results on task vector transport across T5 models evaluated on closed-vocabulary text classification benchmarks. While direct addition of Ta to 03 fails to transfer knowledge effectively, our method substantially closes the gap toward full fine-tuning, confirming its ability to identify and retain task-relevant directions. Notably, the relative improvement over naive transfer is even larger than in the vision setting, suggesting that sign-based filtering is particularly beneficial when source and target pre-training objectives differ more strongly. This is challenging regime, since transferring updates between T5v1. and FLAN- T5 combines both pre-traue traiiriiing and instruction-tuning mismatch. Despite this shift, GradFix remains effective with the same lightweight procedure used in vision. These results confirm that the method architecture-agnosicc in practice and supports reliable task vector transport in the language domain well.

---

Published as a conference paper at ICLR 2026
Table 3: Merging experiments ViT-B/16 in multi-task and multi-source settings.
Pipeline
EUROSAT
SVHN
GTSRB
RESISC45
DTD
AVG
On zero-shot
49.41
50.58
48.29
67.98
55.96
54.44
Multi-task ((TA,))=11-)) 00
BB + TTJ 0B + +63
49.58
50.84
49.31
67.87
56.27
54.77
65.07
70.19
64.33
71.42
58.51
65.90
Task Arithmetic
Baseline
49.31
50.99
48.73
68.05
56.54
54.73
Mask-then-Merge
55.90
71.56
59.65
71.40
57.66
63.23
Merge-then-Mask
65.37
72.10
59.55
71.16
57.07
65.05
Baseline
49.15
50.75
48.95
67.97
56.54
54.67
Meege
50.41
64.28
54.05
69.51
37.13
59.08
Merge-then-Maun
65.62
72.42
62.73
71.57
37.77
66.02
Multi-source
K ((TAa k=1 + +08)
Task Arithmetic
Baseline
36.94
35.09
30.77
50.54
44.73
39.61
Merge-then-Mask
12.52
15.94
4.20
3.97
2.93
7.91
Mask-then-
65.96
72.97
65.80
71.30
61.01
67.41
TIES-Merging
Baseline
12.69
10.39
2.85
5.81
16.33
9.61
Merge-then-Mask
14.46
15.94
3.09
3.35
2.66
7.90
Mask-then-Ment Merge
65.17
72.58
65.36
71.40
61.12
67.13
5.2 TASK VECTOR TRANSPORT FOR MODEL. MERGING
We evaluate GradFix in combination with model-merging methods, specifically Task Arithmetic [Iharcco al., 2023) and TIES-Merging (Yadav al., 2023). We consider settings: multi-task (one source model, multiple tasks) and multi-source (multiple source models, one task).
Multi-fask experiments. For this setting. all task vectors are extracted from the same source pre- trained model A (same pre-training, different downstream tasks) and transported to a fixed target pre-trained model 0g. For task vectors from distinct tasks, we compare two pipelines. Mask-then- Merge: transport each task vector with GradFix and then merge. Merge-then-Maskk first merge task vectors into Tmerged then transport the merged vector using a consensus mask computed by estimating per-parameter anti-gradiem signs on On for each task and selecting the most frequent sign each coordinate. We report results for this setting in Tab. 3 Here, 0n + TA.j and 0B + 8,² denote single-task references evaluated task respectively using naive addition and GradFix. Direct Task Arithmetic and TIES merging without GradFix perform near zero-shot, indicating strong cross- model misalignment, while Merge-then-Mask gives the best results. Masking each vector first can discard coordinates that would complement each other after merging; merging first preserves them and lets GradFix align coherent update with the target loss geometry.
Multi-source experiments. In the multi-source setting, we use K source models (0A. ) that are pre-trained on different data distributions and then fine-tuned on the same downstream task. We transport resulting source task vectors to a single fixed target model 0B, whose pre-training remains unchanged during transport. Here, Merge-then-Mask is not expected to help: because all vectors correspond to the same task, the gradient-sign mask on Op shared across sources, so consensus masking after merging is effectively equivalent masking each source separately. We therefore use Mass--then-Merge: transport each source vector first, then merge the transported vectors. Table 3 shows that this recovers performance from the collapse of direct merging and also improves over single-source transport, suggesting that combining multiple transported updates provides more robust descent direction.
8

---

Published as a conference paper at ICLR 2026
Table Performance 8g with oracle or estimated gradient signs under different mask strategies: sign agreement retains matching signs, sign forcing aligns all signs, magnitude-scaled uses the product of task and gradient magnitudes, and random assigns signs uniformly. Results averaged over seeds with [D 1 on CLIP ViT-B/16. The table includes 0B zero-shot and fg fine-tune as lower and upper reference bounds.
Model
Mask Strategy
EUROSAT
RESISC45
GTSRB
SVHN
DTD
AVG
Bg zero-shot
49.41
67.98
48.29
50.58
55.96
54.45
B fine-tiee
98.70
95.66
98.65
97.45
83.19
94.73
sign agreement
95.06
87.06
82.92
92.04
71.44
85.71
Dp + 8*
sign forcing
97.95
93.51
95.91
96.60
80.59
92.92
magnitude-scalee
49.92
67.94
51.63
50.78
56.01
55.25
sign agreement (ours)
61. .9
70.05
60.89
71.07
58.32
64.45
0g + 84
sign forcing
61.32
70.10
60.91
70.52
58.05
64.18
magnitude-scaled
49.51
68.06
49.20
50.71
56.03
54.70
0g + 8-
random
49.49
67.97
48.41
50.54
56.06
54.50
5.3 MASKING STRATEGIES
To analyze the effect of different mask construction strategies on the transport of the task vector TA+ we compare our primary masking method (sign agreement), with three alternatives: sign forcing, magnitude-scaled,, and random masks. Table reports results using sample per class on ViT- B/16, averaged across multiple random seeds.
For both 8 (Eq. 5 and 8* (Eq. (2)) the mask m determines which directions of тл are retained. Here, i indexes parameter coordinates. In sign agreement, retains only the coordinates whose signs match those the reference as in Eq. (4). In sign forcing, all signs TA are aligned with the signs of the anti-grade anti-gradient estimator 8, obtaining:
m = sign(TA.:) 84,
8/' = a TA.i) = a [TA, 8g.
(11)
This flips entries that disagree with the reference and applics a forced-sign update. When the oracle тв is used as the reference, sign forcing generally outperforms sign agreement, fully leveraging the true task direction maximizes transfer. In contrast, using few-shot anti-gradient-basee estimates from fgg sign agreement performs slightly better than sign forcing. This is consistent with the fact that the anti-gradd gradient estimate is noisy: forcing all directions can propagate errors, while keeping only agreeing entries provides more reliable mask. We also evaluated a magnitude-scaled strategy to determine whether leveraging magnitude information offers additional benefits. This strategy computes the mask based on the magnitude of both the source task vector and the target reference vector p (where TB for the oracle and p = -g for the estimate):
m ms max(0, Lanh(TA,F -P:)),
6.me = a (m TA,:).
(12)
This approach assigns a mask value near 1.0 only when components match in sign and possess significant magnitude, while suppressing mismatches. However, as shown in Tab, 4 this magnitude- aware strategy consistently underperforms sign agreement. Crucially, this failure persists even in the oracle setting (ō*), which suggests that while directions trainings, parameter magnitudes are highly specific to the local loss geometry of each basin. Conse- quently, enforcing magnitude consistency acts as an overly aggressive filter, discarding valid updates where the models agree on direction but differ on scale. In the limited-data regime (8-), this issue further exacerbated by the noise inherent in estimating gradient magnitudes from small datasets D., confirming that the sign structure serves the most robust proxy for alignment.
transferable across different pre-
Finally, we evaluate random mask, where signs are sampled uniformly from U4-1,+ +13. Like magnitude-scaled masking, this approach performs close to the zero-shot baseline. This highlights that although sign-based masking outperforms magnitude-ban based filtering, the gain is not due to masking alone: random sign choices offer useful signal. Thus, effective transfer relies strictly the precise geometric alignment provided by the target anti- tradieras rather than just the sparsity of the update.

---

Published as a conference paper at ICLR 2026
EUROSAT
SVHN
GTSRB
RESCISC45
DTD
70
60
70
58
Accuracy [96)
55
60
56
60
45
50
52
50
53
35
40
48
48
25
30
44
30
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5 Alpha
0.7
0.9
0.1
0.3
0.5 Alpha
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Alpha
Alpha
Alpha
Mean
Maj. Voting
Figure 2: Accuracy across different values for mean and majority-vote sign estimation.
5.4 SENSITIVITY TO THE SCALING COEFFICIENT
We investigate the sensitivity of masked transport to the scaling factor a E (0, 1], providing proxy for how compatible and robust the transported task vector is with the target backbone. In addition to our proposed majority voting strategy, we consider baseline where the estimated sign is taken as the sign of the averaged anti-gradient (mean). Results are reported in Fig. 2
Across datasets, majority voting consistently outperforms the mean strategy for values of a, providing more reliable approximation of the true gradient sign. Notably, majority voting yields smooth performance curves without sudden drops, and maintains higher accuracy over a broader range of a. This difference arises from the aggregation mechanism; averaging gradients before thresholding is highly sensitive to variance and outliers, so even a small subset of misaligned samples flip the estimated sign and destabilize updates only on the relative frequency of signs, which concentrates rapidly around the true direction with increasing samples (as shown in Appendix(A) (A). As result, it inherently more stable and preserves transfer accuracy even in few-shot or noisy regimes.
grows. Majority voting, instead, depends
From a practical perspective, this robustness means that masked transport with majority voting does not require fine-grained tuning of to achieve good performance. The method remains effective across a wide range of scaling choices, which particularly valuable when adapting new datasets where validation data or tuning budgets are limited.
5.5 SUBSET DATA SELECTION
Beyond random sampling, we analyzed whether structured strategies for constructing D, improve anti-gradient-sigh estimation. We compared random selection with feature-based alternatives from CLIP embeddings: herding, k-medoids, and a coresct-style medoid-proximin greedy selection.
Across datasets, structured selectors yield small gains at very low budgets, while random sampling remains competitive and approaches their performance as the number of examples per class in- overhead and does
creases. Because random sampling adds no not require full target-data access, it is often preferable in constrained settings. Full details and curves are provided in Appendix
CONCLUSIONS & FUTURE WORK
In this work, we show that the sign structure of anti-gradient provides powerful and reliable proxy for descent directions the target loss landscape. The strong performance of our oracle-based analysis, which uses signs from the true task vector, validates this core insight and confirms that effective transfer is possible when the transported task vector is aligned with the target model's local geometry. GradFix approximates this oracle with only a handful of labeled samples to estimate anti- gradient signs, yielding large gains over naive transfer. While our approach is effective and robust in low-data regimes, the remaining gap the oracle highlights clear directions for future work. In particular, future research can explore better anti anti-gradee tridieradeett sign estimators, stronger sulset-selection strategies, and extensions to other architectures and transfer settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the PNRR-M4C2 (PE00000013) project "FAIR Future Artificial In- telligence Research". Angelo Porrello was partially supported by the Department of Engineering "Enzo Ferrari" through the FAR2025DIP program (CUP E93C250003700055, We acknowledge the CINECA award under the ISCRA initiative for access to high-performance computing resources and technical support.
REPRODUCIBILIT STATEMENT
We provide the codebase needed to reproduce our results. All hyperparameters used in our exper- iments are detailed the appendix. We also provide complete proofs for all claims made in the paper, that the results and statements can be independently verified.
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basn::: Merging models modulo permutation symmetries. In Proceedings of the International Conference Learning Representations, 2023.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: communication-effiice sgd via gradient quantization and encoding. Proceedings of the Ad- vances Neural Information Processing Systems, 2017.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In Proceedings of the International Conference on Learning Representations, 2018
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Bench- mark and State of the Art. Proceedings of the IEEE, 2017.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive ianguage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling ciatuctiionfinetunae lan- guage models. Journal of Machine Learning Research, 2024.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2014.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2 10. 14 168, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the International Conference on Learning Representations, 2020.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Redueing task interference in model merging. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.

---

Published as a conference paper at ICLR 2026
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Proceedings of the Advances in Neural Information Processing Systems, 2018.
Stanford NLP Group et al. The stanford natural language inference (snli) corpus, 2022.
Nick Harvey. Near-optimal herding. In Proceedings of the International Conference on Learning Representations, 2014.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, R. De- sai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, D. Song. J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. Proceedings of the IEEE International Conference on Computer Vision, 2020.
Wassily Hoeffding. Probability incqualities for sums of bounded random variables. Journal of the American Statistical Association, 1963.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of the International Con- ference on Learning Representations, 2023.
Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer Fusion with Optimal Transport. In Proceedings of the Interna- fional Conference on Learning Representations, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.0836/ 2020.
Sai Pranceth Karimireddy, Quentin Rebjock, Sebastian U. Stich, and Martin Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In Proceedings of the International Conference Learning Representations, 2019.
Leonard Kaufman and Peter J. Rousseeuw. Clustering by means of medoids. North-Holland, 1987.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, and Peng Cheng, Enhancing large language model performance with gradient- based parameter selection. Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- eration: Learning to solve and explain algebraic word problems. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceed- ings of the Advances in Neural Information Processing Systems, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019,
Daniel Marezak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task- specific subspaces. In Proceedings of the International Conference on Learning Representations, 2025.
Anshul Nasery, Jonathan Hayase, Pang Wei Koh, and Sewoong Oh. Pleas-merging models with permutations and least squares. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
12

---

Published as a conference paper at ICLR 2026
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Read- ing digits in natural images with unsupervised feature learning. In Proceedings of the Neural Information Processing Systems Workshops, 2011.
OpenAl, Josh Achiam, and Steven Adler et al. Gpt-4 technical report. arXiv preprini arXiv: 2300.08774, 2023.
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Proceedings of the Advances in Neural Infor- mation Processing Systems, 2023.
Aniello Panariello, Emanuele Frascaroli, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Modular embedding recomposition for incremental learning. Proceedings of the British Machine Vision Conference, 2025a.
Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Bartlomiej Twardowski, Andrew D. Bagdanov, Simone Calderara, and Joost van de Weijer. Accurate and efficient low-rank model merging in core space. Proceedings of the Advances in Neural Information Processing Systems, 2025b.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quie Egoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the Annual Meeting of the Association for Compulational Linguistics, 2016.
Chanho Park, H Vincent Poor, and Namyoon Lee. Signsgd with federated voting, arXiv preprint arXiv:2403. 16372, 2024.
Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega, Monica Millunzi, Simone Calderara, and Rita Cucchiara. A second-order perspective on model compositionality and incremental learning. In Proceedings of the International Conference Learning Representations, 2025.
Angelo Porrello, Pietro Buzzega, Felix Dangel, Thomas Sommariva, Riccardo Salami, Lorenzo Bonicelli, and Simone Calderara. Dataless weight disentanglement in task arithmetic via fronecker-factored approximate curvature. In Proceedings of the International Conference on Learning Representations, 2026.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Gob, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the International Conference on Learning Representations, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert. icarl: In- cremental classifier and representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, Elisa Ficarra, Emanuele Rodolå, Simone Calderara, and Angelo Porrello. Update your transformer to the latest release: Re-basin of task vectors. Proceedings of the International Conference on Learning Representations, 2025.
Ozan Sener and Vladlen Koltun. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 17Xivv/708.0000 0048, 2018
Johannes Stallkamp. Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recog- nition benchmark: multi-class classification competition. In Proceedings of the International Joint Conference on Neural Networks, 2011.
13

---

Published as a conference paper at ICLR 2026
George Stoica, Daniel Bolya, Jakob Brandt Bjomer, Pratik Ramesh, Taylor Hearn and Judy Hoff- man. Zipit! merging models from different tasks without training. In Proceedings of the Interna- tional Conference on Learning Representations, 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations, 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume I (Long Papers), 2018.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without in- creasing inference time. In Proceedings of the International Conference on Learning Represen- tations, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES mergingg Re- solving interference when merging models. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
14

---

Published as a conference paper at ICLR 2026
USE OF LARGE LANGUAGE MODELS (LLMS)
made limited use of a large language model only for light editorial refinement, such as adjusting phrasing, improving grammar and enhancing overall clarity. The model not involved in gener- ating research ideas, designing experiments, interpreting results, or authoring scientific content.
We
APPENDIX
A ANTI-GRADIENT SIGN ESTIMATOR GUARANTEE
We formalize why majority-vok estimation of anti-gras unti-gradient signs provides reliable proxy for true descent directions the limited-date regime.
Lemma Majority-Vote Gradient Sign). Let := Vo, (BBB) denote the true target-loss gradient (n) be i.i.d. per-sample gradients:
at parameter coordinate and let gi
91 (n) 9: 8; (n)
Blc!-] 0,
(13)
where (m) is symmetric noise around zero. Define
Ps P=:= Pr (i sign(g:)
(14)
where p; represents the probability that the sign of a single per-sample gradient g;" matches the true gradient 9;-
Then, for all i with 9. * p: > 1/2. In particular, the majority-vote estimator
sign Σ sign(g) (m))
(15)
recovers the correct sign with probability at least
Pr[s; sign(-9:)] exp (-2N 1/2)3).
(16)
Proof. We divide the proof into two parts.
Step 1: Bias of single-sample signs.
Define the indicator random variable X,, (siig(()) sign(g:)} E {0, 1). The success probability of single sample is Pr[X,, 1] "r[ssigna (")) sign(g:)).
Without loss of generality, assume gi 0. The event of a successful sign match gf > which can be rewritten as + = > 0, or { > -9.. Since the noise e is symmetric around zero, we know that Pr(e (n) > 0] = Pr(e (n) < 0] 1/2. Because gi 0, the interval (-9,, 0) is non-empty.
The probability of the noise falling into this interval, Pr|-g; (m) 0]. is positive; therefore, the total probability of success is:
p Pr(s(") (n) -9i]
(m) Pr(ee(") (n) > 0] Pr[-gi < <,; < <]
= -1/2+Pris PP(-PP(-g gi < (n) < 0).
Since Pr|-9: < <( < < > 0, it follows that > 1/2.
Step 2: Concentration of majority vote.
The majority-vote estimator 8; succeeds if
N Xn > N/2.
Σ
(17)
n=1
15

---

Published as a conference paper at ICLR 2026
Now, we bound the probability of failure, which is the event that the sum of correct sign estimates less than or equal to N/2. We can express this event as a deviation from the expected value of the
sum. The expected value of the sum is E E Xn E E[x,] Npc. Thus, the deviation is:
N
N
N
X
X
Xn
Np.
(18)
==1
==
==1
If rewrite the event of failure, En Xn S N/2, in terms of this deviation obtain:
X, Np: S N/2 Np: - N(p: - 1/2)
(19)
According to Hoeffding's 's inequality (Hoeffding 1963) for sum of i.id. random variables X, E [0, 1], we have:
N Σ n=l
2 ( 1/2)) Σm (1 -1-0)² 0)2
Pr
X,
Np; -N(p: 1/2) s exp
(20)
The denominator simplifies to Σn 1 = N. Substituting this back into the inequality gives:
N
222 (p: - 1/2)3)
Pr
Σ
X S N/2
S exp
cxp (-2N(p,- - 11/2
(21)
N
The probability of correct recovery is the complement of this failure probability:
Pr(s; (99:)] Pr
Σ X,
N/2
Pr Σ X S N/2
(22)
Therefore, obtain the final bound:
Pr(s; sign(-9:)) 2 exp (-2N(p: 1/2)?)
(23)
This result formalizes the intuition that, under mild assumptions on per-sample gradient noise, the majority-vote sign over small batch of samples provides reliable approximation to the true de- scent direction. The probability samples N and the signal-to sgmal-to-noise ratio p, 1/2. In practice, this suggests that even few labeled samples can suffice to construct a gradient-sign mask that preserves most descent-aligned compo- nents of the source task vector.
correct recovery grows exponentially with both the number of
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
B.1 VISION RESULTS
Table 5: Cross-dataset performance of ViT-B/16 (A: datacomp x1 s13b b90k, B: laion2b
8 a34b b88k) models averaged across random seeds.
Model
D
EUROSAT
RESISC45
GTSRB
SVHN
DTD
AVG
BB zero-shot
49.41
67.98
48.29
50.58
55.96
54.45=7.30
0n fine-tune
98.70
95.66
98.65
97.45
83.19
94.73+5.94
0B
49.58
67.87
49.31
50.84
56.27
54.78=7.08
pn + 8" (oracle)
95.06
87.06
82.92
92.04
71.44
85.71+8.30
56.61+6.06
69.25=0.96
56.08+2.87
61.3244.09
56.21+0.85
59.89a6.05
8g + 8
61. 63.94+0.43
70.05=0.56
60.88+2.85
71.07
58.3240.30
64.45+5.47
0
2
59.49=1.43
71.20=1.13
61.70+0.76
62.01
57. 37.0020.54
62.29=5.31
OB 8
2
65.07=1.10
71.42=0.90
64.33+1.05
70.19.4.55
58.51
65, $5.9665.00
5
61. 61.99+7.29
73.01=0.48
63.08
67.03+3.93
59.65+0.80
64.95=5.81
Og 8
5
66.05=1.21
71.57=0.88
66.61#0.42
73.59+0.82
60.02+0.20
67.57=4.96
6ept
10
59.98+3.77
72.27=1.65
64.54+1.12
67.85+1.02
60.9620.3
65. 12+4.77
OB 8
10
66.59#1.83 83
72.05=0.59
6..02+1.59
74.82+1.22
60. 18
67. 67.93+5.38
0ppl
60.59+3.94
74.22+0.66
65.5140.80
67.19+0.65
62.59+0.08
66.02+5.0
On +
20
67. 01.05400.1
72.29=0.14
66.42+0.47
74. 74.11#0.59
60.92+0.00
68. 15
58.80+2.45
75.88=0.83
60.91
67.58+2.91
64. 14.1130.11
66 66.2625500 5.77
On + 8
50
66.9420.46
72.26:0.22
66.03+0.14
74.07+1.52
61.35+0.03
68. 15:4.5
Table Cross-dataset performance of ViT-L/14 (A:: datacomp xl s13b b90k, B: laion2b 33b b82k) models averaged across random seeds.
Model
EUROSAT
RESISC45
GTSRB
SVHN
DTD
AVG
9g zem-shot
82.50
73.12
56.12
37.28
63:35
58.53=12.35
fine-tume
98.95
97.06
99.16
97.80
81..6
95.31e6.13
+ TA
62.77
73.49
56.03
39.09
63.56
58.99211.8
(oracle)
96.75
90.30
88.65
92.00
72.66
19a8.22
64.65e5. 90
74.54=0.57
63.9784.50
62.51s4.58
63. 76a0.22
65.80155.61
35.5040.74
gried
70.76+1.77
78.62+00.2
09.91=1
45.23+1.85
64.97+0..21
65.50=11
Bu
2
74.10.2..0
76. 76.97.00668
71.35a2.75
54.31x2.54
66 1010.59
68 68.614844
69.75al.c
75.1142.94
73.25g0062
67.1112.39
66. 66.7220.13
70.4523.90
75.5942
76.8240.48
73.1421.23
74.4182.22
86.95a00.69
73.31
good
10
70.36+3 82
78.07+1.20
74.11.0.31
60.43+3..8
00.3640..27
70.46+6.45
73.74+1.
77. 77.50
74.94e0.75
15.88
67.41+0.44
73.77+3
good
78.7412.90
80.
74.6540.28
65.99x2.15
10.0...
74.31a5
20
74.8740.71
78. 16.1660333
74.91200.55
75.7940.90
67. 55.55..0.24
74.15g3.83
77.07e1.66
82. 16.6.6.1
73.3820.45
67.833
73.8320.08
75.25
74.754089
78.2740.18
74.61=091
76.79+1.25
67.7740.0
74.27+3.86
17

---

Published as a conference paper at ICLR 2026
C ABLATIONS
C.I SIGN AGREEMENT ANALYSIS
EUROSAT
GTSRO
SVHN
RESISC4S
DTD
KXX 483
027
SA1 464
M3 652
103
140
K30 18
6.1
133
SCT 669
K24 33
444
403
2 1044 RL3 tht 153
14.
0.1
06.8
01.
1: 102 5S8
42 6-1
6.3
853 115
11.3
620
Transforma
6 104 554 ninna
12
130
290 12.
6.1
153
854 423
43.
653
650
124424002 02-02
643 463-665500 M08
130
6.0 445 165-488
451 402
45
550 429
Deoeoo 300
174
1O 48
123 0.
04%
05%
665
15.6
100
12: 0.
667
032
500
1L 0.0 022 118 11A
155
550 662
104
550 823 (0122
035
023
663
6.5 425 424
R55 125
15
5A2 662
30.5 0-1
130
665
142
043
15SD5 6.2 423 427 413
MAX 454
Images per
Images
Images per
Images
Images
Figure 3: Sign agreement per block for ViT-B/16. The heatmaps show the percentage of sign agree- ments between the m and m constructed by computing the signs of the anti-gradients at different [Di] budgets.
To understand the nature of the misalignment between source and target models, show in Fig. a heatmap of anti- gratieraa sign agreement. Specifically, we computed the element-wise agreement between the signs of the target task vector (TB) and the estimated anti-gradients at different data budgets.
This heatmap reveals that agreement is not uniform across different layers, and there is simple, global sign correlation. This observation provides a direct explanation for the failure of naive task vector transfer. Without mechanism correct for this misalignment, simply adding the task vector introduces harmful directions that degrade performance. Our method, in contrast, actively addresses this by using the target model's gradients construct a mask, ensuring that only the useful compo- nents of the task vector are transferred and aligned with the target loss landscape. Tables 5 and 6 demonstrate that 8 consistently matches or outperforms 0 across both ViT-B/16 and ViT-L/14 models, even at larger [De] budgets. Moreover, 8 exhibits substantially lower standard deviation across seeds, confirming its robustness to the random choice of supervision data and its efficiency compared to direct fine-tuning under identical data constraints.
C.2 SUBSET DATA SELECTION
In the main experiments, randomly selected the subset D3 from D. As an ablation study, now evaluate different heuristics-randomm herding, k-medoids, and coreset-for constructing D, estimate anti-gra aati-gradient signs, aiming to understand how subset selection impacts the accuracy and
efficiency of anti-gres gradient estimation. Each strategy is evaluated at b E {1, 5, 10, 20) examples per class. For herding (Rebuffi et al.) 2016; Harvey (0144), k-medoids (Kaufman 1987], and coreset (Sener & Koltun 2018). images are embedded using the frozen CLIP image encoder of the source model AA. Let f denote this frozen image encoder, normalized features are computed as xxx)
Rousseeuw]
and let D := [xx y) D c).
Random. Sample uniformly from without replacement.
Herding. Greedily select representatives D C 'Do of size b to match the class mean feature by minimizing the discrepancy of the running average:
D. arg min Dbj==b
1
Σ
P.
(x).
(24)
D
De
mED:
k-Medoids. Select D. D of size that minimizes the in-class assignment cost under distance d:
Dr
arg arggin min
Σ min (zz(x),(()))
(25)
[D aEDe
sDD;
18

---

Published as a conference paper at ICLR 2026
EUROSAT
SVHN
GTSRB
RESISC45
DTD
68
70
75
67
61
66
72
65
60
66
71
64
63
59
70
61
58
20
2
10
20
10
20
Imagss per Class
Images per Class
Images per Class
Images per Class
Images per Class
Random
Coreset
Herding
KMedoidd
Figure 4: Accuracy across different numbers of images per class for various Dj construction heuris- tics.
Coreset (medoid-proximigg greedy). Adopt prediidprrxximity greedy selection within 2):
i Seed:
$1 = arg min
Σ d (z)))),((k)) REDe
JEDe
(26)
iii) Greedy:
8t arg min jED
min
d (s)))
t= 2, .,
.ED 2,11
where - = (s1,...,8:-1) denotes the set of already selected samples. This strategy em- phasizes prototypical samples reduce variance in small budgets and is closely related to coreset selection for active learning (Sener & Koltun 2018).
For each dataset and budget b, D. = U. DE, anti-gradiemt signs estimated discussed in Sec. 4.3 using majority-vote aggregation, and masked transport is applied to 0g. We report in Fig. the ac- curacy for each strategy as function of images per class, with standard deviation across random seeds for the random baseline. Across datasets and budgets, structured selectors (coreset, herding, k-medoids) often provide small but consistent gains over random selection the few-shot regime. Yet, random selection remains a strong baseline, with performance approaching that of structured methods as b increases, while incurring no memory or computation overhead from embedding or distance pre-computation. Importantly, this shows that approach remains effective even when the subset D, is chosen at random, validating its applicability in strict few-shot settings where so- phisticated selection strategies are infeasible. Trends stable across seeds, with variance shrinking as grows. Finally, structured methods require access to the full target dataset D. which is unrealistic in privacy-constrained large-scale settings.
19

---

Published as a conference paper at ICLR 2026
C.3 RANDOM TASK VECTOR TRANSPORT
Table 7: Ablation study comparing ViT-B/16 models with GradFix (8 against randomized task vector baseline (grandmm). The random vector preserves the and standard deviation of the source task vector but lacks its structural information.
Model
EUROSAT
SVHN
GTSRB
RESISC45
DTD
AVG
OB zero-shot
49.41
50.58
48.29
67.98
55.96
54.44
f fine-tune
98.70
97.45
98.65
95.66
83.19
94.73
0B + TA
49.58
50.84
49.31
67.87
56.27
54.77
0B + 8+
95.06
92.04
82.92
87.06
71.44
85.70
Trans Frimssusiin Fusion
50.12
53.26
50.24
67.99
56.70
55.66
56.61
61.32
56.08
69.25
56.21
59.89
OB grandom
49.86
51.43
48.56
68.06
56.06
54.79
BB J
61.94
71.07
60.88
70.05
58.32
64.45
gont
59.49
62.01
61.70
71.20
57.00
62.28
0B grandom
50.07
51.45
48.64
68.14
56.05
54.87
03 + 84
65.07
70.19
64.33
71.42
58.51
65.90
gopt
61.99
67.03
63.08
73.01
59.65
64.95
B
grandom
50.24
51.79
48.65
68.11
57.74
55.31
8 + 84
66.05
73.59
66.61
71.57
60.02
67.57
To assess whether the effectiveness of GradFix arises from the transfer of meaningful structural knowledge from the source task vector rather than solely from the gradient masking mechanism, we performed an ablation using a random task vector. Specifically, we replaced the original source vector тл with randomly generated vector Trandom drawn from normal distribution matched to the element-wise statistics of TA to ensure a fair comparison:
Trandom
NN o
(27)
where Mra and Tr are the mean and standard deviation of the parameters TA. then applied our proposed gradient-sign masking method to this random vector using the same target gradient mask m. The resulting update defined as:
grami 03 + a(m C Trandom)
(28)
We evaluated this baseline using supervision budgets of [D:] E {1,2,5} samples per class.
The results are reported in Tab. Notably, the performance of the random vector often comparable to, or only marginally better than, the zero-shot baseline (0g). These findings confirm that the gradient-sign sign mask alone insufficient induce strong performance; must act on vector that contains valid task-specific information, as provided by TA-
C.4 GENERALIZATION AND ROBUSTNESS
To ensure that merging the transported task vector does not compromise the robust capabilities of the new backbone, we evaluated performance on a held-out support set, ImageNet-R (Hendrycks) al. 2020). This allows us to assess whether the target model's superior zero-shot capabilities preserved after transfer.
As shown in Tab. 8 transferring the task vector via GradFix (0p + 8-4) substantially improves accuracy on the downstream tasks compared to the zero-shot baseline, while fully preserving the superior zero-shot generalization of the target model. This confirms that method effectively adapts the model to the target task without sacrificing the general robustness the updated pre- training.
20

---

Published as a conference paper at ICLR 2026
Table Generalization analysis. We report accuracy on each downstream task and on the held-out ImageNet-R support set. GradFix improves task performance while preserving the target model's zero-shot capabilities.
EUROSAT
SVHN
GTSRB
RESISC45
DTD
Method
Task
Supp.
Task
Supp.
Task
Supp.
Task
Supp.
Task
Supp.
A zero-shot
49.08
68.80
47.00
68.80
43.37
68.80
58.94
68.80
47.50
68.80
On zero-shot
49.41
79.78
50.58
79.78
48.29
79.78
67.98
79.78
55.96
79.78
ft
98.58
65.93
93.59
67.77
98.23
65.98
92.43
65.35
79.04
65.02
On + +84
67.80
79.77
69.74
79.73
65.23
80.17
72.14
79.88
59.63
79.27
D DATASETS AND SUPERVISION PROPORTIONS
In this section, we provide detailed information about the datasets used in our experiments and compute, for each one, the supervision proportions corresponding that I denotes the number of labeled examples per class used to estimate anti-gradient signs. The resulting percentages indicate what fraction of the full training set those few-shot budgets represent.
subset budgets [D Recall
D.1 VISUAL DATASETS
EuroSAT: A dataset based on Sentinel-2 satellite images covering 13 spectral bands, con- sisting of 27 0100 labeled and geo-referenced samples across 10 classes (Helber al. 2019).
SVHN: A real-world image dataset from Google Street View house numbers, containing 73 257 labeled digits across 10 classes (Netzer et al. 2011).
GTSRB: The German Traffic Sign Recognition Benchmark, comprising 39 209 training images and 12 630 test images across 43 classes (Stallkamp et al. 2011))
RESISC45: A scene classification dataset with 31 500 RGB images 256 x 256 from Google Earth, covering 45 scene classes with 700 images per class (Cheng et al.) 2017).
DTD:
The Describable Textures Dataset, consisting of 5640 images organized into 47 cat- egories inspired by human perception (Cimpoi et al. 2014).
Table 9: Supervision proportions for visual datasets. shows the total dataset percentage.
[De] denotes examples per class. Each cell
IDe
Dataset
# Samples
Classes
1
2
10
20
50
EUROSAT
27,000
10
0.04%
0.07%
0.19%
0.37%
0.74%
1.85%
SVHN
73.257
10
0.01%
0.03%
0.07%
0.14%
0.27%
0.68%
GTSRB
39,209
43
0.11%
0.22%
0.55%
1.10%
2.19%
5.48%
RESISC45
31,500
45
0.14%
0.29%
0.71%
1.43%
2.86%
7.14%
DTD
5,640
47
0.83%
1.66%
4.15%
8.30%
16.60%
41.49%
D.2
TEXTUAL DATASETS
SNLI:
Stanford Natural Language Inference dataset, containing 570 sentence pairs labeled for entailment, contradiction, or neutral (Group et al. 2022).
MNLI: Multi-Genre Natural Language Inference dataset, comprising 433 000 sentence pairs annotated with textual entailment information across various genres (Williams et al. 2018
RTE: Recognizing Textual Entailment dataset, with 2490 examples for training, 277 for validation, and 3000 for testing, divided into two classes (Wang et al.) 2018).
QNLI: Question Natural Language Inference dataset, containing 104 743 training exam- ples divided into classes (Wang et al. 2018
21

---

Published as a conference paper at ICLR 2026
SCITAIL: A science entailment dataset built from science question answering, with 596 training examples divided into two classes (Khot et al. 2018).
Table 10:
Supervision proportions for textual datasets. [
denotes examples per class. Each cell
shows the total dataset percentage.
Dataset
# Samples
Classes
[D]] 50
SNLI
570,000
0.03%
MNLI
433,000
3
0.03%
RTE
2,490
4.02%
QNLI 104,743
2
0.10%
SCITAIL
23,596
0.42%
E HYPERPARAMETER SELECTION
We evaluated the optimal task vector application coefficient using the validation set of each dataset, following standard practice (Ilharco al. 2023 Gargiulo et al. 2025 Marezak al. 2025). For 8", the optimal is equal to 1 across all datasets. In Tab. we summarize the coefficients cor- responding to the optimal performance of 8 for each dataset. 6 is obtained for each dataset by training with the AdamW optimizer (learning rate 1e-5) on the corresponding subset D, used to compute the gradient mask m, with single gradient-descen step.
Architecture
EUROSAT
SVHN
GTSRB
RESISC45
DTD
ViT-B/16
0.3
0.3
0.6
0.2
0.7
ViT-L/14
0.4
0.5
0.5
0.2
0.5
Architecture
SNLI
MNLI
RTE
QNLI
SCITAIL
T5v1.1
0.3
0.1
0.2
0.1
0.7
Table 11: Optimal hyperparameters for 8 across CLIP and T5 architectures.
F COMPUTATIONAL COST ANALYSIS
Following standard scaling-law approximations (Kaplan et al. 2020). we adopt the commonly used per per-parameter FLOP estimates to compare the computational efficiency of our approach against baselines. Let P denote the number of model parameters. We assume the following operation costs:
forward pass: 2P FLOPs
backward pass: x 4P FLOPs
Optimizer update (Adan/AdamW); 10P FLOPSs per step (accounting for momentum updates, squaring, bias correction, and parameter write-back).
Given this, we derive the costs for GradFix, the few-shot baseline 0ort and full fine-tuning.
GradFix cost. GradFix requires single gradient computation on the target model followed by the masking operation.
Forward Backward pass: 6P FLOPs
Mask construction and masked task-vector application (element-wist multiplication and addition): 2P FLOPs
Total GradFix Cost: 8P FLOPs.
22

---

Published as a conference paper at ICLR 2026
&B cost (one-step fine-tuning). The 0 baseline represents one-step fine-tuning update on the target model.
Forward Backward pass: 6P FLOPs
Adam optimizer update: 10P FLOPs
Total 00
Cost: 16P FLOPs.
Full fine-tuning cost. Standard fine-tuning (used to produce the original source task vectors) typically involves 2000 training steps.
Per step (Forward Backward + Optimizer): 6P + 10P 16P FLOPs
For 2000 steps: 16P 2000
Comparing the methods reveals significant efficiency gaps:
goge vs. GradFix: 2x more FLOOPs
Full fine-tuning vs. GradFix: 4, x00 more FLOPs
Full fine-tuning vs. goot, RJ 2, 000x more FLOPs
Euro
23


Reviews:
[{"review_id": "LcMVWQx2jT", "reviewer": "Reviewer_1QUv", "summary": "The paper proposes GradFix, a method to transfer task-specific fine-tuning knowledge from a source pre-trained model to a different target pre-trained model. The authors argue that naive transfer fails due to parameter space misalignment and that the key to successful transfer lies in the sign structure of the target model's gradients. GradFix computes the gradient on the target model using only a few labeled samples. It then creates a binary mask by checking for sign agreement between the source task vector and the target gradient. This mask filters the source task vector and updates the target model parameters without any fine-tuning. The authors provide a theoretical guarantee that this update ensures a first order loss descent on the target model. Empirically, GradFix is shown to outperform naive task vector addition and standard few-shot fine-tuning on vision and language benchmarks.", "questions": "1. Could the authors please justify the \"single step of gradient descent\"  for the $\\theta_B^{opt}$ baseline? How does GradFix compare to a more standard few-shot baseline, such as training $\\theta_B$ on $\\mathcal{D}_s$ for maybe 10 epochs, or training a PEFT method on $\\mathcal{D}_s$?\n\n2. In a practical, no-validation-set scenario, how should $\\alpha$ be set? What is the performance of GradFix if $\\alpha$ is fixed to a default value (e.g., $\\alpha = 0.5$) across all datasets in Table 1?\n\n3. Have you explored hybrid approaches that do not discard magnitude entirely? For instance, what happens if you down-weight sign-mismatched components instead of zeroing them, or use a method that combines signs with magnitudes (e.g, TIES-Merging style?)\n\n4. The related work on TIES-Merging  also focuses on resolving sign conflicts for task vectors. How does GradFix's \"masking\" approach conceptually differ from TIES's \"sign-consistency\" aggregation? How would GradFix compare empirically if TIES was adapted for this transport setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The challenge of updating downstream tasks when a new foundation model is released is a significant and practical problem for the ICLR community.\n2. The core idea of using the target model's gradient signs as a filter for the source task vector is elegant and easy to understand. The method's simplicity of requiring only a single forward-backward pass on a few labeled samples is a major practical advantage.\n3. Figure 1 provides an excellent and unambiguous illustration of the core masking procedure.\n4. While simple, the proof of a first-order descent guarantee (Sec 4.2)  provides a solid theoretical justification for why the method should work, moving it beyond a simple heuristic.\n5. The use of majority voting to estimate gradient signs from a handful of samples (Sec 4.3) is well-justified, and the accompanying concentration analysis (Appendix A)  strongly supports this design choice.\n6. The evaluation spans both vision (CLIP ViT) and language (T5) models, demonstrating the applicability of the approach across domains.", "weaknesses": "# Major Concerns\n\n1. **Over-reliance on a weak baseline $\\theta_B^{opt}$**: The primary few-shot baseline is defined as fine-tuning the target model with the same few samples $\\mathcal{D}_s$. However, Appendix B.2 clarifies that this is achieved with a \"single step of gradient descent\". This is not a standard definition of few-shot fine-tuning. A more conventional baseline would involve multi-epoch training on $\\mathcal{D}_s$ or applying a parameter-efficient method like LoRA on $\\mathcal{D}_s$. As such, the claims of consistently outperforming few-shot finetuning are based on a potentially weak comparison, which may inflate GradFix's perceived gains.\n\n2. **Underspecified hyperparameter $\\alpha$**: The method introduces a critical scaling factor $\\alpha$ in Equation 5, whjich is tuned on a validation set for each dataset (in Table 4). This undermines the \"plug-and-play\" claim in a strict few-shot setting where a representative validation set may not exist. The sensitivity analysis in Section 5.4 confirms that performance is sensitive to $\\alpha$, but it does not propose a way to set $\\alpha$ _without_ a validation set. This is a key practical limitation.\n\n3. **Complete disregard for gradient magnitude**: The method relies _only_ on sign agreement, discarding all magnitude information from both the source task vector $\\tau_A$ and the target gradient $g_B$. This seems like an extreme simplification. A component of $\\tau_A$ with a very large, task-critical magnitude will be zeroed out if it has a sign mismatch. Conversely, a tiny-magnitude component of $\\tau_A$ is preserved as long as its sign matches. The justification for this sign-only choice is not deeply explored.\n\n## Minor Concerns\n\n1. In Table 1, the full $\\theta_B \\textit{fine-tune}$ performance is slightly higher than $\\theta_B + \\delta^*$ (oracle) performance (e.g., EUROSAT: 98.70 vs 95.06; GTSRB: 98.65 vs 82.92). This suggests that $\\tau_A$ is fundamentally missing significant information (or contains conflicting information) that cannot be recovered even by a perfect sign mask based on $\\tau_B$. This information gap is a limitation and needs more discussion.\n\n2. In Section 5.2, the paper compares \"agreement\" with \"force agreement\". The argument is that \"agreement\" is more robust to noisy gradients. However, \"force agreement\" seems more conceptually aligned with the theoretical goal of ensuring a descent direction $g^\\top \\delta^A \\geq 0$. This choice feels non-obvious and the empirical difference (Table 3) is small, making the justification feel post-hoc.\n\n3. he finding that random sampling is a strong baseline compared to structured methods like coreset or herding is a good practical result . However, it is also surprising. This might imply that the gradient sign structure is extremely stable across the data distribution, which is an interesting finding in itself and could be analyzed more deeply.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mdwyT9VkLo", "reviewer": "Reviewer_PYpn", "summary": "This paper proposes GradFix, a simple method to transfer task vectors between different pre-trained models. It masks the source task vector using the gradient sign of the target model, keeping only directions aligned with the target loss. This ensures a first-order descent update without re-training. Experiments on CLIP and T5 show that GradFix effectively transfers fine-tuning knowledge using only a few labeled samples, outperforming naive task-vector addition and few-shot tuning.", "questions": "1. Why can’t we directly use the source model? If the source model is indeed inaccessible, obtaining its task vector still seems almost equivalent to having it, since it only differs from the base pre-trained weights—usually publicly available.\n\n2. Could the authors provide a more convincing motivation or real-world use case where such task-vector transport is necessary or clearly beneficial?\n\n3. Is the task-vector transport still effective in multi-task scenarios—for example, when merging multiple transported task vectors?\n\nI would be happy to raise my score if the authors can address all these concerns convincingly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is simple and easy to understand, yet surprisingly effective for transferring knowledge between models.\n2. The method is lightweight and requires very little data, making it practical and efficient.\n3. Results are clear and consistent across different tasks, showing strong generality and reliability", "weaknesses": "1. The motivation of the setting is not entirely convincing. If the source model has already been fine-tuned on the task, it is unclear why one would not simply use that model directly. Since obtaining the task vector requires access to the fine-tuned source, the need to reapply it to a target model feels less practical, especially when the transferred performance does not surpass the source.\n2. The method assumes that the source and target share the same architecture, differing only in pre-training. This limits real-world applicability, as many practical scenarios involve transferring between models with different structures or sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4AQDxuLX18", "reviewer": "Reviewer_G6x6", "summary": "The paper recognizes that updating checkpoints with new data or a new training pipeline leads to redundant repeated fine-tuning for the same downstream tasks. To address this, it proposes GradFix, which uses gradient-sign masking to transfer task-specific knowledge from the pre-trained model.", "questions": "1.The proposed method appears applicable only to models with the same architecture trained on different downstream datasets, which limits its scope. If the two models share only similar architectures but differ in scale, does the method still apply?\n2.The paper does not consider more complex multi-source transfer scenarios—i.e., simultaneously transferring multiple task vectors from different source models (such as models pre-trained on different datasets but adapted to the same task) into a single target model.\n3.Can your method be transferred to diffusion-based generative models or VLM tasks? These pre-trained models are typically very large yet serve many downstream tasks and encode rich priors. If extending to VLM, what adaptations would be required?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper proposes a fine-tuning-free knowledge transfer strategy that uses the sign of the gradient as a robust surrogate for the descent direction. The method is insightful and reasonably well-motivated.\n2.The paper is clearly structured and well-organized, with high readability.", "weaknesses": "Please refer to the Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5b36VvBoNs", "reviewer": "Reviewer_sW3e", "summary": "This paper addresses an interesting problem to transfer the task vector on a finetuned task across different pretrained checkpoints. The authors propose to mask the task vector based on whether its sign aligns with the gradient obtained with a small target set, which empirically improve the performance consistently. \n\nFurther ablations are needed to convincingly show that the observed improvement is due to cross-model task knowledge transfer rather than exploiting from a random vector based on incidental gradient sign alignment.", "questions": "Typos:\nThe alpha coefficient in Eq. 5 seems to be redundant, as it appears also in Eq. 6.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well-motivated, simple yet effective. \n2. The method demonstrates consistent performance gain compared to zero-shot and other baselines.", "weaknesses": "1. While the proposed leads to consistent performance gain, the performance gap is still quite substantial compared to the fully-finetuned counterpart (e.g., 20%-40% accuracy gap). There seems to be a diminishing effect brought by increasing the size of the small target set to estimate the gradient sign (Table 5), which prevents the proposed method to eventually deliver on-par performance with fully-finetuned checkpoint even with large number of samples.\n\n2. There is no ablation to verify whether improvements truly reflect transfer of task-specific knowledge from the original task vector or the model is just exploiting a random vector by incidental gradient alignment. \n\nIt is non-intuitive to readers that the original task vector can be transferred to the new checkpoint without any parameter permutation. One would expect the original task vector could be just random noise to the new pretrained checkpoint, which is partially verified by the results that naively adding them do not improve performance. Suppose it is actually noise, there is a possibility that the proposed method works because it is selecting the parameters which “luckily align with the right gradient direction” from the original task vector even though it is noise. I think a further ablation experiment is needed to verify whether this is the case. For example, I suggest the authors to test their proposed method on a randomly generated vector (e.g., a Gaussian vector with the same mean and std as actual task vector) and compare it with the performance obtained with the original task vector. The result comparison should verify to which extent the proposed method is exploiting from the actual task knowledge in the original task vector.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S0xZ3tQFa9", "reviewer": "Reviewer_VKR3", "summary": "This paper presents GradFix, which aims to transfer task vectors between different pre-trained models by leveraging gradient sign structures in settings with limited downstream data. Specifically, GradFix computes gradients on the target pre-trained model and uses the resulting gradient signs to mask out harmful update directions that may increase the downstream loss. The paper also provides a theoretical guarantee showing that this procedure yields a first-order reduction in loss. To address the limited data regime, a majority-vote strategy is adopted to robustly estimate gradient signs. Experimental results demonstrate that GradFix outperforms baselines such as naive transfer and related approaches, and reduces the performance gap to full-dataset fine-tuning.", "questions": "Besides the weakness shown in the above section, please also see the following questions: \n\nQ1: From Equations (1) and (4), the mask $m$ seems to take binary values (0 or 1). However, Figure 1 depicts $m$ with positive and negative signs. Why does the figure illustration of $m$ differ from its binary formulation? \n\nQ2: In the limited-data scenario, the performance gap between GradFix and the oracle (i.e., $\\theta_B + \\delta^*$) remains substantial. How much does GradFix improve as more samples become available, for example, when the full dataset is used? Additionally, at what sample size does standard fine-tuning begin to outperform GradFix? Understanding this transition point would help clarify which approach is preferable under different computation, labeling, or storage budgets.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well-written and easy to follow. It investigates an important and practical problem regarding improving the transfer efficiency of task vectors amid the rapid advancement of foundation models. Moreover, the simplicity of the proposed method also makes it easy to understand and implement.\n\n- The experimental results are promising, showing that GradFix consistently outperforms the baselines and related methods, and reduces the performance gap to full-dataset fine-tuning.\n\n- The paper provides a theoretical guarantee that GradFix ensures a first-order decrease in the loss. This also helps explain why and how the proposed method is effective.\n\n- Experiments examine the behavior of GradFix under different types of downstream data subsets and hyperparameter settings, which helps improve the understanding of the method’s applicability.", "weaknesses": "- At Lines 36-38, the paper states that one motivation is to reduce redundant re-training when new models are released by companies or researchers. However, newer and more advanced models often involve architectural changes (e.g., increased model size to accommodate knowledge learned from larger datasets), whereas GradFix appears to require the pre-trained models to share the same architecture. This constraint may limit the applicability of the method in such scenarios.\n\n- At Lines 469-471, the paper states that GradFix does not require fine-grained tuning of $\\alpha$. However, Figure 3 indicates that different downstream datasets achieve optimal performance at different $\\alpha$ values. For example, $\\alpha = 0.3$ performs best for EUROSAT, SVHN, and RESISC45, whereas $\\alpha = 0.7$ is optimal for GTSRB. Moreover, the accuracy differences across $\\alpha$ values on these datasets can be as large as 10-20%. Despite this, the paper does not provide a strategy for selecting $\\alpha$ when adapting to new datasets or pre-trained models in practical use cases.\n\n- Table 1 and Table 2 do not report accuracy variance across different sample selections in the few-shot setting. However, Figure 2 suggests that random data selection can still affect the performance of GradFix to a certain degree. It is therefore unclear how the performance variance of GradFix compares to that of $\\theta_B^{\\text{opt}}$​. For example, are there few-shot sample selections where $\\theta_B^{\\text{opt}}$​ actually outperforms GradFix, and how frequently does this occur?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)