Variational Task Vector Composition
Authors: Boyuan Zhang
Yingjun Du
Xiantong Zhen
Ling Shao
Venue: NeurIPS 2025 poster ()
Keywords: Task vector
Vartiational inference
Spike-and-slab prior
Controllable posterior with gated sampling
Primary area: deep_learning
TL;DR: We reformulate the task vector coefficient prediction as a variational inference problem and propose a novel composition framework.

Abstract: Task vectors capture how a model changes during fine-tuning by recording the difference between pre-trained and task-specific weights. The composition of task vectors, a key operator in task arithmetic, enables models to integrate knowledge from multiple tasks without incurring significant additional inference costs. In this paper, we propose variational task vector composition (VTVC), where composition coefficients are taken as latent variables and estimated in a Bayesian inference framework. Unlike previous methods that operate at the task level, our framework focuses on sample-specific composition. Motivated by the observation of structural redundancy in task vectors, we introduce a Spike-and-Slab prior that promotes sparsity and aims to preserve the most informative components. To further address the high variance and sampling inefficiency in sparse, high-dimensional spaces, we develop a gated sampling mechanism that constructs a controllable posterior by filtering the composition coefficients based on both uncertainty and importance. This yields a more stable and interpretable variational framework by deterministically selecting reliable task components, reducing sampling variance while improving transparency and generalization. Experimental results demonstrate that our method achieves state-of-the-art average performance across a diverse range of benchmarks, including image classification and natural language understanding. These findings highlight the practical value of our approach, offering a new, efficient, and effective framework for task vector composition.

Variational Task Vector Composition
Boyuan Zhang'
Yingjun Du? Xiantong Zhen Ling Shao !M
Al Lab, School of Engineering Science, University of Chinese Academy of Sciences
VVIS Lab, University of Amsterdam #Central Research Institute, United Imaging Healthcare, Co., Ltd. zhangboyuan230mai 1s. ucas. ac.cn
Abstract
Task vectors capture how model changes during fine-tuning by recording the difference between pre-trained and task-specific weights. The composition task vectors, key operator in task arithmetic, enables models integrate knowledge from multiple tasks without incurring significant additional inference costs. In this paper, propose variational task vector composition (VTVC), where composition coefficients are taken as latent variables and estimated in a Bayesian inference framework. Unlike previous methods that operate the task level, our framework focuses on sample-specific composition. Motivated by the observation of structural redundancy in task vectors, introduce Spike-and-Slan prior that promotes sparsity and aims to preserve the most informative components. To further address the high variance and sampling inefficiency in sparse, high- nigh-dimensional spaces, we develop gated sampling mechanism that constructs a controllable posterior by fil- tering the composition coefficients based on both uncertainty and importance. This yields more stable and interpretable variational framework by deterministically selecting reliable task components, reducing sampling variance while improving transparency and generalization. Experimental results demonstrate that method achieves state-of-the-art average performance across a diverse range of benchmarks, including image classification and natural language understanding. These findings highlight the practical value of our approach, offering a new, efficient, and effective framework for task vector composition.
Introduction
Task vectors represent the difference between a model's pre-trained and fine-tuned weights, capturing the changes during fine-tuning on specific tasks. Using task vectors, task arithmetic [17] enables simple and efficient model editing. The composition of task vectors, a key operator in task arithmetic, enables models integrate knowledge from multiple tasks without increasing additional inference costs. Task vector composition has demonstrated strong performance across a variety of domains, including computer vision [14], natural language processing [41, 23, 13], and multimodal learning [16, 26]. This approach provides a practical solution for knowledge integration and model editing [48, 44, 53, 46).
In recent years, the composition task vectors has been widely studied to improve efficiency and controllability [52]. Some works focus on improving task vector representations by redesigning spaces and developing parameterization techniques to better capture the relationships between tasks [30, 9]. Others extend the functional scope of task vectors reduce computational overhead [41, 14, 15]. Advances in composition methods also enable more effective integration of knowledge across multiple
"Correponding Author (ling. shwotieee.orrrg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tasks, enhancing both the controllability and interpretability of models [51, 54]. Meanwhile, in the broader field of meta-learning, researchers have also explored using probabilistic frameworks such as variational inference to learn task-adaptive modules, thereby improving the model's generalization ability on new [55 8]. While recent works have made notable progress, three key challenges remain under-explored. (1) Existing approaches typically rely on deterministic composition schemes, lacking mechanisms to quantify uncertainty. (2) Most methods operate at the task level, which limits their ability to adapt to sample-level variability. (3) Task vector spaces often exhibit redundant structure, which may reduce efficiency in both inference and storage.
In this paper, we propose novel approach for the Variational Task Vectors Composition (VTVC). Our main contributions are as follows: (i) We cast task vector composition as a variational inference problem and introduce an amortized inference network to model sample-specife posteriors over composition coefficients. This formulation enables efficient integration of domain- specific knowledge at the sample level and overcomes the limitations of conventional task-level composition. (ii) To prior that
address the structural redundancy in task vector spaces, we introduce Spike-and-She promotes sparsity by modeling the variational posterior a mixture of zero- valuev spikes and Gaussian slabs. This structured prior guides the variational posterior to focus on essential task components, enhancing interpretability and improving the efficiency of sample-specific composition. (iii) To address the high variance and sampling inefficiency of traditional sampling-bass based inference, we develop controllable posterior via gated sampling, which deterministically filters composition coefficients based on their estimated uncertainty and importance. By deterministically selecting task coefficients based on uncertainty and importance, the proposed framework avoids the high variance of stochastic sampling. highlights the components that contribute to adaptation, and promotes consistent inference, making more stable, interpretable, and reliable.
We evaluate our framework on diverse benchmarks computer vision and natural language process- ing, Experimental results show that VTVC achieves strong performance in various task arithmetic scenarios, including task addition and task negation. Our analyses of the three main contributions- variational composition, Spike-and-Slab priors, and the gated sampling process highlight the method's advantages. We demonstrate its ability to integrate sample-level knowledge, reduce redundancy, and improve inference stability.
2
Related work
Task vector composition and task arithmetic. Task arithmetic is a central technique in the model merging field, which aims to combine multiple specialized models into one [17]. The field has been widely studied to improve efficiency and controllability [52], with theoretical foundations established from perspectives like the Neural Tangent Kernel and generalization analysis [30, 23). Early frameworks like aTLAS introduced anisotropic scaling for more precise control [51]. A key challenge is managing interference between task parameters. Recent work has largely focused on two directions. The first is leveraging sparsity, where methods like DARE merge only the critical, sparse parameters to improve efficiency [50, 43, 11]. The second direction focuses on directly resolving task interference, with methods aligning parameter signs [47] or redesigning task vector spaces to find less conflicting representations [9, 54, 4]. The functional scope of task vectors has also been extended in various applications, from vision to language models [14, 41, 15]. Other approaches achieve sample-leve) adaptation by modifying the model architecture uses Mixture-of of-Experts (MoE) modules to dynamically route inputs to different task vectors [38]. While these methods are effective, they are largely deterministic. contrast, we propose a probabilistic inference framework that learns sample-specific composition coefficients, uniquely allowing it to quantify and leverage uncertainty without requiring architectural changes.
Applications of Spike-and-Slab priors. Spike-and-Slab priors are widely used in Bayesian sparse modeling to distinguish between important and unimportant variables [21, 2]. The foundational work by Mitchell et al. [28] introduced them as mixture distributions for effective variable selection. While traditional estimation relied on MCMC sampling [18], the need for scalability on large datasets has prompted a shift toward more efficient variational inference (VI) approaches [40, 34). In deep learning, these priors have been applied to tasks like unsupervised image classification [10], with recent studies also exploring structured forms to model complex correlations [37, 1, 36, 27]. However, their practical integration with large deep models remains a challenge, especially for sample-specific sparse inference.
2

---

3 Preliminary
In transfer learning and multi-task learning, task-specific knowledge is often captured by modeling changes in network parameters, including models based on CLIP [31], GPT-2 [32], and T5 [33]. In this paper, we focus particularly CLIP, it offers powerful multimodal representations and aligns with experimental settings used in previous work.
Formally, denote the CLIP image encoder as function / x θ - 2, where for input image a A' and parameters 0 E O the output for the input image. Given the pre-trained parameters O and the fine-tuned parameters O in downstream task t, the corresponding task vector is defined as T = O Op. In practice, we follow standard procedures by fine-tuning only the image encoder while keeping the text encoder fixed, which helps maintain feature alignment across tasks.
f(x; 0) represents the learned latent representation
Task vectors can be composed through various arithmetic operations, and the most common operation task addition. In task addition, given a task vector pool 7 with N task vectors {Tr) fine-tuned on different tasks, with global learnable scaling factor X, a unified model can be constructed as:
Onew O A X-E TG. This form of composition has shown strong practical value, enabling effective integration of knowledge from multiple tasks into a single model and demonstrating superior transfer capabilities in image classification and transfer learning benchmarks.
Task vectors can also be decomposed into block-wiss representations. Based on functional parti- tioning, aTLASS [51] divides each task vector т; into . parameter blocks {r. 1 f==1- and assigns an independent scaling coefficient A? to each block. Such modular decomposition also enables the exploration of task relevance at different network layers or functional units, which can be further exploited in downstream analysis. The composed model parameters can then be written
N
M
θ O
Σ
A TH. 1=
(I)
This block-wise formulation enables finer-grainee knowledge integration and provides a foundation for efficient multi-task learning and task transfer. While most existing methods perform task vector composition at the task level with fixed coefficients, they fail to capture fine-grained differences between samples. To address this limitation, we model composition at the sample level, where the coefficients become functions of the input Formally, the parameters of sample-specific merged model (x) can be written as:
N
(z) = Oo +
(x)
(non-block case).
1=1
(x)) O, +
Ziy 2)((x) T/
(block case).
Here, (x) and (i)(x) are sample-specific coefficients produced by inference network that takes r as input. This finer-grained approach allows the model to adapt to input variability and enhances performance in heterogeneous environments, underpinning proposed variational framework.
Methodology
This section introduces our proposed framework for the variational composition of task vectors. In Section 4.1, we reformulate the composition problem as a variational inference process and introduce an amortized inference network to model sample-specins posteriors over composition coefficients. To address the structural redundancy in task vector spaces, we then introduce Spike-and-S prior to promote sparsity by modeling the variational posterior as a mixture distribution in section 4.2. We also propose a controllable posterior with gated sampling that automatically selects informative task vectors in section 4.3, which enhances the model's generalization capability and robustness.
3

---

4.1 Variational Composition of Task Vectors
Conventional task vector composition is typically performed direct and deterministic modifi- cation of model weights, lacking any mechanism for uncertainty quantification. As result, such approaches are less capable of adapting sample-specific variations and cannot estimate uncertainty in the composition process. In our variational inference framework, the composition coefficients z are treated as latent variables, while the input x and labels y are observed. Figure 1 illustrates the probabilistic dependencies. Ac- cording to Bayesian theory, our objective is to compute the posterior distribution p(z) x, y).. Since the posterior is typically difficult to compute directly, variational inference introduces a parameterizable approximate posterior distribution (z) leading to the evidence lower bound (ELBO):
-V (nee)0),
x
Z
logp(y/x; 2 Byy(xx) (y(at))logpp( x, x)] DKL (1)) (4)
y
Here, the first term represents the expected log-likelihood, which promotes the model to make accurate predictions, while the second term is the KL divergence, which regularizes the posterior towards the prior to prevent overfitting. This framework naturally recasts the composition problem as an inference task, allowing us to incorporate flexible prior knowledge, such as sparsity or structural constraints, into the composition process. The detailed mathematical derivation is provided in Appendix A.I.
Figure 1: Variational Composi- tion of Task Vectors.
To efficiently realize sample-level variational inference, we employ amortized inference by de- signing a neural network with parameters & to parameterize the posterior distribution: qo (zlx) ((z; p. (x), (x)x.. This network takes samples x as inputs and predicts the parameters of the composition coefficient distribution z. Amortized inference improves both efficiency and scalability by sharing parameters across samples.
In summary, sample-level composition allows the model adapt composition coefficients indi- vidual inputs, offering finer control over task integration and leading to improved generalization in heterogeneous settings. This represents fundamental difference from task-level approaches. How- ever, conventional variational inference typically uses Gaussian priors, which are unable to capture address structural redundancy in task vector spaces. This lead to inefficient composition and increased risk of overfitting.
4.2 Spike-and-Slab Priors for Sparse Representations
The standard Gaussian prior is widely used, but it fails to capture the structural properties inherent in high- sigh--imensional task vector spaces. As illustrated in Fig. 2, we conduct thorough redundancy analysis of task vector representations. Using t-SNE and singular value decomposition, significant redundancy in task vectors, which leads to inefficient integration and a higher risk of overfitting. For instance, over 95% of the variance components across all datasets, highlighting substantial compressibility. Moreover, when input samples are located near the boundaries of multiple task domains, Gaussian priors are unable to capture the multimodal nature of the underlying composition, thereby limiting model performance on heterogeneous data.
find
be explained by fewer than 40 principal
Our variational composition framework offers a flexible way of incorporating prior knowledge to address the issue of structural redundancy in task vector representations. We introduce Spike-and- Slab prior, which effectively promotes structural sparsity in the compositions of task vectors. The Spike-and-Slae prior is defined as mixture distribution:
p(z) (1 - T) 60(z) + n N(2;0,03),
where (z(z) denotes the Dirac delta distribution at zero, [(z; 0, 2 denotes a Gaussian distribution with zero mean and variance o², and the mixture coefficient that controls the expected sparsity of the coefficients. The Spike-and-She Slab prior allocates probability mass between a point mass at zero (the spike) and continuous Gaussian distribution (the slab). This ensures that coefficients are either exactly zero or drawn from the Gaussian, thereby enabling explicit and interpretable structure selection. By promoting coefficient sparsity, the Spike-and-Sha prior automatically identifies and retains the most important task vector components while effectively climinating irrelevant ones. This

---

VOUE Comporeees
210
Nurberr Singuar lokuss
(a) (-SNE clustering of task representations.
(b) Cumulative singular value energy distribution.
Figure 2: Redundancy and structural analysis of task vectors. (a) t-SNE visualization shows that task representations form distinct clusters corresponding to each dataset, there are notable overlap regions. indicating partial coupling of task features across domains. (b) Cumulative singular value energy plots reveal that fewer than 40 principal components account for over 95% of the variance across all eight datasets. These analyses reveal substantial structural redundancy in task vector representations.
property not only improves model efficiency but also enhances interpretabilityy allowing us to clearly determine which task knowledge is relevant to each input sample.
Under the Spike-and-Slab prior, adopt factorized variational posterior:
qo (z x)
IIII To (w (x) (2) x))
Specifically, for each composition coefficient z,. we introduce a corresponding binary indicator variable w/. The coefficient is retained when w/ 1 and set to zero when w/ = 0.
By performing variational inference, obtain the following ELBO objective:
LELBO (eolleoooool x, z, w)] - (go (xx))l(((,))
(7)
To optimize this framework, we employ amortized inference with Monte Carlo sampling. We design neural network parameterized by & that inputs sample and outputs three sets of parameters: inclusion probabilities n. (x), weight means pay (x), and log variances log of (x). During training, we first sample binary inclusion variables w/ ~ BernoullirT? (x)), then sample weights z/ (N)r((x) (x)), and finally combine them to obtain sparse composition coefficients žद = w/ z/. The complete derivation of the variational objective with Spike-and-Slab prior is available in Appendix A.2.
However, in high-dimensional settings, Monte Carlo-based variational inference often suffers from unstable sparsity patterns and unreliable results [35, 6]. This limitation motivates the development of a more stable and controllable posterior, which we describe in the following section.
4.3 Controllable Posterior with Gated Sampling
To address the instability and high variance of Monte Carlo sampling high dimensions, we propose gated sampling mechanism for controllable posterior. Instead of sampling binary variables from the conventional Bernoulli distribution with probability Txx), our method employs deterministic, continuous gating function to modulate composition coefficients. The continuous gating function replaces discrete sampling with differentiable alternative, enabling gradient- training while preserving the structure of variational inference. Figure 3 illustrates both stochastic and deterministic inference processes, highlighting how our method transitions from random Bernoulli sampling to uncertainty -wwaee, aware, soft-gated posterior construction.
begin by constructing an uncertainty estimate U. which integrates two components: gradient sensitivity and distributional deviation. Gradient sensitivity S quantifies how responsive coefficient small perturbations in the input, while distributional deviation measures the extent which coefficient deviates from the batch mean, where μg and og denote the mean and standard deviation

---

-~ N ((x((())
x
Z
Z
Ω
r
S
X
sec 4.3
+ J
y
T
w
u
V
Z
w~ (ernonlli(((s))
Figure 3: Variational Composition of Task Vectors with Spike-and-Slab Prior. The framework illustrates two inference approaches based on the Spike-and-SS posterior process generates latent variables z from input features x via N ((u (x)). producing sparse representations 2 @ through element-wise multiplication with binary indicators sampled from ((rmoull(T(())).. On the right, the deterministic gated sampling process computes uncertainty 2( from gradient sensitivity S and distributional deviation V. uses an adaptive threshold to generate continuous gating variables Ω, and produces sparse representations 2 = z O Ω. This framework demonstrates the transition from random binary selection to deterministic continuous gated in the inference process.
prior. On the left, the sampling- basgg variational
of the coefficient within the batch, respectively. The overall uncertainty measure is formulated as weighted combination:
z(x)) JBB
U = 1 S (1 (1-1)) V 1 V (11-x))+++111)))) (1-
σB
where 1) is balancing coefficient. Intuitively, coefficients that are highly sensitive to input changes or show large fluctuations among similar samples are deemed unreliable and should be suppressed, while stable and significant coefficients should be preserved.
We then define an adaptive gating function:
(z,4; V) z 1([z) 2 r(u;
where the threshold function r(4; v varies with the uncertainty metric 4, creating an adaptive selection mechanism as follows:
r V 2/1 (1 + uhe -2),
(10)
where V = [11, 222) are learnable parameters, 1/1 is a base threshold, and 4/ controls sensitivity to uncertainty. Under this mechanism, coefficients with high uncertainty are subjected to higher thresholds and thus are less likely to be selected, while those with low uncertainty face lower thresholds, increasing their likelihood of being retained. This adaptive selection ensures that more reliable and informative components are preferentially integrated into the final model composition.
To enable end-to-end differentiable training, we approximate the hard-gated function with a soft- gated function, thereby replacing the Monte Carlo sampling methods described in Section 4.2 with a deterministic point estimation approach:
I(U; Q(2,2)= = [z] ( Vn
(1))
P
where (-) denotes the sigmoid function and is a temperature hyperparameter. During training, the soft-gated function allows for gradient-based optimization, while at inference time we switch to hard thresholding to ensure sparse and interpretable coefficient selection. Based on this design, each coefficient's variational posterior can be approximated
mixture of Spike-and-
priors:
qz(xx) = 2(z. 6u(z) + f2 24) -N(z; ps (x), oz (x)).
(12)
Here, Ω replaces the binary indicator variable for deterministic gating. To further address the high variance of sampling in nigh-dimensional spaces, we shift from sampling the coefficients
to predicting them as a deterministic point estimate from the input instead of modeling distribution over 2, we explicitly compute an uncertainty metric I and use to control the gating function Ω. This leads to a more stable training process. We formalize this deterministic choice for within the variational posterior using a Dirac delta distribution as follows:
(x). In our framework.
44 v (Z, z/x) q (00==(0)0)
(13)

---

This parameterization enables the selection of deterministic coefficients within the variational infer- ence framework, while explicitly modeling uncertainty.
In addition to the primary objective of the ELBO, we introduce auxiliary regularization losses Lrey to further stabilize training and encourage exploration. These include boundary loss, penalizing coefficients near the threshold to encourage more decisive selection decisions, and an exploration loss, promoting threshold-related parameters search in broader space to avoid local optima. Detailed definitions of these regularization terms and hyperparameter settings are provided in Appendix B.
In summary, the final training objective can be expressed as:
[Dul
arg min F.W
Σ (f Oo g (z(x;: D) VD T) Vs) XLreg (2, V)
(14)
(Drl
where À is the balance parameter. Our final training objective comprises two components: the primary prediction loss based the gated sampling, and regularization term that stabilizes the optimization process while enhancing the model's s generalization capacity. By modeling coefficient uncertainty explicitly, our method enhances generalization and interpretability. Overall, this umcertainty-guidee gated composition yields a more robust and interpretable mechanism for knowledge integration across diverse tasks.
5 Experiments
5.1 Experiments Setup
Tasks and Datasets. We evaluate our framework across three distinct scenarios. For multi-task model merging, we follow previous work and use eight image classification datasets: Cars [20], DTD
[5], EuroSAT [12], GTSRB [39], MNIST [22], RESISC45 [3], SUN397 [45], and SVHN [29]. For NLP applications, we use the General Language Understanding EVLl)))ion(GLUU)) benchmark [42]. We also design a task negation experiment to evaluate performance on more complex task arithmetic.
Baselines. We compare method VTVC against comprehensive set of recent model merging methods, including Task Arithmetic [17], aTLAS [51], TIES- [47], DARE [50], AdaMerging [49], and WUDI-Merging [4]. To ensure fair comparison, all methods evaluated under the same experimental conditions.
Implementation Details. We use three pre-trained CLIP Vision Transformer (ViT) architectures: ViT-B/16, ViT-B/32, and ViT-L/14 [31, 7]. For NLP tasks, we use RoBERTa [24] as the backbone model. All experiments are trained with the AdamW optimizer [25]. For our method's 's gated posterior, we set the base threshold(1$)) to 0.05 and the sensitivity parameter((ss) to 1.0. All experiments were conducted on eight NVIDIA A40 GPUs. Further details on hyperparameters and task-specific settings are provided in Appendix B.
Evaluation Metrics. For model merging and image classification tasks, we report the top-1 accuracy. For task negation, we evaluate the accuracy on both the forgotten target task and the remaining control tasks. For the GLUE benchmark, we report the average score across all tasks. We also report the gated ratio of our method, which measures the average proportion of active coefficients, to analyze its sparsity.
5.2 Main Results
Performance Multi-Task Merging. We first evaluated VTVC against current state-of-the- art methods on model merging tasks. The evaluation covered eight visual tasks using ViT-B/32 models and eight NLP tasks using RoBERTa-bau experimental settings. As shown in Table 1 and Table 2, VTVC achieved the highest average accuracy in both domains. For visual tasks, VTVC reached 87.45% average accuracy, with partieularly strong performance on DTD and RESISC45 datasets, improving accuracy by 15.35% and 5. 5.14% respectively compared the previous best method. For NLPP tasks, VTVC achieved 81.93% average accuracy, showing significant improvements on challenging tasks such as COLA, STS-B, and RTE, where accuracy gains reached 80.60%, 74.37%. and 83,80% respectively. These results demonstrate that the sample-level variational approach effectively captures task-specific features across different modalities, making it flexible and powerful method for model merging.
models. All methods were tested under fair
7

---

Table 1: Multi-task performance when merging VIT-B/32 models on eight visual tasks.
Methods
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Avg Ace
Pre-trained
59.63
44.13
45.74
32.60
48.26
60.27
63.53
31.63
48
Task Arithmetic [17]
60.23
55.18
80.26
66.21
95.94
69.17
64.09
76.27
70.92
Ties-Merging [47]
65.17
59.43
78.03
67.55
97.49
73.29
76.98
75.33
AdaMerging [49]
69.90
50.10
83.40
82.40
95.70
73.10
69.80
87.30
76.46
DARE [50]
61.89
55.19
79.89
66.03
95.87
68,94
63.69
75.98
70.94
aTLAS [S1]
71,59
78.93
93.05
88.33
96.87
87.39
67.94
85.47
83,70
WUDI-Merging [4]
70.88
71.53
95.74
93.12
98.01
86.21
72.35
92.01
84.98
VTVC (Ours)
76.02
86.88
95.84
90.57
98.24
91.35
71.19
89.53
87.45
Table 2: Multi-task performance when merging RoBERTa-base model on NLP tasks.
Methods
COLA
STS-2
MRPC
STS-B
QQP
QNLI
MNLI
RTE
Avg Acc
Pre-trained
0.00
53.76
85.01
4.01
37.48
53.05
37.09
71.19
42.70
Weight Averaging
0.00
59.21
85.79
46.99
45.37
63.94
48.00
71.19
52.56
Task Arithmetic [17]
8.35
88.26
89.57
32.84
82.03
85.40
75.54
80.43
67.80
Ties-Merging [47]
31.76
88.86
86.18
10.94
61.05
85,94
83.01
69.56
64.66
DARE [50]
0.00
88.14
86.61
30.19
84.33
79.09
63.95
77.16
63.68
VTVC (Ours)
80.60
90.60
88.48
74.37
80.40
83,20
74.00
83.80
81.93
Performance on Task Negation. Task negation aims to remove specific knowledge from merged model while preserving other capabilities. The goal is to reduce perfor- mance the target task while maintaining accuracy control tasks. As shown in Table 3, VTVC achieves the best task negation performance with the lowest target accu- racy and competitive control accuracy. This demonstrates that VTVC selectively remove task-specifie knowl- edge without significantly affecting other tasks. More detailed results are provided in Appendix C.I.
Table 3: Performance of task negation aver- aged across eight datasets.
Methods
Target([)
Control(t)
Task Arithmetic
22.62
60.73
Ties-Merging
20.93
61.52
TaLoS aTLAS
17.28
61.33
19.42
61.31
VTVC (Ours)
16.75
60.84
5.3 Ablation Studies and Analysis
In this section, we analyze the contribution of each component through ablation experiments. All ablation experiments are conducted under unified setting. To avoid the impact of data augmentation strategies and improve computational efficiency, we use pre-computed feature representations with static data augmentation strategy.
Benefit of sample-specific task vectors. Spmeii- task vector composition consistently outperforms task-level baselines, as shown in Table 4. The improvements are especially clear on Cars, SVHN, and GTSRB datasets. These datasets show greater diversity within the sample and differences between domains. SVHN contains data features that differ significantly from pre-training source data. GTSRB includes various types of traffic signs. Cars features different automobile models and viewing angles. These characteristics make unified task-level vectors insufficient for capturing sample-specific differences. Our approach calculates combination coefficients for each input sample, enabling more accurate and adaptive knowledge integration. As a result, improvements are greatest on datasets with high variability and notable domain gaps. This confirms the effectiveness of our sample-specific task vectors when handling complex data scenarios. All results are based on features encoded by the VIT-B/32 model. More detailed results are provided Appendix C.2.
Benefit of variational compasition of task vectors. Table 4 shows that the variational inference framework enhances the overall performance of the model. This is particularly evident in datasets with stronger differences between domains, such as EUroSAT, MNIST, and SVHN. This improvement is attributed to explicit uncertainty modeling of task vector coefficients. This provides better robustness when processing cross-domain data variations. The KI. divergence regularization effectively reduces overfitting risk, improving model generalization. Deterministic methods perform slightly better on DTD, likely due to the regularity of its texture features. In this case, deterministic inference

---

Table 4: Performance comparison of task-level versus sample-specific methods with deterministic and probabilistic task vectors across cight datasets. Results demonstrate that sample-specific methods consistently outperform task-level methods across all datasets, with the variational inference framework further enhancing model performance.
Method
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISCAS
SUN397
SVHN
Average
Pre-trained
58.56%
42.66%
40.74%
26.45%
25.23%
54.60%
46.46%
9.25%
37.99%
Deterministie-
Task-level [51] 62.99%
73 73.29%
92.67%
81.11%
13%
88.41 %
67.16%
58.01 %
77.60%
Sample-specific
74.75%
85,97
96.07%
85,90%
98.07%
90.81% %
68.01 %
67.17%
83.34%
Probabilistic-
Task-level
73.06%
80.64%
96.
85.66%
98.65%
90.00%
67.95% 72.97% %
64.23%
82.05%
Sample-specific
75.33%
85.71% 1%
96.33%
86.76%
98.72%
90.98 %
67.78% %
84.32%
ViT-B/32
ViT-B/16
ViT-L/14
Care
Cars
Care
FLESKAT SHNDK2
FUrSAAT SUINN
RumdaT
RESOSCES
CTSAR
RESESCES
GTSRR
RESEECAS
G15R3
VNST
Goussion prior
Sp Slab prior
Figure 4: Comparison of Accuracy and Gated Ratio under Gaussian and Spike-and-Slab Priors. We visualize model accuracy and gated ratios for three VIT architectures across eight image classification datasets. The Spike-and-Slab prior consistently delivers higher accuracy while activating fewer task coefficients than the Gaussian prior, demonstrating more efficient and selective task vector composition.
adequately captures the necessary information. The uncertainty of the additional parameter from variational inference may introduce minor interference. The results validate that our variational inference framework is particularly effective for heterogencous and cross-domain scenarios. More detailed results are provided in Appendix C.3.
Normal Gaussian vs. Spike-Slab priors. Figure 4 compares the performance of Gaussian and Spike-and-Slai priors across different VIT models. Spike-and-Shl and-Slab priors achieve substantial perfor- mance gains on most datasets. These improvements are most evident in the datasets DTD, MNIST, and SVHN, which feature complex patterns significant cross-domain differences. Spike-and-Slab priors excel at capturing sparse structures in combination with task vectors. They guide the model to select only the most relevant dimensions for each task. The gated ratio clearly demonstrates this effect. The increased sparsity reduces interference from redundant information and improves both efficiency and performance. It also reduces computational costs during inference. On EuroSAT and RESISC45, Gaussian priors perform slightly better, possibly because these tasks benefit from dense, high-dimensional feature representations. Their tasks likely require comprehensive feature combinations across dimensions. Spike-and-Slaie priors perform exceptionally well on datasets with complex and heterogeneous characteristics. The sparse approach improves cross-domain performance and makes the composition of task vectors more efficient. More detailed results are provided in Appendix C.3.
Effect of controllable posterior with gated sampling. Table 5 evaluates the effectiveness of the gated sampling mechanism. Introducing the gated sampling leads to a clear overall performance improvement over the baseline Spike-Slab prior. This improvement is particularly significant on datasets such as DTD, EuroSAT, and RESISC45. The gated sampling effectively identifies and preserves critical task knowledge while suppressing noise and unreliable components by evaluating the reliability of composition coefficients. The controllable posterior with gated sampling yields more stable and interpretable component selection compared to inference with random sampling. Analysis in different data sets reveals that gated sampling performs more prominently on datasets with complex and variable feature spaces, further confirming its superiority in handling heterogeneous data. These experimental results validate the effectiveness of our theoretical framework, demonstrating the important role of uncertainty- guided adaptive component selection in enhancing model generalization

---

Table 5: Analysis of the adaptive gated sampling. The gated sampling significantly improves model general- ization capabilities across datasets by selectively filtering reliable composition coefficients.
Method
Cars
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
Average
w/o Gated Sampling
75.69%
88. 38.66%
96.41%
87.17%
98.81%
90.71%
72.25%
67.85%
69.6%%%
Gated Sampling
77.78%
93.31%
97.04%
87.16%
98.91%
92.08%
73.47%
68.57%
86.04
capability and robustness. All results are hased on features encoded by the ViT-B/32 model. More detailed results are provided in Appendix C.3. To further verify the effectiveness and transferability of the controllable posterior, we apply the gating module as a filter on the aTLAS method. The evaluation shows performance improvements across all datasets and three ViT architectures. Results are provided in Appendix C.4.
Relationship Between Model Scale and Gated Ratio. Figure illustrates the impact of gated sampling on average accuracy and activation ratios across three Vision Transformer (ViT) models of increasing scale: ViT-B/32, ViT-B/16, and ViT-L/14. The results clearly demonstrate significant accuracy improvements when incorporating gated sampling in all models. The improvements are most pronounced in larger models (ViT-L/14), highlighting the scalability advantage of our method. Notably, as the model scale increases, the activation coefficient ratio with gated posterior rises significantly, while models without gated sampling maintain relatively stable or slightly de- creasing activation ratios. This pattern suggests conventional variational methods struggle to ac- commodate the knowledge integration demands of increasingly complex models. In contrast, our gated sampling dynamically adjusts knowledge integration strategies based on the model scale. It preserves more valuable task vectors in larger models, enabling more efficient and precise model editing. This phenomenon is consistent with the increased representational capacity and flexibility of larger models, which benefit more from adap-
80.%
Accuncy two Sumptaur
Accincy Grusd Suerpliggt
Guard Re Secefing
Qlaed Kanwree Unee Sempling!
Rudo
11.0%
83.8%
VIT-0-2
VI--U/16
WI--/144
Midd Architature
Figure 5: Comparison Accuracy and Gated Ra- tio Across Model Architectures. The gated sampling consistently improves prediction accuracy across all models. As model complexity increases, frame- work retains more coefficients, showing effective- ness gated sampling handling more complex tasks.
tive gating strategies.
Conclusion
In this paper, we proposed a new framework, Variational Task Vector Composition (VTVC), which formulates the composition process as variational inference problem. Our Bayesian approach enables precise, samplc-specific modeling that effectively captures domain-specific knowledge for each input. By introducing a components in task vectors while removing redundancy. This concept inspired our gated sampling mechanism, which further enhances reliability by deterministically and efficiently selecting highly informative task vectors based on uncertainty. Extensive experiments on both vision and language benchmarks demonstrate that VTVC significantly improves performance on various operations, including task addition and task negation. Overall, our framework offers new, probabilistic perspective on task arithmetic and broadens the application potential of task vectors for adaptive model editing.
prior, our method automatically selects important
Limitations and Future Work. A key direction for future research is improving the offline training efficiency and scalability of our framework. While VTVC enables fast online inference, it relies on computationally intensive stage to train the inference network, especially as the number tasks and model scale increase.
Acknowledgements. This work was partially supported by the National Natural Science Foundation of China under Grant No. 62176068.
10

---

References
[1] M. R. Andersen, O. Winther, and L. K. Hansen. Bayesian inference for structured spike and slab priors. Advances in Neural Information Processing Systems, 27, 2014.
[2] R. Bai, V. Ročková, and E. I. George. Spike-and-slab meets lasso: A review of the spike-and-slab lasso. Handbook of Bayesian variable selection, pages 81-108, 2021.
[3] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of IEEE,
1865-1883, Oct 2017.
[4] R. Cheng, F. Xiong, Y. Wei, W. Zhu, and C. Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. In International Conference Machine Learning, 2025.
[5] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1600--613, Columbus, OH, USA, 24- 27 Jun 2014.
[6] A. K. Dhaka, A. Catalina, M. Welandawe, M. R. Andersen, J. Huggins, and A. Vehtari. Challenges and opportunities in high dimensional variational inference. Advances in Neural Information Processing Systems, 34:7787-7798, 2021.
[7] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[8] Y. Du, X. Zhen, L. Shao, and C. G. Snoek. Hierarchical variational memory for few-shot learning across domains. In International Conference on Learning Representations, 2022.
[9] A. A. Gargiulo, D. Crisostomi, M. S. Bucarelli, Scardapane, F. Silvestri, and E. Rodolà. Task singular vectors: Reducing task interference in model merging. arXiv preprint arXiv:2412.00088,, 2024.
[10] I. J. Goodfellow, A. Courville, and Y. Bengio. Large-scale feature learning with spike-and-slab sparse coding. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1387-1394, 2012.
[11] Y. He, Y. Hu, Y. Lin, T. Zhang, and H. Zhao. Socalize-and-stiche Efficient model merging via sparse task arithmetic. Trunsuctions on Machine Learning Research, 2025.
[12] Helber, B. Bischke, A. Dengel, and D. Borth. Introducing euroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium. pages 204-207, Valencia, Spain, 22-27 Jul 2018
[13] R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[14] A. Hojel, Y. Bai, T. Darrell, A. Globerson, and A. Bar. Finding visual task vectors. In European Conference on Computer Vision, pages 257-273. Springer, 2024.
[15] C.-Y. Hsu, Y.-L. Tsai, Y. Zhe, Y.-L. Chen, C.-H. Lin, C.-M. Yu, Y. Zhang, C.-Y. Huang, and J. Sakuma. Badtv: Unveiling backdoor threats in third-party task vectors. arXiv preprint arXiv:2501.02373. 2025.
[16] B. Huang, C. Mitra, L. Karlinsky, A. Arbelle. T. Darrell, and R. Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural Information Processing Systems, 37:22124-22153. 2024.
[17] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Edit- ing models with task arithmetic. In Proceedings of International Conference on Learning Representations (ICLR), Kigali, Rwanda, 1-5 May 2023. OpenReview.net.
11

---

[18] H. ISHWARAN and J. S. RAO. Spike and slab variable selection: Frequentist and bayesian strategies. The Annals of Statistics, 33(2):330-773, 2005.
[19] L. Iurada, M. Ciccone, and T. Tommasi, Efficient model editing with task-localized sparse fine-tuning. In ICLR, 2025.
[20] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3D object representations for fin-ggraiee categorization. In IEEE/CVF International Conference on Computer Vision (ICCV) Workshop on 3D Representation and Recognition, pages 554-561, Sydney, Australia, 1-8 Dec 2013.
[21] Kumar, P. Sarkar, K. Tian, and Y. Zhu. Spike-and-slab posterior sampling high dimensions. arXiv preprinl 2533:22503.02798 2025.
[22] Y. LeCun, C. Cortez, and C. C. Burges. The mnist handwritten digit database, 1998.
[23] H. Li, Y. Zhang. S. Zhang. P.-Y. Chen, S. Liu, and M. Wang. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. International Conference Learning Representations, 2025.
[24] Y. Liu, M. Ou, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis, Zeulemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rX1v:1907.11692, 2019.
[25] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 6-9 may 2019. OpenReview.net.
[26] G. Luo, T. Darrell, and A. Bar. Vision- Iision-language models create cross-modke task representations. In ICML,, 2025.
[27] G. Malsiner- Wallli and H. Wagner. Comparing spike and slab priors for bayesian variable selection. Austrian Journal of Statistics, 40(4):241-264, 2011.
[28] T. MITCHELL,, BEAUCHAMP,L. BERGER, and C. MALLOWS. Bayesian variable selection in linear regression, comments. Journal of the American Statistical Association, 13(44)):1023 1036, 1988.
[29] Y. Netzer, T. Wang. A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems (NeurIPS) Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, 12-17 Dec 2011.
[30] G. Ortiz-Jiménez, A. Favero, and Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 66727-66754 New Orleans, LA, USA, 10-16 Dec 2023. Curran Associates, Inc.
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of International Conference on Machine Learning (ICML), volume 139, pages 18-24 Jul 2021.
Proceedings of Machine Learning Research (PMLR),
[32] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[33] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[34] K. Ray, B. Szabó, and G. Clara. Spike and slab variational bayes for high dimensional logistic regression. Advances in Neural Information Processing Systems, 33: 13:14422-11444 14434, 2020.
12

---

[35] T. Reichell, L. Ong, and T. Rainforth. Rethinking variational inference for probabilistic programs with stochastic support. Advances in Neural Information Processing Systems, 15:15160-15175, 2022.
[36] V. Rockova and K. McAlinn. Dynamic variable selection with spike-and-slab process priors. Bayesian Analysis, 16(1) (6(1)2233-29, 2021.
[37] F. Scheipl, L. Fahrmeir, and T. Kneib. Spike-and-slab priors for function selection in structured additive regression models. Journal of the American Statistical Association, 107(500):158-- 1532, 2012.
[38] L. Shen, A. Q. Tang, E. Yang, G. Guo, Y. Luo, L. Zhang. X. Cao, B. Du, and D. Tao. Efficient and effective weight-ensembling mixture of experts for multi-task model merging. In International Conference on Machine Learning. PMLR, 2024.
[39] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recogmition benchmark: multi-class classification competition. In Proceedings of International Joint Conference on Neural Networks (ICNN), pages 1453-1460, San Jose, CA, USA, 31 Jul-5 Aug 2011.
[40] M. Titsias and M. Lázaro-Gredilla Spike and slab variational inference for mulliitask and multiple kernel learning. Advances in neural information processing systems, 24, 2011.
[41] E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024.
[42] A. Wang. A. Singh, J. Michael, F, Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[43] K. Wang, N. Dimitriadis, G. Ortiz-Jiménez, F. Fleuret, and P. Frossard. Localizing task information for improved model merging and compression. In International Conference on Machine Learning, 2024.
[44] S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li. Knowledge editing for large language models: A survey, ACM Computing Surveys, 57(3):1-37, 2024.
[45] Xiao, K. A. Ehinger. Hays, A. Torralba, and A. Oliva. Sun database: Exploring a large collection of scene categories. Interational Journal of Computer Vision (IJCV), 119(1):3-2,, 2016.
[46] R. Xu, Z. Qi, Z. Guo, C. Wang, H. Wang, Y. Zhang, and W. Xu. Knowledge conflicts for Ilms: A survey. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 8541-8566 2024.
[47] P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal. TIES-Merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 7093-7115. Curran Associates, Inc., 2023.
[48] E. Yang, L. Shen, G. Guo, X. Wang, X. Cao, J. Zhang, and D. Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.0760 2024.
[49] E. Yang. Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning. In Proceedings of International Conference on Learning Representations (ICLR), 2024.
[50] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning. PMLR, 2024.
[51] F. Z. Zhang, P. Albert, C. Redriguez--pazz A. van den Hengel, and E. Abbasnejad. Knowledge composition using task vectors with learned anisotropic scaling. Advances in Neural Information Processing Systems, 19-673,,,,7 2024.
13

---

[52] J. Zhang, J. Liu, J. He, et al. Composing parameter-efffcien modules with arithmetic operation. Advances Neural Information Processing Systems, 36: 12589-12610, 2023.
[53] N. Zhang, Yao, B. Tian, Wang, Deng, M. Wang, Z. Xi, Mao. J. Zhang, Y. Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.0128,,200 2024.
[54] Zhang, R. Zhang, and Z. Nie. Dynamic task vector grouping for efficient multi-task prompt tuning. arXiv preprint ArXiv:2503. 18063, 2025.
[55] X. Zhen, Y. Du. H. Xiong, Q. Qiu, C. G. M. Snock, and L. Shao. Learning learn variational semantic memory. Advances in Neural Information Processing Systems, 165:9122-9134, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper claimed in the abstract. Detailed information can be found in the third paragraph the introduction section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide "limitation" subsection in the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all experimental details the experiment section. The detailed input and output of method can be found in Appendix and supplemental materials.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We provide all experimental details in the experiment section. The detailed and Python codes of our method can be found in Appendix and supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
http:/guddee//ooeeuumsmssssessse onloiicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips "nips.cc/ppppp wwtppblcc/gpppeeccccccccccccccot wewgddee/oeeeeiieeeeeeeeeeeeeeee wwt.SSummissiiopooooooooooiooooi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We follow the standard experimental setup in Variational Task Vector Com- position, where the data splits, set as the same as previous works aTLAS [51]. Detailed information can be found in Sec. 5
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following the standard experimental setup, we repeat each experiment over 3 random seeds and report the mean of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computing resources experiments 5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eerppp/eeeeee/eeeeeeeeeee cc hccpppliic/thhhcccueeeeeeeeeeeee
Answer: [Yes]
Justification:
We reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide the potential broader impacts the conclusion section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and models pose no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the code package and datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of the datasets/code/mooel are provided in the supplemental materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We use LLM solely for grammar checking.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https battps:/mne /ttps/////eui.pp.eeee /00p//////ooeeeeee//22///////// for what should or should not be described.
21

---

A Mathematical Derivation of Variational Composition of Task Vectors
A.1 Derivation of the ELBO Objective Function
In the variational composition of task vectors, our main objective is to infer the posterior distribution p(z|x.y). y). where denotes the composition coefficients of task vectors, denotes the input data, and y represents the labels. According to Bayes' theorem, this posterior is given by:
p(z|x,y) = p(y x, z)p(z)
(15)
p(yix)
where the marginal likelihood ply (x) defined as:
p(ylx) / ply|x, 2)p(z)dz
(16)
Since directly computing the above integral is typically intractable, we introduce variational approximate posterior distribution (z((x) to approximate the true posterior. We derive a lower bound on the log marginal likelihood:
log p(yl() = log / p(ylxx,)) z)p(z)dz
(17)
Introducing the variational distribution g(z)x):
log x) = log
ply ())(2)
(18)
p(y|x, p)p(z)
=
log
(zllx)d2
((z(x)
Since the logarithm a concave function, by Jensen's 's inequality:
p(y|x, 2))(z)
log (yy/() log Bg(z(x)
((2(x)
2 E (x/x)) log ely|x,z)ple
q(zlx)
(19)
log p(z)
(x (ogg x 2)] Bx(()x))
((()x)
1 Dog p(y/p z)] (x((((22)))ll((
This derives Equation 4 from the paper:
logp(y x) 2 (g(())) [log ply z)] - DKL (x(x)(()))(ll(
(20)
This inequality's right-hand side known as the Evidence Lower BOund (ELBO). It comprises two critical components. The first term (x(x))) [lo p(y |x, z]] represents the expected logarithmic likelihood, which guides the model toward accurate predictions. The second term ( the KI. divergence term, regularizing the posterior toward the prior distribution, thus preventing overfitting.
Maximizing ELBO is equivalent minimizing the KL divergence between the variational posterior and the true posterior. Through optimization of the ELBO, we can learn optimal composition coefficients for task vectors while maintaining robust generalization capabilities.
A.2 Derivation of ELBO with Spike-and-Shh Prior
In the Spike-and-Shn prior based variational framework, we introduce binary indicator variables w to determine which composition coefficients retained and which set to zero. We define a joint variational posterior distribution go (w, z/x), where & represents neural network parameters. The ELBO can be expressed as:
LELBO (q plo p w, z)] DeL. (gx z/x) [p(w, z))
(21)
22

---

Using conditional probability decomposition, we can factorize the joint variational posterior as: (w z|x) = (z) x)
(22)
Similarly, the prior distribution can be decomposed as:
p(w, 2) - r(z)p(w/z)
(23)
Based on the above factorization, the KL divergence term can be further expanded:
(yo (w, 2) ()))]/p[w,, x)) (yo (z)x (o) x)(lm())p(((z)))
go (z/x) dz
go (z/x) log
p(z)
(24)
f go (z|x)
//
2o (w/z, x) log to (o x) dw dz
p(w(z)
(eo (p(z)) + (x)xx) [DL (w x) x)][)(wl)))
Therefore, the ELBO can be rewritten as:
(x))) LELBO = E (w.)))) (lg w, 2)] (go (z(x))lp((p)) - Eg (x/x) (Dxu (9 (wlz, x)]lp(( [2))
(25)
In our model, we formulate a comprehensive probabilistic framework by defining the follow- ing distribution structure: The binary variables the variational posterior follow (w |x) =
Bernoulli (x)). where of (x) represents the inclusion probability for each coefficient, while their prior counterparts are distributed according p(w:) Bernouli(TT). with serving as the global sparsity parameter. For the coefficient values themselves, when wf 1, the posterior distribution characterized by qo (z)) buf p(z |oy 1) = (Nx ;, 0, d²), establishing a complete probabilistic specification of our variable selection mechanism.
1, 1, = io, of (x)), and correspondingly, the prior follows
For the binary indicator variables the KL divergence is calculated as:
DKL (Bernoulli(?} (emoulli(T))
Σ (x) log (x) +1 701-2/(00) (x)) log
1 {x)
(26)
(x)
π
1 π
When w/ = 1, the KL divergence for coefficient 2 is:
DKL (z) w 1, 1,x))lp(z]|1d /o 1)) DKL (N M4y (f)
o²
(x + 1i (x)
(27)
log
a²
Combining the above KL divergence calculations, and considering that when w/ 0. the coefficient is forced to zero (thus contributing no additional KL divergence), the complete form of the ELBO is:
LELBO
Eg (w. > x [log (x,,)x/,,,..
(x)
+ Σ (x)) log 1 7:'(xx
Σ Hd (x) log
(28)
o (x) + 22 (x)
Σ (x)
log
o²
This represents the complete expansion Equation 6. The first term is the expected likelihodoood, representing the model's predictive accuracy; the second term is the KL divergence of binary indicator variables, controlling the sparsity pattern; and the third term the conditional KL divergence of non-zero coefficients, weighted by their inclusion probabilities, ensuring reasonable distribution of weight values. By optimizing this objective function, we achieve adaptive sparse composition of task vectors while maintaining the model's generalization capability.
23

---

B Regularization and Hyperparameter Settings
Regularization Overview. In our adaptive gating mechanism, regularization plays a crucial role in ensuring training stability and enhancing model generalization capability. Our proposed regulariza- tion strategy addresses several key challenges: uncertainty in gating decisions, insufficient parameter space exploration, and information integration efficiency. When coefficient values approach thresh- olds, small perturbations can cause unstable selection results; training processes tend converge prematurely to suboptimal solutions; and selected components must effectively represent sample features. The total regularization objective Creg (z. V) comprises multiple carefully designed components, each aiming to jointly improve the robustness and effectiveness of the gating mechanism.
Boundary Loss. The boundary loss La is specifically designed to penalize coefficients near thresholds, enhancing stability by encouraging more decisive selection decisions. When coefficient values lie in ambiguous threshold regions, minor input variations can cause significant fluctuations in component selection, reducing model robustness and generalization capacity. The boundary loss is defined as:
(z(2,
max(0, m |zz, r(Ua, V)1)
(29)
This loss function imposes positive penalty when the absolute difference between coefficient Zi,j and threshold r(U,,; vf falls below boundary width m, otherwise zero, effectively pushing coefficients away from unstable threshold regions to form more decisive binary selection patterns. In our experiments, the default boundary width parameter m 0.1 effectively balances clear decision-making with model flexibility.
Exploration Loss. Exploration loss Le promotes threshold-related parameters (4/1 and x/2) to explore broader parameter spaces, preventing training from converging local optima. By encouraging these parameters to deviate from their initial values, the model explores potentially superior parameter configurations, enhancing the adaptability of the gating mechanism. The exploration loss is defined as:
VV) log (l w + c) log (1 2 + c)
(30)
This loss function decreases as parameters deviate from their initial configurations by taking the negative logarithm of the parameter-initial value difference, thus incentivizing broader parameter space exploration. Here, represents a small constant (typically 10-") that ensures numerical stability. The uncertainty loss fy is defined as:
b
(2)
Us [[Z + 0]
(31)
1=1==1
This loss penalizes selected (non-zero) coefficients with high uncertainty. encouraging the model to reduce uncertainty while retaining coefficients, further enhancing gating decision reliability. The comprehensive regularization objective integrates these components:
Creg (z, Ut; V = \z, V) + (V) + (2)
(32)
Hyperparameter Settings and Sensitivity Analysis. In our experimental implementation, we used a set of meticulously tuned hyperparameters: boundary width m 0.1, boundary loss weight
= -10-4 exploration loss weight de = 10-3, uncertainty loss weight X = 10-2, base threshold initial value wi 0.05, uncertainty sensitivity initial value vy = 1.0, global regularization coefficient A = 10-3 (Equation 13), and gating temperature parameter = 20.0 (Equation 10). Sensitivity analysis indicates that model performance is relatively robust variations in A within the range [10-5,1033] more sensitive to Àe where increasing this value (> 10-2) promotes more aggressive parameter exploration but can lead training instability, while decreasing (< 10 4) can cause the model to converge to local optima. Performance is optimal when A, is within [5 x 10-3 5 x 10-2). 2]. The initial setting of the base threshold vi significantly impacts model performance: excessive values (> 0.1) lead to overly sparse representations, compromising information retention; insufficient values (< 0.01) retain too many redundant components, increasing overfit risk. For different feature dimensions and dataset characteristics, we recommend setting 16 within the range [0.03, 0.08] and automatically selecting optimal configurations through validation sets. Comprehensive experimental results demonstrate that our proposed regularization strategy and parameter configurations signif- icantly enhance the effectiveness of the adaptive gating mechanism, enabling reliable component selection across diverse task vector spaces.
=
24

---

C Supplementary Experimental Results
C.1 Detailed results of task negation
Table 6: Task negation on ViT-B/32 across 8 datasets. Lower is better for Target (Tgt.) and higher is better for Control (Ctr.).
SUN399
Average
Ctr.
Tgt.
Ct..
TgL
Chr
Ctr.
Ctr.
Cr.
Tgt.
Ct:
Ct.
Tgl.
Ct..
Task Arithmetic [17]
35.47 60.73 26.96 60.67
11.59 60.67 7.17 60.40
12,70
60.86
32.08 61.23 47.96 60.67 7.03 60.60
22.62 60.73
TIES Merging [47]
23.28
61.94
oS [19] 28.30
60.88 18.99
$9.76
9.63
62.13
5.20
62.01
12.18
63.00
13.56
60.27
41.87
60.47
8.49
62.10
17.28 61,33
[51]
60.52
60.72
VTVC (Ours)
25.48 60.13
21.19
59.98
9.26
61.59
4.18
60.74
8.57
62.83
15.95
60.36
42.66
60.27
6.74
60.84
16.75
60.84
C.2 Detailed results of task-level vs. sample-specific and deterministic vs. probabilistic
Table 7: Detailed accuracy comparison across VIT models and datasets. report zero-shot, determin- istic (task-level and sample-specific) and probabilistic (task-level sample-specific) performances each model. Highest performance in each dataset is highlighted in bold.
Method
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Avg.
VIT-B/32
Pre-trained
58.56
42.66
40.74
26.45
25.23
54.60
46.46
9.25
37.99
Deterministic-hacee
Task-level
62.99
73.29
92.67
81.11
97.13
88.41
67.16
58.01
77.60
Sample-specific
74.75
85.97
96.07
85.90
98.07
90.81
68.01
67.17
83.34
Probabilistic
based
Task-level
73.06
80.64
96.19
85.66
98.65
90.00
67.95
64.23
82.05
Sample-specine
75.33
85.71
96.33
86.76
98.72
90.98
72.97
67.78
84.32
VIT-B/16
Zero-shot
63.45
43.77
50.63
35.22
19.29
55.70
47.13
15.53
41.34
Deterministic-bonn
Task-level
71.61
75.92
93.56
82.15
97.36
90.37
69.54
66.67
80.90
Sample-speciftc
80.59
90.10
96.96
88.04
98.85
92.17
70.02
74.97
86.46
Probabilistic based
Task-level
81.32
82.44 44
88.18
98.93
91.57
69.95
70.50
84.87
Sample-speciffe
81.52
88.75
96.78
88.76
98.93
92.90
71.81
75.12
86.82
VIT-L/14
Zero-shot
76.40
52.10
55.63
45.61
52.59
63.41
50.41
41.73
54.74
Deterministic-buan based
Task-level
85.39
83.41
96,89
91.55
98.76
93.79
71.05
76.02
87.11
Sample-specific
87.29
96.84
97.91
93.46
99.09
94.30
73.90
82.38
90.65
Probabilistic- bosee
Task-level
88.47
90.08 93.28
97.78
92.87
99.01
93.54
72.33
78.46
89.07
Sample-specific
89.64
97.99
93.55
98.92
95.11
75.05
83.03
90.82
25

---

C.3 Detailed results of Gaussian vs. Spike-and-Slab priors
Table Performance comparison between Gaussian Spike-and-Slab prior models (% across VIT
models and datasets. Highest performance in each dataset is highlighted in bold. Gated ratio values are reported in itaiiss.
Method
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Avg.
ViT-B/32
Gaussian prior
75.33%
85.71
96.33% 96.41% %
86.76%
98.72% 98.81%
90.98%
72.97%
67.78%
84.32% 84.69%
Spike-and- Slab prior
75.69%
88.66
87.17%
90.71%
72.25%
67.85%
ViT-B/I6
Gaussian prior Spike and-Slab prior
81.52%
85%
96.78%
76%76%
95.8518
92.17%
71.81%
74.97%
86.61
82.23 %
90.31 %
96.70%
89.27 %
98.87%
92.40%
74.10% %
75.46%
87.42%
VFT:1/14
Gaussian prior Spike and Slab prior
87.29%
93.22%%
97.67%
93.46%
99.09%
94.30%
73.90%
82.38%
90.38%
97.63%%
93.56
99.14%
94.43%
75.07%
83.09
C.4
Detailed results of gated sampling
Table 9: Accuracy and gating ratio comparison (%) across VIT models and datasets. Performance and
average gated ratio are reported for models with and without gated sampling. The highest performance in each dataset is highlighted in bold. Gated ratio values in italics.
Method
Cars
DTD
EuroAAT
GTSRB
MNIST
RESISCAS SUN397
SVHN
Avg.
VIT-B/32
w/o Gated Sampling
75.69%
88.66%
96.41% %
87.17%
98.81%
90.71% %
72.25%
67.85%
84.69%
Gated Ratio
80.40%
72.60%
58.60%
80.20%
67.30%
77.70%
71.10%
64. 64.30%
71.53%
w/ Gated Sampling
77.78%
93.31 %
97.04%
87.16%
98.91%
92.08%
73.47%
68.57%
86.04%
Gated Ratio
65.62%
21.88%
59.38%
21.87%
60.42%
63.54%
91.67%
64.58%
68.62%
VIT-B/16
w/o Gated Sampling
82.23%
90.31%
96.70%
89.27%
98.87%
92.40%
74.10% %
75.46%
87.42%
Gated Ratio
78.70%
74.50%
10%
63.30%
10%
76, 90%
72.40%
60.20%
70.40%
w/ Gated Sumpling
83,61 %
95.01
97.00%
89.22%
98.93%
93.02%
73.53%
75.47%
88.22%
Gated Ratio
79.17%
73.96%
67.71%
82.23%
81.25%
77.08%
62.08%
75.21%
75.21%
ViT-L/14
w/o Gated Sampling
88.38%
94.35%
97.63%
93.56%
99.14%
94,43%
75.07%
83.09%
90.71%
Gated Ratio
67.00%
76.60%
66.50%
69.70%
65.50%
74.40%
62.50%
80.60%
70.35%
w/ Gated Sampling
89.45%
97.75%
97.96% %
93.67
99.15%
95.49%
75.52%
83.31 %
91.54%
Gated Ratio
69.79%
72.92%
70.83%
85.83%
85.42%
85.42%
67.71%
78.13%
78.13%
C.5
Applying gated sampling to aTLAS for filtering
Table 10: Performance comparison between standard a aTLAS Gated aTLAS methods across three VIT architectures and eight datasets (%). The gated mechanism consistently improves performance across all datasets and models. Highest performance in each dataset is highlighted in bold.
Method
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Avg.
VIT-B/32
w/o Gated 62.99%
73.29%
92.67%
81.11%
97.13%
88.41% 89,84%
67.16%
58.01% %
77.60%
Gated
71.32%
80.24%
94.96%
84.92%
97.96%
68.33%
61.68
81.16%
VIT-B/16
w/o Gated
71.61%
75.92%
93.56%
82.15%
97.36%
90.37%
69.54%
66.67%
80.90%
Gated 80.06% %
82.37%
94,96
86.04%
98.10% %
91.59%
70.29%
68.68 %
84.01 %
ViT-L/14
w/o Gated
85.39% 86.78%
83.41 %
96.89%
91.55%
98.76%
93.79% 94.75%
73.05%
76.02%
87.1 1%%
w/ Gated
88.40 %
97.33%
92.88%
98.90% %
77.19% %
77.79 %
88.59
26


Reviews:
[{"review_id": "QKAOOVN6yY", "reviewer": "Reviewer_TZ6C", "summary": "This paper introduces \"Variational Task Vector Composition\" (VTvC), a framework for model merging. The core idea is to reframe task vector composition as a Bayesian inference problem, where the combination coefficients are treated as sample-specific latent variables. The method employs a variational inference (VI) approach to estimate these coefficients. To promote sparsity and select informative components, the authors introduce a Spike-and-Slab prior and a deterministic \"gated sampling\" mechanism. The stated goal is to improve the stability, interpretability, and generalization of the merged model. The framework is evaluated on eight image classification datasets, with its primary comparison being against the aTLAS method.", "questions": "1.\tBaseline Performance Discrepancy: Can you provide a detailed explanation for the substantial performance drop of the aTLAS baseline (77.60% vs. 84.98% in the original paper) and the zero-shot base model (37.99% vs. 48.14%)? My assessment would significantly improve if you can demonstrate that your experimental setup is fair and the reported improvements are based on a correctly reproduced and competitive baseline.\n2.\tComprehensive Comparison and Trade-off Analysis: To establish the significance of VTvC, it is crucial to compare it against other relevant methods like AdaMerging, and a data-free method like WUDI-merging. Could you provide a discussion and, ideally, empirical results that analyze the trade-offs between your method and these baselines across multiple dimensions: (a) final accuracy, (b) merging cost (time/compute), (c) scalability to larger models, and (d) data dependency?\n3.\tBroadening the Experimental Scope: The current evaluation is limited to task addition on CV datasets. To demonstrate the generality of your method, would it be possible to provide results on other standard task arithmetic benchmarks, such as task negation? Additionally, could you show its applicability to the NLP domain, for instance, by conducting merging experiments on small models like RoBERTa-base? This would greatly strengthen the paper's empirical contribution.\n4.\tJustification for Sample-Specific Complexity: The proposed sample-specific approach introduces significant computational overhead during the merging phase. Could you elaborate on the specific scenarios or types of tasks where this high cost is justified and where simpler, task-level merging methods would fail? A clear justification for this complexity would strengthen the paper's contribution.", "limitations": "The authors include a \"Limitations\" section in their conclusion. However, it does not adequately address the most critical limitations of the work. The paper mentions high GPU memory requirements but frames it as a general issue for combining many task vectors. It fails to explicitly state that its own sample-specific mechanism is the primary driver of this high cost and the core reason for its poor scalability compared to other methods. The discussion should be more transparent about the trade-offs, specifically acknowledging that the proposed method is likely not feasible for large-scale models (e.g., LLMs) and quantifying its merging cost relative to more efficient alternatives.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tOriginality: The paper introduces an interesting perspective by applying variational inference to the task vector composition problem. Framing the combination coefficients as sample-specific latent variables and using a Bayesian framework to estimate them is a departure from conventional deterministic or heuristic approaches in model merging.\n2.\tPrincipled Approach to Sparsity: The use of a Spike-and-Slab prior to induce sparsity is a well-grounded, principled technique. This is a more elegant and systematic approach than simple thresholding and is a clear strength from a methodological standpoint.\n\nWeaknesses:\n1.\tQuality & Reproducibility: The quality of the experimental evaluation is severely compromised by major, unexplained discrepancies in the baseline results.\n     o\tThe paper reports that its primary baseline, aTLAS [1], achieves 77.60% average accuracy on ViT-B/32. However, the original aTLAS paper reports 84.98% for the identical setup.\n     o\tEven more concerning is the 10+ percentage point drop in the zero-shot performance of the base model itself (37.99% in this paper vs. 48.14% in the original aTLAS paper).\n     o\tThese significant and unaddressed discrepancies cast serious doubt on the validity of the entire experimental framework and render the paper's claims of outperformance unconvincing.\n2.\tSignificance & Scope of Evaluation: The paper's contribution is significantly diminished by its insufficient experimental scope and its failure to compare against a sufficient set of relevant, contemporary baselines.\n     o\tLimited Task Diversity: The evaluation is confined to a single task type: task addition for image classification. To demonstrate the general applicability of a model merging framework, it is essential to evaluate it on a broader range of tasks. For example, experiments on task negation, a standard benchmark in task arithmetic, are missing. Furthermore, the evaluation is limited to the vision domain; demonstrating performance on NLP tasks, even on smaller models like RoBERTa, would greatly strengthen the paper's claims of generality.\n     o\tLack of State-of-the-Art Baselines: The comparison is almost exclusively limited to a single method (aTLAS). methods like AdaMerging [2], and data-free approaches like WUDI-merging[3] are completely ignored. This lack of a comprehensive comparison makes it impossible to assess the actual significance and standing of VTvC in the current landscape.\n3.\tQuality & Clarity on Practicality: The paper avoids a critical discussion of its own practical limitations.\n     o\tThe claim of having \"no additional inference costs\" is misleading, as this is standard for most task-vector merging methods.\n     o\tThe paper fails to acknowledge or quantify the prohibitive merging cost of VTvC, which requires a forward pass through an inference network for every sample. This makes the method far more computationally expensive than its peers.\n     o\tDue to this high cost, the method is not scalable to large language models (LLMs). This severe limitation is not addressed, raising questions about the practical utility and significance of the proposed framework.\n[1] Zhang, Frederic Z., et al. \"Knowledge composition using task vectors with learned anisotropic scaling.\" Advances in Neural Information Processing Systems 37 (2024): 67319-67354.\n[2] Yang E, Wang Z, Shen L, et al. Adamerging: Adaptive model merging for multi-task learning[J]. arXiv preprint arXiv:2310.02575, 2023.\n[3] Cheng, Runxi, et al. \"Whoever started the interference should end it: Guiding data-free model merging via task vectors.\" arXiv preprint arXiv:2503.08099 (2025).", "was_revised": true, "final_justification": "During the rebuttal phase, the authors clarifies most of my concerns by more experimental data facts.", "rebuttal": "Thank you for your thoughtful feedback and for acknowledging the sound theoretical formulation involving our variational framework and Spike‑and‑Slab prior. We address your questions as follows.\n\n**Q1: Baseline Performance Discrepancy.**\n\nThank you for mentioning the performance discrepancy, and we elaborate on the reason for you now. First of all, our experimental setup is fair as all models are compared under the same experimental conditions, including the same datasets, the same preprocessing procedures, and the same evaluation metrics.\n\nThe performance discrepancy is mainly due to the data augmentation strategies we used in the original manuscript. As we stated in  our paper (lines 264-266): \n\n> \"To improve computational efficiency, we precompute feature representations for all datasets and apply data augmentation to generate multiple feature versions, thereby constructing a more diverse training set.\" \n\nThis augmentation strategy is static and relatively weak compared to the dynamic data augmentation strategy used by the original aTLAS method (applying random transformations to each sample in real-time during training) and therefore caused accuracy decline. We adopted the static one purely for the sake of computational efficiency, and it presumably affects the performance of all models equally. \nTo demonstrate this, in this response, we conduct experiments using the same dynamic data augmentation method as the original aTLAS paper. The results are shown in the following table:\n\n|Method|ViT-B/32|ViT-B/16|ViT-L/14|\n|-|:-:|:-:|:-:|\n|Zero-shot|48.22|55.33|64.72|\n|aTLAS|84.70|85.99|91.54|\n|VTVC **(Ours)**|**87.45**|**90.97** |**93.89**|\n\nResults show that after restoring the original augmentation strategy, the ViT-B/32, ViT-B/16, and ViT-L/14 models improved by **2.75%**, **4.98%**, and **2.35%** respectively compared to aTLAS. This proves that, regardless of experimental settings, our method consistently performs better.\nWe will supplement the complete experimental results under identical data augmentation conditions in the revised version of our paper. We hope this addresses your concerns about the performance discrepancy.\n\n**Q2: Comprehensive Comparison and Trade-off Analysis.**\n \nWe follow your suggestion and have conducted additional experiments with related methods for a comprehensive comparison. The results are reported as follows: \n\n|Methods|Cars|DTD|EuroSAT|GTSRB|MNIST|RESISC45|SUN397|SVHN|Avg.|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Zero-shot|59.63|44.13|45.74|32.60|48.26|60.27|63.53|31.63|48.22|\n|Task Arithmetic(ICLR’23)|60.23|55.18|80.26|66.21|95.94|69.17|64.09|76.27|70.92|\n|AdaMerging(ICLR’24)|56.90|50.10|83.40|82.40|95.70|73.10|60.80|87.30|73.71|\n|WUDI-Merging(ICML’25)|70.88|71.53|**95.74**|**93.12**|98.01|86.21|**72.35**|**92.01**|84.98|\n|VTVC **(Ours)**|**76.02**|**86.88**|95.48|90.57|**98.24**|**91.35**|71.19|89.53|**87.45**|\n\n**(a) Final accuracy**\n\nOn 8 visual benchmarks, VTVC achieves **state-of-the-art performance** with an average accuracy of 87.45%.\n\n**(b) Merging cost (time/compute)**\n\nThe training and inference costs of the four methods (based on 1000 samples, running on a single NVIDIA A40 with a ViT-B/32 model and batch size 64) are shown in the table below: \n\n|Method|Training cost|Inference cost|\n|-|:-:|:-:|\n|Task Arithmetic(ICLR'23)|~0s|120s| \n|AdaMerging(ICLR'24)|~10s|5.6s| \n|WUDI-Merging(ICML'25)|~1s|5.6s| \n|VTVC **(Ours)**|112s|5.6s|\n\nDue to VTVC's structural design, the training stage requires learning a small number of additional parameters ${\\Phi, \\Psi}$. However, during inference, there is no need to traverse composition coefficients, significantly improving inference efficiency. Overall, VTVC trades a slightly higher one-time offline cost for significantly higher accuracy and sample adaptation capability, while maintaining millisecond-level online latency.\n\n**(c) Scalability** \n\nThe scalability discussion for the four methods is shown in the table below: \n\n+ **Larger Models**\n    + *Task Arithmetic*: Inference cost increases linearly with number of layers, grid search becomes even slower.\n    + *AdaMerging*: Number of parameter blocks increases, linear solver matrix becomes significantly larger.\n    + *WUDI-merging*: Computation scales linearly with parameters.\n    + *VTVC(Ours)*: No additional parameters added, inference latency still <1ms.\n+ **More Tasks**\n    + *Task Arithmetic*: Requires complete grid search for $\\lambda$ for new tasks.\n    + *AdaMerging*: Must re-solve all block-level coefficients.\n    + *WUDI-merging*: Direct SVD merging on $\\tau_{new}$.\n    + *VTVC(Ours)*: Only requires few iterations of fine-tuning $\\Phi$ to quickly absorb $\\tau_{new}$.\n\nThe results in Figure 5 and Appendix C.1 show that our method VTVC achieves better gating effects and model merging performance when transferred to larger backbone networks with only minimal additional cost. For new tasks, there is no need to traverse composition coefficients again, maintaining fast inference time.\n\n**(d) Data dependency**\n\n+ *WUDI-merging*: Completely data-free, most lightweight, but cannot optimize for specific samples.\n+ *AdaMerging*: Requires small amounts of validation data to estimate block-level coefficients, low data threshold, but lacks sample-level adaptation.\n+ *VTVC(Ours)*: Requires rich validation data to learn coefficient mapping, suitable for data-rich or continuous incremental learning scenarios.\nTherefore, if pursuing highest accuracy and cross-domain generalization, VTVC's data investment is reasonable and necessary. In extremely data-scarce scenarios, WUDI-Merging or AdaMerging can be chosen as cost-accuracy trade-offs.\nWe will provide complete experimental configurations, cost calculations, and validation results in the revised version, and highlight the above trade-off analysis in the main text discussion.\n\n**Q3: Broadening the Experimental Scope.**\n\nWe strongly agree with your viewpoint and have promptly conducted supplementary experiments on task negation and the NLP domain. The specific results are as follows:\n\n+ **Task negation**\n\n    In the task negation setting, the model aims to forget the target task as much as possible while maintaining control tasks within a certain accuracy range (>95%).\n\n    For *task negation*, we introduce an asymmetric gated sampling mechanism. This mechanism sets lower thresholds for target task coefficients while protecting important components from being filtered out. The experimental results for task negation using ViT-B/32 are as follows: \n\n    |Methods|Target($\\downarrow$)|Control($\\uparrow$)|\n    |-|:-:|:-:|\n    | Task Arithmetic(ICLR’23)|22.62|60.73|\n    | TIES-Merging(NeurIPS’23)|20.93|**61.52**|\n    | TaLoS(ICLR’25)|17.28|61.33|\n    | aTLAS(NeurIPS’24)|19.42|61.31|\n    | VTVC **(Ours)**|**16.75**|60.84|\n\n    Our experiments show that the asymmetric gating mechanism achieves better forgetting effects on target tasks while keeping control tasks within the required performance range. This demonstrates that our method works well not only for task addition but also for task negation. \n\n+ **NLP domain**\n\n    For *discriminative language tasks*, we use RoBERTa as the backbone and evaluate on the 8-task GLUE benchmark. We will provide detailed experimental settings and more comprehensive results in the revised version. \n \n    |Methods|COLA|STS-2|MRPC|STS-B|QQP|QNLI|MNLI|RTE|Avg.|\n    |-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n    |Zero-shot|0.00|53.76|85.01|4.01|37.48|53.05|37.09|71.19|42.70|\n    |Weight Averaging|0.00|59.21|85.79|46.99|45.37|63.94|48.00|71.19|52.56|\n    |Task Arithmetic(ICLR'23)|8.35|88.26|**89.57**|32.84|82.03|85.40|75.54|80.43|67.80|\n    |TIES-Merging(NeurIPS'23)|31.76|88.86|86.18|10.94|61.05|**85.94**|**83.01**|69.56|64.66|\n    |DARE(ICML'24)|0.00|88.14|86.61|30.19|**84.33**|79.09|63.95|77.16|63.68|\n    |VTVC **(Ours)**|**80.60**|**90.60**|88.48|**74.37**|80.40|83.20|74.00|**83.80**|**81.93**|\n\n    The experimental results further demonstrate that VTVC is also applicable to discriminative NLP tasks, showing cross-modal generality and robustness.\n\n**Q4: Justification for Sample-Specific Complexity.**\n \nWe acknowledge that sample-level composition introduces additional computational overhead, primarily during inference-time coefficient prediction. However, our model amortizes this cost via a lightweight inference network, which outputs coefficients in a single forward pass and does not require iterative optimization. The results in **[Q2]** show that compared to task-level methods commonly used in existing task arithmetic work, VTVC can complete coefficient learning with minimal time during the inference stage after training, without trailing composition coefficients.\nMeanwhile, as illustrated in Figure 2 of our paper, sample-level methods possess the ability to capture and identify differences between samples that task-level methods cannot achieve. Experimental results demonstrate that a more fine-grained use of sample features can significantly improve model merging performance. Based on this finding, we believe the following scenarios are more suitable for deploying sample-level methods:\n1. **Online inference/edge deployment scenarios** with mixed input sources, where sample-level methods can capture distinctive features of new data and provide more accurate predictions.\n2. **High-risk or continual learning scenarios** (such as medical imaging, autonomous driving) that require evaluating decision confidence, where our method can smoothly absorb new domain knowledge while maintaining performance on old tasks.\nBased on the above discussion, we believe the additional cost of sample-level methods is controllable in many real-world complex scenarios and brings significant accuracy and generalization benefits.\n\n**Thank you again for helping us improve the paper, and hope our response can resolve your concerns!**"}, {"review_id": "FQND75wJRt", "reviewer": "Reviewer_HsZa", "summary": "This paper proposes a variational Bayesian framework for sample-specific task vector composition, improving over traditional task-level methods in model merging. Experiments show consistent performance gains across diverse datasets. The key contributions are: \n1. Variational inference formulation with amortized inference for per-sample coefficient estimation. \n2. A sparse Spike-and-Slab prior to eliminate redundant task components and improve interpretability. \n3. A gated sampling mechanism to deterministically select reliable components based on uncertainty and importance, enhancing inference stability.", "questions": "1. In line 112 and Eq.(1) you listed the formulation of the classic task vector composition. Given a task vector pool $\\mathcal{T}$ and composition coefficients $\\mathbf{z}$, for your sample-level composition, what is the formulation of the merged model? \n2. Some common notations are not clearly defined in the context of task arithmetics, making it difficult to check the soundness of work. When you say input $\\mathbf{x}$ and labels $\\mathbf{y}$, is this for the target task’s input and labels or all the tasks that are used to generate the task vector pool $\\mathcal{T}$? Similarly, in equation (12), what is $\\mathcal{D}_t$? If $\\mathcal{D}_t$ is the target task, is it fair to compare formulation in equation (12) where you are allowed to optimize the coefficient specifically for a task in the task pool? In task addition, isn’t it like we are using a single copy of the task coefficients and evaluating the performance of that single merged model on all tasks for evaluation?\n3. Can you explain the motivation of Eq. 11 more? What can you represent $\\mathbf{z}$ as deterministic functions of input $\\mathbf{x}$? Seems like $\\mu, \\sigma, f$ are all parameterized by $\\Phi$, how are these related? Also, isn’t $\\mathbf{z} = f_\\Phi(x)$ very rare to happen if $\\mathbf{z}$ is continuous?\n4. Can you clarify the definition of gated ratio? This makes Figure 4 hard to read, and in Figure 4, how to read “while activating fewer task coefficients than the Gaussian prior” from the figure? The accuracy comparison is easy to understand. \n5. Table 1: how is deterministic-based sample specific different from probabilistic-based sample specific row? Is probabilistic-based task-level row also [42] or something else? It’s better to emphasize which row is your method. Another suggestion is to mention [42] is one of the SOTA methods so you use it as the baseline.", "limitations": "Yes, in the conclusion section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength\n**Quality**: The submission is mostly technically sound and the methods are appropriate. This is a complete piece of work. Authors analyzed both strengths and weaknesses fairly in the paper. Claims are well-supported. \n\n**Clarity**: This paper is well-organized. The authors put some important mathematical derivation in the Appendix which is very helpful for understanding of the methodology. \n\n**Significance**: Once the questions below are sufficiently addressed, I foresee the result in this paper will be very impactful for the community as this paper provides a new formulation of task vector addition and it’s easy for researchers to build on top of it. Besides, for scalability, the authors proposed a controllable deterministic solution that enables gradient based training that should be easily adapted into more modern deep learning frameworks. \n\n**Originality**: This paper provides a novel perspective of the task arithmetic from variational inference. It is very different in nature from all other existing papers to choose task vector coefficients and the use of variational inference is well-motivated by the necessity to include uncertainties. This paper further improves the interpretability of task vector coefficients.\n\n## Weaknesses\n**Clarity & Quality**: Some key components of task arithmetic are not clearly explained. Since this paper provides a sampled based task vector composition framework, which is very different from the mainstream literature, it’s better for authors to clearly and mathematically define the formulation of sample based task vector and analyze the difference in a more straightforward way, potentially not by just text. The weaknesses in the clarity also make it hard to justify the soundness of methodology, thus hurting the quality judgement of the paper. See questions section below for more details.", "was_revised": true, "final_justification": "After discussions with the authors, I still have a concern about the scalability of this methodology due to the prohibitively high training time, but I think the methodology itself by nature, is novel and interesting enough, and the variational framework is technically sound even though the submitted version contains many unexplained notations. However, technical questions are fully addressed in the discussion period. I hope authors include a thorough discussion section about scalability comparison with other methods in the next revision, and mention the limitation of training time in the paper as a warning, which could lead to future follow-ups.", "rebuttal": "We sincerely appreciate your constructive comments and recognition of our method’s novelty, significance, and potential scalability. Below, we provide clarifications regarding the mathematical formulations and soundness of the methodology.\n \n**Q1: In line 112 and Eq. (1) you listed the formulation of the classic task vector composition. Given a task vector pool $\\mathcal{T}$ and composition coefficients \\$\\mathbf{z}\\$, for your sample-level composition, what is the formulation of the merged model?**\n\n**A1:** The merging formulation for our sample-level composition extends the task-level definition by making the composition coefficients \\$\\mathbf{z}\\$ input-dependent, *i.e.*, \\$\\mathbf{z}(x)\\$. Specifically:\n* **Non-block case**:\n$$\n    \\theta(x) = \\theta_0 + \\sum_{i=1}^N \\mathbf{z}_i(x) \\cdot \\tau_i,\n$$where each task vector \\$\\tau\\_i\\$ is assigned a scalar coefficient \\$\\mathbf{z}\\_i(x)\\$ specific to input \\$x\\$.\n* **Block case**:\n$$\n\\theta(x) = \\theta_0 + \\sum_{i=1}^N \\sum_{j=1}^M \\mathbf{z}\\_{ij}(x) \\cdot \\tau_{ij},\n$$where each task vector \\$\\tau\\_i\\$ is divided into \\$M\\$ parameter blocks, and each block \\$\\tau\\_{ij}\\$ has its own coefficient \\$\\mathbf{z}\\_{ij}(x)\\$ conditioned on the input \\$x\\$.\nWe will clarify these formulations in the revised paper (Section 3, after Eq. (1)) to explicitly distinguish the sample-level case.\n\n**Q2: Some notations related to task arithmetics are unclear, making it hard to assess correctness.**\n\n**A2:** Thank you for pointing out the potential confusion in notation. We clarify the notations and design choices below:\n1. **Meaning of $\\mathbf{x}$ and $\\mathbf{y}$**: In our formulation, \\$\\mathbf{x}\\$ and \\$\\mathbf{y}\\$ refer to arbitrary input-label pairs sampled during the composition training stage. They are **not** tied to any specific target task, but are drawn from the joint validation set used for learning composition parameters.\n2. **Definition of \\$\\mathcal{D}\\_t\\$ in Eq. (12)**: \\$\\mathcal{D}\\_t\\$ denotes the **joint training data** used to optimize the variational components \\$\\Phi\\$ and \\$\\Psi\\$ during composition. Specifically, we construct \\$\\mathcal{D}\\_t = \\bigcup\\_{i=1}^8 \\mathcal{D}\\_i^{\\text{val}}\\$, i.e., the union of the validation splits from all 8 datasets. We uniformly sample mini-batches from this set to train the gating components under the variational objective.\n3. **On “optimizing coefficients for a single task”**: Our method does **not** tune coefficients per task individually. The model learns **a single pair of shared networks** \\$\\Phi\\$ and \\$\\Psi\\$ to produce sample-level coefficients based on \\$\\mathbf{x}\\$, using **multi-task supervision** from \\$\\mathcal{D}\\_t\\$. During evaluation, the same shared components and task vector pool \\$\\mathcal{T}\\$ are used for all tasks without any task-specific tuning. Thus, Eq. (12) does not confer any advantage to a particular task. In contrast to classical task addition, where one fixed coefficient vector is shared across all samples, our formulation dynamically generates composition coefficients per input. However, the model remains shared and fair across tasks. We will clarify these points in Section 3.3 and the caption of Eq. (12) for clarity.\n4. **Whether evaluation still uses \"one merged model\"**: Yes. During inference, the model parameter set consists of $\\Theta_0$, task vector pool $\\mathcal{T}$, and trained $\\Phi, \\Psi$—all three are frozen. For any test sample (regardless of which task it belongs to), the model first generates $\\mathbf{z}$ through $\\Phi$, obtains increment $\\Delta\\Theta$ through gating, and completes prediction with one forward pass. **The entire process no longer updates weights**, all tasks share the same network, only $\\mathbf{z}$ varies with samples.\nWe hope this clarification of the training process helps you better understand our work.\n\n**Q3: Can you explain the motivation of Eq. (11) more? What does it mean to represent $\\mathbf{z}$ as a deterministic function of input $\\mathbf{x}$? Since $\\mu$, $\\sigma$, and $f$ are all parameterized by $\\Phi$, how are they related? Also, isn't \\$\\mathbf{z} = f\\_\\Phi(\\mathbf{x})\\$ very unlikely to happen if \\$\\mathbf{z}\\$ is continuous?**\n\n**A3:**\nWe provide a comprehensive clarification for each sub-question:\n\n1. **Motivation of Eq. (11)**.\n+ Eq. (11) transforms the traditional \"**obtain $\\mathbf{z}$ via sampling** $\\rightarrow$ then gate\" approach into \"**first give $\\mathbf{z}$ a deterministic point estimate** $f\\_\\Phi(x)$ → then use gating $\\Omega$ to filter by uncertainty\". The core motivation is: in high-dimensional sparse scenarios, directly performing *Bernoulli-Gaussian mixture sampling* on each dimension has high variance, heavy computation, and difficult convergence. By folding the randomness of the sampling process into a continuous, differentiable gating function, we can achieve **differentiability, stability, and sparse selection** simultaneously.\n\n2. **Why can z be represented as a deterministic function $f\\_\\Phi(\\mathbf{x})$?**\n+ From the variational inference perspective, we freely choose the approximate posterior family. We set: $q\\_{\\Phi,\\Psi}(\\Omega,\\mathbf{z}|\\mathbf{x}) = q\\_\\Psi(\\Omega|\\mathbf{z})\\delta\\_{\\mathbf{z}=f\\_\\Phi(\\mathbf{x})}$ Using Dirac $\\delta$ to represent \"posterior concentrated at point $f\\_\\Phi(\\mathbf{x})$\". This is not a random event of \"continuous variable hitting exactly one point\", but defines a **new point-mass posterior family**. Randomness and uncertainty are handled by $\\Omega$.\n+ $f\\_\\Phi(\\mathbf{x})$ typically takes the mean $\\mu_\\Phi(\\mathbf{x})$ of the original Gaussian posterior. This retains $\\mu,\\sigma$ for subsequent uncertainty measurement while avoiding high-variance sampling of $\\mathbf{z}$.\n\n3. **How are $\\mu, \\sigma, f$ related when all parameterized by $\\Phi$?**\n+ The neural network parameterized by $\\Phi$ outputs three sets of tensors:\n    1.  **$\\mu_\\Phi(\\mathbf{x})$** serves as the value of $f_\\Phi(\\mathbf{x})$, *i.e.*, composition coefficient point estimate\n    2. **$\\sigma_\\Phi(\\mathbf{x})$** used for computing gradient sensitivity + distribution deviation, participating in constructing uncertainty $U$\n    3. $\\pi\\_\\Phi(\\mathbf{x})$ for Bernoulli probability if using stochastic gating\n\n    Since all three share forward features, computational cost is low and gradient updates are consistent, avoiding multiple network calls.\n\n4. **Isn't \"$\\mathbf{z}=f_\\Phi(\\mathbf{x})$\" with zero probability unreasonable?**\n+ The Dirac $\\delta$ here is a **modeling choice for the posterior family**, not an actual sampling event; thus there's no paradox of \"zero probability yet happening\".\n+ The KL term in ELBO can still be computed: \n$$\nD_{KL}(\\delta_{f\\_\\Phi(\\mathbf{x})} || p(\\mathbf{z})) = -\\log p(\\mathbf{z}=f\\_\\Phi(\\mathbf{x})) + const\n$$ \nThis can be viewed as regularization on prior probability density, still providing constraints.\n\n**Q4: Can you clarify the definition of gated ratio?**\n\n**A4:** We clarify as follows:\n\n+ **Definition of gated ratio**: In the Spike-and-Slab prior setting (Sec. 4.2), each coefficient $\\mathbf{z}\\_{ij}$ is associated with a binary gating variable \\$\\omega\\_{ij} \\in \\lbrace0, 1\\rbrace\\$. If \\$\\omega\\_{ij}=1\\$, the coefficient is active and participates in model merging; otherwise, it is masked to zero. The **gated ratio** is defined as the average proportion of active coefficients:\n$$\\text{Gated Ratio} = \\mathbb{E}\\_{i,j}[\\omega\\_{ij} = 1].$$\nFor example, a gated ratio of 0.7 means that only 70% of coefficient blocks are used during merging.\n\n+ **How to interpret Figure 4:**\n\n    + In Figure 4, we plot the accuracy (*blue and orange lines*) and the gated ratio (*green line and shaded area*) across eight datasets for three backbone models. The Spike-and-Slab prior (*orange line*) consistently achieves higher accuracy than the Gaussian prior (*blue line*), while the green curve shows that it uses fewer coefficients (*i.e.*, a lower gated ratio). The precise numbers are reported in Appendix C.2.\n\n   We will revise the main text and caption of Figure 4 to make this explanation clearer.\n\n\n**Q5: The difference between deterministic-based sample specific and probabilistic-based sample specific row and writing suggestions.**\n\n**A5:** To address your confusion, we clarify the meaning of each row as follows:\n\n| Method              \t| Task-level   | Sample-specific  | \n|-|:-|:-|\n| *Deterministic*       \t| aTLAS\t|Det-SS **(Ours)** \t| \n| *Probabilistic*             \t| Prob-TL **(Ours)**\t|Prob-SS **(Ours)** \t| \n\n+ ***Deterministic vs. Probabilistic***\n    + **Deterministic-based**: Directly uses point estimation $\\mathbf{z}=f_\\Phi(\\mathbf{x})$ without random sampling; for sample-level, it then applies gating function $\\Omega$ for sparse pruning.\n    + **Probabilistic-based**: Establishes complete variational posterior $q(\\mathbf{z}|\\mathbf{x})$ for $\\mathbf{z}$, uses sampling/expectation during training, explicitly retains noise terms and KL regularization.\n+ ***Task-level vs. Sample-specific***\n    + **Task-level**: Same task shares one set of coefficients; \"Prob-TL\" is our control implementation that restricts variational inference to task level.\n    + **Sample-specific**: Coefficients vary with input; both \"Det-SS\" and \"Prob-SS\" are sample-level methods proposed in this paper, where **Prob-SS is our main method**.\nWe will clearly identify our method in the revised version and explain why we use aTLAS as the baseline.\n\n**Thank you again for helping us improve the paper and hope our response can resolve your concerns!**"}, {"review_id": "3waHR2bBJD", "reviewer": "Reviewer_Xxkm", "summary": "The paper proposes a variational approach for task vector composition. Given a pre-trained model and target tasks, the main idea consists of sampling the weights’ difference to be added to the pre-trained weights to deliver a stronger set of weights that adapt to multiple downstream tasks. As a baseline, a simple Gaussian parameterization to the posterior is introduced, and then a spike-and-slab prior to enforce sparsity in the parameters to be updated is introduced, to enable task-specific parameter selection. Finally, the paper introduces a gating approach to dealing with the high-dimensionality of the weights and the problems it induces in Monte-Carlo sampling. The experimental setup follows prior work, showing results across 8 datasets, and 3 different ViT models.", "questions": "My questions are all included in the Strengths and Weaknesses section.", "limitations": "The authors mention the high GPU demand as a limitation of their approach, although I would like to see, as I expressed before, a proper breakdown on that limitation. Other than that limitations are properly addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nThe paper extends the prior work of aTLAS ([42] Zhang et al “Knowledge Composition using Task Vectors with Learned Anisotropic Scaling”, NeurIPS’24) by reformulating the task arithmetic in a variational manner. Using this approach, the task vector can be obtained directly in a sample-specific manner, showing that this approach results in better performance than when using task-specific vectors. \n\nThe main idea of the paper is interesting, well-motivated and developed. Although it requires some extra effort to match the pieces, the paper follows a proper story with a clear idea, that is, to have a sampling-based approach, uncertainty-aware, to producing task-specific vectors that enable the same neural network to deliver strong results without the need of retraining in downstream tasks. \n\n\nWeaknesses:\n\nThe first shortcoming of the paper is its writing, with loose narrative and confusing notation. For instance, the authors refer first to z as the latent representation of a CLIP model, to use it immediately after to refer to the task vector coefficients. The paper needs to be polished with proper notation and definitions presented forehand. For me to understand what the authors were trying to accomplished, I had to go read first the aTLAS paper. The paper should be a bit more self-contained.\n\nWhile the contribution is neat, it is not innovative; applying a Bayesian approach to an existing pipeline is a recurrent idea, and considering that it in this case it comes with a marginal advantage over existing alternatives, it is hard to ponder the contribution of the paper.\n\nI am a bit concerned in regards to the experiments. I find the experimental setup a bit ill-defined. In particular, I miss the following details:\n-\tThe training details: while the authors comment the datasets they use, they barely refer to the “standard benchmarks”. They depart from the pre-trained ViT models and my understanding is that they try to learn the network for parameter sampling based on the corresponding datasets. What it is not clear to me is if they trained a single model using all training partitions from the corresponding datasets, or if they trained a single model for each dataset using the corresponding trainining partitions. In the paper it is suggested at some point that gathering a strong set of task vectors to form a single model can result in strong results, however it doesn’t seem to be the case in this scenario. May the authors comment on that? It would be good to see a one-leave-out set of experiments where possible, considering that the model merely consists of a CLIP vision encoder. \n\n-\tA comparison against prior work like aTLAS is missing. \n\n-\tTable 1 shows that having a sample-specific approach clearly outperforms having a task-specific approach. However, the difference between the deterministic and probabilistic approaches is quite marginal, which is concerning. \n\n\n-\tThe same concern shows in Figure 4: the difference between using a simple Gaussian prior w.r.t. using a Spike-and-Slab prior seems marginal to me, putting under question what the contribution of the latter is in the whole framework. Is it indeed a necessary prior? \n\n-\tWhat is the computational complexity of having a per-sample task vector against having task-specific weights? I understand that the former approach will incur in multiple inference steps that are not necessary in the latter approach. A tradeoff or balance between gain and complexity should be also included in the experimental section.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns and provided the other reviewers' comments and consideration for the paper, I am inclining towards accepting the paper. I therefore keep my original rating, hoping for the reviewers to address the gaps in the final version.", "rebuttal": "Thank you for your thoughtful feedback and for noting the clarity of our research motivation and approach. In what follows, we respond to the raised points with clarifications.\n\n**W1: The paper suffers from loose narrative and confusing notation, particularly the use of **z**. It also relies too much on external papers like aTLAS to be understood.**\n\nWe appreciate the reviewer’s feedback and agree that clarity can be improved. In the revised version:\n* We will  **consistently reserve** $\\mathbf{z}$ for task vector coefficients and avoid reusing it for CLIP embeddings (which are now denoted as $h$).\n* We will and a **notation table** and clarify key terms early in Section 3 to make the paper more self-contained.\n* A brief summary of task composition and aTLAS will be included to reduce reliance on external papers.\nThese changes aim to resolve the ambiguity and improve readability.\n\n**W2: The Bayesian angle feels like a standard extension and offers only marginal improvement, making the contribution harder to assess.**\n\nWe appreciate the reviewer’s positive overall evaluation. While we adopt a Bayesian perspective, our motivation is grounded in the specific **challenge of composing task vectors under uncertainty and redundancy**, which is **not addressed in prior works** like aTLAS. Unlike previous methods that rely on deterministic, task-level composition, we introduce a **structured variational framework with Spike-and-Slab priors**, enabling **sample-specific sparse inference** and **controllable coefficient selection**.\n\nThis design is **particularly suited for multi-task transfer with heterogeneous tasks**, where a fixed combination often fails to generalize. Moreover, our **deterministic gating mechanism** avoids the instability of Monte Carlo sampling in high-dimensional weight space—an aspect rarely considered in prior literature. These ingredients not only bring **consistent performance improvements**, but also offer better **sparsity-efficiency trade-offs** and **interpretability**. We hope this clarifies that our contribution is more than a marginal extension.\n\n**W3a: Standard benchmarks.**\n \nWe clarify the task vector acquisition process. We use CLIP-ViT-B/32 as the base model and perform lightweight fine-tuning separately on 8 **public standard datasets** (Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, SVHN), which are the most commonly used visual benchmarks in aTLAS and Task-Vector literature. The fine-tuning settings are consistent with aTLAS, so each dataset corresponds to one task vector.\n\n\n \n**W3b: One-leave-out set of experiments.**\n\nRegarding your question, we have conducted one-leave-out experiments on the ViT-B/32 model(Deterministic-based+task-level), with results shown in the table below:\n\n| w/o Cars | w/o DTD | w/o EuroSAT | w/o GTSRB | w/o MNIST | w/o RESISC45 | w/o SUN397 | w/o SVHN | **Avg.** |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| 73.85  | 74.36 |71.77 |73.89 |70.59 |72.47 |73.04 |72.55 |72.78 |\n\nThe experimental results show that when excluding one dataset, the average accuracy of model merging on the remaining seven datasets decreased by **4.82%** compared to merging eight models. This experimentally demonstrates that expanding the task vector pool can enhance model merging performance. For theoretical supplementation and proof, please refer to Ortiz-Jimenez et al.[5] and Li et al.[6].\n \n \n**Q1: A comparison against prior work like aTLAS is missing.**\n\nThe \"Deterministic-based+Task-level\" results in Table 1 of our paper are actually the aTLAS results. We did not explicitly label them as \"aTLAS\" when writing to distinguish between task-level and sample-level effects, which may have caused confusion in understanding the comparison. We will clearly mark the aTLAS name in the final version and have added comparisons with more prior work including Task Arithmetic[1], AdaMerging[2], and WUDI-Merging[3]. The results are shown in the table below:\n\n| Methods | Cars | DTD | EuroSAT | GTSRB | MNIST | RESISC45 | SUN397 | SVHN | Avg. |\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| Zero-shot   | 59.63  | 44.13 |45.74 |32.60 |48.26 |60.27 |63.53 |31.63 |48.22 |\n| Task Arithmetic[1]   |60.23  | 55.18 |80.26 |66.21 |95.94 |69.17 |64.09 |76.27 |70.92 |\n| AdaMerging[2]  | 56.90  | 50.10 |83.40 |82.40 |95.70|73.10 |60.80 |87.30 |73.71 |\n| WUDI-Merging[3]  | 70.88  | 71.53 |95.74 |**93.12** |98.01 |86.21 |**72.35** |**92.01** |84.98 |\n| VTVC **(Ours)** | **76.02**  | **86.88** |**95.48** |90.57 |**98.24**|**91.35** |71.19 |89.53 |**87.45** |\n\n\n**Q2: The difference between the deterministic and probabilistic approaches.**\n \nThank you for pointing out the superiority of sample-specific approaches in our method. For the probabilistic approach, we can see from Table 1 that there is a **4.45%** improvement at the task level and a **0.98%** improvement at the sample level. However, we notice that on the most challenging dataset SUN397, the ViT-B/32 model shows a **4.96%** improvement with the sample-level probabilistic method compared to the deterministic method.\n\nWe believe one reason for the smaller average improvement at the sample level is that the sample-specific method already fully utilizes each sample's features through learning a lightweight network. However, the probabilistic approach has the natural advantage of quantifying the reliability of output results, which provides the necessary foundation for our subsequent sparse priors and gated sampling approaches.\n\nExperimental results demonstrate that probabilistic methods not only introduce reliability evaluation of results but also further improve model merging performance. Therefore, the introduction of probabilistic methods represents a valuable innovation for model merging.\n\n**Q3: The difference between using a simple Gaussian prior w.r.t. using a Spike-and-Slab prior.**\n\nWe agree that compared to other experiments, the improvement of the Spike-and-Slab prior-based method over the Gaussian prior method is indeed marginal. However, on the most challenging dataset SUN397, the ViT-B/16 model achieves a **2.29%** improvement under the Spike-and-Slab prior, demonstrating that this sparse prior has more potential on more challenging tasks.\n\nTo more clearly show the difference between Gaussian prior and Spike-and-Slab prior results, we present the average accuracy across three models and eight datasets, along with the gated ratio for the Spike-and-Slab prior, as shown in the table below:\n\n| Method              \t| ViT-B/32 | ViT-B/16 | ViT-L/14 |\n|-|:-:|:-:|:-:|\n| w/ Gaussian prior      \t| 84.32\t| 86.61 \t| 90.38 \t|\n| w/ Spike-and-Slab prior          \t| **84.69** \t| **87.42** \t| **90.71** \t|\n| Gated Ratio              | *71.53%* \t| *70.40%* \t|*70.35%* \t|\n\nFrom the table results, we can see that although the Spike-and-Slab prior shows relatively small accuracy improvements compared to the Gaussian prior-based method, it achieves this performance using only around **70%** of activated block coefficients. In contrast, the Gaussian prior learns all block coefficients but causes slight accuracy degradation. This experimentally validates our findings mentioned in Figure 2 that task vector redundancy indeed affects model merging, and also inspires us to design the gated sampling method. Therefore, we believe the Spike-and-Slab prior is necessary.\n\n\n**Q4. The computational complexity and a tradeoff between gain and complexity.**\n\nThank you for your suggestion to evaluate the tradeoff between gain and complexity. We will add this tradeoff analysis in our revision. For intuitive understanding, we compare the complexity between sample-level and task-level approaches as follows:\n\n+ **Parameters**: Sample-level approach adds 0.75M parameters (\\~0.85\\% of backbone); task-level approach only needs to store one set of fixed scalars $\\Lambda$, with negligible parameter overhead.\n+ **Inference FLOPs**: Sample-level requires forward pass through $\\Phi$ first (\\~0.2 GFLOPs), then one ViT forward pass (\\~4.5 GFLOPs), total overhead increase \\~4.4%; task-level only has ViT forward pass.\n+ **Time (A40 GPU, batch 64)**: ViT-B/32 standalone inference 5.6 ms/sample; sample-level composition 5.9 ms/sample, increase of 0.3 ms/sample.\n+ **Benefits**: Average accuracy on eight datasets improved by **5.74%** for deterministic methods and **2.27%** for probabilistic methods.\nThe sample-level approach achieves higher accuracy gains with <5% inference cost, making it a better choice for scenarios requiring the highest cross-domain adaptability.\n\n**References**:\n\n[1] Ilharco et al. Editing models with task arithmetic. ICLR’23.\n\n[2] Yang et al. AdaMerging: Adaptive Model Merging for Multi-Task Learning. ICLR’24.\n\n[3] Cheng et al. Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors. ICML’25.\n\n[4] Ortiz-Jimenez et al. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS'23.\n\n[5] Li et al. When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers. ICLR’25.\n\n**Thank you again for helping us improve the paper and hope our response can resolve your concerns!**"}, {"review_id": "OIdxvtZdfS", "reviewer": "Reviewer_Wknn", "summary": "The paper proposes a method for the integration of task-specific knowledge to be generalized across tasks. The authors formulate a probabilistic model for the task vector composition task and propose a spike-and-slab variational inference framework for efficient training and sample-specific inference. The authors also propose a differentiable and deterministic gated sampling algorithm for training their model. The experiments show that their method improves the classification accuracy across datasets. The authors also empirically discussed the benefits of the gated sampling approach.", "questions": "What is the pretrained model in Fig 2a? \nWhy does MNIST show a very distinct representation but gain a lot from the knowledge integration in Table 1?\nIn the experiments, I understand that the image is the input and the class is the output. Do you finetune the pretrained weights for all the tasks together with the ELBO loss? If so, then coming back to my earlier point, please provide how many trainable parameters you are adding.\nAt the beginning of the experiments, you mentioned, “We select these datasets to evaluate the effectiveness and scalability of our approach on diverse visual and task scenarios.” I think it is incorrect to say that the approach was evaluated on diverse tasks, since only image classification was chosen.\nPlease clarify the difference between a deterministic and stochastic approach, and Task-level vs sample-specific?\nAre the w/o Gated results in Table 2 from the same model and inference approach as the Table 1 final row? If so, then why are the numbers different?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper is clearly written and presents a good motivation through Fig 2 that there are overlaps in task-specific knowledges and a lot of redundant features. The proposed method is intuitive and easy to follow. The experiments show improved overall results for image classification across different datasets over the competitive approach. They further show accuracy improvement due to a sample-specific gated sampling approach, which was the main methodological novelty.\nWeakness:\nI don’t have much experience in the field of knowledge transfer through the task vector arithmetic. I enjoyed reading the paper and the main competitive approach aTLAS. I think the main limitation of this work is the limited novelty of the method over aTLAS. The method is a sparse summation of the task vectors instead of a non-sparse summation in aTLAS. Methodologically, I am not sure how much this change is novel in the ML literature.\nThe notations are confusing. Is z the activations from a neural network? If so, then how can it be added to the parameters in \\Theta_0 in eq 12?\nI liked the deterministic gated sampling algorithm, but it also brings additional parameters to be trained. How is this then beneficial to use in knowledge transfer, where the objective is to fine-une pretrained weights. If one has additional compute budget, then wouldn’t it be easier and accurate to train separate models (classifiers in this case) on separate data?", "was_revised": true, "final_justification": "I am increasing the point as the authors have clarified my doubts about the additional parameter overhead of their approach and also provided additional evidence in support of their method in NLP tasks. However, my other doubt about the incremental novelty of the author's approach vs aTLAS remains.", "rebuttal": "We appreciate your constructive review and for recognizing the clarity of presentation and empirical improvements, we are glad you enjoyed reading the paper. We now offer clarifications and address the concerns raised.\n \n**W1: The main difference from aTLAS seems to be introducing sparsity into the summation. Methodologically, it’s unclear how novel this change is in the ML literature.**\n\nWhile our method indeed builds on aTLAS in spirit, it introduces several **non-trivial methodological departures** tailored for **sample-specific and uncertainty-aware task composition**:\n * We go beyond fixed task-level scaling and instead propose a **variational Bayesian formulation** that **learns input-dependent coefficient distributions** rather than fixed values.\n * Our use of **Spike-and-Slab priors** enables **structured sparsity**, which not only improves interpretability but also reduces overfitting and improves generalization on heterogeneous tasks. \n* We further design a **deterministic gating mechanism** to replace stochastic sampling, which addresses variance and scalability issues commonly encountered in variational inference over high-dimensional parameter spaces.\n* Empirically, this framework consistently outperforms aTLAS while using fewer parameters (see Fig. 4 and App. C.3), indicating that the proposed changes bring measurable benefits beyond architectural variation.\n To our knowledge, these components—**Bayesian sample-level task vector composition with sparse gating**— have not been explored in prior work, including aTLAS. We believe this constitutes a meaningful and task-relevant extension in both the **modeling and inference** aspects.\n\n \n**W2: Notations.**\n \nThank you for pointing out the confusion in our notation. We used $z$ in line 104 to represent the latent representation of input image $x$ from the neural network, and $\\mathbf{z}$ in lines 155-156 to represent the composition coefficients in our variational framework. In Eq. 12, $\\mathbf{z}(\\mathbf{x}_s;\\Phi)$ refers to the composition coefficients for sample $\\mathbf{x}_s$, which are filtered by gating $\\mathcal{G}$ (Eq. 7) and then combined with the task vector matrix $\\mathcal{T}$ to produce weight increments of the same shape as $\\Theta_0$, resulting in the final prediction. We will clarify the symbol definitions in our final revision to ensure a clear distinction of symbol meanings.\n\n \n**W3. Comparison of training separate models on separate data.**\n\nRegarding your concerns about additional computational costs, we provide a detailed discussion:\n+ **Minimal parameter overhead**: The gated sampling model has extremely small parameter requirements. The additional parameters only include $\\Psi=\\lbrace \\psi_1, \\psi_2\\rbrace $ (two scalar thresholds) and a three-layer MLP with parameters $\\Phi$, totaling **<1%** of the backbone model parameters. Compared to training separate complete models for each task, the overhead of our gating mechanism is negligible. Despite this minimal cost, gating significantly improves knowledge transfer performance — achieving **1.35%** improvement over spike-and-slab prior on ViT-B/32, and **3.56%** improvement when applied to the aTLAS method. Moreover, our gated sampling method achieves state-of-the-art results across three models and eight datasets, demonstrating its excellent performance.\n+ **Dynamic knowledge composition**: For your suggestion of training separate models on separate data, multi-task applications often require **dynamically switching or combining knowledge across different domains**. With separate models trained on individual datasets, it is nearly impossible to enable/suppress specific domain features at runtime. Additionally, when adding N new tasks in the future, you would need N large-scale training runs and storage for N complete model weights. In contrast, task arithmetic only requires fine-tuning once per new task and saving task vectors, with minimal storage growth during composition. Our gated sampling mechanism further improves inference efficiency and storage space, providing more flexible choices for task vector composition in multi-task scenarios.\nWe hope this discussion helps you better understand the advantages of gated sampling.\n\n \n**Q1: Pre-trained model in Fig 2a.**\n\n\nThe pretrained model used in Figure 2a is the CLIP-ViT-B/32 image encoder.\n\n\n**Q2: Explanation of MNIST.**\n\nYou made a very careful observation. We believe the possible reason for this interesting phenomenon is as follows: In the t-SNE projection, MNIST clusters appear clearly separated from other datasets mainly because of the huge domain gap between its visual distribution (grayscale, low resolution, handwritten strokes) and natural images. Due to this difference, pretrained features alone perform poorly on MNIST. However, our sample-level composition automatically selects a few task vectors related to \"contours/strokes\" for handwritten digits during inference while suppressing redundant channels. Therefore, MNIST accuracy in Table 1 improves dramatically from 25.23% (using only $\\Theta_0$) to 98.72%, showing the most significant gain.\n\n**Q3: Do you finetune the pretrained weights for all the tasks together with the ELBO loss? If so, then coming back to my earlier point, please provide how many trainable parameters you are adding.**\n\nIn our training process, the pretrained weights are not jointly fine-tuned with the ELBO loss across all tasks. Our process consists of two stages:\n1. **Task vector creation**: We independently fine-tune each dataset to obtain corresponding task vectors, then freeze the pretrained weights $\\Theta_0$ and all $\\tau$.\n2. **Composition stage**: We only train the inference network $\\Phi$(3-layer MLP) and two gating thresholds $\\Psi$ using ELBO loss to predict sample-level composition coefficients $\\mathbf{z}$ and perform gating.\n**Trainable parameter scale**: Using ViT-B/32 as an example, the backbone has approximately 87.8M parameters. The inference network uses a three-layer fully-connected structure, plus two scalar thresholds and a global scaling coefficient $\\lambda$, totaling approximately 0.75M parameters, which is ~0.85\\% of the backbone. The additional FLOPs during inference are also **<5%**. Compared to training $K$ complete separate models for $K$ tasks, our storage and update overhead is significantly reduced.\n\n\n**Q4. Single task scenario.**\n\nThank you for pointing out the imprecise wording in our paper. You are correct – we indeed only selected image classification experiments, and we agree that expanding our method to other task scenarios is important for validating its effectiveness.\nTherefore, we have added experiments on discriminative NLP tasks. The experimental results are as follows: \n\n| Methods           | COLA | STS-2 | MRPC | STS-B | QQP | QNLI | MNLI | RTE | Avg. |\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| Zero-shot   | 0.00  | 53.76 |85.01 |4.01 |37.48 |53.05 |37.09 |71.19 |42.70 |\n| Weight Averaging   | 0.00  | 59.21 |85.79 |46.99 |45.37 |63.94 |48.00 |71.19 |52.56 |\n| Task Arithmetic[1]  | 8.35  | 88.26 |**89.57** |32.84 |82.03|85.40 |75.54 |80.43 |67.80 |\n| TIES-Merging[2]   | 31.76  | 88.86 |86.18 |10.94 |61.05 |**85.94** |**83.01** |69.56 |64.66 |\n| DARE[3]  | 0.00  | 88.14 |86.61 |30.19 |**84.33** |79.09 |63.95 |77.16 |63.68 |\n| VTVC **(Ours)** | **80.60**  | **90.60** |88.48 |**74.37** |80.40 |83.20 |74.00 |**83.80** |**81.93** |\n\nFor discriminative language tasks, we use RoBERTa[4] as the backbone and evaluate on the 8-task GLUE benchmark[5]. We will provide detailed experimental settings and more comprehensive results in the revised version.  \n\n\n**Q5. Clarify the difference between a deterministic and stochastic approach, and Task-level vs sample-specific.**\n\n\n+ **Deterministic vs Stochastic approaches**: Deterministic methods directly provide a single point estimate with unique composition results. Stochastic methods treat coefficients as random variables, explicitly model uncertainty, and complete composition through distributions or sampling.\n+ **Task-level vs Sample-specific approaches**: Task-level methods use coefficients that only depend on the overall \"task\" label, shared by all samples. Sample-specific methods use coefficients that are further influenced by specific inputs, enabling fine-grained adaptation to differences within samples.\n\n\n**Q6. Are the w/o Gated results in Table 2 from the same model and inference approach as the Table 1 final row?**\n \nTable 2 compares method performance under sparse conditions. The w/o Gated results in Table 2 come from the model with spike-and-slab prior, while the final row in Table 1 comes from a sample-level probabilistic model under Gaussian sampling. The difference between these two methods is that **the former uses a spike-and-slab prior to reduce redundancy**. Detailed results can be found in Appendix C.2.\n\n\n**References**:\n\n[1] Ilharco et al. Editing models with task arithmetic. ICLR’23.\n\n[2] Yadav et al. TIES-Merging: Resolving Interference When Merging Models. NeurIPS’23.\n\n[3] Yu et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML’24.\n\n[4] Liu et al. Roberta: A robustly optimized bert pretraining approach. CoRR’19.\n\n[5] Wang et al. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP Workshop BlackboxNLP’18.\n\n**Thank you again for helping us improve the paper and hope our response can resolve your concerns!**"}, {"review_id": "1eBQTfteIS", "reviewer": "Reviewer_ZAnD", "summary": "This paper proposes a novel framework for variational composition of learned task vectors, aiming to improve both performance and efficiency by reducing redundancy and introducing uncertainty-aware combination coefficients.", "questions": "1. How should we implement gated sampling for task negation?\n\n2. If we have irrelevant task vectors, how does a spike-and-slab prior help reduce such redundancy?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The structure and main contributions of this paper are clearly described. The authors provide sufficient background and prior knowledge, which helps the audience better understand the context and motivation behind this work.\n\n2. The proposed methodology is conceptually sound and is empirically validated through experiments on real-world datasets.\n\nWeaknesses\n\n1. The contribution of the gated sampling mechanism appears limited, especially if similar techniques have already been introduced in other frameworks. The paper does not clearly explain any unique challenges or novel adaptations specific to applying this approach in the task vector setting.\n\n2. There is a lack of experimental evaluation on the task unlearning scenario.", "was_revised": true, "final_justification": "The author has addressed my concern regarding the motivation for gated sampling.  I will keep my current rating (4 already as positive) and do not have any further questions.", "rebuttal": "We appreciate the reviewer’s feedback and for highlighting the clear structure and robust empirical validation of our work. Below, we address the questions and concerns.\n \n**W1: The gated sampling mechanism seems limited in contribution. Similar ideas may exist elsewhere, and the paper doesn’t explain what’s uniquely challenging or new about using it in task vector composition.**\n\nWe agree that gating mechanisms have been explored in other works. However, applying such mechanisms to **task vector composition** introduces several **unique challenges** that our method addresses:\n* First, **task vectors operate in a high-dimensional parameter space**, where conventional Monte Carlo sampling leads to high variance and instability[1]. Our gated sampling mechanism avoids this by introducing a **deterministic, continuous gating function**, which allows **gradient-based optimization** while maintaining sparsity and stability.\n* Second, unlike existing gating approaches, our mechanism jointly considers **token-level uncertainty and coefficient magnitude**—a design tailored to the **uncertainty-aware and sparse nature of task composition**.\n* Finally, we show that this design improves **both efficiency and generalization**, especially when combined with Spike-and-Slab priors (see Figure 4 and Appendix C.3), which to our knowledge has **not been explored in prior task arithmetic literature**.\nWe will clarify these contributions and motivations more explicitly in Section 4.3.\n\n \n**W2/Q1: Task negation**\n\nWe agree that evaluating task negation is important for demonstrating the scalability and flexibility of our framework. To address this, we have added new experiments on task unlearning.\nSpecifically, we introduce an **asymmetric gated sampling mechanism** that sets a lower threshold for the coefficients of target (to-be-forgotten) tasks, while preserving important components of control tasks. This allows for effective suppression of target task influence without compromising control task performance. The results on the ViT-B/32 backbone are summarized below:\n\n| Methods           | Target($\\downarrow$)| Control($\\uparrow$)|\n|:-|:-:|:-:|\n| Task Arithmetic[2]   | 22.62%    | 60.73%     |\n| TIES-Merging[3]      | 20.93%    | **61.52%**     |\n| TaLoS[4]             | 17.28%    | 61.33%     |\n| aTLAS[5]             | 19.42%    | 61.31%     |\n| VTVC **(Ours)**       | **16.75%**    | 60.84%     |\n\nThese results show that our method achieves stronger forgetting on the target task while maintaining comparable performance on the control tasks. This demonstrates the method’s applicability to both **task addition** and **task negation** scenarios.\n\n**Q2: If we have irrelevant task vectors, how does a spike-and-slab prior help reduce such redundancy?**\n\nWhen task vectors contain irrelevant vectors or blocks for the current task, the **spike-and-slab prior** reduces redundancy by **pushing unimportant coefficients to exactly 0** through probabilistic modeling. This achieves structured sparsity, reduces redundancy, avoids negative transfer, and lowers computational and storage costs.\n+ The spike-and-slab prior models each combination coefficient as a mixture of **point mass and Gaussian**: $p(\\mathbf{z})=(1-\\pi)\\delta_0(\\mathbf{z})+\\pi\\mathcal{N}(0,\\sigma^2)$,\nhere $\\delta_0$ is the Dirac distribution at $0$ (the \"spike\"), and $\\mathcal{N}$ is the \"slab\". This means the prior **prefers most coefficients to be $0$**, with only a few truly useful dimensions falling in the continuous Gaussian part. As a result, dimensions irrelevant to the current sample/task are naturally pushed to $0$ during learning, effectively \"turning off\" redundant task vectors or their sub-blocks.\n+ In variational inference, each coefficient $\\mathbf{z}_i^j$ has a binary indicator variable $\\omega_i^j$: $\\omega=1$ means keeping that dimension, $\\omega=0$ means setting it to zero. The model uses **factorized posteriors** and jointly infers $\\omega$ and $\\mathbf{z}$. If a dimension contributes little to the likelihood, the optimal posterior pushes mass toward $\\omega=0$ (corresponding to the prior's spike), achieving exact $0$ sparsity. The KL term in the ELBO further penalizes unnecessary non-zeros, compressing redundant dimensions.\nWe hope this clarification helps address your concerns about the spike-and-slab prior.\n\n**References**:\n\n[1] Reichelt et al. Rethinking variational inference for probabilistic programs with stochastic support. NeurIPS’22.\n\n[2] Ilharco et al. Editing models with task arithmetic. ICLR’23.\n\n[3] Yadav et al. TIES-Merging: Resolving interference when merging models. NeurIPS’23.\n\n[4] Iurada et al. Efficient Model Editing with Task-Localized Sparse Fine-tuning. ICLR’25.\n\n[5] Zhang et al. Knowledge Composition using Task Vectors with Learned Anisotropic Scaling. NeurIPS’24.\n\n**Thank you again for helping us improve the paper and hope our response can resolve your concerns!**"}]

Author rebuttal:

Meta-review: This paper proposes a variational Bayesian framework of task vector composition for model merging, where combination coefficients are considered as sample-specific variables. The proposed method uses a spike-and-slab prior for sparsity, by which interpretability and generalization ability is improved. The work is technically solid. The experimental results demonstrate the effectiveness of the proposed method. Applying variational inference to the task vector composition problem is interesting. The scalability of the proposed method is a weakness. By the author response, most of the reviewers’ concerns were addressed. The additional experiments strengthen the paper.
Decision: Accept (poster)