Machine Unlearning via Task Simplex Arithmetic
Authors: Junhao Dong
Hao Zhu
Yifei Zhang
Xinghua Qu
Yew-Soon Ong
Piotr Koniusz
Venue: NeurIPS 2025 poster ()
Keywords: Machine Unlearning
Simplex
VLM
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: VLM unlearning by a closed-form ensemble of infinite number of functions with parameters uniformly sampled from a task arithmetic simplex.

Abstract: As foundation Vision-Language Models (VLMs) unlock fine-tuning on smaller datasets while leveraging large-scale pre-training data, machine unlearning becomes critical in addressing privacy concerns and regulatory compliance. Task vector, representing the difference between parameters of models fine-tuned with and without specific data, is a popular retraining-free unlearning strategy. However, we observe that task vectors exhibit substantial sensitivity to various fine-tuning configurations, resulting in unstable unlearning effectiveness that correlates negatively with the prediction-level variance. While aggregating multiple functions (e.g., VLM with classifier) whose parameters are represented by different task vectors  reduces function variance and improves unlearning, the computational cost of obtaining numerous task vectors and aggregating functions is computationally high. Thus, in order to  capture the space of task vectors induced by diverse fine-tuning strategies, we propose modeling it within the convex hull of $(Q-1)$-simplex whose vertices represent $Q$ task vectors. Although a function ensemble can be formed by sampling numerous task vectors from such a simplex, we derive a closed-form ensemble of an infinite number of functions whose parameters are uniformly sampled from the simplex, enabling efficient function-level task vector ensembling with enhanced unlearning performance. Extensive experiments and analyses across diverse datasets and scenarios demonstrate the efficacy of our method.

Machine Unlearning via Task Simplex Arithmetic
Junhao Dong"?, Hao Zhu?, Yifei Zhang', Xinghua Qu', " and Piotr Koniusz 'Nayyang Technological University, CCARR, IHPC, A STAR, 'Bytedance, ICSI66 'University of New South Wales, Ausstallia National University
Yew-Soon Ong"
Abstract
As foundation Vision-L Models (VLMs) unlock fine-tuning on smaller datasets while leveraging large-scale pre-training data, machine unlearning be- comes critical in addressing privacy concerns and regulatory compliance. Task vector, representing the difference between parameters of models fine-tuned with and without specific data, is popular retraining-fres unlearning strategy. However, we observe that task vectors exhibit substantial sensitivity to various fine-tuning configurations, resulting in unstable unlearning effectiveness that correlates neg- atively with the prediction-leve variance. While aggregating multiple functions (e.g., VLM with classifier) whose parameters are represented by different task vec- reduces function variance and improves unlearning, the computational cost of obtaining numerous task vectors and aggregating functions is computationally high. Thus, in order to capture the space of task vectors induced by diverse fine-tuning strategies, we propose modeling it within the convex hull of (Q-1)-simplex whose vertices represent Q task vectors. Although a function ensemble can be formed by sampling numerous task vectors from such simplex, we derive closed-form ensemble of an infinite number of functions whose parameters are uniformly sam- pled from the simplex, enabling efficient function-lewe task vector ensembling with enhanced unlearning performance. Extensive experiments and analyses across diverse datasets and scenarios demonstrate the efficacy of our method.
Introduction
Vision-Language Models (VLMs), such as CLIP [52] and its derivatives [39, 41] including adversari- ally fine-tuned VLMs [45, 13, 14, 20, 12, 15], enjoy remarkable generalization due pre-training on massive-scale datasets [65]. Given their superior performance on various tasks, including notable robustness single-modal settings [17, 16, 19, 10, 18, 9], significant privacy concerns and regulatory compliance challenges arise regarding data erasure rights mandated by regulations [53, 2]. Conse- quently, developing efficient unlearning models that can effectively "forget" specific data critical to responsible deployment, as simply retraining VL.Ms from scratch is prohibitive [42].
Among machine unlearning approaches [23, 25, 4], so-called task vector arithmetic [29] is very effective plug-and-play strategy. Task vector is defined as the difference of parameters of the model fine-tuned on the target data (e.g., data to be removed) and the original model. The simplicity of task vector arithmetic makes it suitable for large-scale VLMs. Subtracting a task vector from the original model's parameters enables its efficient unlearning without retraining. reduces the model's performance on target dataset (forget ser) and preserves performance on the remainder of the original data. However, the unlearning performance inherently depends on the quality of task vectors. which differ between fine-tuning runs, leading to prediction-level uncertainty due to variance that is unaccounted for in task vector arithmetic.
"Corresponiing authors.
PK also in charge of theory & derivations.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To support our claim, Figure I reveals the relationship between the average unlearning accuracy on the forget
Avg. Forget Set Accuracy [%)
24.50
* ==5
24.25
Q-10
set and the corresponding average prediction-level vari- ance across 8 datasets, computed using function-level en- sembles (VLMs with classifier) with varying numbers of task vectors Q. Specifically, we obtain an ensem-
24.00
11
Q =
23.75
Q=25
23.50
-030
1=1 (1x; 00-0000-AT (see [8]) with vari- ble pa(x) ② -
23.25
23.00
ance (x)) (c)))= E (2;; 0o-0000))) AT;) where f(: 0o) Rin_size + [0, 1 is VLM with original training parameters 0, E _Rpar_ _saze and a SoftMax clas- sifier with C' classes. Let .I...., TQ be Q task vectors obtained by fine-tuning f(, 6o) Q times under different augmentations on the forget set (protocol of [29]). which transforms 0n into 0: for i 1=1,, 1, 1...... Q. Task vectors, de- fined as T. 0 Oo. are constructed by fine-tuning on 8 datasets, and evaluated on the function ensemble by aver- aging the forget accuracy and variance over their test sets. In detail, we fine-tuned/tested on forget datasets (Stanford Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, SVHN). Retain accuracy is obtained on ImageNet. See Section 4. for details.
22.75
22.60
0.16
0.16
0.20
0.22
0.24
Avg. Prediction Variance
Figure Accuracy on forget set vs. class variance w.r.t. the number of task vectors Q used by Q prediction functions that we aggregate. We varied À (defined in text) to sustain the retain set accuracy of 60.4% as per protocol [29].
Our findings indicate a strong negative correlation: increased prediction variance corresponds to poorer unlearning (higher forget set accuracy indicates worse unlearning). Conversely, aggregating additional task vectors into the function-level ensemble consistently reduces prediction variance and improves unlearning efficacy. This finding is supported by the Bienaymé formula under the average
correlation of distinct variables p 0 and a shared variance o², defined as x² x² + (1 (1- (1-2) po²
which dictates that の² decreases as Q grows ifor sufficiently small p. Thus, we note that aggregating multiple functions of different task vectors can lower the prediction variance and improve unlearning.
However, increasing the number of aggregated functions (one per task vector) is costly, as obtaining each task vector requires fine-tuning. Thus, we propose to replace individual task vector arithmetic with the task (Q 1) simipee whose vertices are Q task vectors. Averaging task vectors realizes a multi-task setting [29]. Thus, uniformly sampling a simplex can provide an infinite number of interpolations between task vectors (i.e., interpolation between .. Q task vectors), realizing an infinite number of tasks encapsulated by the task simplex. However, aggregating an infinite number of functions by naively sampling the task simplex is prohibitive. Instead, we leverage Taylor expansion and derive a computationally efficient closed-form expectation over an infinite number of functions realized by infinite uniform sampling of such task simplex. We show that our model leads to the celebrated bias-variance trade-off [33] and we provide the means of penalizing the variance by learning the importance of vertices improving their location within small &a ball of e radius.
Our task simplex unlearning outperforms state-of-the-a art approaches based on task vector unlcarning across 8 datasets. We also show that with task simplex, one can devise a simple incremental unlcarning, allowing gradual removal of data. Our main contributions are summarized as follows:
We analyze unlearning effectiveness vs. function-level (VLM with a classifier) variance in an ensemble of functions of task vectors. As the Bienaymé formula dictates, increasing the number of functions (and thus task vectors) decreases the variance and improves unlearning performance.
ii Motivated by the additive task vector arithmetic, we propose (Q (Q-1 1)-task simplex whose vertices are task vectors. Sampling from such a simplex can yield an interpolated task vector between any number of task vector vertices, facilitating multi-task learning.
iii. As aggregating a large number of functions of task vectors computationally prohibitive, we propose a closed-form ensemble of the infinite number of functions of interpolated task vectors sampled uniformly from the task simplex. To this end, we use 1) Taylor expansion, 2) Jacobian and Hessian of the function evaluated at On. and 3) the expected value over outer products of interpolated vectors sampled from the simplex, obtained from properties of Dirichlet distribution.
iv. We show our closed-form ensemble leads to the bias-variance trade-off by enabling penalizing variance. We also replace the mean aggregation of an infinite number of functions with the probability of at least one success of each class in the infinite trial. Finally, we show that we can distill unlearned parameters from the ensemble (if the final model follows the original VLM).

---

2
Related Works
Machine Unlearning for VLMS facilitates selective removal of specific knowledge or data (forget set) while preserving knowledge of remaining data, and helps address privacy protection and regulatory compliance [2]. This privacy challenge is especially pronounced in VLMs, given their superior generalization due to extensive mre-trained data [42]. Recent efforts in unlearning for foundation VLM include: (i) incremental line-tuning of VLMs on datasets from which the target knowledge is removed [38, 61], (ii) gradient ascent the parameter level to erase target knowledge [63, 57], and (iii) pruning model neurons associated with the target knowledge [43]. Arithmetic on task vector [29], defined as parameter differences between models fine-tuned with and without specific data, enables efficient and scalable unlearning. Ortiz-Jimenez et al. [50] further addressed weight disentanglement within task vectors via model linearization in the tangent space. However, utilizing individual/few lask vectors neglects the models' variability due to diverse fine-tuning, Thus, we propose an efficient ensemble of an infinite number of functions of task vectors captured by the task simplex.
Parameter-level Model Merging can enhance performance by combining parameters from multiple models [64, 32, 22]. A simple strategy involves parameter averaging across fine-tuned models, im- proving task-specific performance without extra computational overhead during inference [56]. Apart from averaging, refined merging of coefficients
be obtained from Fisher information matrices [46] or linear regression [31]. Yadav et al. [59] explored interference among model parameters when merging, addressing by pruning low-magnitude parameter modifications. Yang et al. [60] proposed to adaptively learn merging coefficients in an unsupervised scheme. Related are also Parameter-efficient Fine-tuning (PEFT) models, e.g., LoRA [27], PACE [48], CrossSpectra [66] and BiLoRA [67] whose fine-tuned parameter residuals are task vectors. Distilled//adapee parameters can be also merged [40, 11]. We focus on function-lever ensembles rather than parameter-level merging.
Function-level Ensemble Learning aggregates the function outputs of diverse models, and improves both accuracy and calibration [8, 62]. Simple averaging at the prediction (function) level has been extended to deep learning to enhance generalization across diverse domains under distribution shift [36, 51]. Gontijo-Lope el al. [24] empirically showed that diverse training configurations induce distinct generalization behaviors characterized by uncorrelated errors, thereby improving ensemble performance. Seddriguz--Opazo al. [49] investigated cross- coos-bbcckbone ensemble learning in VLMs based on re-weighting predicted logits. In contrast, we focus on VLM unlearning and lowering the prediction variance by the ensemble of infinite number of functions from the task simplex.
3 Proposed Method
CLIP-V en
0
TAy
CLIP-V
Below, we provide background and introduce a closed-form aggregation of the infinite number of functions (VLMs with classifier) formed from task vectors within the task simplex. We also propose how to distill such an ensemble into one task vector ( VLM with no ensemble).
pre-train
tune
aug.
closed-fom
The task simplex based unlearning. Pa- rameters Og are given from pre-trained model. For Q augmentation ways, we get 0,, PQ on the unlearning set and task vectors T....., To which form slimplex S for function-level ensemble.
Figure 2:
Background. Foundation VLMs, such as CLIP [52] and its recent variants [39, 41] are pre- trained on extensive datasets collected from diverse internet sources. Formally, a VLM with a classifier can be represented as a function f:X' x 0- (0, 1]C, parameterized by set of parameters On R which maps an input x: X to predicted probabilities across C categories. To adapt a pre-trained VLM
specific downstream task i using a single task vector [29], characterized by its dataset D; (e.g.. ImageNet), the pre-trained parameters On are fine-tuned, resulting in task-specific parameters 0,, i> O. Following finnituning, the associated task vector is defined as the parameter difference between the fine-tuned parameters and the original pre-trained parameters: T; = 0, Oo. Depending on the adaptation type, i may refer to specific dataset or the i-th aug. variant of data.
3.1 Problem Formulation
Figure 1 shows that aggregating more task vectors of different fine-tuning configurations on the unlearning set can lower the prediction variance and enhance unlearning performance. We thus propose to capture the space of all possible interpolations of Q task vectors by a (Q 10-11 -1)-task simplex whose vertices are given by (Ti...., TQ). Figure 2 is an overview of our method, detailed below.
3

---

Problem definition. We investigate machine unlearning in the context of VLMs, aiming to remove knowledge of a so-called forget set, while preserving performance across the rest of the data ie., the retain set. Leveraging task vector arithmetic, unlearning can be efficiently achieved by performing task vector negation. Subtracting the scaled task vector from the original parameters pure-training, i.e., 0 = 00 T where À is a scaling factor obtained on a held-out validation set ensure the model's 's accuracy reduction on the forget set while maintaining accuracy the retain set. However, given sample æ Ex,, we are interested in an ensemble of the infinite number of functions (VLM with classifier) of task vectors uniformly sampled from (Q-1) (Q-1)-sipplex Ag:
De /.. م
H (
f(x. 80 (o) dr E [0. 10.
(1)
Task vectors have large dimension, i.e., m = 86M dim. (CLIP ViT-B-16). Given low number of vectors, e.g., Q 30 & m--66M,, due to the curse of dimensionality there exist no obvious way to reliably estimate what distribution type the task vectors follow. However, we choose to model Eq. (1) with the Dirichlet distribution as even for very large m, it facilitates: (i) ease to define its Probability Density Function (PDF) with task vectors as its support parameters, (ii) compact support which does not "exceed" observed minimum or maximum values of individual parameter coefficients, (iii) efficient sampling through our Taylor formulation based on the closed-form 0th, 1st and 2nd order moments.
Closed-form Aggregation of Functions from Task Simplex. To facilitate efficient aggregation in Eq. (1), we employ second-order Taylor expansion of the VLM function f(z; 0g - A around the original parameters Op. Let (Op) E xm denote the Jacobian matrix of [(æ; 0o) for a given input a. Let class-specific Hessian matrices be (H)(0)))...- (H)(0)) c: E mm For sufficiently small scaling factors constraining |10 - - to a small > 0 ball, we have the following second-order approximation: f(x; 0o À r) = f(x; 0o) (Oo)
(1 0 Thus, for a task simplex Ag. the aggregated function-/ ensemble given as:
()(=))= = |
Σ rEDeo
( (o))
Σ TEAp
(00)e A
ΣrrT rEDo
A
Pr.
(o(0o:Ag)
dr
unnosmmablle set
Ae
After simplifying the right-hand part of Eq. (2), we (Ou; Ao) 1 HT However, w
|مo) TEda
in Eq. (2) contains a sum over task vector samples from Ag. Thus, below we derive closed-form W.
Theorem 1. Let S C Tm be a (Q-1)-simples, with vertices II...., TTQ E RR Let matrix H @Rmmm (in our case, a Hessian matrix). Consider an ii.d, uniformly drawing an infinite number of vectors T contained in S. Compute an expected value R HT) which leads to closed-form expression::
Q
w QQQ+1 11)
Σ HT +
H
T.)
(3)
Proof. Every vector 7 enjoys barycentric coordinates T = Ei= a: Ti with a 2 and Σ ai 1. Under the uniform measure on S, the vector a = (an,.... aQ) is distributed as Dir [Q). Thus:
E[as] = .
Eia Q(++)
B[a,a,] = पािना (iff i j).
Now we expand - HT= Ei j==1 0,aj T. HTj, substitute Eq. (4) and compute expectations:
Q Σ
E(T T Σ Ela r HT,
E[a,ay]
-, HT, 1((+1)
T HT: Q(++))
T HT
+/=1
f,j=1
(5)
$
$
"See also expectation over the outer product of vectors S [14].

---

Corollary 1. Let vertices be assigned importance weights, i.e. (wi...., wQ: Ei= W, =1). Let w be a linear interpolation of these weights at location Extending Theorem BLUT H yields:
w' Q(Q + 1)(2 + 2)
Σ (2 + (2:)w)) ( T: HT: +
Σ 1 843
(1+ + wi + w2) ri HT:)
(6)
Proof. See Appendix G.I.
Theorem and Corollary provide closed-form for Eq. (2) under uniform and impor- tance weighting of task vectors, respectively. Weights of importance weighting (or task vectors themselves) may be optimized to lower the variance associated with Eq. (2).
Computing Variance. Using Eq. (2), the variance of ensemble of the infinite number of functions is given as:
o² (x) = (=) (x)
(7)
Advanced Aggregation Scheme. Up to this point, we discussed the average pooling aggregator in Eq. (I) and (2). Another popular aggregator is maximum pooling over a set of votes (outputs of classifier-level functions) [34]. However, the maximum operation cannot be applied over an infinite number of voters. Therefore, we leverage the probability mass function (PMF) of the Poisson Binomial distribution. Firstly, we ask question: what is the probability of zero successes in Q trials? This question is asked under independent Bernoulli trials that are not necessarily identically distributed, i.e,, we treat an output of each function in the ensemble as one probability of success, denoted Pf. For Q trials and class c, readily obtain I pς) and we readily deduce that 1 1- (1-pf) Z max pf is the probability of at least one successes in Q trials and an upper i=l....Q
bound of maximum pooling. Now it remains to extend this result to an ensemble of the infinite number of functions from the task simplex.
Theorem 2. Consider task vector E Ao sampled from task simplex Dg. Let g(x; 0g - XT) = log(1 - f(x: 0o - XT)), where : R - (-o0, 0j converts the prediction into the log space. Notice that | ug (x) (+)== g(0; O, - XT) and thus we have:
(x))) = exp (15olu, = II (1 f(x; - XT)) 2 max f(x; 0p Ar), (8) TE
TED.
E(O.1jC
which is the probability of least one successes in the infinite number of trials, each trial being an output of function f(æ; On Xr) for task vector t E Ag. Volume A is defined in Theorem 3.
Proof. See Appendix G.2.
Theorem proposes an aggregator which returns a high likelihood if (i) many aggregated functions yielded at least weak but consistent belief in class activation across many functions or (ii) at least one aggregated function yielded a very strong belief in class activation. In contrast, the average pooling aggregator may not yield strong response if many voters yield consistent but relatively weak class activations.
Theorem requires evaluation of volume of the task simplex | which provided below.
Theorem 3. Consider (Q (0-11-simples De with vertex list (T), T :T.. and construct a vertex
list matrix offset by Tı, i.e. V= = [T2 T1....: .Q T1] Rmx((21) (Q-1), Then the volume of the simplex Ag is given as:
V det V V
A
(Q 1)!
Proof. See Appendix G.3.
Distillation from Ensemble. Our aggregation of the infinite number of functions requires to be implemented on top of VLM. However, if deployment requires VLM without ensemble, one can distill the task vector from the ensemble as follows:
arg min (opt.)ar st. [|w]]|==1 =1 Ar: s.L. AAArilaSSe,
Σ [[
-
- ,
stop-grad
(x; w;
. ATq))
+
02 x w. ( To +o ATq)) (10)
5

---

where X" is distillation set used without labels, i.e. the training split of unlearning set, 3 0 controls the variance (section below explains importance of bias variinae trade-off in controlling the model complexity), stop_grad means we stop the gradient through computations of f and is single distilled task vector capturing the ensemble. Moreover, "opt." indicates one can optionally optimize over importance weights uz and/or small perturbations AT; of vertices within radius. Notice that we slightly abuse notation of f and o by passing parameters w and ÅT; into them.
Controlling Variance of Ensemble. In order to control the complexity of the unlearned model, one can reduce variance of the model in the so-called bias-variance trade-off [33]. For our ensemble, reducing variance may be achieved as follows:
arg min s.I. | andior AT: ||Atillase,
Σ (
In:
(11)
where optimizing requires early stopping to achieve a desired variance reduction. Notice, Mean Square Loss (MSE) over æ E X" between labels y(æ) and the learner f(x), given as ED,,T [1(x))
f(x;; 0o -0---))333 }|]]
can readily be decomposed into the bias and variance terms:
ED. [1/"(x)) - (x)())) (z) + 72,
(12)
(Bias, ((=)
(f(=()
where (-) is some class function we want to approximate, y(x)= f (x)) where m~,N noise with variance 7² a.k.a. the so-called irreducible error. See Appendix G.4 for further details.
Bias-variance trade-off. Eq. (12) shows that that our ensemble can be interpreted as bias- variance trade-off [33] under different choices of the unlearning task vectors т. While we cannot reduce the bias, we can adjust the variance to improve generalization on the underlying unlearning set. Belkin el al. [1] showed that below the so-called interpolation threshold, high bias and low variance indicate low model complexity. whereas low bias and high variance indicate high model complexity. Above the interpolation threshold, lowering the variance can further increase the function complexity. In either case, variance is key in controlling the ensemble complexity.
Theoretical Analysis. Below we provide additional theoretical analysis of properties of our ensemble. Theorem 4 (Interpolation Bound). Let Rm + RCC be an unlearning function. Let g(-) be L-Lipschitz continuous, i.e., (1g( o(O,) )l1 - L L 0 22, Vi + j.
Given task vectors (T,..... TQ), diameter 1) = max; ||:, T, ll2, and any convex combination == E O,T; where E = =-1,,0 1, 2 0, we obtain:
00(0 - AT) Σ a g AT.) S LAn.
(13)
Proof. See Appendix G.5.
This bound small if (i) task vectors differ by a small diameter v), À is small and g(.) is smooth with low Lipschitz. constant T.. Indeed, these conditions met, e.g., have low 1 due sparse change between task vectors as explained in Appendix H.2, and À < 1. Theorem 4 says that any interpolated choice of (a1, a2) deviates from the expected convex combination by most LAn.
Next, below we compare Taylor expansions (i) Vector Uniform Merge (Uniform Soup) [56], (ii) Function Ensemble [56, 8] and (iii) our infinite number of functions drawn from the task simplex:
Uniform
Vector
(xx Oo
à/ 1=
25= Ti)
Ri fxx: 0p) - (J)( 11) + X2 11 H (00)
(14)
Merge:
kinear
quadratic_ VUM
Function Ensemble:
Σ \=
[(x: 0o -ATT) Rd linear
X2
i=1 Ti (,(O))
(15)
Q
quadvatic_FF FE
Ours:
f(z; 1-00~220 (-T) dr x linear
quadratic_ VUM M + quadratic FE.
(16)
Ag
Interpolation of Vector Uniform Merge and Fune, Ensemble.

---

Table Standard unlearning of CLIP models measured by the average accuracy (%) on the forget
datasets (Stanford Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN) and the accuracy (%) on the retain dataset (ImageNet) across diverse CLIP architectures. Note that the retain
set accuracy is required by the protocol to be fixed around 95% of pre-trained model by cross-val. X.
ViT- ViT-Base/32
ViT-Base/16
ViTl-Lare//14
Method
Forget (1)
Retain (1)
Forget (1)
Retain (1)
Forgel (1)
Retain (1)
Pre-trained Model
48.09
63.33
55.46
68.33
65.22
75.53
Standard Task Arithmetic [29]
24.23
60.74
20.57
64.77
16,66
72.10
Best Model Val. Set [56]
22.69
60.54
19.86
64,43
16.16
71.64
Vector Uniform Merge [56]
23.01
60.64
20.28
64.66
17.30
72.08
Vector Greedy Merge [56]
22.14
60.46
19.28
64.51
16.77
71.67
Vector TIES- TIE--Megging [59]
23.94
60.27
20.19
64.56
17.83
72.50
Vector EMR-Merging [28]
21.83
60.34
19.10
64.52
15.67
71.89
Function Ensemble [56, 8]
22.75
60,32
19.89
64.48
16.32
71.91
Ours Ours (Distillation)
15.20
60.58
12.17
64.93
9.98
72.17
15.66
60.79
12.70
64.72
10.63
72.59
The tth and 1d order terms of Taylor expansions are identical for Vector Uniform Merge, Function Ensemble and approach, and related to the Neural Tangent Kernel of f(-).
However, Eq. (14) and (15) differ in their second-order terms of Taylor expansions. Eq. (16) shows that our approach in fact interpolates between quadratic terms of Vector Uniform Merge and Function Ensemble. For arbitrary concentration parameter a the Dirichlet distribution, we obtain an interpolation with p= aQ+I
Q and Q+1 aQ and Ç aQ+1 For =1, p= Q+1
Experiments
4.1 Experimental Setup
Datasets. Following prior works 50), we adopt the identical experimental setup. The unlcarning evaluation of CLIP performed on eight datasets designated the forget set. To assess performance of retained knowledge, we use ImageNet [7] as the retain set. Further details are in Appendix A.
Compared Methods & Evaluations. In addition to standard task rector-based unlearning [29] for VLMs, we also adapt the notion of task vectors to extend existing model merging approaches [54] for unlearning. Specifically, we reinterpret Uniform Merge [56], Greedy Merge [56], TIES TTES-Merging [59], and EMR-Merging [28] as parameter-level task vector merging strategies for unlearning comparisons. As our proposed method emphasizes function-level ensemble unlearning, we also consider the vanilla function-level ensemble as baseline. Detailed formulations of these methods within the context of vision-language model (VLM) unlearning are provided in Appendix A.2. Unlearning is assessed on both the forget set and the retain set, where lower accuracy on the forget set indicates better unlearning, and accuracy the retain set mandated by the protocol reach fixed retain accuracy.
Implementation Details. Unless otherwise stated, we adopt the standard CLIP fine-tuning protocol for generating the task vectors each forget dataset, in line with prior task arithmetic studies [29, 50]. Specifically, we adopt the AdamW optimizer [44] with peak learning rate of 1 x 10-5, momentum (0.9. 0.999) along with cosine annealing scheduler and a weight decay of 0.1. During fine-tuning, the CLIP text encoder is frozen to retain the integrity the pre-trained textual representations for classification head via zero-shot prompt embeddings. To ensure diversity of task vectors, we generate a pool of 30 fine-tuned CLIP models by varying the data augmentation configurations, e.g., using different hyper- parameters of RandAugment [6]. All VLM unlearning experiments are conducted across diverse CLIP backbones [52], including BTT-Base/32, ViT-Basc/16, and ViT-Large/ 14. For each unlearning scenario. the task vector negation coefficient À is selected
small held-out subset of the training data. Unless otherwise specified, the task simplex is constructed using 30 vertices, and the distillation variance penalty is set to B 2.0. More details are in Appendix A.3.
4.2 Main Results
Unlearning with Task Vectors. Below, we compare our proposed method and its distilled single task vector variant, against prior task vector based unlcarning approaches across three CLIP backbones:
7

---

Table Linear task vector unlearning of CLIP models measured by the average accuracy (%) on the forget datasets (Stanford Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, SVHN and the accuracy (%) on the retain dataset (ImageNet) across diverse CLIP architectures. The protocol mandates the retain set accuracy to be fixed around 95% of pre-trained model by cross-val. X.
ViT- ViT-Base/32
ViT-Base/16
ViT-LLrge//14
Method
Forget (1)
Retain (1)
Forget (1)
Retain (1)
Forgel (L)
Retain (t)
Pre- traitedined Model
48.09
63.33
55,46
68.33
65.22
75.53
Standard Task Arithmetic [29]
11.40
60.53
8.48
64.87
7.76
72.15
Best Model Val. Set [56]
10.98
60.46
8.17
65.10
7.40
72.26
Vector Uniform Merge [56]
10.62
60.37
7.90
64.68
7.48
72.04
Vector Greedy Merge [56]
10.12
60.42
7.55
64.66
7.14
71.93
Vector TIES- TIE--Merging [59]
11.08
60.25
7.82
64.72
7.43
72.28
Vector EMR-Merging [28]
9.53
60.51
7.09
64.35
6.51
72.29
Function Hnsemble [56, 8]
9.97
60).40
7.41
64.62
7.02
72.12
Ours
7.88
60.43
6.46
64.59
5.70
72.38
Ours (Distillation)
8.13
60.83
6.79
65.08
5.94
72.56
Table 3: Incremental unlearning measured by sequential-average forget accuracy (%, lower is better) after each step across eight benchmark datasets using CLIP ViT-Base/32.
Method
Cars
+DTD
+EuroSAT
+GTSRB
+MNIST
+RESISC45
+SUN397
+SVHN
Standard Task Arithmetic [29]
34.8
34.4
28.0
24.3
23,8
26.2
31.0
29.2
Best Model Val. Set [56]
32.0
32.6
26.3
22.9
22.1
24.5
29.4
27.7
Vector Uniform Merge [56]
32.6
33.1
26.6
22.8
22.7
24.3
29.1
28.0
Voctor Greedy Merge [56]
32.0
32.5
26.0
22.5
21.4
23.9
28.7
27.1
Vector TIES-Merging
33.6
34.2
27.4
23.8
23.3
25.8
30.7
28.9
Vector EMR Merging [28]
22MR-Mrrgnn
31.0
31.6
25.4
22.0
21.5
23.9
28.4
26.8
Function Ensemble [56,
32.3
32.8
26.3
22.8
22.3
24.7
29.4
27.7
Ours
28.7
28.3
21.0
17.3
15.2
21.5
20.1
Ours (Distillation)
29.0
21.8
18.3
17.2
17.6
22.8
22.5
ViT-Base/32, ViT-Base/ and ViT-Large/|4, Table lists the average accuracy on both the forget set and the retain set, computed by individually unlearning each of the eight target datasets. One can observe that the classification accuracy on the retain set remains consistently high, preserving -95% of the original performance of the pre-trained model on all methods. Compared to function ensemble our closed-fone solution achieves 7.55%, 7.72% and 6.34% improvement in unlearning on ViT-Base/32 ViT-Base/16 and ViT-Large/14, 4, respectively. Our method also outperforms alternatives when scaled to larger-scale CLIP architectures (e.g., ViT-Large//44, effectively removing targeted knowledge while retaining general capabilities. The distilled variant of our method achieves unlearning accuracy similar to that of our function-level ensemble, highlighting benefits of distilling function-level task simplex ensemble into a single task vector for improved portability. Unlearning performance w.r.t. each dataset is presented in Appendix B.
Unlearning with Linear Task Vectors. Model linearization, grounded in the Neural Tangent Kernel (NTK) theory [30], enjoys strong task arithmetic capabilities [50]. Thus, we leverage linearized task vectors various model merging strategies and evaluate their effectiveness in unlearning tasks. Details of the task vector linearization are provided in Appendix C. Table 2 shows that the linearized task vectors consistently yield substantial reductions in the forget set accuracy across multiple merging methods and CLIP architectures, while maintaining fixed accuracy on the retain set relative to their standard (no--iinearized) counterparts Table 1. Notably, both our method and its distilled variant continue to outperform other baselines, achieving the best unlearning efficacy while preserving most of the retained knowledge on ImageNet.
Incremental Unlearning. While prior literature on unlearning in VLMs focus on single-dataset removal, we propose more realistic and challenging scenario: unlearning datasets incrementally, This setup lets us assess the cumulative unlearning ability and interferences across distribution shifts from the perspective of task vectors. To this end, we fine-tune CLIP ViT-B/32 on each forget dataset, construct task vectors, and sequentially use them in each method to remove datasets one by one: Cars SVHN. After each removal step, we report the average accuracy on all unlearned (forget) datasets so far. For our distillation, after each step we distill the task vector - and set 0g == 0n - > accumulate parameters across unlearning steps. Table shows our method enjoys the lowest average forget accuracy across all steps. See Appendix D for more details.
DTD - EuroSAT

---

VQA Task.
Table 4: CLEAR VQA (multi-modal) results.
Below apply our method on multi- modal CLEAR benchmark [21] (fictional author profiles with face images-captions pairs), Average accuracy (VQA task) with LLaVA-L.5-7B, CLIP ViT-L/14 as vision encoder. Task vectors are derived by fine- tuning on the corresponding target data. Task specificity: forgetting is measured in terms of classification accuracy. This experiment demon- strates that our approach not limited classification tasks. Fine-tuning can be performed on image-text pairs, and the CLIP loss or any other loss can be used for fine-tuning to obtain task vectors.
Method
VQA Acc. Forget ())
VQA Retain (1)
Pre-truined Model
69.2
55.7
Stundard Task Arithmetic
42.7
49.4
Uniform Merge
37.9
50.5
TIES Merr-Mggg
36.1
49.7
EMR- Merging
34,8
49.3
Function Ensemble
35.6
50.0
Ours
31.5
50.2
4.3 Analysis
Below provide further analyses of each component in our closed- form ensemble unlearning.
Ablation Studies. Table 5 compares four key components of our method in unlearning efficacy: (1) Vanilla en- semble of functions from task vectors (first row). (2) Our Task Simplex (Sim- plex), (3) Advanced Aggregation (The-
Table 5: Ablation study of key components in our method for average forget-set and retain-set accuracy (%).
Simplex Adv. Aggregatior
Waightng
Vettios Opt. Furget (.)
Retain (1)
22.75 75
60.32
17.89 77899
60.41
16.87
60.52
15.67
60.30
orem 2) denoted Adv. Aggregator, and
(4) Variance reduction by Vertex Im-
portance Weighting (Corollary I+Eq. (11)) denoted as Weighting, and (5) Variance reduction by Optimization of Vertices AT; (Eq. (11)) denoted as Vertices Opt. We report both the average forget- set accuracy across 8 datasets and the average retain-set accuracy ImageNet based on CLIP ViT-Base/32. Table shows that the task simplex significantly enhances the unlearning efficacy over function ensembling of limited task vectors. Advanced aggregation further reduces the forget accuracy while preserving most of the retained knowledge on ImageNet. The unlearning improve- ment obtained by variance reduction by vertex importance weighting justifies motivation that suppressing prediction variance helps unlcarning performance.
The Impact of the Number of Task Vector Vertices of the Simplex. Figure/ shows that ensembling larger number of functions from task vectors reduces prediction variance, and improves unlearning efficacy. Figure 3 fur- ther validates this claim within the framework of the task simplices. We analyze our closed-form simplex aggrega- tion under varying numbers of task vectors (i.e., simplex vertices). Unlike ensembling functions from limited task vectors, our closed-form solution approximates infinite sampling over the simplex, resulting in consistently lower prediction variance and reduced forget-set accuracy. Fur- thermore, we observe a clear correlation between unlearn- ing performance and prediction-leve variance: modeling higher-dimensional task vector simplices by aggregating more task vectors yields enhanced unlearning effective- ness, further validating the robustness of our method.
Forget Set Accuracy (%)
26
Funccioo-level Ensemble
24
Task Vactor Simplex (Ours)
22
20-
0=5
Q=10 10
18
0 15
20
16
25
wgg
0 30
11.00 0.125 0.150 0.175 0,200 0.225 0.250 0.275
Avg. Prediction Variance
Figure 3: Comparison between function- level ensembles from task vectors and our task simplex w.r.t. the number task vectors Q.
Importance Weighting Strategies. Motivated by the observed correlation between unlearning efficacy and prediction variance, below we study different importance weighting strategies for simplex vertices with the goal of enhancing unlearning.
In addition to importance weighting mechanism from Corollary I+Eq. (11), we also investigate a weighting strategy that measures the impor- tance of each task vector vertex based on the Cross-Entropy (CE) loss on the forget dataset (formulated in Appendix E). Table evaluates
Table 6: Average accuracy (%) comparing variants
of importance weighting of simplex vertices.
Weighting strategy
Forget (L)
Retain (f)
Uniform weighting
16.87
60.52
CE-guided weightting
16.37
60.28
Vuriance-redueing weighting (Ours)
15.20
60.58

---

both sor and retain-set accuracy and shows that the variance reduction by importance weighting consistently outperforms the CE-guided strategy in terms of the forget-set performance. We attribute this improvement to learning weights that reduce variance of ensemble by limiting the impact of functions associated with noisy task vectors.
Comparison of Distillation Schemes. Our dis- tillation in Eq. (10) distills single task vector from the ensemble to facilitate easier deploy- ment of unlearned-se -et model. Table 7 studies the effectiveness of different task vector distilla- tion schemes. Specifically, it compares different
Table 7: Average performance (%) of our method w.r.t. different task vector distillation schemes.
Prediction Alignment Metric
Variance Tuning
Forget
Retain (f)
KL, distaace Eg distance
60.47
1683
Div.
yes 15.66
16.09
61.64
yes
60.79
prediction alignment metrics (e.g., KL Divergence and the l2 distance) and evaluates their variants with or without variance tuning. Without variance tuning, the use of the lg distance already demon- strates improved unlearning efficacy over the KI. Div., potentially due to its stronger impact on logits and smoother gradient behavior. Incorporating importance weighting for variance tuning enhances the effectiveness of both metrics by suppressing high- nigh-variance predictions during distillation.
Concentration Parameter a. The Dirichlet distribution may follow a non-uniform density if the concentration parameter & 1. Up to this point, we employed a 1. However, Eq. (16) shows that a controls the level of interpolation between quadratic terms of Vector Greedy Merge and Function Ensemble in our method. Table (CLIP ViT-B/16) shows that the best unlearning performance is achieved
Table 8: Effect of Dirichlet concentration a.
a
Forget (L)
Retain (1)
0.1
14.65
64.49
0.5
12.83
64.26
0.8
12.05
64.68
for a = 0.8. Intuitively, a < indicates higher sampling density
1.2
12.48
64.85
toward simplex vertices (away from the simplex center) which are
1.5
12.70
64.92
task vectors. Thus, the function ensemble puts more emphasis on contributions closer to Q task vectors.
2.0
13.29
64.81
1.0
12.17
64.93
Sampling Task Vectors vs. closed-form Solution. Our func- tion ensemble enjoys closed- -oseed-ormm solution as an efficient alter- native to naive task vector sampling from the simplex. Figure compares our closed-form task vector aggregation with explicit task vector sampling for function-level ensembling. As the number of sampled vectors increases, the forget-set accuracy gradually improves but saturates near 300 samples, beyond which additional gains are marginal. However, this improve- ment comes at the cost of significantly higher unlearning time, up to 4x longer. In contrast, our closed-form solution achieves similar performance with a dramatically lower time cost (e.g., 0.8 hours 6.3 hours), offering a scalable and efficient learning. The circle radius indicates the number of sampled task vectors used in unlearning, with our method effectively sam- pling the infinite number of task vector interpolations between vertices of the task simplex.
Explet Task Vector Samplng Our Closed form Solution
Avg. Forget Set Accuracy
is 18 1
100 200
300
400
Avg. Unlearning Time (min)
Figure Comparison of explicit task vector sampling vs. our closed- form solution. The circle radius de- notes the number of sampled task vectors, while ours represents an in- finite sampling.
5 Conclusions
We have investigated the function-level ensemble perspective of task bector-based unlearning in VLMs, revealing an inherent connection between unlearning effectiveness and prediction-level variance. Through empirical and theoretical analysis, we have shown that aggregating larger number of functions from several task vectors obtained on unlearning set improves unlearming performance by reducing prediction variance, consistent with the Bienaymé principle. To address the scalability limitations of explicitly generating and ensembling an infinite number of task vectors, we introduce a novel framework based on a high-dimensional task simplex, where each vertex represents a task vector derived from distinct fine-tuning strategy on the unlearning set. We then derive closed-form ensemble over an infinite number of interpolated task vectors uniformly sampled from this simplex. This formulation enables efficient unlearning via function-level aggregation while capturing the bias-variance trade-off. Our framework supports the distillation of unlearned model parameters from the ensemble, ensuring compatibility with the original VLM when required.
10

---

Acknowledgments
Piotr Koniusz and Hao Zhu are supported by CSIRO's Science Digital. This research supported in part by National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative, partly supported through the the under its AI Centre of Excellence for Manufacturing (AIMfg) (Award W25MCMFO14), the Centre for Frontier Artificial Intelligence Research, Institute of High Performance Computing, A AtStar, and the College of Com- puting and Data Science at Nanyang Technological University. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, and Infocomm Media Development Authority.
References
[1] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal, Reconciling modern machine-learning practice and the classical bias- trade-ofT. Proceedings of the National Academy of Sciences, (5432):15849-1554 2019.
[2] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang. David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141-155 159. IEEE, 2021.
[3] Gong Cheng, Junwei Han, and Xianqiang Lu. Remote sensing image scene classification: Benehmark and state of the art. Proceedings of the IEEE, 105(10):1865-1888 2017.
[4] Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7210-7217, 2023.
[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606 3613, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quac V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition workshops, pages 702-703, 2020.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.
[8] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
Junhao Dong, Piotr Koniusz, Junxi Chen, and Yew-Soon Ong. Adversarially robust distillation by reducing the student-teacher variance gap. European Conference on Computer Vision. pages 92-111. Springer. 2024.
[10] Junhao Dong, Piotr Koniusz, Junxi Chen, Z Jane Wang, and Yew-Soon Ong. Robust distillation via untargeted and targeted intermediate adversarial samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28432 28442, 2024.
[11] Junhao Dong, Piotr Koniusz, Junxi Chen, Xiaohua Xie, and Yew-Saon Ong. Adversarially robust few-shot learning via parameter co- so-distillation of similarity and class concept learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 28535-28544, June 2024.
[12] Junhao Dong, Piotr Koniusz, Liaoyuan Feng, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21037-21044 October 2025.
[13] Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I. KDD '25, page 236-247, New York. NY, USA, 2025. Association for Computing Machinery.

---

[14] Junhao Dong, Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. Proceedings of the 42nd International Conference on Machine Learning. volume 267 of Proceedings of Machine Learning Research, pages 14061-14078. PMLR, 13-19 Jul 2025.
[15] Junhao Dong Jiao Liu, Xinghua Qu, and Yew-Soon Ong. Confound from all sides, distill with resilience: Multi Multi-objective adversarial paths to shot robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 624-634, October 2025.
[16] Junhao Dong, Seyed-Mohsen enemy is my friend: Exploring inverse adversaries for improving adversarial training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 24678-24687. June 2023.
Monsavi-Dezfooll Jianhuang Lai, and Xiaohua Xic. The enemy of my
[17] Junhao Dong. Yuan Wang, Jian-Huang Lai, and Xiaohua Xie. Improving adversarially robust few-shot image classification with generalizable representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9025-9034, 2022.
[18] Junhao Dong, Yuan Wang, Xiaohua Xie, Jianhuang Lai, and Yew-Soon Ong. Generalizable and discrimi- native representations for adversarially robust few-shot learning. IEFE Transactions Neural Nenworks and Learning Systems, 36(3):5480-549) 2024.
[19] Junhao Dong, Lingxiao Yang. Yuan Wang, Xiaohua Xie, and Jianhuang Lai. Toward intrinsic adversarial robustness through probabilistic training. IEEE Transactions on Image Processing, 32:3862-3872, 2023.
[20] Junhao Dong, Cong Zhang, Xinghua Qu, Zejun MA, Piotr Koniusz, and Yew-Soon Ong. Robust super- alignment: Weak-to to-s-toggtoogg robustness generalization for language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[21] Alexey Dontsow, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Y Rogov, Ivan Oseledets, and Elena Tutubalina. Clear: Character unlearning in textual and visual modalities. arXiv preprint arXiv:2410.188 18057, 2024.
[22] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259 3269. PMLR, 2020.
[23] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances neural information processing systems, 32, 2019.
[24] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In The Twelfth International Conference on Learning Representations, ICLR, 2022.
[25] Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pages 3832-3842. PMLR, 2020.
[26] Patrick Helber, Benjamin Bischke. Andreas Dengel. and Damian Borth. Eurosat: A novel dataset and deep lcarning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-222 2019.
(27] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR, 2022.
[28] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free hiigh-performance model merging. Advances in Neural Information Processing Systems. 37:122741- 122769, 2024. 2024.
[29] Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, ICLR, 2023.
[30] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernl:: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
12

---

[31] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging, weights of language models. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[32] Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. When do minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16599 2022.
[33] Ron Kohavi and David Wolpert. Bias plus variance decompasition for zero-one loss functions. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning. ICML'99, 96. page 275-283, San Francisco, CA, USA, 1996. Morgan Kaufmann Publishers Inc.
[34] Piotr Koniusz and Hongguang Zhang. Power normalizations in fine-grained image, few-shot image and graph classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(2):591-609, 2021.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grainet categorization, In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[36] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[37] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 36(11)::3 1):2278--2224 1998.
[38] Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi, and Fan Liu. Single image unlearning: Efficient machine unlearning in multimodal large language models, Advances Neural Information Processing Systems, 77:35414-35455 2024.
[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- pre-training for unified pages 12888-12900. PMLR, 2022.
understanding and generation. In International conference on machine learning,
[40] Zhibin Li, Piotr Koniusz, Lu Zhang, Daniel Edward Pagendam, and Peyman Moghadam. Exploiting field de- pendencies for learning on categorical data, IEEE Trans. Pottern Anal. Mach. Intell., 45511)11359-11352, November 2023.
[41] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[42] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearming for large language models. Nature Machine Intelligence, pages 1-14, 2025.
[43] Zheyuan Liu, Guangyao Dou, Xiangchi Yuan, Chunhui Zhang, Zhaoxuan Tan, and Meng Jiang. Modality-aware neuron pruning for unlcarning in multimodal large language models. arXiv preprint a7Xiv:2502.15910, 2025.
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representaatons, ICIR, 2019.
[45] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations..CIR 2023.
[46] Michael S Matena and Colin A Raffel. Merging models with fsher-weightee averaging, Advances in Neural Information Processing Systems, 35: 35:17703-17716, 2022.
[47] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised learning, volume 201 page 4, Granada, 2011.
[48] Yao Ni, Shan Zhang, and Piotr Koniusz. PACE: marrying the generalization of PArameter-efficien fine- tuning with consistency regularization. In The Thirty-eigtth Annual Conference on Neural Information Processing Systems, 2024.
13

---

[49] Cristian Rodriguez Opazo, Ehsan Abbasnejad, Damien Teney, Edison Marrese- Marrese-Taylor, Hamed Damirchi. and Anton van den Hengel. Synergy and diversity in clip: Enhancing performance through adaptive backbone ensembling. In The Twelfih International Conference on Learning Representations ICLR, 2025.
[50] Guillermo Ortiz- Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727- 66754, 2023.
[51] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift, Advances in neural information processing systems, 32, 2019.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8765 PMLR, 2021.
[53] Protection Regulation. General data protection regulation. Intouch, 25:1-5, 2018.
[54] Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, and Jin Lu. From task-specific models to unified systems: A review of model merging approaches. arXiv preprint arXiv:2503.0888 089998 2025.
[55] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
[56] Mitchell Wortsman, Gabriel Ilaarco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning. pages 23965-23995 PMLR, 2022.
[57] Jing Wu and Mehrtash Harandi. Munba: Machine unlearning via nash bargaining. arXiv preprint arXiv:2411.15377 2024.
[58] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE computer suciety conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[59] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
[60] Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo. Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
[61] Tianyu Yang, Lisen Dai, Zheyuan Liu, Xiangqi Wang. Meng Jiang. Yapeng Tian, and Xiangliang Zhang. Cliperase: Efficient unlearning visual-textual associations clip, arXiv preprint arXiv:2410.23330. 2024.
[62] Yongquan Yang, Haijun Lv, and Ning Chen. A survey ensemble leaming under the era of deep learning. Artificial Intelligence Review, 56(6):5545 5589, 2023.
[63] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlcarning of pre- trained large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pages 8403 8419, 2024.
[64] Xu- Xu-Cheng Yin. Kaizhu Huang, Chun Yang, and Hong- Wei Hao. Convex ensemble learning with sparsity and diversity. Information Fusion, 20:49 59, 2014.
[65] Jingyi Zhang. Jiaxing Huang, Sheng Jin and Shijian Lu. Vision-language models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[66] Yifei Zhang, Hao Zhu, Junhao Dong, Haoran Shi, Ziqiao Meng, and Han Yu Piotr Koniusz. CrossSpec- tra: exploiting cross-layer smoothness for parameter-effiiennt fine-tuning, In The Thitt Annual Conference on Neural Information Processing Systems, 2025.
[67] Hao Zhu, Yifei Zhang, Junhao Dong, and Piotr Koniusz. BiLoRA: orthon-orth parameter spaces for continual learning. In 2025 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2262222 2025.
14

---

Machine Unlearning via Task Simplex Arithmetic (Supplementary Material)
Junhao Dongl,, Hao Zhu², Yifei Zhang', Xinghua Qu Yew-Soon Ong" and Piotr Koniusz 'Nanyang Technological University. CCARR, IHPC. A STAR. 'Bytedance, *Data6 CSIROO UUniversity of New South Wales, "Australian National University
A Further Experimental Configurations
Below, we provide details of our experimental setups of Vision-Lannguage Model (VLM) unlearning in terms of dataset description, formulation of compared approaches, and implementation details.
A.1 Dataset Setups
Following prior task vector-based unlearning works [29, 50], evaluate the unlearning efficacy of VLMs across eight datasets that span a diverse range of recognition tasks, collectively forming the forget set. The refain set fixed as the ImageNet dataset [7], which serves assess the preservation of general knowledge during unlearning. The eight datasets used as forget sets are categorized by their recognition scenario follows:
1. Fine-grained classification: Stanford Cars [35].
2. Texture recognition: Describable Textures Dataset (DTD) [5].
3. Remote sensing and aerial imagery: EuroSAT [26], and Remote Sensing Image Scene Classifi- cation (RESISC) [3].
4. Traffic and digit recognition: German Traffic Sign Recognition Benchmark (GTSRB) [55], MNIST [37], and Street View House Numbers (SVHN) [47].
5. Scene recognition: SUN397 [58].
For each dataset (eight datasets in total) in the forget set, we obtain the task vector by fine-tuning the CLIP model on its training split. Unlearning evaluation is then performed using the ImageNet validation set, its test labels are not publicly available. Note that the task vector is obtained through fine-tuning on the training set of each forget set (eight datasets in total), while the unlcarning performance evaluation conducted on the validation set of ImageNet, as the ground-truth labels for its test set are not publicly available. To promote diversity in the task vector space, we vary the data augmentation configuration during fine-tuning by applying Rand/Augeett [6] with different hyperparameter settings. Specifically, we vary the number of sequential augmentation transformations from and the transformation magnitude from 10. This results total of 30 fine-tuned CLIP models per dataset for each architecture, yielding a rich set of task vectors for subsequent unlearning experiments.
A.2 Detailed Formulations Compared Methods
To comprehensively evaluate the effectiveness of our proposed function-level ensemble unlearning framework, compare it with several representative baselines. These include standard task vec- tor based unlearning [29], diverse task vector vector-adapted model merging strategies [56, 59, 28]. and a vanilla function-level ensemble baseline [8]. Note that the model merging is adapted to the task
"Correpoodiig authors. PK also in charge of theory & derivations.
15

---

vector domain by simply treating the task vector as a unique set of network parameters. Below, we outline how each method is formulated and applied in the context of vision-language model (VLM) unlearning.
1. Standard Task Arithmetic [29]. The unlearned model is obtained by subtracting single task vector from the fine-tuned model, i.e., f(;; 0 This method is retraining- and efficient, but may suffer from instability due to sensitivity to fine-tuning configurations.
2. Best Model on the Validation Set [56]. The unlearned task vector selected based on the best unlearning efficacy (the lowest forget set accuracy) from the validation set. The unlearned VLM is obtained via subtracting the selected task vector, i.e., f(x;; arg min; (0AAcc(00AT..).
3. Uniform Merge [56]. First, compute the average of multiple task vectors, and then subtract the resulting average task vector from the base model to obtain the unlearned model, i.e., (a; 1 E Tz). This can also be regarded
parameter-level ensemble.
4. Greedy Merge [56]. Iteratively select and merge task vectors based on greedy criterion (i.e., minimizing unlearning score) to optimize unlearning performance.
5. TIES-Merging [59]. Trim low-magnitude changes in the values of task vectors and then resolve sign disagreements across the task vectors being merged.
6. EMR-Merging [28]. First, elect unified task vector from all the task vectors and then generate their corresponding modulators, including masks and rescalers (coefficients), to align the direction and magnitude between the unified task vector and each specific task vector, respectively.
7. Function ensemble [8]. To isolate the benefit of our proposed function-level closed-form solution, we consider a vanilla ensemble of models whose parameters are fine-tuned with different forget datasets. The predictions from each unlearned model are averaged inference time, i.e., 2/2 E f(x; 0 AT.). This ensemble does not require task vector/model merging.
A.3 Implementation Details
In accordance with previous task arithmetic approaches [29, 50], we employ the standard CLIP fine-tuning protocol to generate task vectors for each forget dataset, unless specified otherwise. For each forget dataset, we fine-tune the CLIP model using the AdamW optimizer [44] with cosine learning rate schedule. The peak learning rate is set to 1 x 10-5 with momentum coefficients (81, (0.9, 0.999) and weight decay of 0.1. The CLIP text encoder is frozen throughout to preserve the integrity of the pre-trained textual embeddings. which serve as zero-shot classification prompts. The vision encoder is updated exclusively. Accordingly, the number of training epochs is set based on dataset characteristics: 70 epochs for Stanford Cars, 100 epochs for DTD, 40 epochs for EurOSAT, GTSRB, RESISC45, and SUN397, and 30 epochs for MNIST and SVHN. We evaluate unlearning performance across three widely used CLIP variants released in [52]: ViT-B/32, ViT-B/16 6, and ViT-L/14. All models are initialized from pre-trained OpenAI weights and kept fixed for fair comparison across methods.
Unless otherwise defined, the task vector simplex is constructed using Q 30 vertices (i.e., 30 diverse task vectors per forget dataset). The coefficient À controlling the magnitude of task vector subtraction is selected per forget dataset based unlearning efficacy measured on small held-out split, i.e., À E {0.0, 0.05, 0.1...., 1.0], mimicking a tuning- free practical scenario. We choose À that achieves the lowest forget accuracy while the unlearned VLM still preserves least 95% of the accuracy of the pre-trained VI.M on the retain dataset (ImageNet). For the closed-form function-level ensemble and its distillation, the bias variance weighting factor is fixed to / = 2.0, which find consistently balances forget and retain performance across datasets. All the experiments in this paper are conducted based on eight NVIDIA H100 GPUs with 80GB of memory.
B Standard Unlearning Performance for Individual Datasets
Unlearning w.r.t CLIP ViT-Base/32 backbone. We here provide the VLM unlearning results across eight datasets using the architecture of CLIP ViT-Base/32 in Table 9. We can easily observe that our method and its distillation variant generally achieve great unlearning efficacy on the forget datascts while preserving most of the retained knowledge other datasets. Furthermore, our
16

---

Table 9: CLIP ViT-Base/32 unlearning measured by the individual aceuracy (%) on each forget dataset and its corresponding accuracy (%) on the retain dataset (ImageNet).
Method
Cars
DED
EaroSAL
GISRB
MNIST
RESISCAS
SUNS97
SVHN
F()
R (1)
(()
(((1)
F.))
R.(1)
F(])
F
R
R(()
F())
F(L))
R()
Pre trained Model
59.5
63.3
$1.4
63.3
64.7
63.33
32.6
63.3
48.3
63.3
60.3
63.3
63.2
63.3
31.6
63.3
Standard Task Aritmerri [29]
34.8
6618
29.6
60.5
12.0
661.9
8.5
60.
16.4
60.6
31.9
61.1
52.1
60.9
8.3
61.1
Best Model Val. Set [56]
32.0
60.4
28.5
60.5
10.7
60.6
60.4
13.6
61.3
29.7
60.3
51.1
60.3
7.7
60.6
Vector Uniform
[36]
32.6
60.7
29.2
60.8
10.5
00.7
16.3
6018
30.1
60.4
61.1
Voctor Greedy Merge [56]
32.0
10.0
60.
11.5
80.7
49.8
Nestor TiEs-Merging [59]
33.6
6003
60.3
10.7
60.1
60.3
15.8
60.6
00.0
52.6
60.4
60.3
Vector EMR-Me Mergiig [28]
31.0
60.]
27.8
60.4
9.8
60.2
60.4
13.9
60.5
29.2
60.4
47.9
39.9
60.8
Function Ensemble (56, $1
32.3
600.
29.0
60.2
10.2
600.1
7.8
00.4
14.5
6006
30.4
00.3
49.9
00.0
7.8
00.7
Ours
28.7
60.7
23.6
60.5
3.2
60.4
1.8
60.6
1.3
60.9
16.0
60.4
44.2
60.7
2.0
60.8
Ours (Distillation)
29.2
60.9
23.9
60.7
3.6
8036
2.5
60.8
1.4
62.1
16.6
00.6
45.5
00.9
2.6
61.0
Table 10: CLIP ViT- VTT-Base/16 unlearning measured by the individual accuracy (%) each forget dataset and its corresponding accuracy (%) on the retain dataset (ImageNet).
Method
DTD
EaroAAT
GISRB
MNIST
RESISCA5
SUNS97
SVHN
(1)
[(])
R()))
[()
R(1)
F())
((
F(.)
R(11
F(L)
R(1)
F())
R1)
F(L)
(1)
Pre-tmined Model
64.7
68.3
68.3
55.3
51.7
68.3
65.5
68.3
51.9
Standard Task Arithmetic [29]
27.8
64.7
24.1
64.3
12.4
65.0
61.5
9.4
65.3
27.3
64.7
19.6
64.8
6.4
61.8
Best Model on Val. Set [56]
30.0
63.9
25.1
64-2
9.9
64.6
7.6
64.3
5.4
65.1
27.7
64.9
48.6
63.9
6.6
64.5
Vector Uniform Morge [56]
28.6
64.6
21.7
63.3
10.6
64.9
8.4
64.9
13.2
65.5
24.0
64.5
49.2
64.7
6.5
64.8
Vector Greedy Morge [36]
30.0
633.9
23.1
64.0
10.2
64.3
7.5
64.3
4.1
05.3
23.8
64.5
49.1
64.6
6.4
64.7
Vector TIES Merging [59]
28.3
64.6
25.7
61.6
9.0
64.4
61.6
13.3
61.9
26.6
61.3
44.1
64.7
7.1
61.6
Vector EMR- Merging (28)
26.8
64.5
64.6
8.5
64_1
64.7
12.6
42.0
64.5
64.7
Function Ensemble (56,
27.9
K99
13.1
43.7
Ours
20.3
64.9
64.9
1.4
64.77
64.9
088
64.6
42.0
65.1
Ours (Distillaaion)
21.5
64.7
15.5
64.7
1.7
64.5
1.0
64.7 7
0.9
65.1
16.7
64.4
43.1
64.8
64.7
Table 11: CLIP ViT-Large/14 unlearning measured by the individual accuracy (%) on each forget dataset and its corresponding accuracy (%) on the retain dataset (ImageNet).
Method
Cars
DTD
EaroSAT
GTSRB
MNIST
RESISCIS
SUN397
SVIIN
F()
R (t)
F(])
(()
F())
R <t)
F())
(t)
F(.)
R
<t>
F())
(())
F())
R
F(L)
(t)
Pre trained Model
77.9
75.5
55.4
75.5
63.6
75.5
50.5
75.5
76.3
75.5
71.3
75.5
68.2
75.5
58.4
75.5
Standiand Arithmeric [20]
16.9
70.7
9.5
71.9
3.4
50.5
73.3
Best Model
Set
21.7
70.5
5.4
71.2
3.4
73.9
71.3
Victor Uniform Morge [56]
21:7
71.4
71.8
9.7
71.7
8.7
74.0
71.7
49.8
71.7
72.3
Victor Grealy Merge [16]
20.6
70,6
22.8
71.1
1044
72.2
71.3
5.3
73.4
11.0
70.8
50.4
71.1
72.9
Vector TIES Merging [59]
22.4
72.6
23.9
72.5
10.0
72.3
5.5
72.4
9.0
73.1
13.7
71.3
51.3
72.6
6.9
72.5
Vector EMR Merging [28]
19.7
71.8
21.0
72.0
888
71.7
4.8
71.8
7.9
72.0
12.0
72.0
45.1
71.7
6.1
72.1
Function Ensemble (56,
20.5
71.9
21.9
72.0
9.2
71.7
5.0
71.9
8.2
72.0
12.5
71.9
47.0
71.8
6.3
72.0
Ours
14.0
72.2
11.6
72.1
0.9
72.0
0.7
72.1
0.4
72.4
9.5
72.2
41.3
72.3
1.6
72.2
Ours (Distillation)
1.0
72.8
43.3
72.8
distillation approach maintains comparable effectiveness, indicating the utility of function-level ensemble compression into a single task vector for efficiency.
Unlearning w.r.f. CLIP ViT-Base/16 backbone. Our method also shows consistent performance gains with CLIP ViT- 6 backbone, as shown in Table 10. Specifically, we achieve more uniformly low forget-set accuracy, which reflects stable task vector disentanglement at the function level. Moreover, we observe minimal degradation in the ImageNet accuracy, further confirming that the retain-set set supervision/knowledge
well preserved.
Unlearning w.r.t. CLIP ViT-Large/14 backbone. With a stronger-capacity CLIP ViT-Large/ 14 backbone, our proposed method continues to outperform all other VLM unlearning approaches (see Table 11). We can easily observe that the forget-set accuracy across all the datasets is substantially reduced compared to lightweight CLIP architectures. In the meantime, the larger model capacity appears to amplify the benefits of our approach, as our closed-form solution is able to separate spec-spe information from generalizable knowledge.
C Further Details of Task Vector Linearization
We have investigated an additional VLM unlearning scenario (see Table 2) regarding linearized task vectors grounded in the Neural Tangent Kernel (NTK) theory [30], which has recently demon- strated improved unlearning effectiveness. In the tangent space (linear) approach, the VLM function f( 0u +T) is linearized via a first-order Taylor expansion at 0.. This yields a lin- earized VLM in which weight changes produce linear changes in the output: fl(x; + T')
f(;; 0o) + Ve. (z; 0o), where T' is defined as the linearized task vector. By simply replacing
17

---

standard task vectors with their linearized counterparts and adopting the linear approximation f", we achieve VLM unlearning in the context of paagent-space task vectors.
Thus, for each forget dataset, we linearly fine-tune a diversity of linearized task vectors as the standard unlearning cases. We then build the corresponding linearized task vector simplex for closed-fore function ensemble. During evaluations, we follow the same protocol as the standard unlearning detailed in Appendix A.3, reporting both forget-set and retain-set accuracy. More details are in [50].
D Further Details of Incremental Unlearning
To comprehensively evaluate incremental unlearning performance, we follow an unlearning protocol across a predefined sequence of diverse datasets: Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN (see Table 3). Each dataset represents distinct distributional characteristics, enabling an assessment of how cumulative distribution shifts influence unlearning efficacy. We employ the CLIP ViT-B/32 model and follow the standard fine-tuning setup as detailed in the main text. Each forget dataset is individually fine-tuned to derive corresponding task vectors, subsequently subtracted sequentially from the current model to simulate incremental forgetting.
To quantify cumulative unlearning effectiveness, after each sequential subtraction of a task vector, we evaluate the VLM on all datasets that have been unlearned up that step (forget dataset). Specifically, we compute the forget-set accuracy for each previously evaluated dataset and the current one, reporting their average as the average forget-set accuracy. This sequential-average forget accuracy clearly indicates how effectively information from multiple tasks is incrementally erased. Lower values indicate superior unlearning capability.
E
Task Vector Re-weighting Guided by Cross-entropy Loss
Recall that we investigate an adaptive re-weighting alternative based on the Cross-Entropy (CH) loss evaluated the forget dataset (see Table 6), which servers a performance proxy for each task vector. A higher CE loss indicates that the VLM predictions are significantly deviating from the forget samples, thus reflecting better forgetting performance. Let forget denote the CE loss value for the forget dataset corresponding to the i-th task vector. To adaptively quantify each task vector's relative importance, we adopt softmax normalization defined as: w, exp(( (orget) / /E, exp(()
While CE-guided re-weighting provides an effective alternative to prioritizing task vectors that induce stronger forgetting (i.e., higher CE loss on the forget set), its effectiveness remains limited. As shown in Table 6, the CE-based strategy yields marginal improvements over uniform weighting, yet falls short of the performance achieved by our learned re- weiihigtting scheme. This discrepancy stems from the fact that CE losses are computed independently for each task vector, lacking the capacity model interactions or correlations across different forget directions.
In contrast, our learned re-weighting approach benefits from being jointly optimized with the overall objective, allowing to adaptively capture complex interdependencies among task vectors. Notably, the learned weights are implicitly regularized by the prediction variance minimization objective, enabling more effective machine unlearning.
F
Broader Impact and Limitations.
F1
Broader Impact
The growing deployment of large-scale VLMs consistently brings increasing societal pressure to support machine unlearning, enabling systems to forget specific information upon request. This need arises from regulatory requirements such as the right to be forgotten and ethical concerns over data misuse or retention. Our proposed task vector simplex unlearning framework provides an efficient alternative to re- training, enabling scalable removal of targeted knowledge from pretrained VLMs with minimal computational overhead.
The task vector formulation also promotes a compositional and interpretable unlearning mechanism, where users can flexibly combine, subtract, or re- re-weight different task- behaviors. This capability facilitates responsible model editing and personalized deployment.
18

---

The broader impact of our work includes:
Data compliance and privacy. Our proposed method supports data removal scheme without full retraining, providing a feasible path toward compliance with modern data privacy regulations and ethical unlearning requirements.
Resource-efficient unlearning. By enabling efficient unlearning through task vector simplex, our method lowers the barrier for deploying unlearning mechanisms the context of VLMs.
Modular Al systems. The compositional structure of task vectors introduces way toward plug-and-play model updates, supporting dynamic behavior editing foundation models.
F.2 Limitations
Despite the strong unlcarning performance demonstrated by our flseddform task vector simplex approach for VLMs, several limitations still remain, which we aim address or mitigate throughout the paper, as outlined below:
Dependence forget data availability. Our method assumes access to representative/parial data from the target forget task to extract task vectors. While this requirement may appear restrictive, it is often reasonable in real-world scenarios where users explicitly request the removal identifiable data or task-specific knowledge.
Funetion-level ensemble dependeney. Our main method operates the function level by aggregating multiple task vectors through a simplex-based ensemble. While this strategy is flexible and effective, may raise concerns about architectural compatibility or deployment constraints. To address this, we provide an extension that distills the ensemble's 's behavior (predictions) into a single set of model parameters, yielding fully unlearned model that remains structurally identical to the original VLM.
G Proofs.
G.1 Corollary
Proof. Let each r be parameterized in barycentric coordinates T Σ a₁ T, with a, 2 0 and Σe a; = 1. Let each w be parameterized as w Σの O; wi.
Notice a aq)) R is uniformly distributed over the (Q-1)- i.e., Dir (12) Moreover, we have the following expectations:
Eia Q(Q +
2
Baaja,] Q(Q + 1)(Q 2)
iff if j,
E Bassaaau] Q(Q + 1) +2
if kij
(17)
which are based on the following formula:
P1 p2! PQ!
Blapt PI
(18)
QQQ
(Q + - pi)
because we have density:
f(an...., Beta
a 1-1 (Q 1)! fa. 20. a.= 1=1
(19)
where Beta(a),..... ,2) = Ih r(a.)
is the multivariate Beta function. In order to get raw
rE2 oz)
moments, computes:
E B[aft ... 201
of ... ...... Beta(l...., (I... da.
(20)
Σ "0,=,,
19

---

where Beta(1..... 1) I(Q) 1Q-1)!
(1)(11)
E[ m. ..
p0l
Beta(1 +p1, 1 1+ +2, 1+ po))
(21)
Beta(1.....
p1! ... po! T(Q + E Ps) Q( ... (Q + (2 (2 -1)
[ [(1 + p:)
(Q 1)!
(22)
which is exactly Eq. (18).
For the choice of three variables a; Oj and Og., Eq. (18) can simply be expanded as follows
Blaiayav)=B[aa- ... al- a2
PQ 01
... Q QQ+1) (Q43-1) Q(++)
ii. For the choice of two variables až and aj Eq (18) can simply be expanded as follows B[afa,] 0f 0f 1:00 of
Q(++1) (Q+3-1) Q(0+1)(0+2)
iii. For the choice of one variable
a. Eq. (18) can simply be expanded follows B[a 01 .1. 0r
E[af... ..f.....a aaaf..-aa-aaao a ...aa
pQl
((++1) (Q+311) Q[Q+1)(++2)
Now we expand w HT Ej Ed a.ajawweT] H substitute Eq. (17) and com- pute expectations:
E[wT] T HT) 1)( 2)
Σ (2+ (2+ T Hr:
(1 + wi + +w;) r HT,
1m1
kJ=
(23)
This completes the proof.
G.2 Advanced Aggregation
Proof. Eq. (8) emerges from the following set of transitions:
II
(1 f(x:; 00 -rr))
TE
1 exp
log
II [1= XT)))
TEDo
1 exp
Σ
log (1 fxx;
10:500-11p - -x
TEAn
1 exp
A
Σ
log (1- f(x; - AT)I]
Ap
TEAo
1 exp | Hgg (z)].
(24)
Moreover, - (1-pf) is the probability of zero successes in Q independent Bernoulli trials that are not necessarily identically distributed (Wadycki et al. [71]), thus - I (1 -f) is the probability of least successes in Q independent Bernoulli trials that are not necessarily identically distributed.
Therefore, it follows that infinitely sampling trials from the simplex Ae. in 1 - (1 fæ; O, XT)), yields the probability of least one successes in the infinite number of independent Bernoulii trials that are not necessarily identically distributed, and are sampled from simplex Ag- O
Notice that as function varies due to changes of task vectort, underlying trials are not necessarily identically distributed, and that is why employed the PMF of the Poisson binomial distribution.
20

---

G.3 Volume of Simplex
For detailed derivations of the volume formula of simplex refer to the work by Stein [70].
G.4
Bias-variance Trade-off'
Proof. Let y()) = (x) + with ~ (0,, Let T = (()) be a task vector learned by fine-tuning some augmented dataset D derived from unlearning dataset Do. Let predictor be f(x) = f(;; 0o- AT).
Consider the following expression
f(x) = (f((z) + - f(x) (f" f(e)) +1
(25)
and observe that
1y(x)) f()) =| f(x) -( f(x)) n) 1mml
(26)
Consider the following expectation:
ED.,T
By -(x)) Do: T]
(27)
expectation
Now, expand its inner expectation as follows:
Ey [ Do. -| [ +2 [(]))) +E, [[]]]]]]]]]
= -=lf + 1
(28)
Subsequently, Eq. (27) reduces to the following expression:
Ep. [x) ( 1
(29)
which we expand below into the squared bias and variance. Let (2)( = ED,,7 [f(;; 0n Ar) and
consider that fa; 0o - Xr) changes due to changing task vector and dataset Do. whereas f (a)) does not change w.r.t. changes of T or Do-
Substitute u f(æ; Oo - AT) (x) and v - 12(())) (xx), and apply the following expansion:
Iu F11² = | 1/2 2(u, 0) + ||001
(30)
= f=; 0g Xr) - - (x) 2 2 (f P, XT) (1))) + - (x)))
Incorporating expectations into Eq. (30) leads to:
Ep., (1l((x - Ar) fz || ED..T f \τ) - (zy
=Boo
110 (x)
(f()))
+ D (J AT) - (1, (
+ ED,
Er - f(x)) f(w))
(31)
Including 1/>112 from Eq. (29) into the result in Eq. (31) completes the proof.
The importance of the above proof that our closed-form ensemble of infinite number of functions with task vectors sampled from task simplex Ae has the associated with it closed-form variance. Such a variance term tells us how much function f(x:; On - AT) fluctuates around μ (x:) over random draws of 7 from the task simplex Ap and dataset Do. High variance may result from task vectors overfitting to the unlearning dataset so controlling the variance controls the generalization of our unlearning algorithm.
21

---

G.5
Interpolation Bound
Proof. Let 0 = E a0 XT:) be the convex combination of perturbed parameters.
Note that:
Q
a.(00 - AT,) = 0, - Σ
0;T; = 00 - AA.
(32)
1=1
Next, for 1 = I....., C', we write:
Q
(a(00 - AT) Σ ai g1 -AT1) 9 (0)
Σ
a g XT.)
(33)
where gr(-) is simply the 1t output of the multivarraae-outtpu unlearning function g(-).
Let gi(-) be L--Lipschtzz continuous, i.e., (9:(O:) gx(,,) = 0, - -0y112, Vi + j. Then g(.) is
L- L-Lipschitz continuous with L= = L1 as:
1 (((0)) s(0,) Σ (0.).) 91(0,) - Σ Li 0: 0/11 L 0: 00 2 vi j. (34)
Next, we write:
(00 -AT;) (0)
Σ
0 (0u -Ar,) - [
(35)
Q
03 (0) -AT)) - (((
(36)
(.Li AT - -0112
(37)
Q
= 4
as ||[[T Ti)
(38)
= LA
0s T Till2
(39)
1=1
Since T= Σÿ Ji have:
e
Q $
| T T: = Σ
cs(T,
Ti) -
Σ
aj Tj
T:
s max
T, =:/2
(40)
1=1
j=l
Therefore:
(000 - -XT) - Σ (: (00-i < Led Σ max ||;;-| -Till2 < L maxmax/f ||: T5l12- (41)
is
Now using Eq. (34) follows that Eq. (41) can be further transformed as:
|0000 - -T) - Σ a.g(O0 -ATi) = Σ gi(O0 -AT) - Σ
=Σ
aggi -ATi)
1=1
$ Σ
L,d max || - T, 112 45
LA Aama ||T Tyll2- (42)
22

---

H Additional Analyses and Considerations
H.1
Computation of the First-order Taylor Term For the standard expansion without weighting, we have:
Σ
1
(43)
A
Q
TEAm
i=1
while for Corollary we have:
WT
WiTi-
(44)
Aol
2j=1 Wy
(B,T)EA, X
H.2 Sparse Parameter Change and Active Parameter Subset
Few-epochs fine-tuning On on unlearning dataset toward 0; yields sparse change
m
100,00 O3iil e) < m
(45)
for small e and m = 86M (CLIP ViT-B-16). He et al. [68] and Zeng et al. [72] observe sparsity between fine-tuned and original models. For le-4 (CLIP ViT-B/16), about 4.6% parameters change.
Thus, in this work we assume that interpolating between such sparse differences in 0, and 0,. i captures parameter subset responsible for unlearning. Interpolating on such a subset of coefficients serves as choosing varying magnitudes of parameters that are active in unlearning.
H.3 Distributions with Non-compact Support
We assert that the compact support of Dirichlet distribution is beneficial sampling from it provides a sample of parameter vector obeying the support, and thus not activating parameters irrelevant for the task.
As a counterexample, let task vectors be Nor- mally distributed, i.e., 0' [N, o²). Let off-diagonal elements be set to O as one can- not estimate for very large m. Let
Table 12: The Normal distribution vs. the Dirichlet distribution ensemble.
Distribution
Forget (1)
Retain (t)
m dimensional covariance
Q Σ 0 and
Normal (S=100)
16.80
64.32
2² = - i=1 (0; =1)² which produces
Normal (S=300)
15.98
64.54
Dirichlet (Ours)
12.17
64.93
parameter samples 0' p ~~ 1).
where
Table 12 shows that the Normal distribution performs worse than the Dirichlet distribution model as for the Normal distribution the slopes of PDF decay slowly toward co whereas the Dirichlet distribution has compact support.
H.4 Parameter Augmentation Perspective
As averaging all task vectors gives viable unlcarning (Vector Uniform Merge), and ensembling functions individual task vectors is viable unlearning (Function Ensemble), interpolation between the mean & x Ti and individual T, can be considered a parameter augmentation strategy.
Let
1
p(x; ②2
f(x; Gu
T) dr.
(46)
[Ω TEN
Based on Theorem 4, if f() changes fast in Dins and slow in Sslow (within the same neighborhood sizes) then intuitively the entropy of prediction Ent(p(x; Pstow)) is lower than (nt(p(x; Itm)))
23

---

as smooth changes contribute coherently to the vote (ensemble), whereas chaotic changes are incoherent leading to cancellation of class peaks; (nt(vec(L/C)) is max. Hence our method leverages smoothness of [()) w.r.t. augmentation. See Izmailov et al. for related considerations [69].
H.S Proximity of Task Vectors with the Simplex
Task vectors do not follow the Dirichlet distribution per However, They located close the simplex. We take Q = 10 out of 30 task vectors fine-tuned on SUN397 and build a simplex. We count-sketch dimension down to Q=d+l=+++ -+11-11 11 for tractability. The remaining 20 task vectors within (.9(1 radius of the simplex while task vectors of Cars exceed 1.6(n/2) distance (cluster around their own simplex) for diameter 7) defined for Theorem
H.6 Varying Fine-tuning Numbers Q Table 13: Scaling with # task vectors Q (CLIP ViT-B/16).
Forget
Retain
Fine-tuning Time
Unlearning Inferenee Time
In our experiments, we fine-tuned the
10
14.06
64,77
1.9h
372s
model over mere 6-35 epochs. We
13.29
64.62
2.9h
458s
can reduce Q = 30 down to Q = to achieve faster speed if parallel fine- tuning not permitted.
30
12.17
64.93
5.7h
716s
Table 13 provides sequential fine-tuning time and unlearning inference time together with the forget and retain performance.
Table 14: Effect of reducing fine-tuning epochs.
tuning Epach
Forget ())
(t)
tine-tuning
1/3 of 6-35 epochs
15.54
65.
2.4h
Table 14 provides sequential fine-tuning time and the forget and retain performance under the epochs range (6-35) reduced by 1/2 or 1/3.
1/2 of 6-35 epocls
14.28
65.06
3.2h
epochs
64.93
Table 15: Effect of vary- ing À (CLIP ViT-B/16, SUN397).
H.7 Impact of A on Taylor Expansion
Task vectors are combined with weight 0 Á 1. Thus, the X² term in the second-order part of the Taylor expansion decays quadratically, while
A
Forget (.)
Retain (1)
0.1
61.4
67.6
the first-order term À linear. practice, À is chosen by cross-validation. For very low 2 method reverts the 1st-order expansion. Table
0.2
57.5
66.7
0.3
53.6
66.2
15 investigates results w.r.t. varied X (CLIP ViT-B/16) on SUN397. Bold indicates the À value achieving the best performance in cross-validation.
0.4
65.7
0.5
45.9
64.8
0.6
42.0
65.1
0.7
40.1
62.8
0.8
39.2
60.3
H.8 Variance (x) Exx)
In experiments we assumed each μ4, for =1 1,..., C is independent for simplicity and f.(-) produces likelihood of the 7th class. As some task vectors may be noisy for some classes, targeted limiting class-wise variance only. the future, we will consider reducing covariance (off-diagonal terms) to decorrelate the model. This will require rethinking the Taylor expansion to produce the outer product of shape C x C..
H.9 Details of the Bienaymé Formula
For fixed sample one may think of f() as transformation of random variable (task vector sampled from the Dirichlet distribution (simplex)) into another random variable living in the class space (surface of simplex) which follows some class distribution resulting from this transformation. The Bienaymé formula tells what happens with variance of each class (treated independently) as the number of ensembled functions grows. That number depends on Q task vectors. However, is extreme case, one
required to be low. The opposite means each Q functions are not independent. could have Q identical functions which indeed cannot reduce the variance if they all are identical. To help reduce the variance, we investigated optimizing w; or T; in a small radius. Reducing off-diagonal terms of covariance could strengthen this effect.
24

---

References
[68] Yifei He, Yuzheng Hu, Yong Lin. Tong Zhang, and Han Zhao. merging via sparse task arithmetic. Transactions Machine Learning Research, 2025.
Efficient model
[69] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. The Conference on Uncertainty in Artificial Intelligence, pages 876-885, AUAT Press, 2018
[70] P. Stein. note on the volume of simplex. The American Mathematical Monthly, 73(3):299-33) 301, 1966.
[71] Walter J. Wadycki, B. K. Shah, P. D. Ghangurde, Edward J. Dudewicz. Nathan Mantel, Charles C. Brown. Harold Larson, Donald Barr, James Frane, Bernard Saperstein, Good, Howard Jones. Letters to the editor. The American Statistician, 27(3):123-227, 1973.
[72] Siqi Zeng, Yifei He, Meitong Liu, Weiqiu You, Yifan Hao, Hao-Hung Hubert Tsai, Makoto Yamada, and Han Zhao. Task vector bases: A unitied and scalable framework for compressed task arithmetic, 2025.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the main contributions and scope of the paper, providing well-motivated rationale for pursuing machine unlearning. The presented claims are substantiated by both rigorous theoretical analysis and comprehensive experimental evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Appendix F.2.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: The paper presents thorough theoretical results, with each theorem accompa- nied by explicitly stated assumptions. Detailed formal proofs provided in the appendix, while intuitive explanations are included in the main text to enhance clarity.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Detailed experimental setups are provided in Appendix A. For each experi- ment/anaylsis, we also provide its corresponding configuration for clarity.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Detailed experimental setups for reproducing results are provided in Appendix A. All the datasets used in the paper are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: (oodeSummissioonoo
/ //niss. wwtpcccpppcc.aecccccccccccccttlt http://pww.ggcppe/////e/eeeeea/e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Full details of training and evaluations are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeat each experiment five times under different random seeds and report the mean performance. Consistent trends across runs indicate the robustness of our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The required computing resources our setting are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://nnuris... http://eprppp..ppppiiiicccc////t httpppbii//tthhitcddeleeeeeeeeee ines?
Answer:
[Yes]
Justification:
We have thoroughly checked the NeurIPS Code of Ethics for our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader positive impacts are discussed in Appendix F.I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the external resources we used are properly mentioned and credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
31

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttps/////euu..eeeeeee www.urrps..c/oooffeeeeeeeeeeeeee /tt/////errnccee/225//////////// 20LL/LLM for what should or should be described.
32


Reviews:
[{"review_id": "kfMA9tglQf", "reviewer": "Reviewer_uLHj", "summary": "This paper addresses the problem of machine unlearning in Vision-Language Models (VLMs) by proposing a closed-form function ensembling method based on the Task Arithmetic Simplex. Traditional task vector approaches suffer from unstable unlearning due to sensitivity to fine-tuning configurations, with prediction-level variance negatively correlating with unlearning performance. The authors model task vectors as vertices of a (Q-1)-dimensional simplex, deriving a closed-form ensemble of infinite functions via Dirichlet distribution and Taylor expansion, which effectively reduces variance and enhances unlearning. Experiments on 8 visual datasets validate the method’s superiority, especially in incremental unlearning and linear task vector scenarios.", "questions": "1. **Theoretical Justification for Simplex Modeling**: Please provide mathematical proof or empirical evidence (e.g., t-SNE visualization of task vector convexity) to support the simplex modeling assumption. How do non-linear task dependencies affect this framework?\n2. **Cross-Modal Generalization**: Are there plans to validate the method on language or multi-modal models? Please include cross-modal experiments to demonstrate generalizability.\n3. **Large Perturbation Robustness**: Could you provide performance data under varying λ to validate the Taylor expansion’s accuracy beyond small perturbation scenarios?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. **Comprehensive Ablation Studies**: The paper conducts rigorous ablation experiments to decompose the contributions of key components (e.g., advanced aggreagation, vertex importance weighting), clarifying the impact of each module on unlearning performance. \n2. **Theoretical Innovation and Practical Efficacy**: Modeling task vectors as a simplex and deriving closed-form ensembles using convex geometry and Dirichlet distribution addresses the high computational cost of traditional methods. Experiments demonstrate significant accuracy gains on forgetting sets (e.g., 9.98% on ViT-Large/14) with stable retained set performance, validating both theoretical rigor and practical utility.\n\nWeaknesses:\n\n1. **Unjustified Simplex Modeling of Task Vector Space**: The authors propose modeling task vectors within the convex hull of a simplex but lack rigorous justification for this assumption. There is no mathematical proof or empirical analysis (e.g., geometric visualization of task vector distributions) to validate that task vectors indeed form a convex simplex. This raises questions about whether linear combinations of task vectors remain within the valid task space, especially given the non-linear dependencies inherent in real-world fine-tuning scenarios .\n2. **Computational Cost and Dataset Limitations**: While closed-form solutions avoid infinite sampling, generating Q task vectors (e.g., Q=30) requires multiple fine-tunings, imposing high costs for large models. Additionally, experiments are confined to vision datasets, lacking validation on language or cross-modal tasks.\n3. **Theoretical Constraints on Taylor Expansion**: The method relies on small perturbation assumptions (λ), but the paper omits comparative experiments under large perturbations, where second-order approximation validity may decline.", "was_revised": true, "final_justification": "The authors' response has addressed many of my concerns, so I am raising my recommendation to Borderline Accept.", "rebuttal": "# Response to Rev. uLHj\n\nWe thank Rev. for constructive feedback.\n\n### **1. Validate task vectors form a convex simplex.**\n\nThank you. This is a complex question.\n\\\n\\\n**Kindly note that we do not claim task vectors follow any specific distribution.** Task vectors have $m=86M$ dim. (CLIP ViT-B-16). For low number of vectors ($Q=30\\ll 86M$), **there exists no technique** that can reliably estimate the distribution to tell its kind due to the curse of dimensionality - one needs $Q\\gg 86M$ to reliably estimate. Our $Q=30$.\n$\\let\\b\\boldsymbol\\let\\th\\theta\\let\\al\\alpha\\let\\la\\lambda\\let\\ta\\tau\\let\\da\\downarrow\\let\\ua\\uparrow$\n$\\newcommand{\\fg}{Forget (\\da)}\\newcommand{\\rt}{Retain (\\ua)}\\newcommand{\\tz}{{\\b\\th}\\_0}\\newcommand{\\ti}{{\\b\\th}\\_i}\\newcommand{\\tj}{{\\b\\th}\\_j}$\n\\\n\\\nAs rebuttal does not allow figures, we will add t-SNE in revision. But we chose Dirichlet (educated choice) as under large $m=68M$ dim., we have:\n- simplicity to set/estimate PDF  (for Dirichlet, our task vectors set PDF)\n- ability to sample from it (our Taylor derivation does that)\n- compact support (not exceeding observed min-max values of individual params.) (**1a**)\n- ability to exploit 0th, 1st, 2nd order moments (simplex mean, implicit cov.) in modeling - Dirichlet lets us use of low-order moments in Taylor expansion (**1c**)\n\n### **1a. Intuitive reasoning.**\n\nNotice that:\n- Few-epochs fine-tuning $\\tz$ on unlearning dataset toward $\\ti$ yields sparse change: $\\sum\\_{j=1}^{86M} {\\mathbb 1}( |\\th\\_{j,0}-\\th\\_{j,i}|>\\epsilon)\\ll m$ for small $\\epsilon$. [A] confirms sparsity of $\\b\\ta$. **For $\\epsilon=1e-4$ (CLIP ViT-B/16), we checked that only 4.6% parameters changed.**\n  \\\n  \\\n  **[A].** *Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach, ICLR'25.*\n  \\\n  \\\n  Thus, **we assume that interpolating between such sparse differences in $\\ti$ and $\\tj$** captures a feasible parameter subset pertinent to the unlearning dataset, and realizes several magnitudes of \"active\" parameters.\n\n- **As counterexample, let task vectors be Normally distributed**:  ${\\b\\th}'\\sim\\mathcal{N}(\\b{\\mu},\\b{\\sigma}^2)$ (off-diagonal is 0; cannot estimate $86M\\times 86M$ dim. covariance) but $\\b{\\mu}=\\frac{1}{Q}\\sum\\_{i=1}^Q\\ti$ and $\\b{\\sigma}^2=\\frac{1}{Q-1}\\sum\\_{i=1}^Q (\\ti-\\b{\\mu})^2$ and so we obtain ${\\b\\th}'=\\b{\\mu}+\\b{\\sigma}\\odot\\bf{v}$ where ${\\bf v}\\sim \\mathcal{N}(\\bf{0},\\bf{1})$.\n  \\\n  \\\n  Table below (CLIP ViT-B-16): function-level ensemble $\\frac{1}{S}\\sum_{i=1}^S f({\\bf x};{\\b\\th}'\\_i)$ under the Normal dist. ($S$ task vectors ${\\b\\th}'$ sampled from it): \n\n  **TABLE**\n  |Distribution|$\\fg$|$\\rt$|\n  |-|-|-|\n  |Normal ($S=100$)|16.80|64.32|\n  |Normal ($S=300$)|15.98|64.54|\n  |**Dirichlet (Ours)**|12.17|64.93|\n\n  The Norm. dist. performs worse than our Dirichlet model as **PDF arms decay slowly toward $\\infty$ whereas Dirichlet dist. has compact support**: it produces finite values of filter parameters constrained by simplex.\n\n### **1b. Theory/analysis.**\nWe provide a rigorous bound: interpolated task vector model deviates from the expected convex combination by at most $L\\la\\eta$. \n\\\n\\\nThe bound is small if:\n- task vectors differ by a small diameter $\\eta=\\max_{i\\neq j}\\lVert{\\b\\ta}\\_i, {\\b\\ta}\\_j\\rVert_2$\n- $\\la$ is small\n- $g$ is smooth (low Lipschitz const. $L$)\n\nThese conditions are met in our paper (low $\\eta$ due to sparse change between task vectors, low $\\la<1$).\n\n**Theorem: Interpolation Performance Bounds.**\n\nLet $g:\\th\\to\\mathbb{R}$ be the unlearning function. Let $g$ be $L$-Lipschitz continuous:\n$$\\lVert g(\\ti)-g(\\tj)\\rVert_2\\leq L\\lVert\\ti-\\tj\\rVert_2\\quad\\forall i\\neq j$$\n\nGiven task vectors $\\{\\b\\ta_1,\\ldots,\\b\\ta_Q\\}$ and any convex combination $\\b\\ta=\\sum\\_{i=1}^Q\\al_i\\b\\ta_i$ where $\\sum_{i=1}^Q\\al_i=1, \\al_i\\geq 0$, we have:\n\n$$\\bigg\\lVert g(\\tz-\\la\\b\\ta)-\\sum_{i=1}^Q\\al_i g(\\tz-\\la\\b\\ta_i)\\bigg\\rVert_2\\leq L\\la\\eta$$\n\n**Theorem's Importance: any interpolated choice for $(\\al_1,\\ldots,\\al_Q)$ deviates from expected convex combination by less/equal $L\\la\\eta$.** We can provide Proof in discussions (rebuttal space limits).\n\n### **1c. Analyzing Taylor orders (how do non-linear task dep. affect framework).**\n\n-  Vector task aggregation model [A] has the same 1st order Taylor expansion as our method (**it equals also to NTK**):\n $$\\underbrace{f\\bigg({\\bf x};\\tz-\\frac{\\la}{Q}\\sum_{i=1}^Q\\b{\\ta}\\_i\\bigg)}\\_{\\text{approach [A]}}\\approx f({\\bf x};\\tz)-\\la\\bigg\\langle J\\_f(\\tz), \\underbrace{\\frac{1}{Q}\\sum\\_{i=1}^Q\\b{\\ta}\\_i}\\_{\\b\\mu\\_\\ta}\\bigg\\rangle= f({\\bf x}; \\tz)-\\la\\bigg\\langle J\\_f(\\tz,\\underbrace{\\frac{1}{|\\Delta\\_\\th|}\\sum\\_{\\b\\ta\\in\\Delta\\_\\th}\\b{\\ta}}\\_{\\b\\mu\\_\\ta}\\bigg\\rangle\\approx\\underbrace{\\frac{1}{|\\Delta\\_\\th|}\\sum\\_{\\b\\ta\\in\\Delta\\_\\th} f\\bigg({\\bf x}; \\tz-\\la\\b{\\ta}\\_i\\bigg)}\\_{\\text{our method}}$$\n  As 0th \\& 1st order moments for our technique are identical with [A] \\& [B], **integration over simplex is not an issue under this expansion** by virtue of methods [A] \\& [B].\n\n   **[B].** *Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML'22*\n\n- Now we compare 2nd order expansions:\n  - (approach [A]):  $\\qquad\\qquad\\qquad\\\\; f\\big({\\bf x};\\tz-\\frac{\\la}{Q}\\sum_{i=1}^Q\\b{\\ta}\\_i\\big)\\approx f({\\bf x};\\tz)-\\la\\langle J\\_f(\\tz), \\b\\mu\\_\\ta\\rangle+\\la^2 \\b\\mu\\_\\ta^T H\\_f(\\tz)\\b\\mu\\_\\ta$\n  - (basic function ensemble [B]): $\\frac{1}{Q}\\sum\\_{i=1}^Qf({\\bf x};\\tz-\\la\\b{\\ta}\\_i\\ )\\approx f({\\bf x};\\tz)-\\la\\langle J\\_f(\\tz), \\b\\mu\\_\\ta\\rangle+\\la^2 \\frac{1}{Q}\\sum\\_{i=1}^Q\\b\\ta\\_i^T H\\_f(\\tz)\\b\\ta\\_i$\n  - (our Simplex ensemble): $\\\\;\\\\;\\\\;\\frac{1}{|\\Delta\\_\\th|}\\sum\\_{\\b\\ta\\in\\Delta\\_\\th} f({\\bf x};\\tz-\\la\\b{\\ta}\\_i)\\approx f({\\bf x};\\tz)-\\la\\langle J\\_f(\\tz), \\b\\mu\\_\\ta\\rangle+\\underbrace{\\rho\\la^2\\b\\mu\\_\\ta^T H\\_f(\\tz)\\b\\mu\\_\\ta+\\zeta\\la^2\\frac{1}{Q}\\sum\\_{i=1}^Q\\b\\ta\\_i^T H\\_f(\\tz)\\b\\ta\\_i}\\_{\\text{interpolation between [A] and basic func. ensemble}}$ for $\\rho=Q/(Q+1)$ and $\\zeta=1/(Q+1)$.\n  \\\n  \\\n  As 0th and 1st order terms are identical, and our **Simplex ensemble interpolates between two known approaches [A] (\"Vector Uniform Merge\" in our paper) and [B] (\"Function Ensemble\"  in our paper)** for 2nd order expansion (convex functions), **which is why it is OK to integrate functions over task vectors uniformly drawn from the Dirichlet dist.** (our simplex ensemble).\n  \\\n  \\\n  Use of concentration parameter $\\al$ of Dirichlet leads to $\\rho=\\al Q/(\\al Q+1)$ and $\\zeta=1/(\\al Q+1)$ which **lets smooth interpolation** between [A] \\& [B].\n\n    **TABLE** (CLIP ViT-B-16)\n    |$\\al$|$\\fg$|$\\rt$|\n    |-|-|-|\n    |0.1|14.65|64.49|\n    |0.5|12.83|64.26|\n    |**0.8**|12.05|64.68|\n    |1.2|12.48|64.85|\n    |1.5|12.70|64.92|\n    |2.0|13.29|64.81|\n    |**1.0**|12.17|64.93|\n\n    If we use Eq. (6) (paper), weight $w_i$ per vertex customizes interpolation (help reduce var.)\n\n### **1d. Parameter augmentation.**\n\nAs averaging all task vectors gives viable unlearning (model [A]), and ensembling functions on individual task vectors is viable unlearning (model [B]), interpolation between the mean $\\frac{1}{Q}\\sum_{i=1}^Q\\b{\\ta}\\_i$ and individual $\\b{\\ta}\\_j$ is \"parameter augmentation\". Let\n$p({\\bf x}; \\Omega)=\\frac{1}{|\\Omega|}\\int\\_{\\b{\\ta}\\in\\Omega} f\\big({\\bf x};\\tz-\\b{\\ta}\\big)d\\b{\\ta}.$\nBased on **1b**, if $f$ changes fast in $\\Omega_{fast}$ \\& slow in $\\Omega_{slow}$ (same neighborhood sizes) then intuitively entropy of prediction $Ent(p({\\bf x}, \\Omega_{slow}))$ is lower than $Ent(p({\\bf x}; \\Omega_{fast}))$ as smooth changes contribute coherently to vote - chaotic changes are incoherent leading to cancellation of class peaks: $Ent(vec(1/C))$ is max. So our method  leverages smoothness of $f$ w.r.t. augmentation. See also [C]:\n\n**[C].** *Averaging Weights Leads to Wider Optima and Better Generalization, UAI'19*\n\n### **1e. Do task vectors form simplex?**\n\nThey do not have to so long they are close to simplex. We take $Q=10$ out of 30 task vectors fine-tuned on SUN397 and build simplex. We count sketch dimensions to $Q=d+1=11$. Remaining 20 task vectors are in $0.9(\\eta/2)$ radius of the simplex while task vectors of Cars exceed $1.6(\\eta/2)$ distance (cluster around their own simplex).\n\n### **2a. Cost: Many fine-tunings.**\n\nWe fine-tune over mere 6-35 epochs (not 1000). We can reduce $Q=30$ to 10 for speed or fine-tune in parallel.\n\nSee sequential fine‑tuning time/unlearning inference time (our paper's setup):\n\n**TABLE** (CLIP ViT-B-16) \n|Task Vector Number $Q$|$\\fg$|$\\rt$|Fine-tuning Time|Unlearning Inference Time|\n|-|-|-|-|-|\n|10|14.06|64.77|1.9h|372s|\n|15|13.29|64.62|2.9h|458s|\n|30|12.17|64.93|5.7h|716s|\n\nBelow we reduce epochs range (6-35) by 1/2 or 1/3rd:\n\n|Fine-tuning Epoch Percentage|$\\fg$|$\\rt$|Fine-tuning Time|\n|-|-|-|-|\n|1/3 of 6-35 epochs|15.54|65.14|2.4h|\n|1/2 of 6-35 epochs|14.28|65.06|3.2h|\n|5-35 epochs|12.17|64.93|5.7h|\n\n### **2b. Lang. or cross-modal task.**\n\nWe apply our method on multi-modal CLEAR benchmark (Dontsov et al., 2024: fictional author profiles with face images-captions pairs). See the avg. accuracy (VQA task) (LLaVA‑1.5‑7B, CLIP ViT‑L/14 as vision encoder). Task vectors are derived by fine‑tuning on the corresponding target data.\n\n**TABLE**\n|Method|VQA Acc. $\\fg$|VQA $\\rt$|\n|-|-|-|\n|Pre-trained Model|69.2|55.7|\n|Standard Task Arithmetic|42.7|49.4|\n|Uniform Merge|37.9|50.5|\n|TIES-Merging|36.1|49.7|\n|EMR-Merging|34.8|49.3|\n|Function Ensemble|35.6|50.0|\n|**Ours**|31.5|50.2|\n\n### **3.Constraints on Taylor.**\n\nKindly note $0\\leq\\la\\leq 1$: $\\la^2$ in 2nd order term decays quadratically; 1st order term $\\la$ is linear. In practice, $\\la$ is chosen by cross-validation. **For low $\\la^2$ our method reverts to 1st order expansion (as [A] \\& [B]).** Below we vary $\\la$ (CLIP ViT-B/16) on SUN397. Bold: $\\la$ best in cross-val. \n\n**TABLE**\n|$\\la$|$\\fg$|$\\rt$|\n|-|-|-|\n|0.1|61.4|67.6|\n|0.2|57.5|66.7|\n|0.3|53.6|66.2|\n|0.4|49.8|65.7|\n|0.5|45.9|64.8|\n|**0.6**|**42.0**|**65.1**|\n|0.7|40.1|62.8|\n|0.8|39.2|60.3|"}, {"review_id": "XXE43WGYHP", "reviewer": "Reviewer_5xSJ", "summary": "This paper addresses the problem of machine unlearning in large vision-language models (VLMs) like CLIP. Machine unlearning refers to efficiently removing or “forgetting” the influence of certain training data (the forget set) from a model without retraining from scratch, to meet privacy regulations (e.g. the “right to be forgotten” under GDPR). The authors build on the concept of task vectors – differences in model parameters when fine-tuning with vs. without the target data (as used in Editing Models with Task Arithmetic by Ilharco et al., ICLR 2023). In prior work, subtracting a task vector from the model could forget a dataset in a plug-and-play way, but the effectiveness varied greatly depending on fine-tuning specifics. This paper’s key idea is to mitigate that variance by using ensembles of many task vectors. They propose modeling the space of possible task vectors (obtained under different fine-tuning conditions) as a simplex (convex hull) and derive a closed-form solution that aggregates an infinite ensemble of models sampled from this simplex. In simpler terms, instead of relying on one fine-tuned model difference, they analytically average out an infinite number of fine-tuning outcomes, which dramatically reduces prediction variance and improves forgetting performance.", "questions": "Computing a full Hessian for CLIP seems infeasible - could you clarify how Equation (3) was implemented? Also, how important are the second-order terms versus first-order?\n\nCLIP ViT-L/14 is large, but how about something like CLIP with Vision Transformer Huge or a future model with billions of parameters – would the method scale? The concern is memory/time for fine-tuning many large models. Do you foresee any obstacles in scaling up, or does the approach parallelize well enough (fine-tuning can be embarrassingly parallel for each task vector)? Also, for very large output spaces (say thousands of classes), does the advanced aggregator (Theorem 2) or variance computation become too slow? It might be helpful to discuss any scalability tricks or limitations.\n\nYour approach makes it easier to remove knowledge of data, which is positive for privacy. Could it be misused in any way? For instance, one could intentionally “unlearn” important facts from a model (for example, sabotage a facial recognition system by making it forget certain identities). This would require access to the model and data, so it’s not a big threat model, but it’s worth considering if making models so flexible could have downsides. Another angle: might repeated unlearning degrade a model in unforeseen ways?", "limitations": "A notable limitation is the computational cost and workflow complexity of the approach. It requires multiple fine-tuning runs and careful orchestration (especially if doing sequential unlearning, one must manage updating the base model). In environments with limited computational resources, this could be challenging. However, this is a trade-off: any effective unlearning that is easier than full retraining will have some cost, and here it’s parallelizable fine-tuning jobs which many organizations can manage.\n\nAs discussed, the approach focuses on classification benchmarks. It’s not directly tested on, say, open-vocabulary CLIP tasks or retrieval tasks. So one limitation is task specificity: forgetting is measured in terms of classification accuracy. If the requirement was to forget certain image-text pair associations in CLIP (like CLIP should not associate a particular image with a caption because the caption is private), one might need to adapt the method (maybe fine-tune CLIP’s multimodal head on that association). The method should conceptually extend, but it may require careful setup for different objectives.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Paper empirically demonstrate that increasing the number of task vectors in an ensemble improves unlearning, showing a clear negative correlation between ensemble prediction variance and forget-set accuracy. Authors introduce the task simplex, a geometric construct capturing a diverse set of task vectors, and show that sampling within this simplex effectively yields interpolations corresponding to “multi-task” unlearning. Additionally, there is  derive a closed-form ensemble method using a second-order Taylor expansion and properties of the Dirichlet distribution, to integrate over infinitely many models from the simplex without brute-force sampling. This yields an analytic solution that can be applied to the original model to forget the data. Moreovere, the paper extend this to a probabilistic aggregator (based on probability of “at least one” model predicting a class) to further boost reliable forgetting. This work also demonstrate how to distill the effects of the infinite ensemble into a single model (a single “unlearning” task vector), making deployment practical. \n\nExtensive experiments on 8 image classification datasets and the ImageNet retain set show that their method outperforms state-of-the-art unlearning methods, including single task vector subtraction (Ilharco et al. 2023), model weight merging techniques (e.g. Wortsman et al.’s Model Soup, ICML 2022), and other ensemble or linearization baselines. They also show their approach supports incremental unlearning – sequentially forgetting multiple datasets one after another – with strong results. The results are impressive. For example, on CLIP ViT-B/32 (Table 1), the average forget-set accuracy (lower is better, since 100% would mean the model still remembers everything) for a single task vector was 24.2%. Their method brings this down to 15.2% – a large improvement (the forget sets are small, so a lower accuracy means the model is doing poorly on them, which is good for forgetting). This outperforms even the strongest baseline (EMR merge got ~21.8%, function ensemble of 30 models ~22.7%). They achieve similarly large gains on the larger CLIP models: e.g., on ViT-L/14, they go from ~16.7% (best prior) down to ~9.98% forget accuracy – essentially halving the residual accuracy on sensitive data, which indicates a very thorough forgetting. Notably, their distilled single model version only slightly regresses (e.g., 15.66% vs 15.20% on ViT-B/32), showing that you don’t actually need to keep multiple models around — one can reap the benefits in a single model after distillation.\n\nOverall, the paper introduces a novel ensemble-based unlearning strategy that is both theoretically grounded and empirically effective, pushing forward the capabilities for efficient post-hoc removal of training data from large models.\n\n===\n\nIf any, one might point out that the method’s computational overhead is front-loaded (fine-tuning 30 models per task). The paper frames this in a positive light (no need to retrain the large model from scratch, which is indeed much more costly than 30 fine-tunings). But 30 fine-tunings with augmentations could still be heavy for very large models or very many forget requests. The authors don’t explicitly report how long those fine-tunings take; however, since they did it for 8 datasets and multiple architectures, it was clearly feasible.", "was_revised": true, "final_justification": "The authors' rebuttal comprehensively addresses my points: HVP enables feasible second-order computation (3s per HVP), second-order terms are key (+2.2% over first-order), LoRA scales to ViT-Huge (8.74% forget acc.), aggregator remains fast (seconds, not hours), and new VQA results show multimodal applicability (31.5% forget vs. 34.8% best prior). Ethical misuse is now discussed, and repeated unlearning preserves ~90% retain acc. These additions (with tables) mitigate the front-loaded cost limitation and extend beyond classification, reinforcing the method's novelty, rigor, and impact in VLM unlearning. The authors did address all the queries with the rebuttal. This is a technically solid paper with high impact in AI privacy/unlearning, excellent evaluation, and no ethical issues. I stand by my original accept recommendation.", "rebuttal": "# Response to Rev. 5xSJ\n\nWe thank Reviewer for constructive feedback.\n\n### **1. Computational overhead is front-loaded (fine-tuning 30 models per task).**\n\nThank you. Kindly note we can vary $Q=10,15,30$, etc. Moreover, we can vary number of fine-tuning epochs (currently in range 6-35) to 1/2 or 1/3rd (e.g., range 2-12) which is significantly lower than model pre-training/training from scratch. Finally, obtaining $Q$ tasks can be parallelized. \n\nTables below show results w.r.t. $Q$ and the number of fine tuning epochs, respectively.\n\n**TABLE** (CLIP ViT-B-16) \n|Task Vector Number $Q$|Forget ($\\downarrow$)|Retain ($\\uparrow$)|Fine-tuning Time|Unlearning Inference Time|\n|-|-|-|-|-|\n|10|14.06|64.77|1.9h|372s|\n|15|13.29|64.62|2.9h|458s|\n|30|12.17|64.93|5.7h|716s|\n\n\n|Fine-tuning Epoch Percentage|Forget ($\\downarrow$)|Retain ($\\uparrow$)|Fine-tuning Time|\n|-|-|-|-|\n|1/3 of 6-35 epochs|15.54|65.14|2.4h|\n|1/2 of 6-35 epochs|14.28|65.06|3.2h|\n|6-35 epochs|12.17|64.93|5.7h|\n\n### **1b. One must manage updating the base model.**\n\nOnly fine-tuning from $\\boldsymbol\\theta\\_0$ to $\\boldsymbol\\theta\\_i$ task vectors. However, unlearning itself is performed by mere Taylor expansion -- unless distillation is desired to \"compact\" the model.\n\n### **2. Could you clarify how Equation (3) was implemented?**\n\nEq. (3) uses so-called Hessian-Vevtor Product (HVP) which requires the use of **vmap**, **jacrev** and **hvp** from torch.func and torch.autograd.functional. HVP never computed the Hessian matrix but directly and efficiently obtains $\\bf{v}=(\\bf{H}\\cdot\\boldsymbol\\tau)$. Vector $\\bf{v}$ then can be vector-multiplied with $\\boldsymbol\\tau$. HVP takes 2-4x the Jacobinan computation which makes is extremely efficient. In our case, one HVP takes approx. 3 seconds.\n\n\n### **3. How important are the second-order terms versus first-order?**\n\nResp. 1 to Rev. uLHj shows that under 1st order Taylor expansion, our approach degrades to the 1st order Taylor expansion of [A] (\"Vector Uniform Merge\" in our paper) and [B] (\"Function Ensemble\"  in our paper)**, and to the NTK model:\n\n**[A].** *Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach, ICLR'25.*\n**[B].** *Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML'22*\n\nHowever, the 1st order Taylor expansion is not the same as running [A] (think full expansion). Table below compares results (CLIP ViT-B/16) between the 1st and 2nd order solutions:\n\n**TABLE**\n|Method|Forget ($\\downarrow$)|Retain ($\\uparrow$)|\n|-|-|-|\n|Ours (1st order only)|17.19|64.52\n|Ours (1st & 2nd order)|12.17|64.93|\n\n### **4. Would the method scale to ViT Huge with billions of parameters ?**\n\nYes, in a way it is always cheaper to run few epochs of fine-tuning than retrain entire ViT-H. Indeed, $Q$ task vectors can be obtained in parallel. To save memory, ViT-H could be equipped with LORAs, i.e., each $\\boldsymbol\\tau_i={\\bf AB}^T$ for some tall (low rank) matrices ${\\bf A}$ and ${\\bf B}$. For ViT‑H with LoRA adaptation (rank‑16), fine‑tuning reduces to ~12h, while inference remains ~14 minutes, since LoRA adds negligible overhead to the forward pass.\n\n**TABLE**\n|Backbone|Forget ($\\downarrow$)|Retain ($\\uparrow$)|\n|-|-|-|\n|ViT-H (Pre-trained)|67.40|79.13|\n|ViT-H (LoRA)|8.74|73.59|\n\n ### **5. Does the advanced aggregator (Theorem 2) become too slow?**\n\nThank you. No, the main cost is in obtaining $Q$ task vectors. The overall Taylor approximation takes seconds not hours. \n\nGenerally, Theorem 2 requires $2C$ runs of **jacrev** and $2QC$ **hvp** runs compared to the basic approach that uses $QC$ **hvp** runs. Variance minimization does not require extra computations as **jacrev** and **hvp** are pre-computed and shared with the basic method. Storing $30 (Q)\\times 86M (params)\\times 8 (double)$ requires 19GB memory which can be kept on the GPU until we move to process the next class in $1,\\ldots,C$. Where needed, for larger $Q$ or ViT-H (632M), vectors can be stored in CPU RAM and moved around, they can be computed on several GPUs (e.g., one **hvp** per GPU), or LORA can be used to limit the size of parameters.\n\nBelow is table showing computation time excluding task vector fine-tuning for several variants of our method:\n\n**TABLE** (1st, 2nd order, Theorem 2, variance tuning) (CLIP ViT-B/16)\n|Method|Forget ($\\downarrow$)|Retain ($\\uparrow$)|Unlearning Inference Time|\n|-|-|-|-|\n|1st order w/o variance tuning|17.19|64.52|393s|\n|1st & 2nd order w/o variance tuning|14.36|64.79|471s|\n|1st & 2nd order w/ variance tuning (Ours)|12.17|64.93|716s|\n\n ### **6. Removing knowledge is positive for privacy. Could it be misused in any way?**\n\nThank you. Absolutely, as any AI tool, there exists always a possibility of misuse. We will make it clearer in the **Broader Impact and Limitations** section. Indeed, a rouge actor with access to the model and unlearning data could intentionally “unlearn” important facts from a model. Indeed, our approach provides mere functionality but this is not a model that can monitor fairness of unlearning tasks.\n\n ### **7. Might repeated unlearning degrade a model in unforeseen ways?**\n\nYes, this is a very interesting question. Unlearning could degrade the ability of $\\boldsymbol\\theta\\_0$ to deal with some class labels which have strong semantic correlation with classes we unlearn. However, as unlearning is merely obtained by task arithmetic, we can always store $\\boldsymbol\\theta\\_0$ and consecutive sets of task vectors to be able to unroll changes.\n\nBelow is the retain set performance (forget set accuracy can be found in Table 3 in the main text) on the retaining accuracy under incremental unlearning using CLIP ViT-Base/32:\n\n**TABLE**\n|Method|Cars|+DTD|+EuroSAT|+GTSRB|+MNIST|+RESISC45|+SUN397|+SVHN|\n|-|-|-|-|-|-|-|-|-|\n|Ours|60.7|60.4|60.0|59.4|58.8|58.1|57.6|57.0|\n\nNote that the pre‑trained model achieves 63.3% accuracy on the retain set; our incremental unlearning preserves roughly 90% of this performance on the test set and 95% on validation set.\n\n### **8. Open-vocabulary CLIP tasks or retrieval tasks.**\n\nTo demonstrate cross‑modal applicability, we additionally validate our method on the multimodal CLEAR benchmark (Dontsov et al., 2024), which involves fictional author profiles with paired face images and captions. As shown in the table below, we report forgetting and retention accuracy on the VQA task using LLaVA‑1.5‑7B (with CLIP ViT‑L/14 as the vision encoder). Our approach achieves the best forgetting while maintaining comparable retention performance.\n\n**TABLE**\n|Method|Forget VQA Acc. ($\\downarrow$)|Retain VQA Acc. ($\\uparrow$)|\n|-|-|-|\n|Pre-trained Model|69.2|55.7|\n|Standard Task Arithmetic|42.7|49.4|\n|Uniform Merge|37.9|50.5|\n|TIES-Merging|36.1|49.7|\n|EMR-Merging|34.8|49.3|\n|Function Ensemble|35.6|50.0|\n|**Ours**|31.5|50.2|\n\nConceptually, our task‑vector framework extends to other multimodal settings, including open‑vocabulary CLIP tasks and image‑text retrieval. Adapting to those scenarios would require defining task vectors on the corresponding objective (e.g., fine‑tuning CLIP’s multimodal head to capture specific image‑caption associations) and applying our unlearning procedure in the same manner. We will explore these directions in future work.\n\n### **9. Task specificity: forgetting is measured in terms of classification accuracy.**\n\nThank you. Certain image-text pair associations in CLIP would require obtaining task vectors on lager number of image-text unlearning pairs by just fine-tuning with the CLIP loss. Then $f$ could be replaced with the feature output of CLIP and the rest follows. However, Taylor expansion can operate on feature spaces too.\n\nKindly see Resp. 8 above where we work with VQA instead of classification."}, {"review_id": "nt7NXmt2XG", "reviewer": "Reviewer_vtZ9", "summary": "This work proposes a method for machine unlearning, which is an interesting and practically important problem. The authors point out that task vectors, a popular unlearning strategy, exhibit substantial sensitivity to various fine-tuning configurations, leading to unstable unlearning effectiveness. Obtaining and aggregating multiple task vectors can reduce prediction-level variance and improve unlearning results; however, this comes at a high computational cost.\n\nThis manuscript presents a method that captures the space of task vectors induced by diverse fine-tuning strategies. Specifically, it models this space through the convex hull of a \n(𝑄−1)-simplex whose vertices correspond to 𝑄 task vectors. Instead of sampling task vectors directly from the simplex, the authors derive a closed-form ensemble representing an infinite number of functions whose parameters are uniformly sampled from the simplex, thereby achieving enhanced unlearning performance in a computationally efficient manner.\n\nThe method appears to be supported by solid theoretical foundations. Furthermore, the experimental results and accompanying analyses demonstrate the effectiveness of the proposed approach.", "questions": "Overall, I find this paper interesting and well-motivated. Minor clarifications and improvements, however, would further strengthen the work.\n(1) This manuscript appears to be very theoretical. Providing more intuitive explanations or illustrative examples could significantly improve accessibility for a broader audience.\n(2) Regarding the organization of the “Proposed Method” section: there is only one numbered subsection titled “Problem Formulation.” This structure may lead to the misunderstanding that all subsequent content falls under problem formulation. It would be clearer to divide this section into multiple meaningful subsections.\n(3) In the paragraph beginning with \"Unlearning with Task Vectors\", the sentence that \"Moreover, our closed-form solution achieves the lowest accuracy (the lower the better) on the forget set, surpassing the state-of-the-art approach by 3.3% on ViT-Base/32-based CLIP.\" is not directly supported by the results in Table 1. Additionally, in the paragraph starting with “Unlearning with Linear Task Vectors,” Another sentence in the paragraph beginning with \"Unlearning with Linear Task Vectors\", the sentence \"Table 1 shows that the linearized task vectors consistently yield substantial reductions in the forget set accuracy 258 across multiple merging methods and CLIP architectures, while maintaining fixed accuracy on the 259 retain set relative to their standard (non-linearized) counterparts in Table 1.\" appears to mistakenly reference Table 1 twice, whereas the first mention likely should refer to Table 1.", "limitations": "Yes, but they put the discussion on limitations in Appendix F.2 instead of the main text.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-written and clearly organized.\nThe proposed method is theoretically sound.\nThe experimental results and analyses are thorough and convincing.\n\nWeaknesses:\nThe section titled “Proposed Method” contains many abstract equations, which may be difficult for readers who are not deeply familiar with this specific area to fully understand. Providing more intuitive explanations or illustrative examples could greatly improve accessibility.", "was_revised": true, "final_justification": "The authors have provided a clear rebuttal that addresses my main concerns. I believe the contribution is sufficient for acceptance.", "rebuttal": "# Response to Rev. vtZ9\n\nWe thank Reviewer for constructive feedback.\n\n### **1. Providing more intuitive explanations.**\n\nAbsolutely. The main idea is to devise a distribution easy to construct and sample give $86M$ parameter space (ViT-B). This can be easily achieved by an ensemble of functions based on Taylor expansion which leverages 0th, 1st and 2nd order statistics of these task vectors together. Taylor expansion is devised at $\\boldsymbol\\theta\\_0$.\n\n- We will illustrate such a Taylor expansion and resulting const, linear and quadratic terms. The linear term depends on the mean of task vectors, and the quadratic term on specific second order statistics which we will illustrate. Kindly notice rebuttal does not permit figures (alas).\n- We will also illustrate the ensemble in terms of the mean and variance for which we can adjust outliers using weights $w_i$.\n\n### **1. Divide \"Proposed Method” into multiple meaningful subsections.**\n\nAbsolutely. In general, we have the core \"Problem Formulation\":\n- Problem definition with Closed-form Aggregation of Functions from Task Simplex,\n\nfollowed by \"Extensions\" and \"Theoretical Analyses\":\n- Advanced Aggregation Scheme. \n- Distillation from Ensemble.\n- Computing and Controlling Variance of Ensemble.\n\n### **2. Sentence with \"the state-of-the-art approach by 3.3% on ViT-Base/32-based CLIP\" is not directly supported by Table 1.**\n\nAbsolutely. We had a typo due to changes in the paper. We meant that compared to *function ensemble [8]* we achieve 7.55\\%, 7.72\\% and 6.34\\% improvement in unlearning on ViT-Base/32, ViT-Base/16  and ViT-Large/14, respectively.\n\n### **3.  \"Table 1 shows...\" repeats Table 1 twice.**\n\nThank you. We have revised accordingly."}, {"review_id": "B57ClTW9Ir", "reviewer": "Reviewer_wwLa", "summary": "The paper proposes to improve unlearning by sampling task vectors from a simplex, which is composed of Q pivot task vectors. Authors discuss the potential of the method for reducing variance. Experiments are conducted on standard unlearning benchmarks with different VLMs.", "questions": "- I would think that ensemble is a general approach that can be applied to other pretrained models (e.g., LLM). Furthermore, by reducing variance, it may also help other task arithmetic operations besides unlearning (e.g., learning instead of unlearning, adding task vectors from different tasks). Can we have a brief discussion on different models and tasks?\n- On the variance of the ensemble\n  * Line 44, the notation of \\sigma(x) is somewhat unclear. In my understanding, the randomness of the ensemble is from the subscript i (i.e., randomly selecting f \\in R^C from Q possible task vectors), thus it is a multivariate random variable, and we should discuss the covariance rather than a scalar.the Does the notation there assume independence among f's components, or is f is a scalar (the probability of ground truth)?  \n  * Line 59, can you further explain the reference to Bienaymé formula? It basically says that the variance of a sum of independent variables is the sum of each r.v.'s variance. Here, the ensemble is a sum of f \\in R^C (each of which is a predicted C-dimensional distribution), but each prediction is not a random variable (as my understanding above, the randomness is from selecting Q task vectors, not from the classifier's output). Would you please clarify which random variable is studied here?  \n- Line 151, the Hessian term may contain subscript c.\n- Equation 2, I would suggest using integration, rather than summations,  over \\tau. For completeness, it is also helpful to specify how to compute the first-order term.\n- Equation 10/11, the \\sigma^2 term contains unclosed parentheses. \n- On experiments:\n  * Figure 3, I would suggest improving the function ensemble baseline by further increasing Q \n  * Table 3, in the incremental unlearning setting, does the ensemble method still keep the retain set performance?\n  * It would also be better to discuss limitations of the proposed methods (e.g., additional costs of computing/storing Hessian)\n  * Regarding line 604 (paper checklist), it would be better to open the code to help reproduce the results.", "limitations": "Please see the question section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Making task arithmetic more robust is an important question for unlearning (and model merging). The paper is well-motivated, and the empirical results are good.\n\nWeaknesses\n- Some notations are not clear (e.g., variance of function ensemble) \n- Some baselines can be improved (the proposed method is a general approach and can be applied to other scenarios besides VLMs)\n- Please see details in the question section.", "was_revised": true, "final_justification": "The authors' response has addressed some of my concerns, and I keep my original score.", "rebuttal": "# Response to Rev. wwLa\n\nWe thank Reviewer for constructive feedback.\n\n### **1. Ensemble is a general approach that can be applied to other pre-trained models (e.g., LLM).**\n\nThank you. Below is the table showing performance on a multimodal LLM, LLaVA‑1.5‑7B (with CLIP ViT‑L/14 as the vision encoder). We additionally validate our method on the multimodal CLEAR benchmark (Dontsov et al., 2024), which involves fictional author profiles with paired face images and captions. As shown in the table below, we report forgetting and retention accuracy on the VQA task. Our approach achieves the best forgetting while maintaining comparable retention performance.\n\n**TABLE**\n|Method|Forget VQA Acc. ($\\downarrow$)|Retain VQA Acc. ($\\uparrow$)|\n|-|-|-|\n|Pre-trained Model|69.2|55.7|\n|Standard Task Arithmetic|42.7|49.4|\n|Uniform Merge|37.9|50.5|\n|TIES-Merging|36.1|49.7|\n|EMR-Merging|34.8|49.3|\n|Function Ensemble|35.6|50.0|\n|**Ours**|31.5|50.2|\n\nConceptually, our task‑vector framework extends to other multimodal settings, including open‑vocabulary CLIP tasks and image‑text retrieval. Adapting to those scenarios would require defining task vectors on the corresponding objective (e.g., fine‑tuning CLIP’s multimodal head to capture specific image‑caption associations) and applying our unlearning procedure in the same manner. We will explore these directions in future work.\n\n### **2. Improving other task arithmetic operations besides unlearning.**\n\nWhile we focus predominantly on unlearning in this work, below we demonstrate incremental learning scheme. The reason why we focus on unlearning is that we model $\\boldsymbol\\theta\\_1,\\ldots,\\boldsymbol\\theta\\_Q$ as variations in task vectors for a given unlearning dataset. Therefore, we assume that:\n- ${\\boldsymbol\\theta}\\_i$ results only in sparse changes: $\\sum\\_{j=1}^{86M} {\\mathbb 1}( |\\theta\\_{j,0}-\\theta\\_{j,i}|>\\epsilon)\\ll m$ for small $\\epsilon$ (86M is the number of parameters for ViT-B).\n- maximum diameter between any two task vectors $\\eta=\\lVert\\boldsymbol\\theta_i-\\boldsymbol\\theta_j\\rVert_2\\quad\\forall i\\neq j$ is small.\n\nThese assumptions are required for modeling with the Dirichlet distribution as per Resp. 1 to Rev. uLHj. **For $\\epsilon=1e-4$ (CLIP ViT-B/16), we checked that only 4.6% parameters changed** (for unlearning) so as long as these assumptions are met (they are easily met for unlearning), we can also consider multi-task learning based on task vector simplices by adding the sum of the task vectors of 8 tasks (datasets): Cars , DTD, SUN397, EuroSAT, GTSRB, MNIST, SVHN, and RESISC45 following Task Arithmetic (Ilharco et al., ICLR 2023). We report the average absolute accuracy (%) w.r.t. CLIP ViT-B/16 in the table below. \n\n**TABLE**\n|Method|Absolute Acc. ($\\uparrow$)|\n|-|-|\n|Pre-trained Model|55.2|\n|Standard Fine-tuning|75.5|\n|Uniform Merge|78.2|\n|Function Ensemble|79.3|\n|Ours|83.6|\n\n ### **3. Line 151, the Hessian term may contain subscript.**\n\nAbsolutely. Thank you.\n\n ### **4. Equation 2, I would suggest using integration, rather than summations.**\n\nAbsolutely.\n\n  ### **5. It is also helpful to specify how to compute the first-order term.**\n\nAbsolutely, for the standard expansion we have:\n\n$$\\frac{1}{|\\Delta\\_\\theta|}\\sum\\_{\\boldsymbol\\tau\\in\\Delta\\_\\theta}\\boldsymbol{\\tau}=\\frac{1}{Q}\\sum\\_{i=1}^Q\\boldsymbol{\\tau}\\_i,$$\n\nwhile for Corollary 1 we have :\n\n$$\\frac{1}{|\\Delta\\_\\theta|}\\sum\\_{(w,\\boldsymbol\\tau)\\in\\Delta\\_w\\times \\Delta\\_\\theta}\\\\!\\\\!\\\\!w\\boldsymbol{\\tau}=\\frac{1}{\\sum_{j=1}^Q w\\_j}\\sum\\_{i=1}^Q w\\_i\\boldsymbol{\\tau}\\_i.$$\n\n### **6. Parentheses in Eq. 10/11.**\n\nDuly noted. \n\n### **7. Increase $Q$ in Figure 3.**\n\nAbsolutely. In Figure 3, we provide the comparison between function-level ensembles from task vectors and our task simplex w.r.t. the number of task vectors Q for CLIP ViT-B/32. In the following table, we further include the performance for $Q=40, 50$. We can notice the convergence of unlearning performance..\n\n**TABLE**\n|Method|Forget Set Acc. ($\\downarrow$)|\n|-|-|\n|$Q=20$|15.58|\n|$Q=30$|15.20|\n|$Q=40$|15.07|\n|$Q=50$|14.98|\n\n### **8. In Table 3, does the ensemble method still keep the retain set performance?**\n\nYes, within 95\\% of the original performance. \n\n### **9. Discuss limitations of the proposed methods (e.g., additional costs of computing/storing Hessian).**\n\n### **10. Regarding line 604, it would be better to open the code.**\n\nAbsolutely.\n\n### **11. The randomness of the ensemble is from the subscript i: should we have a full covariance?**\n\nThank you. This is very interesting question. We assumed each $\\mu_i$ for $i=1,\\ldots,C$ is independent for simplicity and $f_i$ produces likelihood of the $i$-th class. As some of $Q$ task vectors may be noisy for some classes, we assumed limiting that variance alone a bit may be sufficient - and improvements validate that. However, reducing covariance (off-diagonal terms) could indeed help also decorrelate the model. However, that would require some serious rethinking of the Taylor expansion in Eq. (2) to produce the outer product as a closed form solution because of the integration operator.\n\n### **12. The Bienaymé formula.**\n\nFor a fixed sample ${\\bf x}$, one may think of $f(\\cdot)$ as a transformation of random variable (task vector sampled from the Dirichlet distribution (simplex)) into another random variable living in the class space (surface of a simplex) which follows some resulting from this transformation class distribution. The Bienaymé formula tells us what happens with variance of each class (treated independently) as the number of ensembled functions grows. That number depends on $Q$ task vectors. However, $\\rho$ is required to be low. The opposite means each $Q$ functions are not independent - say in extreme case we have $Q$ identical functions, clearly they cannot reduce the variance because they all are identical. For that reason we investigated the idea of learning $w_i$ or even moving $\\boldsymbol\\tau_i$ in a small radius to help reduce the variance. We agree as well that reducing cross-terms in covariance could strengthen this effect."}]

Author rebuttal:

Meta-review: The paper considers machine unlearning in vision language models, where the problem is to 'forget' certain training examples. Prior works, which introduced task vectors, were computationally inefficient, unstable and highly dependent on fine tuning configurations.
The current work models the variation in task vectors by considering an ensemble of task vectors on a simplex to model variations in fine tuning configurations. This leads to a tangible boost in unlearning performance. 

Machine unlearning is a very important topic in the field and principled approaches such as the ones presented in the paper can be very impactful in practice. After the rebuttal phase, the authors have been positive about the paper. Thus, I recommend accepting the paper, and encourage the authors to make improvements proposed by the reviewers.
Decision: Accept (poster)