NegMerge: Sign-Consensual Weight Merging for Machine Unlearning
Authors: Hyo Seo Kim
Dongyoon Han
Junsuk Choe
Venue: ICML 2025 poster (2025)
Keywords: Machine Unlearning
Image Classification
Model Merging
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: NegMerge is a novel machine unlearning method that computes task vectors from multiple models, combining only elements with consistent signs to effectively induce forgetting while preserving the model's retained knowledge.

Abstract: Machine unlearning aims to selectively remove specific knowledge from a trained model. Existing approaches, such as Task Arithmetic, fine-tune the model on the forget set to create a task vector (i.e., a direction in weight space) for subtraction from the original model's weight. However, their effectiveness is highly sensitive to hyperparameter selection, requiring extensive validation to identify the optimal vector from many fine-tuned candidates. In this paper, we propose a novel method that utilizes all fine-tuned models trained with varying hyperparameters instead of a single selection. Specifically, we aggregate the computed task vectors by retaining only the elements with consistent shared signs. The merged task vector is then negated to induce unlearning on the original model. Evaluations on zero-shot and standard image recognition tasks across twelve datasets and four backbone architectures show that our approach outperforms state-of-the-art methods while requiring similar or fewer computational resources. Code is available at https://github.com/naver-ai/negmerge.

NegMerge: Sign-Consensual
Weight Merging for Machine Unlearning
Hyo Seo Kim
Dongyoon Han Junsuk Choe !
Abstract
challenging task. Machine unlearning (Koh & Liang, 2017; Golatkar et al., 2020; Thudi et al., 2022) offers solution by enabling models to erase specific knowledge without the need for full retraining.
Machine unlearning aims to selectively remove specific knowledge from a trained model. Ex- isting approaches, such Task Arithmetic, fine- tune the model on the forget set to create task vector (i.e., a direction in weight space) for sub- traction from the original model's weight. How- ever, their effectiveness is highly sensitive to hy- perparameter selection, requiring extensive vali- dation to identify the optimal vector from many fine-tuned candidates. In this paper, propose novel method that utilizes all fine-tuned mod- els trained with varying hyperparameters instead of a single selection. Specifically, we aggre- gate the computed task vectors by retaining only the elements with consistent shared signs. The merged task vector is then negated to induce un- learning on the original model. Evaluations on zero-shot and standard image recognition tasks across twelve datasets and four backbone architec- tures show that our approach outperforms state-of- the-art methods while requiring similar or fewer computational resources. Code is available at htts://github.com//// www/o//werai//eeennneeeeeeeeeeee
Despite promising results, many existing methods struggle to remove only the target knowledge while preserving the rest. This challenge arises because fine-tuning, in its effort to erase knowledge from the forget set (i.e., the data to be forgotten), often disrupts the knowledge preserved in the relain set (i.e., the remaining data) (Chen et al., 2023; Fan et al., 2023). A known method robust to this issue is Task Arithmetic (Ilharco et al., 2022a), which avoids directly fine-tuning the original model. It calculates task vector by determining the parameter- wise difference between the original model and separately fine-tuned model on the for- get set. The task vector is then subtracted from the original model. This process, referred to as forgetting by negution, has demonstrated strong unlearning performance while pre- serving the model's 's knowledge.
However, our findings reveal that achieving effective un- learning performance with this method requires careful hy- perparameter selection through validation process. This process is both time-consuming and computationally expen- sive, as it involves evaluating multiple fine-tuned models trained with different hyperparameters. Moreover, we argue that selecting only a single model, as in existing methods (II- harco et al., 2022a; Ortiz-Jimenez et al., 2024), does not guarantee optimal performance. To address this, we propose leveraging all candidate models to effectively utilize the information they contain, rather than selecting just one and discarding the rest.
1. Introduction
The Right to be Forgotten regulation (Hoofnagle et al., 2019) allows individuals to request the deletion of their personal data. However, applying this concept to machine learning models is challenging because the training process deeply embeds the data into the model's parameters, making it difficult to remove its influence. The most straightforward solution is to remove the data from the training set and retrain the model from scratch, which requires enormous computational resources. As result, ensuring that mod- els selectively forget specific learned patterns becomes a
In this paper, we present NegMerge, novel method that enhances the process of forgetting by negation. Our ap- proach computes a final task vector by merging task vectors derived from multiple fine-tuned models. During this merg- ing process, we preserve elements with consistent signs across the task vectors and mask out elements with inconsis- tent signs, setting them to zero. The subsequent steps align with the standard forgetting by negation process, where the final task vector is subtracted from the original model to induce forgetting (Ilharco et al., 2022a). Our method inspired by model merging techniques (Wortsman et al., 2022; Yang al., 2023; Jang al., 2024), which utilize
*Work done during an internship at NAVER AI Lab. 'Equal ad- vising. 'Sogang University "NAVER AI Lab. Correspondence to: Junsuk Choe <jschoe @ sogang.ac.krr Dongyoon Han <dongy- oon.han @ navercorp.com: com>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
multiple fine-tuned models produced during the validation process. Building on this concept, we adapt it to the context of machine unlearning.
attempt to induce forgetting by relabeling the forget set to different classes and fine-tuning the model accordingly. However, they often suffer from catastrophic forgetting, as the retain set is not used during fine-tuning, leading to the loss of its knowledge.
We demonstrate the effectivvness of our method in two ex- perimental settings. The first setting aims to make a classifi- cation model with zero-shot recognition capabilities, such as CLIP (Radford et al., 2021), unable to recognize specific knowledge. The second setting focuses on removing knowl- edge associated with specific subset of the training data in standard image classification model (Chen et al., 2023; Fan et al., 2023). We validate our method on ViT (Dosovit- skiy et al., 2021), ResNet (He et al., 2016), VGG (Simonyan, 2014), and Swin-T (Liu et al., 2021) architectures across a total of 12 datasets. Our approach achieves new state-of-the- art performance while utilizing computational resources that are comparable to or fewer than those of existing methods.
We propose a novel approach based on Task Arithmetic (II- harco et al., 2022a) that leverages multiple fine-tuned mod- els to tackle the aforementioned challenges. By incorpo- rating insights from these models, our method computes more effective task vector, enhancing unlearning perfor- mance while preserving retain set knowledge. Notably, our approach requires only the forget set and is effective in both image classification tasks.
Model Merging. Model Soups (Wortsman et al., 2022) ad- dresses the inefficiency of discarding many models during the validation process, where only single best model is se- lected. They argue that merging the weights of all generated models can improve generalization performance without ad- ditional computational overhead. Task Arithmetic (Ilharco et al., 2022a) introduces task vectors, showing that desired knowledge can be effectively added to or removed from models through simple arithmetic operations with these vec- tors. AdaMerging (Yang al., 2023) autonomously learns model merging coefficients at the task or layer level and per- forms this process without relying on the original training data. TIES-Merging (Yadav et al., 2024) incorporates a trim- ming stage that retains only elements with large magnitudes during the merging process and resolves sign conflicts be- tween elements through a voting mechanism, merging only elements corresponding to the selected sign. MagMax (Mar- czak et al., 2024) selects and merges only the elements with the largest magnitude from the task vectors. Several approaches further improve merging methods by localiz- ing task-relevant parameters. Skill Localization (Panigrahi et al., 2023) selects parameters with the largest changes during fine-tuning, Socalize-and-Sticee (He et al., 2024) re- tains the top-k% parameters based magnitude, and TALL Mask (Wang et al., 2024) masks out parameters with small magnitudes.
2. Related work
Machine Unlearning for Image Classification. Exist- ing methods have been applied mainly to two tasks. The first focuses on reducing zero-shot recognition performance for specific knowledge in vision-language models, such as CLIP (Ilharco et al., 2022a; Ortiz-Jimenez et al., 2024). The second involves unlearning knowledge tied a subset of the training data in standard image classification models (Chen et al., 2023; Fan et al., 2023). These two tasks have tradi- tionally been treated as separate research areas.
In the first task, the negation method proposed in Task Arith- metic (Ilharco et al., 2022a) is applied for unlearning. More recently, a linear negation method based on the Neural Tan- gent Kernel (Jacot et al., 2018; Ortiz-Jimenez et al., 2024) has been introduced. This method enables arithmetic in the linear space by fine-tuning the model in the tangent space. Both approaches depend on a single fine-tuned model to calculate task vectors, selecting the single best model from numerous fine-tuned models derived during the validation process.
Machine unlearning for standard image classifier usually involves fine-tuning the original model. Fine-tuning (War- necke et al., 2021) and fi-sparse (Jia et al., 2023) aim to overfit the model only on the retain set to erase the knowl- edge of the forget set. Influence (Koh & Liang, 2017) and SalUn (Fan et al., 2023) utilize both the retain and forget sets to selectively degrade performance on the forget set while maintaining on the retain set.
Our proposed technique merges task vectors by selecting only elements with the same sign while masking the re- maining elements with zero. This approach has not been explored in existing methods, and our evaluation shows that it is highly effective in machine unlearning.
3. Method
In many cases, the size of the forget set is significantly smaller than that of the retain set. In such scenarios, ma- chine unlearning methods that require the retain set can be inefficient. This challenge has driven the development of approaches that perform unlearning using only the forget set. Existing methods (Golatkar et al., 2020; Chen et al., 2023)
3.1. Background
Task Arithmetic involves defining a task vector Tr, by sub- tracting the parameters of a pre-trained model Opre from those of a fine-tuned model 0%, for a specific target task t:
Te = 8/, - 8pre. For handling multiple tasks simultane-
2

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
(a)
(b)
that preserve retain set performance tend to exhibit poor unlearning performance, and vice versa. To address this, we propose method that leverages multiple models to combine their strengths, which single model alone cannot achieve. Figure 1 (a) shows that our method overcomes this trade- off. In contrast, existing merging methods such as Uniform Merge and Greedy Merge (Wortsman et al., 2022), TIES- Merging (Yadav al., 2024), and MagMax (Marezak et al., 2024) struggle with this trade-off, highlighting the need for merging strategy designed for unlearning. The full results for all eight datasets are provided in Figure B1.
73
Sizgl paza
Tartem Megg
1- on forget set (%)
Greddy blege This Meeppe
50
Acc on forget set (%)
70-
Mtugtss
35
Oucc
30
67
25
64-
15
10
61
60.0
60.5
61.0
Cas
DTD EuroSAT
GTSRB
MNIST
RESISCAS
NNHAS
Second, unlearning performance is highly sensitive to the hy- perparameter settings used during fine-tuning. As shown in Figure (b), accuracy on the forget set can vary by up to 15 percentage points depending on the hyperparameters.. This sensitivity not only affects performance stability but also makes the tuning process time-consuming and computation- ally expensive, as it requires additional tuning of the scaling hyperparameter À for each fine-tuned model. However, our method mitigates the sensitivity by merging all fine-tuned models obtained from diverse hyperparameter configura- tions. This strategy inspired by Model Soups (Wortsman et al., 2022), which shows that averaging the weights of inde- pendently fine-tuned models can improve performance and robustness. The key idea is that different hyperparameter choices introduce independent variations, and merging helps cancel out noise from any single run. As a result, Figure I (a) shows that our method achieves stronger unlearning per- formance, while significantly reducing the time needed for scaling hyperparameter tuning by merging models derived during validation. Discussions of the computational costs are described in Section 3.3.
Acc on retain set (%)
SUNSS
Figure 1. Hyperparameter Sensitivity in Negation Methods. Each point in (a) represents the accuracy on the forget and re- tain sets, with the error rate (calculated as - accuracy) used for the forget set to improve visibility. Green points indicate re- sults from models fine-tuned with various hyperparameter settings, while points in other colors show results from different methods. Ours breaks the tradc-off between unlcarning and retaining perfor- mance. Panel (b) presents the accuracy distribution on the forget set across different hyperparameter choices, which vary by up to 15 percentage points. Both (a) and (b) are based on the CLIP ViT-B/32 model, where (a) focuses the Cars dataset, and (b) includes experiments across eight datasets.
ously, task vectors from individual tasks can be combined as 7 = Ee T/. This combined vector enables the model to be adjusted in a desired direction by modifying the original model's s weights. The updated model weights are computed as: 6new = pre + År, where À a scaling hyperparameter that controls the magnitude of adjustment.
A key application of Task Arithmetic (Ilharco et al., 2022a) is to make model forget certain capabilities. This can be achieved through the negation of a task vector from the orig- inal weight, which decreases performance on a target task. For example, Task Arithmetic can be applicd to unlearn- ing in models such as CLIP (Radford et al., 2021), which is known for its strong zero-shot recognition capabilities. Task Arithmetic demonstrated that task vector derived from CLIP model fine-tuned a specific dataset (e.g.. Cars) can reduce the model's accuracy on that dataset while maintaining its overall accuracy on a general dataset (e.g., ImageNet). While Task Arithmetic has shown promise in machine unlearning, limited research has explored how to compute a task vector specifically optimized for unlearning. Our research aims to address this gap.
3.2. The Proposed Method: NegMerge
We propose method to achieve effective unlearning by integrating task vectors, given that multiple models are fine- tuned on the forget set under various training configurations. A detailed description of each step is provided below, and Figure 2 illustrates the overview of our method.
Step 1) Calculating Diverse Task Vectors. As previ- ously mentioned, machine unlearning based on Task Arith- metic (Ilharco et al., 2022a) is highly sensitive to hyperpa- rameters, making it essential to identify optimal hyperparam- eters through validation. In standard validation processes, various training configurations are employed, such as ad- justing hyperparameters like the learning rate or applying additional data augmentation techniques, to derive multiple models. Our study emulates such validation procedures to construct model pool, which is then utilized to calculate diverse task vectors. Importantly, we restrict the number of models to the typical range used in hyperparameter vali- dation (10 to 30 models), ensuring that the use of multiple
Through our pilot study, we identified two major challenges in obtaining effective task vectors for un- learning. First, it is challenging to balance reducing accu- racy on the forget set while maintaining accuracy on the retain set. As shown in Figure (a), hyperparameter sets
Motivation.
3

---

NegMerge: Sign-Consensun
Weight Merging for Machine Unlcarning
Original weight
Improved task vector
Unlearned weight
Task vector
Task vector 2
Task vector 3
Layer
(4x4 weights)
L Layer 2 (dvd waights)
Layer L (4ord weights)
Figure 2. Illustration of the proposed method. NegMerge enhances Task Arithmetic by computing an improved task vector. Specifically, I) task vectors derived from multiple fine-tuned models trained with different hyperparameters are utilized. 2) We compute the improved task vector by merging (#) only the elements that retain consistent sign to zero. 3) This refined task vector is used for negation from the original model. The color intensity in the cells reflects the magnitude of the task vector elements; darker blue represents larger positive values, lighter blue indicates smaller positives, while darker red represents larger negative values, and lighter red indicates smaller negatives.
task vectors while masking elements with differing signs
models does not introduce additional computational over- head. Detailed information on this approach is provided in Section 4.1.
signs across all task vectors contribute to the final task vec- tor, while those with differing signs are excluded by being set to zero. We then perform unlearning by negating this final task vector to the original model (Ilharco et al., 2022a), Theoretical analysis provided in Appendix E.
Step 2) Identifying Task Vector Elements for Forget Set. After deriving task vectors from the fine-tuned models, we analyze them to determine which elements correspond to the forget set. We conjecture that elements that consistently show the same sign across task vectors are attributed to the forget set, as each model is trained to align with this set, regardless the training configurations. On the other hand, components that exhibit inconsistent signs are considered less related the forget set, as their variations are more likely a result of different training configurations rather than supervision from the forget set. Our conjecture regarding sign conflicts is supported by the unlearning performance reported in Table 3 and qualitative results in Figure 3.
3.3. Analysis on Computational Cost
We analyze the computational cost from four perspectives: inference time complexity, merge time, storage, and runtime memory. All compared methods, including Task Arith- metic (Ilharco et al., 2022a), share the model pool, ensuring additional computational overhead from using multiple models. To simulate realistic validation process, we vary the number of fine-tuned models (n) between 10 and 30. As shown in Table 4, our method consistently demon- strates robustness to the number fine-tuned models.
Step 3) Final Task Vector for Negation. We compute the final task vector using the following formulation:
Inference Time Complexity. The standard setup for task vector negation- regation-based methods (Ilharco al., 2022a; Ortiz- Jimenez et al., 2024) involves performing inference with m different scaling coefficients X for each of the n fine- tuned models. Specifically, this process requires m = 20 inferences per model prior works (Ilharco et al., 2022a: Ortiz-limenez al., 2024), resulting in a significant compu- tational cost of (mm)). In contrast, our method performs inference only times single merged task vector, re- ducing the computational cost to ((m). While Task Arith-
Tmerged Σ (Tk sign-consiseentt.
where is the number of task vectors, @ denotes the Hadamard product (element-ww) wise multiplication), and the vector Lignncoonsitenn acts like a filter, containing for ele- ments where the signs of the corresponding components across all task vectors тą are the same and O where the signs differ!. As result, only the components with consistent 'This operation based on sign unanimity and could
justed with additional hyperparameters to allow partial consensus; however, opt for simpler approach.

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
metic employs a single model during its final stage, achiev- ing optimal performance with this approach entails greater computational demands compared to our method. This demonstrates the computational advantages of our approach over competing methods.
metic paper (Ilharco et al., 2022a). We assess unlearning performance on eight datasets: SUN397 (Xiao et al., 2016), Cars (Krause et al., 2013), RESISC45 (Cheng et al., 2017), ET (Helber et al., 2019), SVHN (Yuval, 2011), GT- SRB (Stalikamp et al., 201 1), MNIST (LeCun, 1998), and DTD (Cimpoi et al., 2014), while using ImageNet (Deng et al., 2009) as the retain set to evaluate retaining perfor- mance. All experiments are conducted using pre-trained CLIP ViT- B/16, L/14) models (Radford al., 2021). In the standard classifier scenario, we evaluate unlearning performance on CIFAR-10 (Krizhevsky et al., 2009), CUB- 200-2011 (Wah et al., 2011), and Tiny ImageNet (Le & Yang. 2015) using ResNet-18 (He et al., 2016), VGG-16 (Si- monyan, 2014) and Swin-T (Liu et al., 2021) models.
Our method requires checking the sign of elements across task vectors, making it slower than sim- pler approaches such as MagMax (Marezak et al., 2024), which identifies only the maximum values, or Uniform Merge (Wortsman et al., 2022), which averages task vectors. However, it is much faster than methods that rely on more complex operations, such as TIES- HEE--Mggging (Yaday et al., 2024) or Greedy Merge (Wortsman et al., 2022). Detailed comparisons are provided in Table 1.
Merge Time.
Baselines and Metries. For the CLIP scenario, we com- pare our method with five existing methods: Task Arith- metic (Ilharco et al., 2022a), Uniform Merge and Greedy Merge (Wortsman et al., 2022), VIES-Megging (Yadav et al., 2024), and MagMax (Marczak et al., 2024). For Greedy Merge, we rank models by their loss on the retain set and merge them in a direction that minimizes this loss. We eval- uate performance by measuring accuracies on the forget set Dy and the retain set D,.. In the standard classifier scenario, we follow SalUn (Fan et al., 2023) to compare our method against eight unlearning techniques: Fine-tuning (Warnecke et al., 2021), Random Labeling (Golatkar et al., 2020), Gra- dient Ascent (Thudi et al., 2022), Influence Unlearning (Koh & Liang, 2017). sssprse (Jia et al., 2023), Boundary Shrink and Expand (Chen al., 2023), and SalUn. In addition, we include merging-based baselines Task Arithmetic, Uniform Merge, TIES- and MagMax. Greedy Merge is in- feasible for comparison in this scenario when using only the forget set. The objective is to match the unlearned model's s performance to that of fully retrained model. We use the accuracies of the retain set D,, forget set D,, and test set Dtest to evaluate performance. To assess privacy protec- tion, we employ the Membership Inference Attack (MIA) metric (Carlini et al., 2022), following the MIA-Efficacy metric from (Fan et al., 2023; Jia et al., 2023). Higher MIA- Efficacy implies more unlearning, as it measures how much less information the model retains about the forget data.
Storage. Our approach might initially seem storage- intensive, it involves storing all fine-tuned models for merging. However, this is not the case in practice. Rather than saving every fine-tuned model, we dynamically update a mask, consiscot based on the sign consensus of each element when a new model is trained. This climinates the need to store all fine-tuned models. Consequently, the pro- posed technique achieves more effective unlearning while maintaining the same storage requirements as traditional single-model-based methods. By contrast, methods like TIES Merging cannot utilize such a mechanism, further underscoring the advantages of our approach.
Runtime Memory. Our method provides a clear advantage in runtime memory efficiency. As demonstrated in Table 4, a large proportion of the weights in the task vector are zeroed out during the merging process, with only 5-10% of the total weight elements remaining active. This significant sparsity enables lightweight memory management techniques, like weight lookup tables, which store only the active weights and further reduce runtime memory consumption. As a result, our method achieves superior efficiency compared to baseline approaches.
4. Experiments
4.1. Experimental Setups
Two Unlearning Scenarios. We examine two distinct un- learning scenarios. The first involves a vasion-language model, such as CLIP (Radford et al., 2021), being made to forget the knowledge associated with a specific dataset. The second scenario focuses on standard image classification network trained with cross- cross-entropy loss, where the model is instructed to forget knowledge of particular data points within its training set. The specific implementation details for each scenario are provided in Appendix A.
4.2. Experimental Results
CLIP Unlearning Scenario. Table 1 presents the evalua- tion results across three CLIP models (ViT-B/32, ViT-B/16, and ViT-L/14). Note that the retain set (D,) accuracies for all methods remain around 60%, as we follow Task Arith- metic (Ilharco et al., 2022a) to ensure the model retains at least 95% of the pre-trained model's original accuracy (66.66%) on the validation set. This allows for a direct comparison of forget set accuracies.
Datasets and Backbones. the CLIP scenario, we follow the training and evaluation protocols from the Task Arith-
Our method achieves the best reduction in accuracy on the
5

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Table Unlearning Performance on CLIP VIT Models. Results are shown for CLIP ViT-(B/32, B/16, 1/14), reporting average accuracy (%) the eight datasets we wish forget (Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN), and the general dataset to retain (ImageNet). indicates that the numbers are borrowed from the original papers. denotes the best performance achieved through hyperparameter search. combines models descending order of losses. Time denotes the merging time, measured in seconds, taken to merge 30 models on the Cars dataset using CLIP ViT-B/32, which is averaged over three runs. NegMerge consistently achieves the lowest forget accuracy across all backbones, indicating strong unlearning performance.
VIT-B/32
VIT-B/16
ViT-L/14
Time (sec)
Method
Acc (y())
Acc D,
Acc Dy(()
Acc D,
Acc D,(1)
Acc Dr
Pre-trained
48.13
63.33
55.49
68.32
65.19
75.54
Task Arithmetic
Paper number'
24.00
60.90
21.30
65.40
19.00
72.90
Single Best Model
23.63
60.60
20.64
64.04
19.17
72.09
Uniform Merge
22.50
60.55
21.51
64.60
18.10
71.91
12
Greedy Merge'
23.31
60.75
21.34
64.54
17.71
71.99
607 +2.6
TIES- MEE-MMrggnng
26.21
61.08
23.78
64.72
22.70
72.41
128
MagMax
25.24
60.95
24.45
64.78
21.71
72.55
24
NegMerge (ours)
20.76
60.36
19.24
64.54
17.32
72.08
37
Linear Task Arithmetic
Paper number'
10.90
60.80
11.30
64.80
Single Best Model"
8.88
60.16
6.92
64.62
Uniform Merge
9.12
60.47
6.81
65.26
1912.3
Greedy Merge*
8.73
60.27
6.80
64.72
1696
TIES-Merging
10.66
60.38
8.44
65.12
378
MagMax
11.33
60.67
8.65
65.17
164
NegMerge (ours)
8.03
60.58
6.60
65.10
194
forget set D across all backhones, which demonstrates its generalizability regardless of model size. Specifically, for CLIP ViT-B/32, our method reduces the forget set (D₂) ac-
for ViT-L/14. Our method achieves the best unlearning per- formance, while the second-best method, Greedy Merge, requires significantly more time for merging (1696.5 and 194.2, respectively).
curacy 20.76%, outperforming Task Arithmetic (23.63%),
Uniform Merge (22.50%), and Greedy Merge (23.31%) 1%). Our method maintains strong performance across different model variants. For CLIP ViT-B/16, it reduces forget set ac- curacy to 19.24%, outperforming Task Arithmetic (20.64%) Similarly, for CLIP ViT-L/14, our approach achieves the best forget set performance, lowering accuracy 17.32%.
Full results for all eight datasets and three CLIP models provided in Appendix B.I.
Standard Classifier Unlearning Scenario. Table 2 presents a comparison of various unlearning techniques on CIFAR- 10 using ResNet-18. 18. In this task, we randomly select 10% of the training set as the forget set. The goal is to make the model forget the knowledge associated with the forget set while maintaining its performance the retain set. The fully retrained model serves as the ideal benchmark for forget, retain, and privacy tasks. Following SalUn (Fan et al., 2023), we report the Avg. Gap metric to evaluate how closely each unlcarning method replicates the performance of the retrained model across key metries such as Ace D,- (accuracy the retain set), Acc D, (accuracy on the forget set), Acc Dtest (accuracy on the test set), and MIA score.
In contrast, MagMax and TIES- TIES-Merging exhibit weaker unlearning performance.
Regarding merging time, our method requires slightly more time than Uniform Merge and MagMax but offers signif- icantly better effectiveness. Furthermore, while Greedy Merge (Wortsman et al., 2022) and TIES-Merging (Yadav al., 2024) are considerably slower, our method outper- forms them by large margin accuracy.
To provide more comprehensive evaluation of our method, we employ Linear Task Arithmetic, where Neural Tangent Kernel (NTK) (Jacot et al., 2018; Ortiz-Jimenez et al., 2024) is applied to the standard Task Arithmetic (Ilharco et al., 2022a). The experimental results are presented in the lower part of Table 1, where we conduct evaluations using the CLIP VIT-B/32 and ViT-B/16 backbones. Due computa- tional resource constraints, unable include results
Our method achieves an average gap of 1 1.07, effectively unlearning the required knowledge while causing minimal degradation to the overall model performance. In compar- ison, SalUn, which utilizes all data splits for unlearning, achieves a slightly higher average gap of 1.15. Notably, our approach, relying solely the forget set, outperforms SalUn, demonstrating superior efficiency in unlearning with-
6

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Table Unlearning Performance for 10% Random Data Forgetting on CIFAR-10 using ResNet-18. The results are reported as atb, representing the mean (a) and standard deviation (b) across three independent trials. The Avg. Gap is the average performance difference between each unlearning model and the Retrain model, measured across Acc D,, Ace Dj, Ace Dreat and MIA. (-) denotes that smaller performance differences from the Retrain model preferred. indicates that the numbers are borrowed from (Fan et al., 2023). denotes the best results achieved through hyperparameter search, NegMerge achieves the lowest Avg. Gap, indicating performance
closest to the Retrain model, which serves as the idcal benchmark.
Method
Used Splits
Acc D,(*)
Ace D,(=)
Acc (ze)
MIA(==
Avg. Gap(I)
Retrain *
retain
100.00.a.00
94.76+0.69
94.26+a.u2
12.88
0.00
Random Labeling
99.67=0.14
92.39
92.83
37 37.3640.06
7.15
Influence
2022
98.93+0.28
93. 1..20
2.67
4.06
SalUn
99.62 02.112
97.15
93.93
14.39
1.15
Finetune
99.88 g0.
99.37
94.06-0.27
2.70
3.78
retain
4--ppare
97.7420.33
95.81
91.5920.57
9.84+0.00
2.26
Gradient Ascent
99.50g0.58
99.31 #0.54
94.01
1.70
4.12
Boundary Shrink
98.2922.50
98.22 2..52
92.69
8.96
2.67
Boundary Expanding
forget
99,4220.s3
99.41
93.85=1.00
7.47
2.76
Random Labeling
99.99=0.00
99.9820.02
95.04-0.11
2.15+1.94
4.19
SalUn
99.88 g9.044
99. 90.8
94.42
9.51
2.20
Task Arithmetic
Single Best Model'
98.36.00.5
94.85
91.49 1.......0
10.91
1.62
Uniform Merge
forget
98.70
95.83 17
92.36
10.14
1.75
TIES-Merging
98.38
15.15
92.23
9.36
1.96
MagMax NegMerge (ours)
g8.38
97.97
91.53=+0.00
8.45+22.0
3.00
99. 99.15g0.22
96.63 96.63
92.71
12.87
1.07
out depending on the retain set. In contrast, Task Arithmetic and merging methods, including Uniform Merge, TIES- Merging, and MagMax, exhibit larger gaps of 1.62, 1.75, 1. 1.96, and 3.00, respectively. These results highlight that our method achieves a better balance between unlearning and preserving knowledge in the retain set. Furthermore, our approach ensures strong privacy protection, achieving an MIA score of 12.87, nearly identical to that of the retrained model. This demonstrates that the model effectively forgets targeted data without introducing privacy vulnerabilities.
Table 3. Impact of Sign Consensus Across Task Vectors. The results present unlearning performance across multiple datasets, comparing three different methods. "Al'", Uniform Merge, uses all indices without regard to sign conflict, "Confticr" uses only indices with conflicting signs, and "Consensus", , our proposed method, uses only indices with consistent signs across task vectors. The Conflict method degrades unlearning performance, while Con- sensus performs hest, confirming the effectiveness of using only sign-consistent elements.
Cars
DTD
SUN397
Method
Additional experiments are provided in Appendix B.3. We evaluate unlearning methods on CUB (Wah et al., 2011) and Tiny ImageNet (Lee & Yang, 2015), demonstrating the ef- fectiveness NegMerge under diverse datasets (Table B8, Table B9). We also assess generalizability across back- bone architectures by evaluating unlearning methods VGG-16 (Simonyan, 2014) and Swin-T (Liu et al., 2021), where NegMerge consistently performs well (Table B 10, Table 1). Finally, NegMerge rge maintains its advantage in different unlearning scenarios, including 50% random for- getting and class-wise forgetting, confirming its scalability and robustness (Table B12, Table B13).
Dy
Dr
Dy(L)
Dr
Dy(L)
All
31.7
60.4
29.6
60.6
51.4
60.5
Conflict
40.2
60.2
31.9
60.3
58.3
60.9
Consensus
27.4
60.4
27.2
60.5
47.2
60.6
set, while elements with conflicting signs are less relevant to the forget set. To verify this, we compare unlcarning performance when our method is applied reverse. The experimental results are shown in Table 3. We use the CLIP VYT-B/32 model and the standard Task Arithmetic (Ilharco et al., 2022a). The All method refers the Uniform Merge approach, which uses all elements without considering sign consensus. The Conflict method uses only elements with conflicting signs, while our proposed Consensus method uses only elements with consistent signs. The results show that the Conflict method significantly degrades unlearning performance, while the All method performs better than Con-
4.3. Ablation Studies
Effect of Sign Conflict on Unlearning Performance. We argue that elements with consistent signs across multiple task vectors correspond knowledge related the forget

---

NegMerge: Sign- Sign-Consensual Weight Merging for Machine Unlcarning
Table 4. Sparsity and Unlearning Performance. The results re- port the sparsity (i.e.. the percentage of zeroed elements in the final task vector т. denoted as %). Ace D, and Acc D.., averaged over three runs using ViT-B/32. # indicates the number of task vectors used for merging. As more task vectors are merged, sparsity in- creases and Acc Dy decreases, indicating improved unlcarning performance.
Cars
DTD
SUN397
#
%
Dy(1)
D.
%
Dy(1)
D.
%
Dy())
D.
(a) Image
(h) Original model
(c) Conflict
(d) Consensus
30
90.3
27.4
60.4
92.9
27.2
60.5
92.2
47.2
60.6
25
89.7
26.6
60.2
92.2
27.3
60.5
91.7
47.0
60.4
Figure 3. Visualization of the Impact of Sign Consensus. Grad-
20
88.8
26.1
60.1
91.5
27.1
60.4
90.9
47.8
60.6
CAM visualizations on the RESISC45 datasct compare the Con- Slict, Consensus. and original models. The first and second rows correspond to the cloud and airplane classes, respectively. Red areas indicate strong class relevance, while blue areas indicate weaker relevance. The Consensus model exhibits lower activation in class-relevant regions, indicating more effective unlearning.
15
87.5
26.0
60.0
90.2
27.7
60.4
89.6
17.7
60.5
10
84.9
26.6
59.9
87.5
27.8
60.4
87.2
48.7
60.6
5
77.1
30.5
60.4
79.9
28.8
60.5
81.5
49.7
60.5
Table S. Performance of Task Vector Addition. The evaluated models obtained by adding task vector 7 the original model. denotes our reproduced results based on the configurations from (Mharco ct al., 2022a). represents the single model's best results achieved through hyperparameter tuning, including adjustments to data augmentation. $ combines models descending order losses. NegMerge achieves high performance on both the forget and retain sets.
flict but is outperformed by our Consensus method. These experimental results indicate that the design choice of merg- ing only sign-consistent elements is effective. Full results are provided in Table B6.
In Figure 3, we demonstrate the effectiveness of our method using Grad-CAM (Selvaraju et al., 2017) visualizations on the RESISC45 dataset. We compare the Conflict and our Consensus methods, and include visualizations of the orig- inal model as a baseline. The red areas represent regions where the model strongly associates with the class label, while the blue areas indicate regions with less relevance. In the first row (cloud class), we observe that the Conflict method directs the model's attention to the cloud's loca- tion, resembling the behavior of the original model. In contrast, our method does not highlight the cloud's 's area, which suggests that the model has successfully forgotten its knowledge of the cloud. The same pattern appears in the second row for the airplane class. These visual results demonstrate that our proposed method more effective for machine unlearning.
Method
Cars
DTD
SUN397
Dy(t) D,(1) Dy(t) D,(1) Dy(t) D,(1)
Pretrained
59.6
66.7
43.9
66.7
63.3
66.7
Task Arithmetic
Paper config"
85.0
58.6
78.7
49.3
74.9
59.8
Single Best
86.6
52.7
76.9
48.4
76.5
55.7
Uniform Merge
87.2
55.3
79.0
52.8
76.0
57.1
Greedy Merge*
87.5
55.2
79.3
52.8
76.2
57.1
NegMerge (ours)
87.1
61.7
76.3
63.0
76.3
63.4
original model via negation, and interestingly, as sparsity increases, unlearning performance also improves. This sug- gests that leveraging a larger number of task vectors enables more accurate identification of specific elements correspond- ing to the forget set, allowing us to minimize performance degradation on the retain set. The full results are provided in Table B7. It is noteworthy that other merging methods have significantly fewer zeroed-out elements, resulting in lower unlearning performance, as shown in Table C2.
Ratio of Zeroed Elements in the Merged Vector. The proposed method masks out all elements that do not exhibit consistent signs across multiple task vectors, leaving only those with consistent signs. In Table 4, we investigate the proportion of zeroed-out elements in the final task vector T. where "zeroed-out" includes both elements that were already zero (e.g., due to freezing during fine-tuning) and those masked during merging. Our observations show that as the number of merged models increases, the proportion of masked elements also increases, meaning that the sparsity of the final task vector t increases with the inclusion of more models. Ultimately, the task vectors generated by our method modify only 5-10% of the weight elements in the
Results of Task Vector Addition. To better understand the characteristies of task vectors derived from our method and comparative techniques, we add these task vectors to the original model. The experimental results, presented in Table 5, reveal several key observations.
Most existing methods achieve high performance on the forget set but suffer significant degradation on the retain set. In contrast, our method maintains high performance on the retain set, which we consider a key factor in effective
8

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
size that this is because general knowledge, parameterized within shallow layers, is more susceptible distortion dur- ing fine-tuning, In contrast, deeper layers with task-specific knowledge are more robust to such variations (Morcos et al., 2018; Kornblith et al., 2019). These findings motivate future research on layer-wise unlearning strategies.
Table Sparsity by Layer Depth. Transformer layers are grouped into three depth ranges, and the average sparsity (%) reported for each range across 30 models trained on the Cars dataset. Shallower layers exhibit higher sparsity.
VIT-B/32
ViT-L/14
Layer Group
Range
%
Range
%
We further examine the sparsity across different components in Table C3 We also discuss the broader applicability of our method to LLMs and VLMs in Appendix F.
Shallow Middle
0-3
73.4
0-7
83.5
4-7 7
68.4
8- 15
76.4
Deep
8 11
62.9
23-23
73.3
5. Conclusion and Limitation
machine unlearning- Specifically, for effective machine unlearning, the task vector must adjust the knowledge of the forget set while minimizing its impact on the retain set. Our experimental results demonstrate that our method effectively meets this requirement.
We propose NegMerge, a novel machine unlearning tech- nique that merges and leverages all fine-tuned models gener- ated during validation, rather than discarding all but one. By introducing sign consensus approach identify and iso- late parameters associated with the forget set, NegMerge modifies only the parameters strongly related to the for- get set, achieving effective unlearning while preserving the knowledge of the retain set. NegMerge achieves state-of- the-art performance across 12 datasets and 4 architectures. Additionally, by improving the validation process of existing techniques, NegMe effectively leverages multiple mod- els without requiring additional computational resources. Although empirically validated, NegMe rge currently lacks theoretical justification. Future work will focus on theoreti- cal validation and analytical insights for method extension.
This characteristic of our approach appears to be closely related to the high sparsity of the task vector. As shown in Table C2, while other techniques modify 50-60% of the parameters, our method adjusts only 5-10%, enabling precise modification of performance on the forget set while minimizing the impact on the retain set. This finding is expected to aid future research in machine unlearning, and we plan to explore it further.
NegMerge with Different Model Pools. To evaluate the robustness of the proposed method to the composition of the model pool, we observe its performance using model pools constructed with various training configurations. Specif- ically, we conduct experiments by creating model pools with hyperparameter search spaces different from the origi- nal experimental setting. The results demonstrate that the proposed technique consistently outperforms the baseline, regardless of the model pool composition method. This indicates that the proposed method is robust to the compo- sition of the model pool. The full experimental results are provided in Appendix D.
Acknowledgments
Most experiments were conducted on the NAVER Smart Machine Learning (NSML) platform (Sung et al., 2017). This work was supported by the National Research Founda- tion of Korea (NRF) grant funded by the Korean government (MSIT) FRS-2024-0035044 NAVER AT Lab, particularly Taekyung Kim, Changdae Oh, and Yong- Hyun Park, for their invaluable feedback and the VRL Lab at Sogang University for their support.
We thank the researchers at
Different Merging Operations. Our method computes the final task vector by averaging the task vectors, as shown in Equation (1). The experimental results of using max or min n operations instead of averaging are presented in Table C1. The results indicate that both max and min operations also improve unlearning performance compared to the baseline. However, our avg operation achieves the best performance. We hypothesize that averaging smooths out potential outliers in individual models during merging, resulting in a more stable and effective task vector.
Impact Statement
This paper contributes to advancing the field of machine unlearning. Our method offers positive societal impacts in terms of privacy and data management. Notably, unlike existing approaches, it utilizes all the fine-tuned models generated during the validation process instead of discarding them, making it more resource-friendle This could lead to various positive effects, such as reducing carbon emissions. However, if malicious users gain direct access to the model, there is risk of adverse effects, such as deleting knowledge that should not be erased. This risk can be mitigated by implementing additional security protocols to ensure that only authorized individuals have access to the model.
Ratio of Zeroed Elements by Layer Depth. As shown in Table 6, sparsity, defined as the ratio of zeroed elements, decreases with layer depth in both ViT-B/32 and ViT-L/14. Shallower layers show higher sparsity, suggesting that they are more sensitive to hyperparameter variations and thus experience greater conflict across task vectors. We hypothe-
9

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
References
He, Y., Hu, Y., Lin, Y., Zhang, T.,, and Zhao, H. Localize- and-stitch: Efficient model merging via sparse task arith- metic. arXiv preprint arXiv: 1305. 2024.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and
Tramer, F. Membership inference attacks from first prin- ciples. In 2022 JEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat:
A novel dataset and deep learning benchmark for land and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, (2(7):211-22222) 2019.
Chen, M., Gao, W., Liu, G., Peng. K., and Wang, C. Bound- ary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Hoofnagle, C. J., Van Der Sloot, B., and Borgesius, F. Z. The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law, 28(1 28(1):65-99) 2019.
Cheng. G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10):1865-1 1883, 2017.
Ilaarco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2022a.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In CVPR, pp- 3606-3613, 2014.
Ilharco, G., Wortsman, M., Gadre, Song. S.,, Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, Patching open- vveabvocyy models by interpolating weights. Ad- vances in Neurul Information Processing Systems, 35: 29262-29277, 2022b.
Deng, J., Dong, W., Socher, R., L.-J., Li, and Fei-Fei, L. Imagenet: A scage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Jang, D.-.., Yun, S., and Han, D. Model stock: All we need is just few fine-tuned models. In European Conference Computer Vision, 2024.
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectiv- ity of neural networks. In International Conference on Learning Representations, 2021.
Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584 51605, 2023.
Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2023.
Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N. Linear connectivity reveals generalization strategies. In The Eleventh International Conference on Learning Rep- resentations, 2022.
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259- 3269. PMLR, 2020.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In International conference on machine learning, pp.
PMLR, 2017.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object rep- resentations for fine- fine-graine -rained categorization. In Proceed- ings of the IEEE international conference on computer vision workshops, pp. $54-561, 2013.
10

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlearning
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images, 2009.
machine learning platform that enables you to focus on your models. arXiv preprint arXiv:
2017.
Le, Y. and Yang, X. Tiny imagenet visual recognition chal- lenge. CS 231N, 7(7):3 2015.
Tang, A., Shen, L., Luo, Y., Yin, N., Zhang, L., and Tao, Merging multi-task models via weight-ensembling mixture of experts. Forty-first International Conference on Machine Learning, 2024.
LeCun, Y. The mnist database of handwritten digits. htlp://yann. lecun, com/exul//mist// 1998.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) pp. 303-319. IEEE, 2022.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset, 2011.
Marczak, D., Twardowski, B., Trzcifskii, T., and Cygert, S. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision, pp. 379-399- -995. Springer, 2024.
Wang. K., Dimitriadis, N., Prtiz-Jimenez, G., Fleuret, F., and Frossard, P. Localizing task information for improved model merging and compression. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Morcos, A., Raghu, M., and Bengio, S. Insights on repre- sentational similarity in neural networks with canonical correlation. Advances neural information processing systems, 31, 2018.
Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Machine unlearning of features and labels. arXiv preprint arXiv:2108. 01022100.1157... 11577, 2021.
Ortiz-Jimenez, G., Favero, A., and Frossard, P. Task arith- metic in the tangent space: Improved editing of pre- trained models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-opes, R., Morcos, A. S., Namkoong. H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning. pp. 23965- 23998. PMLR, 2022.
Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task- specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp. 27011-27033, PMLR, 2023.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 1 119:3-22, 2016.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient- gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 61 318-626,77 2017.
Yang, E., Wang. Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2023.
Simonyan, K. Very deep convolutional networks for large- scale image recognition. arXiv preprini 44X 2014.
Ye, P., Huang, Y., Tu, C., Li, M., Chen, T., He, T., and Ouyang, W. Partial fine-tuning: A successor to full fine- tuning for vision transformers. CoRR, 2023.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, pp. 1453-1460. IEEE, 2011.
Yuval. N. Reading digits in natural images with unsuper- vised feature learning. In NIPS Workshop on Deep Learn- ing and Unsupervised Feature Learning, 2011.
Sung. N., Kim, M., Jo, H., Yang, Y., Kim, J., Lausen, L., Kim, Y., Lee, G., Kwak, D., Ha, J.-W., al. Nsml: A

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Appendix
This appendix includes the following materials: 1) Implementation Details (Appendix A), 2) Further Experimental Results (Appendix B), 3) Additional Ablation Studies (Appendix C), 4) Results in Diverse Model Pool (Appendix D), 5) Theoretical Analysis (Appendix E). 6) Applicability Beyond Image Classification (Appendix F).
A. Implementation Details
In the CLIP scenario, for fine-tuning, we the batch size 128 and use a learning rate of le-5 with a cosine annealing schedule. We utilize the AdamW optimizer, applying weight decay of 0.1. Following Task Arithmetic (Ilharco et al., 2022a), we freeze the final classification layer of CLIP's & text encoder during fine-tuning. With the observation that freezing the classification layer does not affect accuracy (Ilharco et al., 2022b), we do not consider unfreezing the final layer of CLIP's text encoder. Attention layers are also frozen (Ye et al., 2023; Tang et al., 2024). We fine-tune the models by adjusting the configurations of RandAugment. Specifically, we vary the number of sequential augmentation transformations (ranging from to 3) and the magnitude of these transformations (ranging from to 10). total of 30 models are fine-tuned. Unlike previous works, we incorporate data augmentation directly into the fine-tuning process, which requires adjusting the number of training epochs better accommodate the augmented data. Consequently, the number of training epochs is set as follows: 70 epochs for Cars, 100 epochs for DTD, 40 epochs for EuroSAT, GTSRB, RESISC45, SUN397, and 30 epochs for MNIST and SVHN.
In the standard image classifier unlearning scenario, we fine-tune models with varied hyperparameters. For CIFAR-IO, we use ResNet-18 with a batch size of 256 and learning rate of 0.05, and VGG-16 with a batch size of 64 and a learning rate of 0.01. Instead of data augmentation, we adjust training settings, setting the number of epochs to 40, 50, or 60, weight decay to le-4, 5e-5, and le-5, and label smoothing with ResNet-18, 18, 5e-5, and le-5, and label smoothing to 0, 0.05, and 0.1, producing another 27 models.
0.05, or 0.1, resulting in 27 fine-tuned models. Similarly, for CUB batch size of 64, learning rate of 0.01, and vary epochs to 5, 10, 20, weight decay to le-4,
B. Further Experimental Results
B.1. Full Results
Table BI, Table B2, and Table B3 show the full accuracy results for the eight datasets (Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN) and the three CLIP models we examine. Similarly, Table B4 and Table B5 provide accuracy results for Linear Task Arithmetic (Ortiz-Jimenez et al., 2024) these datasets for CLIP models.
Table BI. VIT-B/32 Task Arithmetic Results.
Cars
DTD
EurOSAT
GTSRB
MNIST
RESISC4S
SUN397
SVHN
Method
Dy(L)
D,
Dy(L)
D,
Dy(L)
D,
Dy(L)
D.
Dy(L)
Dr
Dy(L)
D,
Dy(L)
D.
Dy(1)
D.
Task Arithmetic
29.0
59.9
30.4
60.8
10.4
60.9
9.1
60.9
21.2
60.6
30.7
60.8
50.6
59.9
7.6
60.9
Uniform Merge
31.7
60.4
29.6
60.6
8.9
60.8
7.0
60.0
20.5
61.4
23.8
60.1
51.4
60.5
7.3
60.7
Greedy Merget
31.0
60.3
29.5
60.6
9.4
60.8
8.4
60.5
21.3
62.0
28.3
60.7
51.4
60.4
7.2
60.7
TIES MEES-Merging
34.0
60.3
33.1
61.3
11.6
61.1
10.2
61.3
26.1
62.4
33.4
61.0
53.8
60.3
7.5
60.9
MagMax
35.6
60.6
31.9
61.1
10.5
60.7
8.4
60.8
20.1
60.7
30.7
60.6
55.4
61.1
9.3
62.0
NegMerge (ours)
27.4
60.4
27.2
60.5
7.9
60.2
6.2
60.0
20.5
59.9
22.6
60.5
47.2
60.6
7.2
60.9
Table B2. VIT-B/16 Task Arithmetic Results.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Method
Dy(L)
D,
Dy(4)
D,
Dj(L)
D.
Dy(1)
D.
Dy(L)
D.
Dy(1)
D,
Dy(1)
D,
Dy(1)
D.
Task Arithmetic'
31.6
63.8
26.1
63.8
7.6
64.3
7.7
64.5
8.9
64.0
27.2
64.4
49.1
63.7
6.9
63.9
Uniform Merge
32.9
64.6
26.3
64.5
9.8
64.8
7.0
64.1
13.9
65.0
25.6
64.7
49.7
64.6
64.7
Greedy Merget
32.9
64.6
25.0
63.7
9.9
64.7
7.0
64.1
12.4
64.8
25.6
64.6
51.1
65.1
6.9
64.7
TIES- TIES-Merging
39.4
65.0
27.4
64.0
10.2
64.8
8,6
64.6
11.1
64.9
33.6
65.3
53.2
64.8
6.7
64.3
MagMax
38.4
64.8
26.6
63.9
10.2
65.0
9.0
64.9
14.6
64.4
36.6
66.0
53.5
65.0
6.7
64.3
NegMerge (ours)
28.8
64.8
25.2
64.5
9.8
65.9
7.1
64.4
10.7
63.8
20.3
63.9
45.2
64.4
7.0
64.6
12

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlearning
Table B3. VIT-L/14 Task Arithmetic Results.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC4S
SUN397
SVHN
Method
Dy(1)
D,
Dy(
D.
Dy(1)
D.
Dy(1)
D.
Dy(L)
D,
Dy(1) D,
Dy(1)
D,
Dy(1)
D.
Task Arithmetic*
34.6
72.2
24.7
71.3
5.4
72.5
3.0
71.6
10.3
73.6
17.0
71.7
51.6
71.9
6.7
71.9
Uniform Merge
29.1
71.8
23.5
71.4
8.2
72.1
3.1
71.5
9.9
72.4
13.9
71.5
50.5
72.3
6.7
72.2
Greedy Merget
28.2
71.5
23.9
71.5
7.3
73.0
3.1
71.7
9.9
72.8
11.5
71.0
51.1
72.3
6.8
72.1
TIES Merging
48.2
73.1
25.5
71.5
9.2
72.4
4.1
72.6
10.3
73.0
21.0
72.0
56.6
72.8
6.8
71.9
MagMax
39.2
72.0
28.7
72.7
9.9
73.6
4.2
72.5
10.7
73.5
20.6
72.2
53.7
72.1
6.7
71.9
NegMerge (ours)
32.7
71.9
23.9
71.9
72.1
2.8
71.3
10.9
73.6
8.8
70.9
43.6
72.1
72.8
Table B4. ViT-B/32 Linear Task Arithmetic Results.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Method
Dy(1)
D,
Dy({)
D.
Dy(L)
D,
Dy({)
Dr
Dy(L)
D,
Dy(L)
D,
Dy(L)
D,
D2(1)
D.
Task *
13.5
60.2
15.2
59.7
60.3
0.2
60.2
61.0
2.6
38.8
59.9
0.7
60.5
Uniform Merge
14.2
60.4
15.3
60.2
0.0
60.3
0.2
60.8
0.0
60.2
2.6
60.2
39.7
60.4
0.8
61.3
Greedy Merget
14.4
60.3
15.8
60.2
0.0
60.4
0.2
60.2
0.0
60.5
2.6
60.3
36.3
59.7
0.7
60.6
TIES- Merg-ngg
19.3
60.4
16.5
60.0
60.5
0.2
60.5
60.4
5.6
42.8
60.3
0.8
60.5
MagMax
22.6
61.0
16.5
60.1
0,2
60.6
0.2
60.8
0.1
61.5
4.2
60.1
46.1
60.9
0.7
60.4
NegMerge (ours)
12.1
60.6
15.6
60.4
60.9
0.2
0.0
60.9
34.0
59.8
Table B5, ViT-B/16 Linear Task Arithmetic Results.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC4S
SUN397
SVHN
Method
Dy(l)
D,
D2())
D,
D2(4) D,
Dy(L)
Dr
Dy(L)
D,
Dy(L) D,
Dy(L) D,
D2()
D.
Task Arithmetic
64.4
10.2
63.8
64.8
0.0
64.5
67.0
37.4
64.7
0.5
Uniform Merge
5.0
64.7
10.1
64.2
0.1
66.0
0.0
66.0
0.1
67.4
1.6
64.0
37.5
65.0
0.4
64.8
Greedy Merget
5,0
64.8
10.3
64.1
0.0
64.5
0.0
64.5
0.1
66.8
1.5
63.9
37.1
65.0
0.4
64.2
TIES- TIE--Merging
64.3
11.7
65.3
0.0
66.9
4.1
64.4
43.8
65.7
MagMax
8.8
64.6
12.3
64.9
0.0
64.9
0.0
64.9
0.1
67.2
5.1
64.9
42.5
65.4
0.4
64.6
NegMerge (ours)
65.9
10.4
64.5
65.9
0.0
66.9
1.1
34.1
64.7
Table B6 extends Table 3 by showing results for all eight datasets, analyzing the impact of sign conflicts in weights during unlearning. Table B7 expands on Table 4, providing zero ratios, forget set accuracy (Acc Dy), and retain set accuracy (Ace D,) across different numbers of task vectors used for merging.
Table B6. Impact of Sign Conffict in Weights for Unlearning. This table presents unlearning performance across various datascts using CLIP ViT-B/32, comparing three different methods. "All," Uniform Merge, uses all indices regardless of sign conflict, "Conflict" uses only indices with conflicting signs, and "Non-conflict," proposed method, uses only indices with consistent signs task vectors.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC4S
SUN397
SVHN
Method
Dy())
D,
Dy4)
D.
Dy()
D,
Dy())
D.
Dy(1)
D.
Dy())
D.
Dy(
D.
D
Dr
31.7
60.4
29.6
8.9
60.8
60.0
20.5
61.4
23.8
51.4
60.5
60.7
Conflict
40.2
60.2
31.9
60.3
11.1
60.7
9.1
60.6
24.0
61.9
32.3
60.2
58.3
60.9
8.8
60.6
Non-conflict
27.4
60.4
27.2
60.5
7.9
60.2
6.2
60.0
20.5
59.9
22.6
60.5
47.2
60.6
7.2
60.9
B.2. Full Charts of CLIP Unlearning Scenario
We provide the complete trade-off graphs illustrating the forget set's accuracy (i.e., 1 accuracy) versus the retain set's accuracy (Figure B1), extending the partial illustration presented in the main paper (Figure 1). Each graph denotes the trade-offs for different datasets, including Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN. Our method enjoys the best trade-off's among competing methods across most of the datasets.
13

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Table B7, Ratio of Zeroed Elements based on the Number of Merged Models, The results, averaged over three runs with standard deviations (std, =), were obtained using ViT-B/32 Task Arithmetic.
(a) Part I: Cars, DTD, EuroSAT, GTSRB
Cars
DTD
EuroSAT
GTSRB
%
Acc D)( Acc Dr
%
Acc Dy({)
Acc D,
%
Acc Dy({) Acc D.
%
Acc Dy(()
Acc D,
30 90.3
27.4
60.4
92.9
27.2
60.5
94.1
7.9
60.2
94.8
6.2
60.0
25 89.7+0.u4
26.640.06
60.2+a.02
92.2+o.10
27.2
60.54..10
93. 93.44
8.340.00
60.3
40.1
6.5
60.1
20 88, 88.
26.
60.1
91.5+0.13
91
27.1+0.13
60.4
92.7
8.840.06
60.640.03
g3.3
6.7
60.3
87.5
26. 26
60.0
90.2
27.7+0.09
60.440.03
90.8
8.3
60.4
01.9
6.9
60.3
10 84
26. 06.6....2
59. 59.920.14
87 510.11
27.8:0.008
60.4 12 88 1..51
8.94..22
60.610.11
89. 19
8,110.08
60.610.04
5 77.1 10.30
30.5:0.42
60.4
79.910.60
28.810.39
60.510.14 14 81.5
9.310.35
60.610.10 81.910.32
8.3
60.5
(b) Part 2: MNIST, RESISC45, SUN397, SVHN
MNIST
RESISC45
SUN397
SVHN
%
Acc Dy(()
Acc D,
%
Acc Dy(4)
Acc D,
%
Acc D)(4)
Ace D,
%
Acc Dy(L)
Acc Dr
30 94.0
20.5
59.9
92.9
22.6
60.5
92.2
47.2
60.6
92.4
7.2
60.9
25 93.3
20.5
60.2a0.08
92.310.04
23.7+0.06
60.640.02
91.7
47.0
60.410.04
91.7+.0.2
7.140.01
60.7
20
92.5
19.6
59.9
91 91.5
22.7:0.08
60.540.02
90.9
47.8
60.640.04
90.7
7.0+0.02
60.4
91.00..06
20.8
00.30
40.09
23.2+0.18
60.540.03 00.6
47.7
60.5
89.2
7..04
60.7
88. 20.42
19.9
60.5
15 87.6 640.28
22.640.79
60.4
20.31
48.7
48.74
60.64..11
86.
20.022.20
7.2+0.01
60.820.08
5 81.420.25
23.0 23.020.36
62. 02. 80.2 20.52
24. 24.1+1.24
1.24
60.4
20
10.510.10
49.
19.7
60.5
28.
7.2+0.03
60.610.08
73
Uniiee Merge
Sagl paica Moge
Siegl
A-Acc on forget set (%))
1- Acc on forget set (%)
Goonly Mlnge
Meeping
Ace on forget set (%)
Qredty Hire
1- Acc on forget set (%%
Gooby Nange
kingie
70.
Maghas
Mights Oue
Cer
71
91
67-
89
$7
89
61
60,0
60.5
61.0
60.0
60.5
61.0
61.5
60.0
60.5
61.0
61.5
60.0
60.5
61.0
61.5
Acc on retain sel (%)
Acc on refain set (%)
Acc on relain set (%6)
Ace on relain set (%)
(a) Cars
(b) DTD
(c) EuroSAT
(d) GTSRB
53
Ueilaa Mlage
Caidie Mage
Cauacy More
foowly Megn
Gemils Minee
Ace on forget set (%)
77-
Ace on forget set (%6
Magllt
ties Mtnalee
1- Ace on forget set (%)
Napita:
You Magie
I Ace on forget set (%)
Maghas
Minging
92-
50
73-
71-
91
Heglooou fomil
67
Mighle
90
60.0 69.5 61.0 61.5 62.5
60.5
61.0
61.5
62.9
39.5
60.0
60.5
61.0
61.5
60.5
61.00
61.5
62.0
62.5
Acc on retain set (%)
Acc on retain set (%)
Acc on retain set (%)
Acc on retain set (%)
(c) MNIST
(f) RESISC45
(g) SUN397
(h) SVHN
Figure BI. Comparison of Merged Models on ViT-B/32. Performance metrics for merged models showing accuracy on the retain set and forget set across different models. Methods positioned towards the upper right corner are generally considered to be better performers.
14

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlearning
B.3. Additional Standard Classifier Unlearning Scenario Results
Performance Different Datasets. Table B8 compares unlearning methods CUB (Wah al., 2011) with ResNet-18 for 10% random data forgetting. This experiment is important as it validates the effectiveness of Ne rge in fine-grained image classification. Table B9 shows similar results Tiny ImageNet (Le & Yang, 2015) using ResNet-18.
Table B8. Unlearning Performance for 10% Random Data Forgetting on CUB using ResNet-18.
Method
Used Splits
Ace D,(*)
Ace Dy(*)
Ace Diaa(=)
MIA(~)
Avg. Gap(I)
Retrain
retain
78.55
56.43
74.61
80.47
0.00
Gradient Ascent
66.75
57.26
66.60
67.61
8.38
Boundary Shrink
66.88
61.60
64.14
100.00
11.71
Boundary Expanding
forget
65.32
61.60
58.80
73.62
10.27
Random Labeling
64.13
57.43
59.54
71.79
9.79
SalUn
66.69
59.60
63.88
74.46
7.94
Task Arithmetic
Single Best Model
74.68
58.60
70.56
100.00
7.41
forget
Uniform Merge
73.94
56.93
69.78
100.00
7.37
NegMerge (ours)
74.64
58.26
70.69
100.00
7.30
Table B9, Unlearning Performance for % Random Data Forgetting on Tiny ImageNet using ResNet-18.
Method
Used Splits
Acc (=(=))
Acc Dy(=))
Ace Dieat(=)
MIA(==
Avg. Gap(L)
Retrain
retain
99.98
63.60
63.67
63.77
0.00
Random Labeling
76.43
76.09
58.09
32.33
18.27
forget
SalUn
73.61
73.81
56.68
30.59
19.19
Task Arithmetic
Single Best Model
forget
77.54
73.63
59.62
30.14
17.54
NegMerge (ours)
75.95
71.87
58.58
31.34
17.46
Performance on Different Models. Table B10 and Table B11 show results on CIFAR-I0 using VGG-16 (Simonyan, 2014) and Swin-T (Liu et al., 2021), respectively. In both cases, NeqMerge rge effectively unlearns the forget set D, using only the forget set itself. These results demonstrate that our method generalizes well across different model architectures.
Table B10. Unlearning Performance for 10% Random Data Forgetting CIFAR-10 using VGG-16.
Method
Used Splits
Acc (=
Acc Dy(=)
Acc Diest(2<) MIA(==)
Avg. Gap(L)
Retrain
retain
99.99
94.02
93.06
10.36
0.00
Gradient Ascent &
99.37
99.07
93.63
1.36
3.81
Boundary Shrink
forget
99.40
99.20
93.68
1.38
3.84
Boundary Expanding
99.39
99.20
93.68
1.42
3.84
Task Arithmetic
Single Best Model!
forget
97.26 98.00
94.90 95.74
90.10 91.01
10.34
1.64
NegMerge (ours)
10.10
1.50
Table BII. Unlearning Performance for 10% Random Data Forgetting on CIFAR-10 using Swin-T.
Method
Used Splits
Acc (,(=)
Acc Dy(=))
Acc Drea(
MIA(>)
Avg. Gap(L)
Retrain
retain
100.00
97.76
97.67
4.74
0.00
Random Labeling
99.96
99.96
97.71
0.64
1.60
forget
SalUn
98.98
99.04
96.40
3.36
1.24
Task Arithmetic
Single Best M
forget
98.50
97.84
95.98
0.99
NegMerge (ours)
98.79
97.80
95.93
4.60
0.78
15

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Performance Different Scenarios. Table B12 presents a comparison of various unlearning techniques for 50% random data forgetting CIFAR-10 using ResNet- 18. Table B13 compares various unlearning techniques for class-wise forgetting on CIFAR-10 using ResNet-18. These results highlight the scalability of NegMerge across diverse unlearning scenarios.
Table B12. Unlearning Performance for 50% Random Data Forgetting on CIFAR-10 using ResNet-18.
Method
Used Splits
Ace D,(=)
Acc Dy(=)
Acc Dieat
MIA(=)
Avg. Gap(L)
Retrain
retain
100.00
92.10
91.70
19.30
0.00
Random Labeling
99.80
99.90
94.70
2.20
7.03
forget
SalUn
99.60
99.60
94.20
4.40
6.33
Task Arithmetic
Single Best Model'
forget
98.40 96.80
97.90
92.60
5.60
5.50
NegMerge (ours)
96.50
91.50
6.30
5.20
Table B13. Unlearning Performance for Class-wise Forgetting on CIFAR-10 using Net-18.
Method
Used Splits Ace D,(2)
Ace Dy(*>)
Acc Drea
MIA(c=)
Avg. Gap(L)
Retrain
retain
100.00
0.00
92.50
100.00
0.00
Random Labeling
83.00
10.10
70.90
99.50
12.30
Forget
SalUn
86.50
10.80
74.10
100.00
10.68
Task Arithmetic
Single Best Model
forget
95.30
0.10
80.60
100.00
4.18
NegMerge (ours)
96.80
0.80
81.80
99.80
3.73
C. Additional Ablation Studies
Table C1 compares ways to derive the improved final task vector Tmerged. We found that our originally proposed averaging method performed the best. This is likely because averaging helps smooth out potential outliers in individual models during merging, resulting in a more stable and effective task vector.
Table CI. Results When Different Merging Operations Are Used. NegMerge (min), NegMerge (max), and NegMerge (avg) represent merging minimum, maximum, and average of task vectors elements, respectively. The experimental results are obtained using CLIP ViT-B/3..
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Method
Dy(1)
Dr
Dy(1)
Dr
Dy(1)
Dr
Dz(1)
D.
Dy(L)
Dr
Dy(1)
D,
Dy(l)
D.
Dy(L)
Dr
Task Arithmetic
29.0
59.9
30.4
60.8
10.4
60.9
9.1
60.9
21.2
60.6
30.7
60.8
50.6
59.9
7.6
60.9
NegMerge (mim) n)
26.5
59.9
27.9
60.6
10.3
60.8
8.3
60.7
25.2
61.0
20.1
60.0
46.8
60.2
8.2
61.0
NegMerge (max)
28.2
60.3
27.4
60.4
10.7
60,9
7.5
60.4
28.7
60.2
26.0
61.0
48.3
60.7
7.3
60.8
NegMerge (avg)
27.4
60.4
27.2
60.5
7.9
60.2
6.2
60.0
20.5
59.9
22.6
60.5
47.2
60.6
7.2
60.9
Table C2 shows that larger zero-out values with uniformly merged sparsified task vectors lead to improved unlearning results. TIES-merging and MagMax exhibit fewer zero-out values, and their performance expected be outperformed by NegMerge.
Table C2, Ratio of Zeroed Elements and the Unlearning Performance. The table reports the zero ratio and the accuracy of the forget set (Ace Dy), comparing these values across several baseline methods. Results are obtained using ViT-B/32 Task Arithmetic. Accuracy on D, omitted as methods remain around 60% accuracy on the retain set.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Method
%
Dy(()
%
Dy(()
%
Dy(()
%
Dy(L)
%
Dy(()
%
Dy(L)
%
D2(()
%
Dy(L)
Task Arithmetic
47.55
29.00
47.55
30.42
47.55
10.44
47.55
9.09
47.55
21.15
47.55
30.71
47.55
50.58
47.55
7.61
MagMax
47.55
35.59
47.55
31.86
47.55
10.51
47.55
8.42
47.55
20.14
47.55
30.69
47.55
55.36
47.55
9.33
TIES-Merging
51.59
34.03
50.62
33.09
51.62
11.56
50.72
10.21
51.94
26.09
51.00
33.41
50.58
53.80
52.06
7.48
NegMerge (ours)
90.34
27.40
92.94
27.18
94.05
7.85
94.76
6.20
93.96
20.50
92.86
22.61
92.20
47.19
92.40
7.18
16

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Table Sparsity by Component Type. The results report the sparsity (%) for Stem and MLP layers across models trained on the Cars dataset, Std. Dev. denotes variability across layers within each component; not applicable to the single-layer Stem.
ViT-B/32
ViT-L/14
Component
Std. Dev.
%
Std. Dev.
Stem MLP
98.3
99.6
55.1
0.3
77.6
0.2
Table C3 examines sparsity across different architectural components. Specifically, we compared the Stem layer (i.e., the patch embedding) and MLP layers (c_fe and c-proj weights/biases).
D. Results in Diverse Model Pool
D.1. Robustness on Diverse Model Pool
In the CLIP Unlearning Scenario, we conducted additional experiments to analyze the impact of varying hyperparameters, such as weight decay, learning rates, and label smoothing, as shown in Table D1. Also, in Table D2 we present the effects of modifying training hyperparameters and RandAugment configurations. Furthermore, we evaluated using seven different model pools, which are detailed in Table D3. In the Standard Classifier Unlearning Scenario, we enhanced the diversity of the fine-tuned models by adjusting RandAugment configurations, as summarized Table D4.
As shown in Table D1-Table D4, unlearning performance improves further compared to the baseline, as expected, with diverse types of hyperparameters. These results highlight that, while the degree of improvement may vary depending on the model pool, using multiple models consistently provides more stable performance gains compared to using single model.
Table DI. Learning Rate, Weight Decay, and Label Smoothing Configuration Pool on ViT-B/32 Task Arithmetic in CLIP Unlearning Scenario. Results are obtained by evaluating 16 models created from pool of configurations using the following hyperparameter settings: learning rates of le-4, 5e-5, le-5, and 5e-6; weight decay values of 0.01 and 0.1: and label smoothing to and 0.1.
Cars
DTD
SUN397
Method
Acc Dy(1)
Ace D,
Ace Dy(()
Acc D,
Acc Dy(L)
Acc D,
Task Arithmetic
33.52
60.29
29.14
60.38
$1.36
60.55
NegMerge (ours)
30.33
60.16
26.43
59.95
47.94
60.33
Table D2. RandAugment, Learning Rate, and Weight Decay Configuration Pool on VIT-B/32 Task Arithmetic CLIP Unlearning Scenario. Results are obtained by evaluating 8 models created from pool of configurations using the following herparrameeer settings: RandAugment with n 1, 2 and m 1,5, 10, learning rates of le-5, 5e-6, and le-6, and weight decay values of 0.01 and 0.1.
Cars
DTD
EuroSAT
GTSRB
MNIST RESISC4S
SUN397
SVHN
Method
D2(1) D. D2(1) D. D2(1) D, D2(1) D. D2(1) D. Dy(1) D. D2(1) D. D2(1) D.
Task *rithmetic*
28.62 60.17 28.03 60.15 7.66 60.46
5.31 60.22 14.56 60.55 27.19 60.72 51.38 60.32
6.75
61.30 60.29
NegMerge (ours)
27.42 60.03 26.80
60.08
7.03 60.39
4.82 59.50
12.89 59.94
18.23 59.81
48.73
60.38
6.71
D.2. Strategy to Create Variants
NegMerge relies on the knowledge encoded in each task vector. Based on our observations, poorly constructed task vectors, such as those trained with unreasonable weight decay, can result unreliable knowledge and introduce noise into the process. To mitigate this, we recommend constructing task vector pool using reasonable hyperparameters, ensuring the vectors are reliable and contribute effectively unlearning.
Furthermore, our method inherently produces model with a well-optimized retain loss. This aligns with one of our core assumptions discussed in Section 4.3: the merged model should preserve performance on the retain set. Practitioners can leverage this property by using the retain loss as guiding signal during the model generation phase, enabling more effective model merging through better monitoring and optimization of retain set performance. While we have not yet explored this idea, we view it as an exciting direction for future research.
17

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
Table D3. Average Seven Different Model Pools. To evaluate robustness across diverse model pools, we assessed performance on seven different model pools. The results show that leveraging multiple models consistently leads to more stable performance improvements compared to relying on a single model.
Pool
RandAAgment (n,
Learning Rates
Weight Decay
Label Smoothing
Pool 1
7 1-3, m 1-10
30
Pool 2
1e-4, 1e-5, 5e-5, 5e-6
0.1
0,0.1
16
Pool
le-4, 1e-5, 5e-5, 5e-6
0.01,0.1 0.1
Pool 4
le-5, 5e-6, 5e-5
0.1
Pool 5
n = m= 10
=
le-4, le-5, 5e-5, 5e-6
0.01,011
0,0.1
64
Pool 6 n 1= 5, 10
le-4, le-5, 5e-5, 5e-6
0.01,0.1
32
Pool
1=1,2, 1,,5,10
le-5, 6 1e-6
0.01,0.1
Pool
Pool (Dy 4)
Pool 2 (Dy 4)
Pool (Dy 4)
Pool (D₃ 4) Pool 5 (Dy 4)
Pool 6 (Dy 4)
Pool (Dy 4
Task Arithmetic!
23.63
22.80
23.21 21.56
23.21 22.23
24.05
21.31 19.37
21.19
NegMerge (ours)
20.76
21.69
22.65
19.08
Table D4. RandAugment Configuration Pool on ResNet-18 in Standard Classifier Unlearning Scenario. Results are obtained by evaluating 5 models created from pool of configurations using the following hyperparameter settings: RandAugment with n. = 1 and
m 1,
Method
Acc D,(~)
Acc Dy(>)
Acc Diea(>>
MIA(~)
Avg. Gap(J)
Retrain
100.00
94.76
94.26
12.88
0.00
Task Arithmetic
97.79
95.88
91.31
9.58
2.40
NegMerge (ours)
97.81
95.76
91.03
10.76
2.14
E. Theoretical Analysis
Let fori and ff denote the weights of the pre-trained model and a fine-tuned model, respectively. We have the formulation i Juntearn = fori meerged. where Tmerged is from Equation (1) our consensually merged task vector. We argue that achieving larger zero-out values with sparsified consensus editing signals in Tmerged could lead to unlcarning performance improvements, based on the following fundamental claims: (1) Weight-wiss unanimous consensus merging reduces non-zero values and gives a robust mmerged; the larger zero-out values in Tmerged contribute to stable merging. (2) There exists stable merged point 6 uulteann which enjoys better unlearning results as the number of task vectors тą increases.
As the number of task vectors Tk in Tiergee increases, the non-zero values decrease because our consensus operation performs like an AND operation. The robustness of Tmerged increases upon merging, as sparse weights are merged uniformly, which enjoys inherently more robustness than an individual weight as revealed in (Wortsman et al., 2022; Jang et al., 2024), where the first term of Tmerged = 1
x Σn Of: - fori conducts uniform merge. Ownteorn moves closer to Oori which likely stays in lower loss regions due to weights that generally hold linear mode connectivity (LMC) (Frankle al., 2020; Juneja al., 2022; Entezari al., 2021). It also mitigates issues that cause fluctuating high losses (i.e., loss barriers) on certain loss surfaces, when weights deviate significantly from Pori- Therefore, task negation at an improved 0 Mittearn is likely to reside in lower loss regions, leading to better results. While we do not specify the exact 0 Oiniearn increasing the number of merged task vectors allows the process to approach a closer-to-opiee -ootimmal point as more sparsified merged weights are merged uniformly (Jang et al., 2024).
Our empirical evidence, shown in Table demonstrates that larger zero-out values from uniformly merged sparsified task vectors lead to improved unlearning results. Additionally, as indicated in Table C2, TIES- HES-megggngg and MagMax exhibit fewer zero-out values, which explains their lower performance compared method.
F. Applicability Beyond Image Classification
While our study focuses on image classification tasks, we believe NegMerge is broadly applicable to other modalities, including large language models (LLMs) and vision- vision-language models (VLMs). Our approach leverages the consistency of directional changes in parameter space across multiple fine-tuned models, a principle not specific to visual tasks or
18

---

NegMerge: Sign-Consensual Weight Merging for Machine Unlcarning
architectures. Given that LLMs and VLMs often built transformer backbones, which our method already demonstrates strong performance on (e-g., ViT-B/32, ViT-I/14), expect the sign-consensual merging strategy to extend to these domains.
19


Reviews:
[{"review_id": "eMSu0Mz9iS", "reviewer": "Reviewer_aohJ", "summary": "This paper treats the unlearning problem as a task arithmetic, where they conduct task vector by finetuning on the forget set, then subtract it from the original weight. To avoid sensitivity to hyperparameter selection, the authors create a finetuned model pool by various hyperparameter settings and aggregate them into a final task vector by considering their signs. They claim that the proposed method requires similar or fewer computational complexity than existing methods, and achieves superior performance compared to state-of-the-art methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive feedback on the clarity and intuitiveness of our method. We hope to address the concerns regarding generalizability and scalability in our responses below.\n\n### **Weakness 1: No theoretical claims were provided**\n\nWe provide a theoretical claim for our method. We would like to note that an informal version was presented in Appendix E.\n\n**Theorem.** Consensus-based merging of multiple task vectors yields a robust and effective unlearning direction by inducing sparsity and guiding the model toward a low-loss region.\n\n**Lemma 1.** As the number of task vectors $\\tau_k$ increases, the merged vector $\\tau_{merged}$ becomes sparser.\n\n**Proof.** Consensus merging keeps each element only if all task vectors agree in sign, acting like an AND operation. With more vectors, agreement decreases, increasing sparsity.\n\n**Lemma 2.** If $\\theta\\_{ft}$ is centered around $\\theta\\_{ori}$, then $\\theta^*\\_{unlearn} = \\theta\\_{ori} - \\tau\\_{merged}$ lies closer to $\\theta\\_{ori}$.\n\n**Proof.** Given $\\tau\\_{merged} = \\bar{\\theta}_{ft} - \\theta\\_{ori}$,\n\nwe have $\\theta^*\\_{unlearn} = \\theta\\_{ori} - (\\bar{\\theta\\}_{ft} - \\theta\\_{ori})$. \n\nIf $\\bar{\\theta}_{ft} \\approx \\theta\\_{ori}$, then:  $\\theta^*\\_{unlearn} \\approx  \\theta\\_{ori}$.\n\nLemmas 1 and 2 imply that consensus merging improves robustness (via sparsity) and effectiveness (by staying near $\\theta_{ori}$ under linear mode connectivity), yielding a reliable unlearning direction.\n\n\nWhile our theoretical analysis may offer a reasonably plausible justification, its effectiveness was most clearly demonstrated through strong empirical results across diverse datasets and architectures. We consider a more rigorous theoretical analysis as future work.\n\n### **Weakness 2: Inability to conduct experiments on ViT-L/14**\n\nViT-L/14 results for Task Arithmetic were actually included in Table 1. For Linear Task Arithmetic, we ran a scaled-down experiment on ViT-L/14 using six models on the Cars dataset to complete it within the rebuttal period. As shown in the table below, ours outperforms the baselines, achieving the lowest forget accuracy (Df ↓) while maintaining comparable retain accuracy (Dr).\n\n| Method             | Df ↓  | Dr    |\n|--------------------|-------|-------|\n| Single Best Model  | 28.68 | 71.45 |\n| Uniform Merge      | 34.68 | 71.96 |\n| Ours               | 24.47 | 71.66 |\n\n\n### **Weakness 3: Evidence is not enough (50% random forgetting and class-wise unlearning)**\n\nWe extend Table 2 with additional experimental results under 50% random data forgetting and class-wise unlearning, using only the forget set. These experiments were conducted on the CIFAR-10 dataset with the ResNet18 architecture. As shown below, our method outperforms baselines in unlearning performance. For class-wise unlearning and 10% forgetting on TinyImageNet and Swin-T, please see our response to Reviewer zcBE (Weakness 1).\n\n**50% Random Data Forgetting Results:**\n\n| Method                 | Dr    | Df    | Dtest | MIA  | Avg.Gap ↓ |\n|------------------------|-------|-------|-------|------|-----------|\n| Retrain                | 100.0 | 92.1  | 91.7  | 19.3 | 0.0       |\n| Random Labeling        | 99.8  | 99.9  | 94.7  | 2.2  | 7.0       |\n| SalUn                  | 99.6  | 99.6  | 94.2  | 4.4  | 6.3       |\n| Task Arithmetic (Best) | 98.4  | 97.9  | 92.6  | 5.6  | 5.5       |\n| Ours                   | 96.8  | 96.5  | 91.5  | 6.3  | 5.2       |\n\n### **Weakness 4: Lack of retain set (Dr) performance in Table 4**\n\nIn Section 4.2, we mentioned that the retain set (Dr) accuracies for all methods remain around 60%, following the implementation details of Ilharco et al. (2022) for the CLIP unlearning scenario. Below is the extended table (Table 4) including both Df and Dr results. We will provide further clarification in the final version.\n\n| #   | Cars (Df↓ / Dr↑) | DTD (Df↓ / Dr↑) | SUN397 (Df↓ / Dr↑) |\n|-----|------------------|------------------|---------------------|\n| 5   | 30.5 / 60.4      | 28.8 / 60.5      | 49.7 / 60.5         |\n| 10  | 26.6 / 59.9      | 27.8 / 60.4      | 48.7 / 60.6         |\n| 15  | 26.0 / 60.0      | 27.7 / 60.4      | 47.7 / 60.5         |\n| 20  | 26.1 / 60.1      | 27.1 / 60.4      | 47.8 / 60.6         |\n| 25  | 26.6 / 60.2      | 27.3 / 60.5      | 47.0 / 60.4         |\n| 30  | 27.4 / 60.4      | 27.2 / 60.5      | 47.2 / 60.6         |\n\n\n### **Suggestions 1: Minor typos**\n\nWe appreciate the reviewer’s notice. We will carefully proofread again before the final submission.\n\n### **Questions 1: Is the proposed method efficient when the pool size is large?**\n\nAlthough our method scales with the number of fine-tuned models, it does not incur additional overhead, since all methods—including single-task arithmetic—share the same underlying model pool. Comparable cost increases also arise in single-task arithmetic due to its broader hyperparameter search space. This is discussed in detail in Section 3.3, Analyses on Computational Cost.", "claims_and_evidence": "The evidence is not enough to convince the effectiveness of the proposed method. When compared with existing unlearning methods, the authors only conducted an experiment on 10% random data forgetting on CIFAR-10. However, SALUN [1] has demonstrated its effectiveness on <10,20,30,40,50>% Random Data Forgetting on CIFAR-10, CIFAR-100, and Tiny-Imagenet, whereas Boundary Unlearning [2] has demonstrated its effectiveness on class-wise unlearning setting.\n\n[1] Fan, Chongyu et al. “SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation.” ArXiv abs/2310.12508 (2023): n. pag.\n[2] Chen, Min et al. “Boundary Unlearning: Rapid Forgetting of Deep Networks via Shifting the Decision Boundary.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 7766-7775.", "methods_and_evaluation_criteria": "The proposed methods are reasonable for unlearning. Also, evaluation criteria are the most popular benchmark and metrics, which are used in current state-of-the-art models.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The baselines are verified to be effective on class-wise unlearning and various scenarios of instance-wise unlearning, however, the authors did not conduct sufficient experiment settings, making the comparison less convincing.", "relation_to_broader_scientific_literature": "Sign consistency is an interesting approach to understand the contribution of each parameter in a model. It is feasible to apply to model pruning, model compression, model editing, etc.", "essential_references_not_discussed": "No essential reference is not discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n-\tUnlearning is an important problem in practice.\n-\tThe proposed methods are intuitive and well-presented.\n-\tThe paper is well-written and easy to follow.\n\nWeaknesses:\n\n-\tNo theoretical claims were provided to understand the effectiveness of the proposed method rigorously.\n-\tThe inability to conduct experiments on ViT-L/14 raises concerns about the feasibility of applying these to large models and real-world applications.\n-\tThe experiments are weak to make the claims. When compared to existing unlearning methods, the authors only conduct experiments on the CIFAR10 dataset with the configuration of 10% Random Forgetting. It should be done on 50% random forgetting and class-wise unlearning, where current methods have demonstrated their effectiveness in those settings.\n-\tIn the study of sparsity, only performance on the forget set has been reported. The performance on retain set is also important, so it should not be ignored from the experiment analysis.", "other_comments_or_suggestions": "There are a few typos can be improved.", "questions_for_authors": "-\tIs the proposed method efficient when the pool size is large?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELVpHjcrYw", "reviewer": "Reviewer_zcBE", "summary": "This paper proposes a new approach to machine unlearning focusing on instance-based unlearning, where instead of forgetting samples corresponding to a specific class, it can forget samples throughout all the classes while preserving model performance on the rest of the samples, i.e., retain-set.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive comments on the clarity of our paper and for recognizing the potential of our method as an alternative to parameter-based unlearning approaches. The reviewer has expressed concerns regarding the generalizability of our work and questions regarding instance-wise unlearning. We hope the following feedback addresses these concerns.\n\n### **Weakness 1: Limited evaluation: class unlearning, more datasets, and different architectures**\n\nWe provide additional results for class-wise unlearning and 50% random data forgetting on CIFAR-10 using ResNet18, using only the forget set. As shown below, our method consistently surpasses baseline methods. Please refer to our response to Reviewer aohJ (Weakness 3) for results on the 50% random forgetting setting.\n\n**Class-wise Unlearning Results:**\n\n| Method                 | Dr    | Df    | Dtest | MIA   | Avg.Gap ↓ |\n|------------------------|-------|-------|-------|-------|-----------|\n| Retrain                | 100.0 | 0.0   | 92.5  | 100.0 | 0.0       |\n| Random Labeling        | 83.0  | 10.1  | 70.9  | 99.5  | 12.3      |\n| SalUn                  | 86.5  | 10.8  | 74.1  | 100.0 | 10.7      |\n| Task Arithmetic (Best) | 95.3  | 0.1   | 80.6  | 100.0 | 4.2       |\n| Ours                   | 96.8  | 0.8   | 81.8  | 99.8  | 3.7       |\n\nAdditionally, we present new results on 10% random data forgetting conducted on TinyImageNet and Swin-T. With this, we now evaluate instance unlearning across three datasets—CIFAR-10, TinyImageNet, and CUB—using three architectures: ResNet-18, VGG-16, and Swin-T. Our approach consistently outperforms baseline methods across various datasets and architectures, including TinyImageNet and Swin-T.  Please find our new results below.\n\nOn the **TinyImageNet** dataset:\n| Method                 | Dr    | Df    | Dtest | MIA  | Avg.Gap ↓ |\n|------------------------|-------|-------|-------|------|-----------|\n| Retrain                | 100.0 | 63.6  | 63.7  | 63.8 | 0.0       |\n| Random Labeling        | 76.4  | 76.1  | 58.1  | 32.3 | 18.3      |\n| SalUn                  | 73.6  | 73.8  | 56.7  | 30.6 | 19.2      |\n| Task Arithmetic (Best) | 76.9  | 73.6  | 59.4  | 29.3 | 18.0      |\n| Ours                   | 76.0  | 71.9  | 58.6  | 31.3 | 17.5      |\n\nUsing **Swin-T**:\n| Method                 | Dr   | Df   | Dtest | MIA | Avg. Gap ↓ |\n|------------------------|------|------|-------|-----|-------------|\n| Retrain                | 100.0| 97.8 | 97.7  | 4.7 | 0.0         |\n| Random Labeling        | 100.0| 100.0| 97.7  | 0.6 | 1.6         |\n| SalUn                  | 99.0 | 99.0 | 96.4  | 3.4 | 1.2         |\n| Task Arithmetic (Best) | 98.5 | 97.8 | 96.0  | 4.0 | 1.0         |\n| Ours                   | 98.8 | 97.8 | 95.9  | 4.6 | 0.8         |\n\n### **Weakness 2: Is instance unlearning truly happening?**\n\nIn the standard classifier unlearning scenario, the Retrain model is trained from scratch using only the retain set and excluding the forget set, and has been regarded as the (ground-truth-like) upper bound of performance, as none of the forget set is used during training. The goal is for the Unlearn model to closely match the Retrain model's performance on each of Dr, Df, Dtest, and MIA individually, which has been the standard evaluation protocol in this scenario. In instance-level unlearning, classification accuracy on the forget set often remains similar, as observed in the corresponding table. This is presumably due to the generalization effect: even though the Retrain model is not trained on the forget set, it can still perform well on it by leveraging class-level information learned from the retain set.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes. The proposed approach in Section~3.2.", "relation_to_broader_scientific_literature": "The proposed approach could provide an alternative to the important parameter-based unlearning approach.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "**Strenghts**\n\n1. The paper is well written, and the problem setup is clear.\n2. The proposed approach could provide an alternative to the important parameter-based unlearning approaches, which suffer from performance degradation on the retain-set samples while enabling unlearning on the forget-set samples.\n\n**Weaknesses**\n\n1. One of the key weaknesses of this paper is limited evaluation. The authors do not provide any empirical evaluation on the class unlearning setup. Furthermore, the authors only provide instance unlearning on CIFAR10 with Resnet. I believe it is crucial to include comparisons involving various datasets, CIFAR100, ImageNet-1K, and TinyImageNet is crucial. Moreover, the comparison involving different vit architecture involving Swin transformer, ViT-B/16, ViT-L/16 using aforementioned datasets is also necessary.\n\n2. In Table~2, if we check the column Acc $(D_{f})$, it is almost similar to Acc $(D_{r})$, which raises a crucial question: is this approach really enabling instance unlearning?", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Please refer to the weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WawV0lzMk8", "reviewer": "Reviewer_xEch", "summary": "The paper proposes a novel framework for machine unlearning that takes advantage of multiple fine-tuned models (10 to 30) to localize the parameters that have the same sign across all task vectors. The idea is that these parameters are characteristic of the task, while others might be noise. The procedure results in a new task vector which is then subtracted from the zeroshot model. The authors include results on CLIP unlearning and on CIFAR with ResNets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our method’s originality. Below, we address concerns on LLM experiments and related work.\n\n### **Weakness 1: Lack of LLM experiments**\n\nWe agree that unlearning in LLMs is an important direction. We believe our method can generalize to transformer-based LLMs given its demonstrated effectiveness on transformer architectures, although the current study focuses on image classification tasks. We appreciate the suggestion and will include LLM experiments and discussion on our method’s applicability to LLMs in the final version of the paper.\n\n### **Weakness 2: Additional references and deeper intuition (CCA, task localization)**\n\n#### **1. Connection to Canonical Correlation Analysis (CCA)** \n\nAs the reviewer suggested, CCA-based methods [1,2] have been used to analyze similarity between layers of neural networks. Indeed, it would be interesting to adapt such a viewpoint to identify which layers are most effective for unlearning. Our work, however, starts from the assumption that certain parameters—rather than entire layers—are more central to the given task. We thus focus on parameter-level unlearning, where we isolate specific parameters that are consistently implicated in the task across multiple models. We agree that layer-level unlearning via CCA is a promising direction, and will add discussion.\n\n#### **2. Relation to Task Localization / Merging Methods** \n\nWe appreciate the additional pointers to task localization methods [3,4,5]. In our Related Work section, we cite some approaches (e.g., TIES-Merging, AdaMerging, MagMax) that also address how to combine or localize tasks. However, these methods focus largely on task addition, attempting to merge models trained on different tasks to maximize the differences that define each new task. For instance, the Tall Mask method [3] seeks “taller” vectors for merging.\n\nIn contrast, our goal is to identify parameters shared across models trained on similar tasks. Rather than negating the most diverse features, we look for the parameters that share a consistent effect (sign) across models. These shared parameters most effectively capture the task vectors that are attributed to the forget set and are therefore the prime candidates for unlearning. We will further clarify this difference in the revised paper.\n\n### **Suggestions 1: Lack of retain set (Dr) performance in Table 4**\n\nWe have added the full Table 4 results (please see Reviewer aohJ, Weakness 4) and will clarify further in the final version.\n\n### **Suggestions 2: “Avg gap” metric not clearly defined**\n\nFollowing SalUn’s evaluation protocol, we report the Avg. Gap, which is the average difference between the Unlearn and Retrain models across Dr, Df, Dtest, and MIA. We will add further clarification in the final version.\n\n### **Suggestions 3: Sign consensus analysis**\n\nWe conducted several analyses using ViT-B/32 and ViT-L/14 on the Cars dataset with 30 models. First, we examined the block-wise consensus ratio. As shown in the tables below, the consensus ratio increases in deeper blocks, in both architectures. This suggests that more parameters are being negated as the blocks go deeper. Here, we grouped blocks into three ranges and reported the average consensus ratio per range. Each block corresponds to one transformer block.\n\n**ViT-B/32:**\n| Block Range | Consensus Ratio |\n|-------------|------------------|\n| 0–3         | 76.55           |\n| 4–7         | 81.64           |\n| 8–11        | 87.08           |\n\n\n**ViT-L/14:**\n| Block Range | Consensus Ratio |\n|-------------|------------------|\n| 0–7         | 66.52           |\n| 8–15        | 73.62           |\n| 16–23       | 76.68           |\n\nAdditionally, we compared the Stem layer (i.e., the patch embedding), MLP layers (c_fc and c_proj weights/biases), and Attention layers (query, key, value, and output projection weights/biases). The standard deviation refers to the variability across different layers within each category (MLP or attention). Surprisingly, the Attention layers consistently showed 100% sign consensus in both architectures. We hypothesize that this is because the query-key-value mechanism plays a fundamental role in capturing contextual relationships, which remain stable across different fine-tuning configurations.\n\n**ViT-B/32:**\n| Layer     | Consensus (%) | Std Dev |\n|-----------|----------------|---------|\n| Stem      | 1.75           | 0.00    |\n| Attention | 100.00         | 0.00    |\n| MLP       | 44.87          | 0.31    |\n\n**ViT-L/14:**\n| Layer     | Consensus (%) | Std Dev |\n|-----------|----------------|---------|\n| Stem      | 0.36           | 0.00    |\n| Attention | 100.00         | 0.00    |\n| MLP       | 22.41          | 0.23    |\n\nWe will include these findings, which we believe enhance the paper’s depth and quality. We sincerely thank the reviewer for the helpful suggestion that led to this.\n\n### **Suggestions 4: NTK reference**\n\nWe will ensure that our next revision will correct it.", "claims_and_evidence": "The claims of the paper are clear and convincing.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria do make sense for the problem at hand. The evaluation is thorough and well documented on the selected benchmarks. The paper, however, does omit experiments on machine unlearning of LLMs, which is an important topic. \n\nA minor issue is the lack of retain performance on table 4.", "theoretical_claims": "There are no formal theoretical claims. The paper does include an appendix about a theoretical analysis but it is more about intuition rather than formal claims.", "experimental_designs_or_analyses": "The experimental design is sound. The “Avg gap” metric is not clear from the text.", "relation_to_broader_scientific_literature": "The paper misses some related works that can better place the work in the broader scientific literature. See weaknesses.", "essential_references_not_discussed": "* Machine unlearning of LLMs is not discussed\n* See Weaknesses for some more references.", "other_strengths_and_weaknesses": "## Strengths\n\n* The idea of leveraging multiple fine-tuned models in machine unlearning is original. The core of the idea has been seen before, e.g., model soups etc, but afaik this is the first work to apply these concepts to machine unlearning.  \n* The paper is easy to follow and well written  \n* The experimental validation is thorough, ablations on the number of models are done (see also below for suggestion)\n\n## Weaknesses\n\n1. Lack of LLM experiments. Machine unlearning for language is an important aspect of the literature (if not the most important) and is not discussed or used in experiments  \n2. The paper could be further improved if more intuition is provided. Specifically, L146-158 (step 2\\) provides some intuition for why the proposed method works but it is very limited. Imo, the method is interesting because it reveals that certain directions are characteristic of the task and hence appear in all models regardless of configurations (as mentioned in the paper), while everything else is noise. Hence, connecting to literature such as \\[1,2\\]  would be very interesting. Similarly, the proposed method performs some kind of task localization (especially in the task addition section) and, therefore, works on this area should also be cited \\[3,4,5\\]. I believe more intuition and connecting to these areas would improve the paper.\n\n\\[1\\] Morcos, Ari, Maithra Raghu, and Samy Bengio. \"Insights on representational similarity in neural networks with canonical correlation.\" Advances in neural information processing systems 31 (2018).\n\n\\[2\\] Kornblith, Simon, et al. \"Similarity of neural network representations revisited.\" International conference on machine learning. PMLR, 2019\\.\n\n\\[3\\] Wang, Ke, et al. \"Localizing task information for improved model merging and compression.\" arXiv preprint arXiv:2405.07813 (2024).\n\n\\[4\\] He, Yifei, et al. \"Localize-and-stitch: Efficient model merging via sparse task arithmetic.\" arXiv preprint arXiv:2408.13656 (2024).\n\n\\[5\\] Panigrahi, Abhishek, et al. \"Task-specific skill localization in fine-tuned language models.\" International Conference on Machine Learning. PMLR, 2023\\.", "other_comments_or_suggestions": "1. See weaknesses  \n2. It would be useful to include a graph of consensus as a number of available checkpoints. This can be done per layer, since previous works have shown that different types of layers (e.g. attention vs mlp layers) have different levels of importance. For x models, there are (30 choose x) combinations, hence subsampling could work.   \n3. NTK reference is Jacot et al. The cited one Ortiz-Jimenez et al applies NTK ideas in task arithmetic. Please fix.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GcN9GQT6cG", "reviewer": "Reviewer_hSw6", "summary": "This paper presents NegMerge, which enhances the process of forgetting by negation for machine unlearning. NegMerge computes a final task vector by merging task vectors derived from multiple fine-tuned models, during which it preserves elements with consistent signs across the task vectors and masks those with inconsistent signs to zero. The results demonstrate that NegMerge achieves state-of-the-art performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive questions. We have done our best to address the concerns raised, as detailed below.\n\n### **Weakness 1: Why Model Merging for Machine Unlearning?**\n\nThis is because models fine-tuned with different hyperparameters tend to specialize in either unlearning or retaining  (as shown in Figure 1). Therefore, while searching for effective hyperparameters is a prerequisite for approaching optimality, we argue that merging them provides a way to combine their strengths and resolve the trade-off that a single model cannot, which also holds in unlearning. As shown in Figure 1(a), no single model achieves high performance on both objectives. \n\nWe believe model merging is effective for unlearning because it provides access to multiple optimization trajectories, which allows us to identify parameters that are related to the forget set (e.g., via sign consensus). This information is not available from a single model. By selectively negating these parameters while preserving others, we can more precisely remove unwanted information without degrading performance on the retain set. As shown in Table 1, the merged model consistently outperforms all single models. We will clarify this point in the final paper.\n\n### **Weakness 2: Mitigating Hyperparameter Sensitivity via Merging**\n\nDue to the sensitivity issue, obtaining a model with strong unlearning performance inevitably requires a hyperparameter search process. However, rather than selecting a single best model from this search, we mitigate the sensitivity by merging all fine-tuned models obtained from diverse hyperparameter configurations. This strategy is inspired by the model soups framework (Wortsman et al., 2022), which shows that averaging the weights of independently fine-tuned models can improve performance and robustness. The key idea is that different hyperparameter choices introduce independent variations, and merging helps cancel out noise from any single run. As a result, the final model is less sensitive to specific hyperparameter choices, while still capturing the shared structure needed for effective unlearning. We will highlight this point in the final paper.\n\n### **Weakness 3: Clarifying Metrics in Table 2 (Acc Dr, Acc Df, MIA, etc.)**\n\nEach metric (Acc Dr, Acc Df, Acc Dtest, MIA) is favorable when its value is equal to that of the Retrain model (“≃”), indicating that the ideal Unlearn model needs to behave similarly to the Retrain model. This notation will be clarified in the caption of Table 2.\n\n### **Weakness 4: The “Conflict” Ablation in Table 3**\n\nThe weaker performance of the \"Conflict\" method suggests that using conflicting-sign parameters is not effective. As shown in Section 4.3, it degrades unlearning performance. In contrast, the \"Consensus\" method, which uses only sign-consistent elements, performs best, indicating that consistent signs better capture changes tied to the forget set, while conflicting signs likely reflect noise from different training setups.", "claims_and_evidence": "The paper claims that the effectiveness of current methods is highly sensitive to hyperparameter selection, which is presented in Figure 1.", "methods_and_evaluation_criteria": "Yes, it follows the previous literature.", "theoretical_claims": "In Section E, Theoretical Analysis, the paper provides a partial theoretical analysis.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The primary contribution of this paper lies in the application of model merging techniques to the field of machine unlearning.", "essential_references_not_discussed": "The related work described in this paper primarily encompasses model merging and machine unlearning.", "other_strengths_and_weaknesses": "1. One of the motivations of this paper is that \"hyperparameter sets that preserve retain set performance tend to exhibit poor unlearning performance, and vice versa.\" The reasoning behind why this motivation leads to the use of model merging to address issues in machine unlearning is not clearly explained. In other words, it is not well-articulated why model merging techniques are particularly suitable for the direction of machine unlearning.\n\n2. The issue that \"unlearning performance is highly sensitive to the hyperparameter settings used for obtaining fine-tuned models\" is presumably widespread in the field of machine unlearning. How does this paper mitigate this phenomenon, or which aspect of model merging technology alleviates this issue?\n\n3. In Table 2, several metrics including Acc D_r, Acc D_f, and MIA are presented. It is not clearly stated whether higher values are better or lower values are preferable for these indicators.\n\n4. In Table 3, the \"Conflict\" ablation version is inferior to the \"All\" version. Does this suggest that the method of selecting parameters using conflicting signs is not very effective?", "other_comments_or_suggestions": "none.", "questions_for_authors": "Please see the \"Other Strengths And Weaknesses\" part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an unlearning method based on merging task arithmetic vectors. The idea is to merge task vectors obtained from various hyperparameters, instead of using a hyperparameter search process, and by that to reduce the computational load of unlearning. The merging is based on the signs of the task vector components, where only components with sign agreement are used for the merged task vector. 
The paper shows that the proposed unlearning method performs well for both vision-language models such as CLIP ViT and for image classification. 

Following comments by Reviewers zcBE and aohJ, the authors added in their rebuttal results for 50% random forgetting, class-wise unlearning, TinyImageNet, some results for ViT-L/14, and Swin transformers. This makes the experimental evaluation of the proposed method to be sufficiently extensive. 

It is worth to mention here the following two weaknesses: 
* The authors' theoretical claim is not rigorous or extensive enough, as pointed out in the review. However, while it is always good to have some theory, it is not strictly necessary in a paper that proposes a new unlearning method based on a sound idea and showcases its performance empirically.
* Reviewer zcBE wrote that having a forget accuracy closer to the retain accuracy than to the test accuracy is a weakness of the proposed unlearning method, even if the unlearning method does not use the forget set samples. Table 2 shows that there are many other unlearning methods that have this weakness, although there might be exceptions like the method in the paper mentioned by Reviewer zcBE. While this is indeed a weakness to mention, it is not necessarily a reason to not accepting this paper.

In summary, this paper has novelty and strengths that outweigh its weaknesses and my recommendation is to consider it has a candidate for acceptance (weak accept).
Decision: Accept (poster)