Low Compute Unlearning via Sparse Representations
Authors: Vedant Shah
Frederik Träuble
Ashish Malik
Hugo Larochelle
Michael Curtis Mozer
Sanjeev Arora
Yoshua Bengio
Anirudh Goyal
Venue: Accepted by TMLR ()

Abstract: Machine unlearning, which involves erasing knowledge about a \emph{forget set} from a trained model, can prove to be 
costly and infeasible using existing techniques. We propose a low-compute unlearning technique based on a discrete representational bottleneck. We show that the proposed technique efficiently unlearns the forget set and incurs negligible damage to the model's performance on the rest of the dataset. We evaluate the proposed technique on the problem of class unlearning using four datasets: CIFAR-10, CIFAR-100, LACUNA-100 and ImageNet-1k. We compare the proposed technique to SCRUB, a state-of-the-art approach which uses knowledge distillation for unlearning. Across all four datasets, the  proposed technique performs as well as, if not better than SCRUB while incurring almost no computational cost.

Published in Transactions on Machine Learning Research (09/2025)
Low Compute Unlearning via Sparse Representations
Vedant Shah
wedamtshach20112@mmall.co com
Mila, Université de Montréal
Frederik Träuble
MPI, Tübingen
Ashish Malik
University of Oregon
Hugo Larochelle
Mila, Université de Montréal
Michael Mozer
University of Colorado, Boulder
Sanjeev Arora
Princeton University
Yoshua Bengio
Mila, Université de Montréal
Anirudh Goyal
aniruddi@1119@@maal.com com
Mila
Reviewed on OpenReview: https: openreview. net/ formm? id= GyKXemkk333
Abstract
Machine unlearning, which involves crasing knowledge about a forget set from a trained model, can prove lo be costly and infeasible using existing techniques. We propose a low- compute unlearning technique based on a discrete representational bottleneck. We show that the proposed technique efficiently unlearns the forget set and incurs negligible damage to the model's performance the rest of the data set. We evaluate the proposed technique on the problem of class unlearning using four datasets: CIFAR-10, CIFAR-100, LACUNA- 100 and ImageNet-lk. We compare the proposed technique to SCRUB, a state-of-the-art approach which uses knowledge distillation for unlearning. Across all four datasets, the proposed technique performs as well as, if not better than SCRUB while incurring almost no computational cost.
1 Introduction
Machine Unlearning (Cao & Yang, 2015; Nguyen al., 2022; Zhang al., 2023; Xu al., 2023; Kurmanji et al., 2023; Warnecke et al., 2021) may be defined the problem of removing the influence of subset of the data on which model has been trained. Unlearning can be essential component in addressing several problems encountered in deploying decp-learning approaches in practical scenarios. Neural networks such as Large Language Models (LLMs), trained on massive amounts of commonly available data, can exhibit harmful behaviors in the form of generating misinformation, demonstrating harmful biases, or other undesirable ble charactoristics. A major culprit behind these behaviors is the presence biased or corrupted instances in the training data of these models. To ensure safe model deployment, necessary to remove these instances. Another reason to remove instances and make model behave if it had not been trained certain data

---

Published in Transactions on Machine Learning Research (09/2025)
is concerns about data privacy and the right of end users to expinge their data (Mantelero, 2013; Dang, 2021). For example, an individual might want their data removed from face recognition system that was trained on their faces such that it is no longer able to identify them. Several regulations are being put in place order to safeguard the "right to be forgotten' (Pardau, 2018; Magdziarczyk, 2019). All the above problems can be addressed by unlearning the specific subset. of the training data which gives rise to the harmful behavior of the model the former cases and an individual's private data in the latter cases. Apart from these concerns, unlearning can also serve other purposes such removing outdated data from model to free up network capacity for more recent relevant data. With increasing concerns about Al safety and the increasing ubiquity of deep learning models in real-world applications, the problem of unlearning is of critical importance.
Decade
Decade
Decoder
Decoder
dg
dj
Output Logits
Retrieve codes
Retrieve cocles
Fixed vvlue mappings
Fixed key-value meppings
Processing via Key VVllue Bottlenack =C=1)
Processing via Key VVllue Bottlenack (C
Key-value pairs selected during inference
Model with a DKVB
2
4
Key codes
Key codes
Inference exampies from forget set
Encode
Encoder
Examples from set
Encode
Encoder
Figure 1: A summary of the proposed unlearning approach. Left: The encoder is frozen and pre-trained and R1 is random projection matrix. The values corresponding to the selected keys are retrieved to be used by the decoder. The gradient is backpropagated through the decoder into the values during training. The figure depicts the case with codebook in the DKVB. However, in practice we use multiple codebooks. Center: Examples from the forget set are passed through the trained model and the key-value pairs selected during the forward pass are recorded. Right: The recorded key-value pairs are then masked from the bottleneck. As result, the key selection is redirected to other keys, with non-informative corresponding values leading to other predictions.
The structure of a key-value bottleneck.
The main challenge in unlearning maintaining the performance of the model on the data that needs to be retained, called the retain set, while unlearning the forget set. The naive way ensure that model has no information about the forget set is train from scratch on the retain set. Unlearning techniques aim to achieve the same goal but at much lower computational cost compared to full retraining. Unlearning
2

---

Published in Transactions on Machine Learning Research (09/2025)
in pretrained network is difficult, especially in densely connected neural networks, since the value of one parameter may affect the output for all the input examples given to the neural network. A possible solution is to fine tune the model we wish to unlearn only on the retain set. While this would ensure that the performance of the model on the retain set is maintained, it can be computationally infeasible in practice. Other more effective solutions include retraining the model on the training data with a negative gradient for the forget set (Golatkar et al., 2020a; Kurmanji et al., 2023), or using knowledge-distillation---e objectives capture information about the retain set while filtering out information about the forget set (Kurmanji et al., 2023; Chundawat et al., 2023). Nevertheless, all of these approaches require some form of substantial additional compute in order to facilitate unlearning. Moreover, some of the existing approaches additionally require access to the original training data to facilitate unlearning, which may not be possible in many practical applications, ions, e.g., for a model in production which is being trained online on an incoming data stream. The use of large models is becoming more popular and prevalent with the advent of general purpose transformer modcls. The roquirement for additional compute can quickly become impractical in the context of these large models, especially in cases where a model is deployed and needs to be redeployed quickly as possible after making the necessary changes.
training
In this articlo, we argue that specific kinds of discrete neural information bottlemecks are highly suited for very efficient and specific unlearning. Neural information bottlenecks have emerged as useful components in neural network architectures, providing numerous benefits such as improving out-of-distribution (OOD) generalization capabilities and robustness to noisy data (Goyal et al., 2021; Jaegle et al., 2021; Liu et al., 2021; 2023), facili- tating large scale unsupervised pre-training and generative modeling (Esser et al., 2021; Oord et al., 2017), and more recently, helping in continual learning (Träuble et al., 2023). In particular, we build upon Discrete Key- Value Bottleneck (DKVB) proposed in Träuble al. (2023). DKVB induccs sparse reprosentations in the form of vey-value pairs which are trained in a localized and dependt- manner. Since these representa- tions are sparse, we hypothesize that it is possible to remove the information about a subset of the training data without damaging the information about the rest of the data the primary desiderata for a useful unlcarning method. Moreover, since the representations are discrete, this may be achieved without requiring any addi- tional compute in the form of retraining or fine tuning, by directly intervening on individual representations.
We investigate the above-mentioned idea of low compute unlcarning in the Discrete Key-Value Bottleneck. Specifically, focus on the problem of class unlearning in multi-class classification tasks, where the aim is to remove information about a specific class, called the forget class, from trained model. We use the term retain classes to refer to the classes other than the forget class that are present in the training data. More specifically, we wish to remove the influence of the forget class on the model. We measure this influence using the performance of the model on held-out test datasets corresponding to the forget class and the relain classes.
We propose two approaches for compute efficient unlearning in DKVB Unlearning via Examples and Unlearning via Activations. We show that the proposed methods achieve unlearning of the forget class while incurring negligible damage to the model's performance on the retain classes. We compare the proposed methods to SCRUB (Kurmanji et al., 2023), a recent state-of-the-art approach that requires additional compute to unlearn, on four datasets: CIFAR-10, CIFAR-100, LACUNA-100 and ImageNet-1k, The novelty of our work lies in the largely under-explored idea of using a model architecture with inherent sparse representations.
2 Related Work
The problem of unlearning has been studied in different forms for over two decades. Early works such as Tsai et al. (2014), Cauwenberghs & Poggio (2000) and Duan et al. (2007) study the problem of decremental learning in linear models, where a small number of samples need to be removed from a model. Ginart et al. (2019) considers unlearning as a problem of deleting individual data points from a model. They give a probabilistic definition, formalize the notion of efficient data deletion, and propose two deletion efficient learning algorithms. Guo al. (2019) introduces certified removal- a theoretical guarantec of indistinguishabiity between a model from which data was removed and a model that never saw the data. Izzo et al. (2021) distinguishes between exact unlearning and approximate unlearning and proposes compute-efficient approximate data
3

---

Published in Transactions on Machine Learning Research (09/2025)
deletion method, and a new metric for evaluating data deletion from these models. Golatkar et al. (2020a)
and Kurmanji et. al. (2023) cast unlearning into an information theoretic framework. Golatkar et al. (2020b) proposes Neural Tangent Kernel (NTK) (Jacot et al., 2018) theory-based approximation of the weights of the unlearned network. Multiple works also delve into the more philosophical, ethical, and logal aspects of unlearning and the "right to be lorgotten' (Kwak et al., 2017; Villaronga et al., 2018). Chundawat, el al. (2023) and Tarun et al. (2023) learn error minimization and error maximization-based noise matrices which are used to finetune the trained model in order to do unlearning. Chundawat et al. (2023) further uses a generator that generates pseudo data points for unlearning in order to operate in data-free regime. Recent works have also explored unlearning in the context of Large Language Models. Liu et al. (2024); Mekala et al. (2024)
Kurmanji et al. (2023) introduces SCRUB, a knowledge distillation based unlearning method. SCRUB considers the original model as a teacher model and trains a student model to obey the teacher model on the retain set and disobey on the forget set. This is done by computing the KL Divergence between the output distributions of the two models and training the student model to maximize it on the forget set (called a mar-step) and minimize it on the retain set (called min-step). The student model is simultaneously also optimized for minimizing the task loss on the retain set. The training consists of mstep mar-steps. The ster-steps and min-steps are executed alternatively. Chen et al. (2023), similarly to us, focuses on class unlearning rather than unlearning specific instances in the data. Unlearning is done by destroying the decision boundary of the forget class. The authors propose two boundary shift methods termed as Boundary Shrink and Boundary Expanding.
Unlearning and Model Sparsity. Jia et al. (2023) and Mehta et al. (2022) investigate unlearning in context of model sparsity. Jia et al. (2023) leverages the Lottery Ticket Hypothesis, Frankle & Carbin (2018) leverages using parameter pruning on a trained dense model to identify the token subnetwork. They observe that applying standard unlearning approaches to a sparsified networks is better as compared to doing unlcarning directly on the dense network. Mchta et al. (2022) identifies the Markovian Blanket of parameters corresponding to the examples to be unlearnt and updates those parameters. Fan el al. (2024) identifies the most solient weights with respect to the forget set and unlearns by finetuning only the salient weights with random labels. These approaches can be seen as applying sparse unlearning updates to the network instcad of full finetuning. We point out that sparsity is critical dimension that determines the effectiveness of unlearning: extremely sparse representations make unlearning trivial, whereas fully distributed represent ations intertwine knowledge in a way that makes compute-efficient unlearning serious challenge. Previous methods studying sparsity in the context of unlcarning such as Jia et al. (2023) and Mchta et al. (2022) propose the use of pruning techniques to first sparsify the network. These approaches start with dense trained models and leverage sparsity for unlearning. In contrast, we propose using sparsity as an in-built inductive bias in the model during the initial training which makes the model suitable for unlearning involving minimal compute requirements. On the other hand, Jia et al. (2023) sparsify the model after it has been trained. Mehta et al. (2022) involves sparse updates to the model parameters as discussed previously. However, these sparse updates are utilized during unlcarning as opposed to during training of the original model in the proposed approach. We identify a sweet spot on the continuum between local and distributed learning that allows for both, compute-efficint unlearning and simultaneously obtaining the same generalization performance.
Xu et al. (2023) introduced taxonomy that categorizes existing rescarch on unlcarning based on different approaches and aims. In this classification, our methods fall within the Model Pruming category by means of disabling specific (key, value) pairs within the bottleneck. Although one could argue that our methods lean towards weak unlearning stratogy-given the pre-trained backbonc might retain some information about the forget set-our approach deviate from the strict definition of weak unlearning as outlined by Xu et al. (2023). As an example, when considering a non-parametric decoder, our methods affect intermediate rather than final model activations. Zhao et al. (2024) empirically shows that unlearning difficulty grows with entanglement between the embeddings of the forget set and the retain set. This analysis reinforces our design choice. A DKVB ensures minimal entanglement between the representations of the forget and retain sets.
Similar to the proposed approaches, Foster et al. (2024) and Schoepf et al. (2024) propose approaches for finetuning-rres free unlearning. The proposed approaches involve performing one backward pass to compute Fisher-inffe information ratios, followed by algebraically scaling the handful of weights most specialised to the
4

---

Published in Transactions on Machine Learning Research (09/2025)
forget set. While most of the approaches discussed above improve upon the naive and intractable baseline of retraining the retain set, most of them (except (Foster et al., 2024) and Schoepf et al. (2024) require a substantial amount of additional computation in the form of optimizing an objective function for unlearning. This additional compute requirement can quickly become infeasible whenever large models are involved. The approach proposed this work, on the other hand, requires negligible computation for unlearning. Any computation that may be required is the form of running inference on the forget set. Further, most of the existing approaches for unlearning are optimized for models with continuous representations. We propose the first approach for unlearning in models with discrete representations. The proposed approaches show that discrete representations while providing several other advantages (such as better generalization), can also be leveraged for effective and compute efficient unlearning.
3 Background and Notations
Let. Dirain = (ari, yo J be training dataset and Drest be the corresponding test dataset. Tn our experiments, we consider the setting of class unlearning, wherein we aim to unlearn a class from a model trained with a multiclass classification objective on Dtrain. is called the forget class or the forget set. Given C, we obtain D train C Dirrin such that truin = {(z.)) E Derain = c}. The complement of Diorget is train i.c., subsct of Dirain that we wish to retain. Thus Dretoin U Diergett Darain- Similarly, from Drest; we have D.arget = ((x, y) E Drestly = =} and its complement Dretain, We refer to Dirai and Dredein as the retain set training and test data; and D Dforget and Doorger as the forget set training and test data, respectively.
Unlearning:
Discrete Key- Bottleneck: A discrete key-value bottleneck (DKVB) (Träuble et al., 2023) consists of discrete set of coupled key-value codes. The bottleneck contains C codebooks with each codebook containing M key-value pairs. Models with DKVB use pre-trained and frozen encoder to encode the input into continuous representation. This input representation is then projected into C' lower dimension heads and each head is quantized to the top k nearest keys in the corresponding codebook. The values corresponding to the selected keys are averaged, and used for the downstream task. The keys in the codebooks are frozen and initialized to cover the imput data manifold whereas the values are learnable. The mapping between the keys and values is non-parametric and frozen. Thus, the gradient is not propagated between the values and keys during training of the model. Since the values are retrieved and updated sparsely, and all the components except the value codes and the decoder sparse and localized representations (i.e., the value codes). These inductive biases allow the framework to exhibit improved generalization under distribution shifts during training, as shown empirically as well as theoretically in Träuble et al. (2023). Figure 1 (Left) shows an overview of a model with a DKVB where C= 1, M = 5 and top- 1. While typically limited to multi-class image classification settings, Diera et al. (2024) discusses the application of DKVB in the context of encoder-only language models.
frozen, DKVB stores information in the form of input-dependent,
4 Unlearning via Sparse Representations
Learning a Discrete Key Value Bottleneck. A Discrete Key Value Bottleneck (DKVB) model is first trained on the given dataset using the standard negative log-likelihood (cross-entropy loss) training objective for multi-class classification. We use a con-paramttric average pooling decoder and test the proposed approaches on two pretrained backbones: 1.) a CLIP (Radford et al., 2021) pre-trained ViT-B/32 (Dosovitskiy et al., 2020) and 2.) a ResNet-50 pretrained on ImageNet in a supervised fashion. Then we proceed to unlearn specific subset of data from these models. Before training with the classification objective, we do key initialization for the DKVB models the same dataset.
Key Initialization in DKVB models. After being forward propagated through the pre-trained encoder, the representations of the input are mapped to the top-k closest keys in the information bottleneck. The mapping between keys and values in the discrete key-value bottleneck is non- commparmettric and frozen. As a result, there no gradient propaatronn from the values to the keys, and hence the keys are not modified during training. Thus, it becomes essential for the keys to be initialized before learning the values and decoder, such that they broadly cover the feature space of the encoder. This initialization helps the
5

---

Published in Transactions on Machine Learning Research (09/2025)
model represent different concepts effectively. As in Träuble et al. (2023), (EMA) updates (Oord et al., 2017; Razavi et. al., 2019) to initialize the keys of the DKVB models. The key-initialization is done on the same train dataset Derain which we want to train the model on. The key initializations depend solely on the input encodings of the backbonc and hence do not require access to any labeled data.
exponential moving average
Inference for Unlearning. We propose achieve unlearning in DKVB models by excluding key-value pairs from the bottleneck such that they cannot be selected again. Numerically, this masking is done by setting the quantization distance of the selected keys to 'infinity.. Figure 1 (center and right column) shows an overview of the proposed methods. More specifically, we experiment with two methods, Unlearning via Activations and Unlearning vio Examples, described as follows.
Unlearning via Examples. In this method, we analyze the effect of unlearning a subset of Ne examples belonging to the forget set. N₂ examples are randomly sampled from the forget set training data (Dferget) and are input into the model having a DKVB. All key-value pairs that are selected during forward propagation tion across the N₂ examples are flagged. These key-value pairs are then masked out from the bottleneck. Technically, this approach requires access to the original training data corresponding to the forget class. However, it is also possible to carry out this procedure with proxy dataset that has been sampled from a distribution close enough to that of the forget set. For more discussion on this, refer to Appendix A.1.
Unlearning via Activations. In this second method, we analyze the effect on the quality of unlearning by deactivating different numbers of key-value pairs corresponding to the forget set. We refer to the key-value pairs that have been selected inputs to the decoder as activations. The entire forget set is forward- propagated through the DKVB model and all the key-value pairs selected across all examples of the forget class are recorded. Next, we mask the lop-N, most frequently selected key-value pairs from the bottleneck. The requirement of accessing the original training data for this method can be avoided by caching all the activations corresponding to the forget set during the last epoch of training. Further, similar to the previous case, unlearning via activations may also be performed given access to data that has been sampled from a distribut ion close enough to the distribution of the forget set.
Both approaches are two different ways of achieving common objective: to exclude subset of activations corresponding to the forget set. However, using one approach over the other may be more practical or even necessary, depending on the task at hand. In both the above approaches, we do not do any form of retraining or fine-tuning. The only computation which may be necessary is incurred during the inference stage for recording the key-value pairs which have been utilized for the forget set. Hence both approaches require negligible addit ional compute. Moreover, the requirement of access to original training data of the forget class can also be circumvented under appropriate assumptions, making the proposed approaches zero-shot unlearning methods.
5 Experiments and Results
The goal of our experiments is two-fold. First, we validate that proposed methods of the Unlearning via Activations and Unlearning via Examples in models with a DKVB (Section 5.2), and show that the proposed methods are competitive with the baselines (Section 5.2.1) in unlearning the forget class while incurring minimal damage to the performance of the models the retain class. Second, we compare the compute efficiency of the proposed methods against that of the baselines. More specifically, we report the number of floating-point operations (FLOPs) required during the procedure of unlcarning. Before presenting these results, we describe our experimental setup (Section 5.3). Before presenting these results, we describe our experimental setup.
5.1 Experimental Setup
Benchmark datasets We validate the proposed methods using experiments across four base datasets: CIFAR-10 with 10 distinct classes, CIFAR-100 (Krizhevsky et al., 2009) with 100 distinct classes, LACUNA- 100 (Golatkar al., 2020a) with 100 distinct classes and ImageNet-lk 1k (Russakovsky et al., 2015) with 1000 distinct classes. LACUNA-100 is derived from VGG-Faces (Cao et al., 2018) by sampling 100 different.
6

---

Published in Transactions on Machine Learning Research (09/2025)
celebrities and sampling 500 images per celebrity, out of which 400 are used as training data and the rest are used as test images.
Models On the aforementioned three datasets we study the following types of model architectures:
(a) Backbone + Discrete Vey-Value Bottleneck (Ours): Overall, this architecture consists of three components: 1) the frozen pre-trained backbone 2) the Discrete Key-Value Bottleneck (DKVB) and 3) decoder, as shown in Figure 1. For the DK we use 256 codebooks, with 4096 key-value pairs per codebook (approximately 1M pairs overall) in Träuble et al. (2023).
(b) Backbone + Linear Layer (Baseline): As a baseline, we replace the Discrete Key Value bottleneck and the decoder in the above model architecture with a linear layer. Thus, the two components of this model are 1) a frozen pre-trained backbone and 2) linear layer. This model will be used for all the bascline meth- ods. We replace the DKVB with a linear layer in the baseline since observe that existing unlearning approaches are designed for continuous representation architectures and thus perform better on them.
In each model, we use a pre-trained frozen CLIP (Radford et al., 2021) ViT-B/32 and ImageNet supervised pre-trained ResNet-50 as our encoder backbones. We refer the reader to the appendix for additional implementation details.
Training the Base Models We then train both model architectures the full training sets of each dataset. Since the backbone is frozen, for the baseline models, only the weights of the linear layer are tuned during initial training (and later unlearning). Since we use only one linear layer, we do not do any pre-training (beyond the backbone), unlike in previous works (Kurmanji et al., 2023; Golatkar et al., 2020a;b). Table 3 shows the performance of these trained models on the train and test splits of the complete datasets. Starting from these base models trained on the full datasots, we will validate the ability to unlearn previonsly Icarned knowledge.
Key-Inittillizaiions. To remain consistent with the literature DKVB (Träuble et al., 2023), we apply EMA-based koy-initializations to the DKVB models in all cases. The key-initializations lcad to a clean class-wise separation of the keys in the DKVB as discussed in (Träuble et al., 2023). Consequently, they enable high-quality unlearning by making sure that different classes have minimal number of shared keys. We on the quality of unlearning in Appendix A.4.
empirically explore the effect of key-initializationn
Unlearning We aim to make the problem of unlearning as challenging as possible. In order to bias the comparison against the proposed methods we select the forget class to be the class best learnt by the models with the DKVB in each case (see Appendix A.3). The intuition behind this experimental design choice is that the class best learnt by the model would be associated with the largest number of representations in the DKVB (leading to high values of N, and Ne for unlearning). thus increasing the likelihood of it sharing representations with other classes. Thus, unlearning these classes would lead to a higher drop in the retain set performance as compared to unlearning other classes. Nevertheless, we experimentally demonstrate that the proposed approaches perform competitively even when the forget set is randomly chosen in Appendix A.9.
Objective & Metrics We report our results the test data of retain classes and forget class, i.c. Dretain and D.orget, Typically in unlearning, the performance of the model on training examples from the forget set is studied. We study unlearning in a more broader sense and aim to study whether the model unlearns general "information" about the class, i.e., given forget set, is the model's performance on all similar data points (and not just those belonging to the forget class training data) drops; i.e. whether the unlearning generalizes. Thus, we report the performance of the models on the forget class evaluation set. Note that this practice would not be valid in case of instance specific unlcarning, wherein the aim is to unlearn specific examples of the training data from the model.
Although the proposed approaches facilitate unlearning up to arbitrary extents, in our experiments, we aim to achieve complete unlearning- achieving minimal accuracy on the forget set while incurring minimal damage to the performance on the retain set. We do this in order to stress test the proposed approaches. The greater the extent of unlearning, the larger the extent of damage incurred on the performance of the model on the retain classes
7

---

Published in Transactions on Machine Learning Research (09/2025)
Rotain Set Accuracy [96)
CIFAR-10
Retain Set Test Accurary [%)
CHEAR-100
Retain Set Test Accuracy 6%]
LACUNA--10
Redain Sel Teol Accuracy [%%
ImaggeNeet
04.0
70.0
90.00
(2:.
03:2
78.2
00.2
08.2
92.4
17.4
89.4
07.4
91.6
76.6
88.6
66.6
600
5080
2001
50.0
30.8
10000
13808
76.8
87.8
1804
63.8
10300
12300
20000
2400
2000
2000
30.0
75.0
87.0
66.0
100)
50
60
20
100
80
60
20
1001
50
100
20
100
90
40
Forget Set Test Accuracy (%6
Forget Set Test Accuracy [%6
Forget Set Test Accuracy [%]]
Forget Set Test Accuracy [%)
Hetain Sict Taast Accuracy 196]
84.0
CIFAR-10
Retain Sel Ted. Accuracy [9]
63.0
CIFAR-100
Retail Set Accuracy 19
68.5
LACUNA-100
Retain Set Toot Acorncy [99)
77.0
ImngsNet-ik
70.6
82.4
61.8
05.3
70.2
81.6
61.2
647
70.5
1200
13500
12000
1030
64.1
70.4
24000
8.00
2000
80.0
00.0
03.5
75.0
100
50
60
40
20
100
80
80
00
20
100
80
60
30
20
100
80
60
00
20
Forget Set Test Accuracy [96]
Forget Set Test Accuracy [%6]
Forget Set Test Accuracy [99]
Forget Set Test Accuracy [%)
Figure 2: Unlearning via Activations. Performance on the retain set test data vs. Performance on the forget set test data across various datasets for (a) CLIP pretrained ViT/B-31 in the top row (b) ImageNet pretrained ResNet-50 backbones in the bottom row as the value of N₂ is increased which is indicated by the color of the markers. The relative performance on the retain set test data as compared to the original models increases after unlearning in the case of CIFAR-10 and ImageNet-Ik Ik and drops for CIFAR-100 and LACUNA-100 in the case of ViT /B-32 and increases for all four datasets in the case of ResNet-50 (see Table 1).
While achieving complete unlearning may not always be desirable, such as in the case of Membership Inference Atlacks (MIAs) the proposed methods can be easily extended to defend against MIAs (We refer to Appendix A.11 for further discussion on MIAs and the proposed methods). We report mean values across 5 random sceds in all cases.
For comparing the compute efficiency of different approaches, we report the approximate FLOPs (Floating Point Operations) required for the procedure of unlearning. The total mumber of FLOPs are calculated as number of FLOPs required during the forward passes + number of FLOPs required during the backward passes. We use the fvcore" library for computing the nunber FLOPss required during the forward passes. FLOPs required using backward passes are approximated as number of operations used for gradient computations number of operations used for weight updates². Since only the linear head weights are trainable, the number of computations required for calculating the gradients would be the same the mumber of parameters in the linear layer. Further, since we use Adam optimizer, the number of operations required for the weight updates would be equal to 18 times the number of parameters.
To calculate the final number of FLOPs, we first calculate the FLOPs required for one example (one forward + backward pass) and then multiply them with the total number of examples and the total number of epochs. For SCRUB, the forward and backwards FLOPs are multiplied with different scalars depending on the msteps parameter.
5.2 Unlearning via the Discrete Key-Value Bottleneck
We will now discuss the results of unlearning via activations and examples, iee. the two approaches proposed in Section 4 on all four benchmark datasets.
Unlearning via Activations. Unlearning via activations requires us to set the hyperparameter N., reflecting the top-N, most frequently activated key-value pairs which will be masked out after inference on the forget
/
Phttps//rgpochaa.or//bbbog wwwwwwaowwwwwwdddddaao.......a forwardd FL.OP
8

---

Published in Transactions on Machine Learning Research (09/2023)
Set Accuracy [76)
CIFAR-10
Retain Set Test Accurary [%%
CHEAR-100
Retain Set Test Accurary 6%]
LACUNA-110
Redain Sel Teol Accuracy [%6
ImaggeNee
04.0
70:0
90.00
(2:.
03:2
78.2
00.2
92.4
17.4
89.4
07.4
91.6
76.6
88.6
66.6
180
30.8
76.8
220
63.8
303
24
420
30.0
75.0
87.0
66.0
100)
50
600
90
201
100
800
60
20
1001
90
180
40
20
100
90
40
Forget Set Test Accuracy (96
Forget Set Test Accuracy [%]
Forget Set Test Accuracy [%]]
Forget Set Test Accuracy [%6
Hetain Sict Tast Accuracy 196]
84.0
CIFAR-10
Retain Sel Ted Accuracy [9]
63.0
CIPAR-100
Retain Set Accuracy 19
68.5
LACUNA-IOI
Retain Set Toot Acouncy [99]
77.0
ImnggNet-ik
70.6
82.4
61.8
05.3
76.2
81.6
156
61.2
64.7
70.5
180
230
2014
64.1
70.4
2000
1000
80.0
00.0
63.5
75.0
100
50
60
40
20
100
80
80
00
20
100
80
60
30
20
100
80
60
00
20
Forget Set Test Accuracy [96]
Forget Set Test Accuracy [%6]
Forget Set Test Accuracy [9]]
Forget Set Test Accuracy [%6
Figure 3: Unlearning via Examples. Performance on the retain set test data vs. Performance on the forget set test data across different datasets for (a) CLIP pretrained ViT/B-32 in the top row and (b) ImageNet. pretrained ResNet-50 backbones in the bottom row as the value of No is increased which is indicated by the color of the markers. The relative performance on the retain set test data as compared to the base model increases for CIFAR-10 and drops for all other datasets in the case of ViT/B-32, whereas it drops for CIFAR-10 and CIFAR-100 and increases for LACUNA-100 and ImageNet-1k in the case of ResNet-50 (sce Table 1)
set. We therefore start by analyzing its role over a wide range of values to probe its choice and effect with N₂ =0 0 being the limit without any unlearning. Figure 2 summarizes the unlearning and effect of N₂ on the retain vs forget test set. In the case of CIFAR-10 and a ViT/B-32 backbone, the initial accuracies on the retain and forget test set are 92.61% and 96.50% respectively. As N₂ increases, the forget class test accuracy decreases, slowly for small No and rapidly for larger No. The model reaches random accuracy (i.e. 10% for CIFAR-10) on the forget class test data at N₂ = 150000. At this point the retain set test accuracy is 92.97%. The model unlearns the forget class completely between No 170, 000 (0.4%) and N. 180, 000 (0%). That is, the forget class test accuracy of the DKVB model drops to 0% between these values of Na. The exact value of No at which the accuracy drops to zero, represents the minimum number of activations that need to be discarded in order to destroy all information about the forget class from the modol. At this point, the retain set test accuracy is 92.94%, which is almost identical to the initial accuracy. Further increasing N, up to N₂ = 200, 000, i.e. about 20% of all key-value pairs, leads to an additional increase in retain set test accuracy to 93%. On the contrary, in the case of CIFAR-10 and a ResNet-50 backbone, the decrease in the forget set test accuracy is rapid for small Na and it slow for higher values of No. Complete unlearning in this case happens between N. = 160000 (0.1%) and N, = 190000 (0%). These differences in trends can be attributed to how the information is factorized among the representations. For eg. there is a steep decline in the forget set test accuracy between N₂ = 50000 (41%) and N. = 55000 (13%) in the case of LACUNA-100 and ResNet-50 backbone. This behavior may be attributed to the presence of high information but less frequently selected key-value pairs between the two values of No. Nevertheless, as can be seen from the equivalent analysis on the CIFAR-100, LACUNA-100 and ImageNet-Ik models in Figure 2 and Figure 3, the same trend of maintaining the initial retain accuracy while minimizing the forget accuracy up to a minimum holds across all four datasots and both backbones validating its meaningful unlearning capability.
Unlearning via Examples. For the second method unlearning via examples N₂ examples are sampled randomly from the training data of the forget class, and subsequently used for unlearning by the mechanism described in Scction 4. Similar before, we aim to assess the effect on the choice of N, over a wide range for each dataset, including N₂ = 0 being the limit without any unlearning. Figure 3 summarizes the unlearning and effect of N₂ on the retain forget test set. We again begin by focusing on the results with CIFAR-10
9

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Comparison between the proposed methods and the baseline across CIFAR-1O, CIFAR-100, LACUNA- 100 and ImageNet- 1k datasets and CLIP pretrained ViT /B-3 and ImageNet pretrained ResNett-0 backbones. We compare the relative change in performance on the retain and forget set test data relative to the originally trained models. The proposed methods are able to unlearn the forget sets completely in all cases while causing minimal changes in the performance of the models on the retain set test data.
CIFAR-10
CIFAR-100
LACUNA-100
ImageNet-1k
Backboae
Meshod
prctuin
furget
Drcluin
p.orget
Dretain
/forget
pretain
/orget
via
(eee
-100%
-0.17%
-100%
VIT/R-32
DKVB Fxamplec
-100%
-0.09%
-100%
Lineat Layer SCRUB
1.6276
-100%
-0.51%
-100%
-1.10%
-100%
7.81%
-1007
Lincar
Finetusing
1.94%
-100%
-1.91%
-96.33%
-2.21%
-100%
0.88%
-100%
Linear
Retraining
1.82%
-100%
+0.39%
-100%
-2.03%
-1007
5.16%
-100%
Lioerr
NegGrad+
0.49%
-100%
+0.63%
-100%
+1.34%
-100%
2.45%
-100%
DKVB via Activasions (coc 5.2)
0.04%
-1 -100%
0.26%
-100%
0.91%
-100%
0.04%
100%
Re<Net-50
DKVB via Examples (sec 5.2)
-0.07%
-1 -100%
-0.31%
-100%
0.17%
-100%
0.01%
100%
lanear Layer SCRUB
-0.07%
-98 63%
-0.91%
-88.79%
-0.26%
-99.67%
0.74%
100%
Linear Layer Finatuning
0.48%
-100%
-0.48%
-99.59%
-2.96%
-100%
-2 25.25%
1100%
Lisear Layer Retraining
8.06%
-100%
1.70%
-100%
1.15%
-100%
-1.14%
-100%
Lioear Layer NegGrad+
2.13%
-100%
-0.85%
-100%
6.73%
-100%
+0.90%
-100%
and ViT/B-32 backbone. Here, the forget set Diorget contains 5000 examples. We start off with retain set and forget set test accuracies of 92.61% and 96.50% respectively. Similar to the previous approach unlearning via activations the test accuracy the forget set decreases with increasing Ne. The accuracy on the retain test set, on the other hand, increases monotonically, although only slightly overall. The model achieves random accuracy on the forget class around Ne = 2500. The accuracy on retain set test data is at just under 93% at this transition. Finally, the accuracy on the forget set drops to 0% (i.c. complete unlearning) between N₂ = 3000 and N₂ = 3400 with a retain set test accuracy of just above 93% at N₂ = 3400. Further increasing N₂ does not affect the retain set test accuracy notably. Similarly to the case of unlearning via activations, the forget set test performance decreases rapidly at first and then slowly with N₂ in the case of CIFAR-10 with ResNet-50 backbone. The retain set test accuracy increases at first and then decrcases, albeit marginally. An equivalent analysis on the CIFAR-100, Lacuna-100 and ImageNet- 1k models in Figure 3 exhibit similar behavior of successful minimization of the forget accuracy up to a minimum while roughly maintaining the retain set test accuracy, validating unlcarning via examples as another option for unlearning using discrete key-value bottlenecks.
Summary. Both methods, Unlearning via Activations and Unlearning via Examples, successfully demonstrate unlearning of the forget class while having a negligible effect on the models' performance on the retain set, Importantly, this is achieved without any form of training, retraining, or fine-tuning is usually required by other methods. The retain set test accuracy remains more or less constant for all four datasets except. for a few minor fluctuations. This is a result of the fact that due to localized and context-dependent sparse updates during the initial training of the model, discrete tegerepprsnntttionn corresponding to different classes in the dataset are well separated from each other, an important prerequisite discussed in Träuble et al. (2023). Henco, all the information about a class can be unlcarned by forgetting only a subset of the forget class training data in the case of Unlearning via Examples, making it very data-efficient. While the aforementioned experiments are conducted in the context of unlearning a single class, Appendix A.8 further discusses the performance of the proposed approaches in multi-class unlearning scenarios.
5.2.1 Comparison with Baselines
We now compare the results of both the proposed methods, which require Backbone + DKVB models against several baseline methods, which are optimized for models without such bottleneck. For this, will use the Backbone + Linear Layer models described in 5.1. On these models, we run SCRUB (Kurmanji et al., 2023), finetuning finetuning the model to be unlearnt on the retain sct, retraining- - training the model from scratch on the retain set only and NegGrad+ (Kurmanji et al., 2023) and compare the performance changes on the forget and retain classes against the performance changes after unlearning with the two proposed methods. Table shows the comparison between the two previously reported methods and the basclines. We can see that one of the two proposed approaches always results in the least change in the performance of the base model on the retain classes, while at the same time achieving complete unlearning of the forget class. The baselines on the
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Comparison of FLOPs for various methods across CIFAR-10, CIFAR-100, LACUNA-100, and ImageNet- 1k datasets using ViT/B-32 and ResNet-50 backbones. We report both forward and backward FLOPss for each method.
CIFAR-1O
CIPAR-100
LACUNA-100
InngeNet-Jk
Barkhhone
Method
Horward (TFLOPs)
Hackward (GPLOPs)
Forward (TFLOPs)
Backward (GPLOPs)
Verward (TP1OPP)
Hackward (GFLOPs)
Forward (TFLOPs)
Hackwward (GFLOPs)
DKVB via Activations (see 6.
21.95
2:10
1.75
5.65
DKVB vis Examples (asc 5.2)
14.91
0.75
3.40
2.90
VIT/B-52
Linear Layer Finetuning
Linsar Layer + SCRUB
855. 13
186 54
14.03
4. 1.38
6455.8 87
1518.83
1644.00 61
293, 50
5188 68
527. 60
1155.00
117.01
39188. 26
11178 75
87230 98
24868
Lincur Layez Retraliiia
196-54
1080.50 1080.08
240.77
504.
192.61
5589.87
NegGrod
24898.22
nKyn via Actrustions (ene 5.2)
16 16-65
1:07
1-33
5:31
ReeNes-50
Linear Layer Winettiiing
Layer SCRUB
19:99
1498.
61.
1618 66
1488.79
8653 07
2007
121.
2159.02
77.05
31578.25
5304.25
209227.11 .17
09796 43
Linear Layer Betsaining
655 47
385. 23
1618.60
883 07
2242:18
1.309 7M
5386.25
ca756 411
Lineur Layez NogGrod
$20. 78
192.01
10505.49
89082. 35
263. 79
164.00
5504.25
40796. 48
other hand, occasionally fail to achieve complete unlearning. Finally, it is important to re-emphasize that the proposed methods achieve the shown performance without requiring any additional gradient-based training for unlearning. In the case of baselines, we stop the unlearning procedure when the forget set is completely unlearned or the forget set test accuracy has converged with minimal damage to the performance on the retain set. Morcover, while we report results for the case of complete unlearning, the proposed methods can be easily used for achieving unlearuing of the forget class to different extents by tuning the Na and Ne hyperparameters. We refer to Appendix A.13.3 and A.13.2 for further training and implementttin details.
5.3 Proposed Methods achieve Unlearning in Compute Efficient Manner
In this section, we compare the proposed approaches against the baselines in terms of the amount of compute required in order to achieve complete unlearning. To facilitate this comparison, we report the number of FLOPs required for the unlcarning procodure for cach case. The FLOPs are calculated following the rules described in Section 5.1. Table 2 compares the FLOPs required for unlearning in each case.
We report the forward and backward FLOPs separately to highlight that the proposed approaches do not require any gradient based updates. Additionally, while the scale of backward FLOPs may seem insignificant against the forward FLOPs, it can easily blow up in cases where complex parametric decoders are used on top of the DKVB. In our experiments, the decoder is simply an average pooling layer. Nevertheless, we can see that the proposed approaches require significantly less forward FLOPs as compared to the baselines. This can be explained by the fact that the proposed approaches require only one forward pass through the models per example of the forget class training data, in order to cache the activations. The baseline methods on the other hand, require multiple forward passes, each corresponding a single training epoch.
We also provide a runtime comparison of the proposed approaches with SCRUB in Appendix A.5, showing that the proposed approaches are at least 20x more compute efficient than SCRUB, further demonstrating the efficacy of the proposed approaches.
6 Limitations and Future Work
The proposed methods inherit the limitations of the DKVB (Träuble et al., 2023) such as the reliance of DKVB on pre-trained encoders which can extract meaningful shared representations and trade-offs in downstream performance due to the use of an information bottleneck. Extensions to the model may involve training sparse representations inducing discrete bottleneck end-to-end. Further, in our experiments, we consider the setting of multi-class classification in a supervised learning setting where the forget set can be easily identified and isolated. However, this may not always be sufficient for a given task and more complicated approaches might be needed to identify the data that needs to be removed from the model. Scaling the proposed framework and evaluating its effectiveness in more complex scenarios such as generative modeling remains to be explored. While the methods introduced in this work are currently not designed for selective unlearning as outlined in Appendix 1.1,, there are various directions for future adaptations. These directions include enhancing the forget set isolation, and addressing limitations related to information retention, for instance by further fine-tuning on the encoder backbone.
Further, while we use the performance of the model on the test set of the forget class as a proxy for unlcarning quality, it is important note that it is not the only indicator of the unlearning quality. The forget class test accuracy could be deteriorated simply by attemuating the weights of the final linear layer corresponding to the
11

---

Published in Transactions on Machine Learning Research (09/2025)
forget class, in the baseline models. However, it would be trivial to relearn the classifier head compared to relearning information unlearned from a feature extractor. For complex tasks, the DKVB would typically be a part of the feature extractor. This distinction is not captured by the forget class test accuracy and is not reflected in our experimental setup due to the simplicity of the tasks considered. We leave extending the Tramework to more complex lasks for future work.
7 Conclusion
In this work, we proposed a new approach to unlearning that requires minimal computation in order to unlearn a subset of data. This approach is based on the use of a discrete architectural bottleneck which induces sparse representations. These sparse representations facilitate unlearning a subset of data from the model with minimal to no performance drop on the rest of the data. We focused on the setting of class unlearning and our experiments show that the proposed approach, while being at least 20x compute efficient, performs competitively with or in some cases better than state-of-the-art approach which requires additional compute to perform unlearning. Consequently, excising the activated key-value pairs from the model is a highly evfective means of unlearning the forget set without disrupting the retain set.
Acknowledgements
This research was enabled by compute resources provided by Mila quile. VS would like to thank Aniket Didolkar and Moksh Jain for useful discussions, feedback on the initial versions of the paper and proof reading the paper. VS would also like to thank Nanda H Krishna and Moksh Jain for helping with the figures included in the paper and Thomas Jiralerspong for proof reading and giving feedback on the paper.
Broader Impact Statement
This paper presents work whose goal is to advance the field of Machine Unlearning, Machine Unlearning has emerged as a very important problem with the recent advances in Artificial Intelligenee With the advent of large, compute intensive neural networks, out work can have significant societal impact by addressing the applications of machine unlearning such as privacy preservation and mitigating harmful among many others discussed in the paper.
References
Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition, 2018.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463 480, 2015. doi: N.1100//SP:005..33
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in newral information processing systems, 13, 2000.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. 2023 IEEE/CVF Conference on Computer Vision and Pallern Recogrition (CVPR), pp. 7766 7775, 2023. URL https: 257636742.
hatpaattticccollaao//oopoppppppe Crg/CorpusII:
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 2023.
Quang- Vinh Dang. Right to be forgotten in the age of machine learning. In Tatiana Antipova (ed.), Advances in Digital Science, pp- 403-411, Cham, 2021. Springer International Publishing. ISBN 978-3-030-71782-7
Andor Diera, Lukas Galke, Fabian Karl, and Ansgar Scherp. Continual learning for encoder-only language models via a discrete key-value bottleneck. ar preprint 24 12..2/222..... 18.8828, 2024.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Hua Duan, Huan Li, Guoping He, and Qingtian Zeng. Decremental learning algorithms for nonlinear langrangian and least squares support vector machines. 2007. URL https://api https////pp.eemmiicccccolaaaaa org/
C319US1D:1398624. 13986244.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12873 12873-12883, 2021.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference Learning Representations, 2024. URL https://openreeiexx forum?id=gnOmIhog
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. Tn Proceedings of the AAAT conference on artificial intelligence, volume 38, pp. 12043 12051, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Xiviv preprint orXic:1803.03633 2018.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 9304 1941-9312 2020a.
Aditya Golatkar, Alessandro Achille, and Stelano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision ECCC 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part XXIX 16, pp. 383-398 Springer, 2020b.
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. Xiv preprinl arXiv:2103.0119 2021.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint Xrxi 1ix 191999100000 00100000,, 2019.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2008 2016. PMLR, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018
Andrew Jacgle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zissorman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651 4664. PMLR, 2021.
Jinghan Jia Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934 2023.
Alex Krishevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafilouu. Towards unbounded machine unlearning. Xix preprint Xxx:223000000880 2023.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Chanhee Kwak, Junyeong Lee, Kyuhong Park, and Heeseok Lee. Let machines unlearn machine unlearning and the right to be forgotten. In Americas Conference on Information Systems, 2017. URL https: aaapi..smanticcccoooo
Dianbo Liu, Alex M Lamb, Kenji Kawaguchi, Anirudh Goyal ALIAS PARTH GOYAL, Chen Sun, Michael C Mozer, and Yoshua Bengio. Discrete-valued neural communication. Advances in Neural Information Processing Systems, 34:2109-2122 2021.
Dianbo Liu, Vedant Shah, Oussama Boussif, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Curtis Mozer, Nicolas Heess, and Yoshua Bengio. Stateful active facilitator: Coordination and environmental heterogeneity in cooperative multi-agent reinforcement learning. In The Eleventh International Conference Learning Representations, 2023. URL http::///operreeeeeeeeeeeeeeeeee ntt./
Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang, Revisiting who's harry potter: Towards targeted unlearning from a causal intervention perspective. arXiv preprint arXiv:2407.16997 2024.
Malgorzata Magdziarezyk. Right to be forgotten in light of regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec. 04 2019. doi: 10.5593 10.5593/sgemsociin Aggemsocial20199/1L1. /802.022.
Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229 235, 2013. ISSN 0267-3649, doi: https://doi..rgg 10. 10.6 20130.0.00000....000000 URL attps://wwa http://ww..eeeecdeeoo.ooooo////s pi1/80288734491300654.
Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVI Conference on Compuler Vision and Pullern Recognition, pp. -04311, 2022.
Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, and Elita Lobo. Alternate preference optimization for unlearning factual knowledge in large language models. Xiv preprint arXiv:2409.13/4 13474, 2024.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee- Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXie preprint arXiv:2209.0228 2022.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. arXiv preprint xrXit:1711.00937, 2017.
Stuart Pardau. The california consumer privacy act: Towards a european-style privacy regime in the united states. J. Tech. Pol'y, 23:68, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin,, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Stefan Schoopf, Jack Foster, and Alexandra Brintrup. Paramoter-tuning-fee data entry error unlearning with adaptive selective synaptic dampening. arXiv preprint srXiw:2402.1009, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEFE symposium on security and privacy (SP), pp. 3-18. IEFE, 2017.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. In International Conference on Machine Learning, pp. 34431 34455. PMLR, 2023.
Cheng-Hao Tsai, Chich- Cen Lin, and Chih-Jen Lin. Incremental and decremental training for lincar classification. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, pp. 343 352, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450329569, doi: 10.1145 10.1145/262338 222222222333222 2623661. URL https: h/tp::///ddiie httpp:::///ii..///...../ 1145/2623330 2623661.
Eduard Fosch Villaronga, Peter Kieseberg, and Tiffany Li. Humans forget, machines remember: Artificial intelligence and the right to be forgotten. Computer Law & Security Review, 34(2):304 313, 2018. ISSN 0267- 3649. doi:
httt://///...............rrrrrr atticlle/ps//002876667700000
URL http::///www.eceemmerrrrt.t..oo
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint orXiv:2108.11577 2021.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip Yu. Machine unlearning: A survey. ACM Computing Surveys, 56(1):1-30 36, 2023.
Haibo Zhang, Toru Nakamura, Takamasa Isohara, and Kouichi Sakurai. A review on machine unlearning. SN Computer Science, 4(4):337, Apr 2023. ISSN 2661-8907. doi: 1011007/42279941---07777744 URL attps://doi.org/ 10t0//...0000/4277-000-001111110
Kairan Zhao, Meghdad Kurmanji, (eorge-Octavine Bärbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 01112293-12333 2024.
A Appendix
A.1 Unlearning in the absence of original training data
The main condition for unlearning the forget class in the bottleneck that the keys which are closest to the encoder representations of the forget set examples in the forget class are being removed. One way to do this described above, is by recording which keys get selected for the forget class examples and subsequently removing them from the bottleneck. However, in the absence of the forget class training data, the same could also be done by passing examples not directly belonging to the forget set but drawn from a distribution that is close enough to the forget set. This will result in approximately the same set of keys being selected as would have been if the examples belonged to the forget set.
A.2 Initial Performances of the Models
We train both: the models with DKVB and the baseline models (i.e. backbone + linear layer) to achieve similar performances on the test datasets in order to ensure a fair comparison. Note that, due to these the models are not necessarily trained to achieve the maximum possible performance on the datasets. Table 3 shows the initial performances of the originally trained models on different splits of the datasets.
A.3 Deciding the Forget Class
We assume that this class should be the most difficult one for the model to forget. Figures 4(a) - 4(c) show the number of mis-classfiicatios per class on the test data, for CIFAR-10, CIFAR-100 and LACUNA-100 for the ViT-B/32 backbone. For CIFAR-10, class #1 is the best-learned class with the lowest number of mis-classifications. Thus, select class #1 the forget class for the dataset. For CIFAR-100 class 58 is the
15

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Performance of the models on different sets of data after the initial training on the four datasets. We use two kinds of models: (a) models having a Discrete KV Bottleneck which are used for the proposed methods and (b) models where the DKVB and the decoder are replaced by a Linear Layer These are used for the baseline. We wish to reduce the accuracy these models on Deet to 0% while maintaining the accuracy on Dretain
(a) Backbone + DKVB
V/T-B/32
ReaNet-50
Dateset
Perain
D Irain
2 forget
Evoin
Deeat
Ceat
Text
Devoin
trach
D
tracs
Deeat
tcaf
D Sext
CIFAR-10
10038
100%
1007
05:011%
661
80 50%
100%
100%
100%
92 04.96
82 0415
0.000%
09-9856
24%
99.917%
61.
LACUNA-100
94.00%
ImaggeNet-13
5.%%
80 08.535
100%
64.94%
22%
82
90.37%
99.30%
100%
76 1135
76.41%
100%
(b) Backbone + Linear Layer
Peratn
2h forogge
Pieal
2r forget
Peroin
Dretein
forget
Prcus
Dretsin
forget
CHRAI--10
94 93.27%
u2 8255
UT.32%
93.02%
55%
US 90%
63.80%
55.425%
87 27.20%
12. N2.045%
81.74%
91. 70%7%%
CHEAHL-20
86.75 86.745%
86 86.6115
80 000.00%
78.815%
78.35 78.3555
96.0033 00.000%
78.02%
77.82%
96. 26.20%%
82 69%
62.41%
Sto D11%
LACUNA-ICO
95 6816
86 5.98
1007%
6815
50%
100%
80.45%
60%
09.20 20.20%
66 40%
65. 10%5
95, 0..00%
ImageNet-1X
best-learned class and for LACUNA-100, class 48 is one of the best-learned classes with mis-classifications. Hence, select classes #58 and #48 the forget classes for CIFAR-100 and LACUNA 100 respectively. We determine the forget. class in all other cases using the same method. We use the same forget classes for experiments on the models with linear layer in place of the DKVB (i.e., the baseline) as well. Table 4 shows the forget class for all the cases discussed in our experiments.
Number Misclasihaatios
CIFAR-10
Number of Misclusslication
CIFAR-100
Number of Miscnsiifictions
LACUNA-100
100
40
30
20
80
20
Class
Class
Class
(a)
(b)
Figure 4: Number of mis-classifications per class for the test data. The rod bars correspond the class with
the least number of mis-classifications (a) CIFAR-10: Class 1 has the least number of missllassification (b) CIFAR-100: Class 58 has the least number of mis-classifications (c) LACUNA-100: Classes 34, 48, 65, 76, 82 and 85 have 0 mis-classifications and hence, do not have a bar
Table 4: Forget classes for the different sccnarios presented in the paper
Forget Classes
CIFAR-10
LACUNA-100 CIFAR-100
ImageNet-lk
ViT-B/32
58
48
ResNet-50
1
94
34
16

---

Published in Transactions on Machine Learning Research (09/2025)
A.4 Importance of Velue Initializations in the DKVB
The effectiveness of the proposed approaches depends on the quality of key-inilia (i.e. how well the keys corresponding to different classes are separated), which also affects the overall performance of the DKVB model. EMA is used for key-initializations and thus plays a erucial role in the approaches. The quality of unlearning would deteriorate in the absence of seey-iitialliations. We demonstrate this with an experiment where we compare the retain set accuracy of ViT-B/32 backbone based DKVB models trained with and without key-initializaaiion when they undergo unlearning via activations for CIFAR-10, CLFAR-100 and LACUNA-100 and report the results in Table 5. We can that the retain set accuracy incurs significant damage upon complete unlearning when the models are trained without key-initialixations.
Table 5: Comparison of the effectiveness of Unlearning via Activations with and without EMA based key- initialization for ViT-B/32 backbone based models
DKVB via Activations
CIFAR-10
CIFAR-100
LACUNA-100
Dretain
forget
Diest
Dust
Dretain
D Forget test
w/ Key-Init w Key-Init
0.36%
-10%%
-0.20%
-100%
-0.17%
-100%
2.96%
-100%
-13.66%
-100%
-9.46%
-100%
In order to stay consistent with the literature on DKVB, we use EMA, exactly as used in (Träuble et al., 2023), using the same parameter values (i.e. gammaa = 0.95). We discuss the implementation in detail in the Appendix A.12.1.
A.5 Comparison of Runtimes
In this section, we present comparison of runtimes of the proposed approaches against the baseline, SCRUB (Kurmanji et al., 2023) as a proxy for comparing the compute requirements of the two approaches. Table presents the results of this comparison across all cases. We observe that in all the cases, the runtime of the baseline multiple orders of magnitude greater than the runtime of the proposed approaches, with proposed approaches being at-least. 20x faster than the baseline." Targe runtimes for SCRUB are result. of the gradient-based parameter optimization required for the approach.
A.6 Comparisons fully trained baselines
We demonstrate perform equally well in comparison to baselines that are fully trained by training the baselines to completion on CIFAR-10 and CIFAR-100 for ResNet-50 backbones, and comparing unlearning on them via SCRUB with unlearning in the proposed approaches.
We first report the performances of the fully trained baselines and corresponding DKVB models in Table 7. For the baseline models, we run an extensive hyperparameter sweep and select models according to the best test performance across all the combinations.
Next, we run SCRUB (Kurmanji et al., 2023) on the fully trained baseline models and compare the performance with the proposed approaches in Table 8.
Clearly, unlearning on fully trained basclines is more difficult than on models trained to the same extent as the DKVB models.
A.7 Unlearning in SCRUB
Figure 5 plots the retain class test accuracy forget class test accuracy for running SCRUB on (CLIP pretrained and then finetuned on CIFAR-100) 100) VTT-B B /32 backbone in the case of CIFAR-100 (similar to Figures 2 and 3). The forget set accuracy drops to 0% after the first opochs. We run the unlearning procedure for 10 epochs, each epoch consisting of either one or two optimization steps, depending on the msteps
Note that the runtimes ImageNet- Ik are low since use pre-computed hackbone embeddings run these experiments.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Table Comparison of runtimes (in seconds) between the proposed methods and the baseline on CIFAR-10, CIFAR-100, LACUNA-100 and ImageNet-11: The proposed methods are at-least 20x faster than the baseline across all cases. Between the two proposed methods, Unlearning vio Activations is faster.
Backbone
Method
CIFAR-10
CIFAR-100
LACUNA-100
ImageNet-1k
DKVB via Activations
5.02
1.57
2.74
0.84
ViT/B-32
DKVB via Examples
13.98
6.65
13.03
2.36
Linear Layer SCRUB
288.80
921.56
553.31
181.04
DKVB via Activations
4.88
0.84
1.50
1.02
ResNet-50
DKVB via Examples
12.26 1703.73
2.39
2.20
4.30
Linear Layer SCRUB
1902.89
4192.96
295.45
Table Performance the models on different sets of data after the initial training on the four datasets.
(a) Backbone +
DKVB
ResNet-50
Dataset
D Dtrain
Dretain train
D'orget train
Dtest
Dretain test
Dforget test
CIFAR-10 CIFAR-100
100% 99.98%
100% 99.98%
100% 100%
82.94%
82.04% 61.81%
91.00% 92.00%
62.11%
(b) Backbone + Linear Layer (Fully Trained)
ResNet-50
Dataset
D Dtrain
Dretain train
forget train
Dtest
Dretain test
forget
test
CIFAR-10 CIFAR-100
86.68% 81.61%
85.96% 81.45%
93.08% 97.2%
84.28% 63.3%
83.50% 63. 63.11%
91.3%
82.00%
parameter. As explained in Section 5.2.1, SCRUB until the damage the retain set test accuracy is minimal.
Retain Accuracy [96]
CIFAR 100
79.0
78.2
77.4
76.6
75.0
100
80
60
40
20
0
Forget
Set Test Accuracy [%)
Figure 5: Retain Class Test Accuracy vs Forget Class Test Accuracy. The markers are color coded to represent the number of epochs.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Comparison of the proposed approaches with unlearning via SCRUB on fully trained baseline models.
F-T stands for fully trained baselines.
10
CIFAR-100
Backbone
Method
Dretain Lest
Dforget
Dretain
D.orret
leat
teat
DKVB via Activations (sec 5.2)
0.04%
-100%
0.26%
-100%
ResNet-50
DKVB via Examples (sec 5.
-0.07%
-100%
-0.34%
-100%
Linear Layer SCRUB
-0.07%
-99.67%
-0.94%
798.9%
Linear Layer + SCRUB (F-T)
-1.15%
-99.99%
-2.35%
-98.37%
A.8 Multi Class Unlearning
We attempt to investigate the effects of unlearning multiple classes at once by performing experiments on CIFAR-100 for both ViT/B-32 and ResNet-50 models. We unlearn upto 10 classes using both Unlearning via Activations as well as Unlearning via Examples as well as one of the baselines SCRUB (Kurmanji et al., 2023) and run each experiment for 5 seeds. The classes to be forgotten are chosen randomly for each seed. Figure 6 plts relative change in performance of the unlcarnt model on the retain class (with respect to the original model) vs the number of classes unlearnt al approxinately the point of complete unlearning.
RosNot50
CIRA--10
VIT CIFAR-100
Roaativ Chonge Ronan Toe ant Arrancy (50
Unkominn its Artronnee
Uskuning its Artromice
Unkorung Emazrçdon
Unkunang Exurcclos
SCRUB
Laeaz
Namvier Cheon Regeem
Nemther of Clewe Regeinm
Figure 6: Multi Class Unlearning for CIFAR-100
We can that Unlearning via Activations performs relatively better compared to Unlearning via Examples. Further, SCRUB outperforms both the proposed methods significantly in the case of ViT backbone, keeping the percentage change in the retain class test. accuracy less than 1% in all cases. However, in the case of a ResNet-50 backbone, SCRUB surprisingly performs the worst for low number of unlearnt classes and competitively for higher number of unlearnt classos. The proposed methods perform comparatively better on the ResNet-50 backbone as compared to the ViT backbone. We can also clearly see that the relative error compared to the original model the retain set performance higher than single class unlearning, expected. Noticeably, the performance starts to significantly deteriorate when forgetting 6 classes in the case of Unlearning via Examples, and > 8 examples in the case of Unlearning via Activations for both the backbones.
A.9 Choosing the forget class randomly
To ensure that the effectiveness of the approach is not class specific, we perform experiments on CIFAR-100, where the class to be forgotten is randomly chosen, and compare the performance of the proposed approaches against SCRUB [2]. We run each experiment for random scods, wherein the forget class is randomly chosen for each seed. Rest of the experimental setup remains the same as described in Section 5 of the paper. We report the results in Table 9
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table Selecting the forget class randomly for CIFAR-100
CIFAR-100
Backbone
Method
Dretain test
Dforget test
DKVB via Activations (sec 5.2)
-0.27 + 0.07 %
-100% + 0 %
ViT/B-32
DKVB via Examples (sec 5.2)
-0.47 + 0.03 %
-100 + %
Linear Layer + SCRUB
-1.56 + 0.35%
-100 + 0%
DKVB via Activations (sec 5.2)
0.58 + 0.12 %
100 + %
ResNet-50
DKVB via Examples (sec 5.2)
0.28 + 0.11 %
-100 - %
Linear Layer + SCRUB
-1.43 + 0.18%
-99.94 + 0%
Clearly, even with the forget class chosen randomly, the proposed approaches perform competitively to SCRUB. This demonstrated that the effectiveness the proposed approaches is not class dependent.
A.10 Unlearning beyond the compute free setting
We investigate the effect of using additional compute to the proposed methods. As shown previously, the proposed methods perform competitively to SCRUB. To the best of our knowledge, SCRUB is the most competitive and relevant unlearning approach. However, it has the inherent drawback of requiring compute for unlearning. Nevertheless, for a fair comparison, we additionally explore the implications of this additional compute for the proposed two methods for a ViT/B-32 backbone on CIFAR-10, CIFAR-100 and LACUNA-100. Specifically, we retrain the DKVB models after the (compute efficient) unlearning, on the training data of the retain set (i.e., Dirtaim) for 10 epochs. For the baseline, in Section 5.2.1, except we run it for 10 epochs instead of stopping when either the forget set has been completely unlearned or the performance has converged. Figure 7 and figure 8 highlight the effect of retraining of the proposed methods compared to SCRUB across multiple epochs, for all three datasets.
use the same experimental setting
Retraining the unlearned models on the retain set does not affect their performance significantly. The performance of the baseline on the other hand increases after an initial drop in case of CIFAR-100 and LACUNA-100. The initial drop may be attributed to the damage the retain set performance caused by the initial maz-steps. The subsequent increase can be attributed to the fact that the SCRUB training objective also optimizes the task loss on the retain set. Thus, once the model unlearns the forget set, SCRUB shifts the model capacity towards better learning the retain set. For CIFAR-10 this results in the model performing better than the DKVB models on the retain set as the retain set test accuracy after unlcarning is higher than the original model. However, the baseline is unable to recover original performance for CIFAR-100 and LACUNA-10N
For the forget set, in all three cases, the bascline completely unlearns the forget set quickly within the first few epochs, shown in Figure 8.
A.11 Using the proposed Methods against Membership Inference Attacks
Depending on the application, complete unlearning of the forget set may not always be the final goal of unlearning. For several use cases such removing information about corrupted data from the model removing harmful biases exhibited by the model, maximal error on the forget set is desirable. However, for applications such as Differential Privacy, it is more desirable lo achieve forget set error that is similar to that of a model trained from scratch only on the retain set. Otherwise, makes the unlearned model susceptible Membership Inference Attacks (MIA) (Shokri et al., 2017). Although we do not explore this sotting detail in this work, the proposed method can also be used for applications where complete unlearning is not desirable. This can be done by following procedure similar to SCRUB+R (Kurmanji et al., 2023), wherein
20

---

Published in Transactions on Machine Learning Research (09/2025)
CIFAR 10
CIFAR 100
LACUNA 100
95.0
78.5
Rotain Set, Test Accuracy [56
Retain Set Test Accuracy [96]
Hetain Sec. Test Accuracy [56]
94.5
78.1
94.0
77.7
77.3
Unlearaing va Activation
93.0
76.9
88.5
Unisarning via Ecomples
Linear Layer
92.5
76.5
88.0
2
8
10
2 4
10
2
6
10
Epochs
Epochs
Epochs
(a) CIEAR-10
(b) CIFAR-100
(c) LACUNA-100
Figure 7: Comparison between the performance of proposed methods with added compute and the baseline the retain set test data. For the proposed methods, the plots start from after the initial zero shot unlcarning. For the baseline, the plots start from the original models. Retraining the models unlearned using the proposed models does not lead to any significant improvements in performance.
CIFAR 10
CIFAR-100
LACUNA 100
Forges Set. Tbat Accuracy [%6
100
Forget Set Test Accuracy [%)
Forget Set Test Accuracy [96]
100
Unlearning va Adtiaaiiss
Unlearning via Exapless
80
80
Linear Layer SCRUB
60
60
40
20
20
20
10
Epochs
Epochs
Epochs
(a) CIFAR-10
(b) CIFAR-100
(c) LACUNA-100
Figure 8: Comparison between the performance of proposed methods with added compute and the baseline on the and forget set test data. Note that for the proposed methods, the plots start from after the initial zero shot unlcarning. For the baseline, the plots start from the original models. The green line occludes the red line since both of them stay at 0% throughout the training.
instead of selecting particular model checkpoint, one select the model corresponding to particular values of Na N₂ such that the error the forget set test data is similar to the reference point as defined in Kurmanji et al. (2023).
First,it is important to clarify that the proposed approaches are not a-priori suited for selective unlearning, i.e., the setting where we want the model to forget specific examples or a small subset. of examples instead of removing the information about an entire class. The KV bottleneck essentially induces clusters of representation, where the members of a particular cluster correspond the representations belonging to the same class (see Figure 2 in Träuble et al. (2023)). When we try unlearn the representations corresponding to one particular example belonging to particular class, the KV bottleneck routes the selection to other (eee-)rrprsssnlatiions within the same cluster since those keys would be the next closest to the representation of the encoder. Since these representation also contain information about the same class as the examples we intend to unlearn, the model would still predict the class to be unlearned.
Due the same reason the proposed approaches are also not designed for working against traditional Membership Inference attacks. According to the basic attacks setup as explained in Kurmanji et al. (2023),
21

---

Published in Transactions on Machine Learning Research (09/2025)
the objective is to obtain model that has unlearned small subset of specific examples (i.e. selective unlearning) such that the loss of the model on the unlearned subset of examples should be indistinguishable from loss on examples that the model never saw during training.
Nevertheless, since the proposed approaches are designed for class unlearning specifically, we attempt to evaluate them on a modified version of the above. We call this "Class Membership Inference Attacks (CMTA)". In CMIA, the goal is to defend against an attacker whose aim is to determine whether a model that has undergone unlearning ever a particular class a part of its training data. Thus, want the model to unlearn a particular class such that the losses posse/ppere performance of the model the unlearned class as whole, is indistinguishable from those on a held-out class that the model never saw during its training. We describe the experimental setup and results below.
Experimental Setup We perform the experiment for CIFAR-10 with a ViT backbone. We divide the dataset into training data (Drrain). validation data (Dvai) and test data (Drest). Training Data consists of 4000 examples per class; validation and test data consist of 1000 examples per class. We first trained a model the first 9 classes of CIFARIO. Thus, class number 10 is the held-out class. Next, we unlearn class 1 from the model using the Unlearning via Activations approach introduced in the paper. We unlearn the model until the loss of the model on the validation sets of the forget class and the held-out class are similar. In our experiments, we find that we reach this point approximately at N₂ = 240000. The loss l(,, y) in our case is the eross-entropy loss.
Next, we label the losses corresponding to the validation and test set of the forget class as 1 and those corresponding to the validation and test set of the held-out class as 0. We train binary classifier the validation losses of the forget and held-out sets and evaluate it the test losses. We follow similar sctting for the baseline model, where obtain model suitable for MIA defense by using SCRUB+R (Kurmanji et al., 2023). For a successful defense, we would want the accuracy of the classifier to be close to 50% on the test losses, indicating that it is unable to distinguish between the unlearned class and the held-out class. Same as Kurmanji et al. (2023), we use sklearn.
as our attacker (the binary classifier). We call the approach described above Partial UvA (Partial Unlearning via Activations). We run experiments for random seeds, and the mean of the attacker performance is reported. Note that similar procedure can also be followed using Unlcarning via Examples.
Observations and Results: We report the results of the experiment described above in the table given below. We observe that although the baseline performs slightly better, the proposed approaches perform competitively, even though we have not intended to develop the method for this scenario.
Table 10: Comparison on Class Membership Inference Attacks between the proposed approach and the baseline. A binary classifier is trained the validation losses of the forget and held-out sets and is evaluated the test losses. The proposed approach performs competitively to SCRUB + R
Approach
Attacker Accuracy
Partial UvA
53.50%
Linear Layer SCRUB + R
51.50%
A.12 Mathematical and Algorithmic Formulations
In this section, we provide mathematical formulations for the proposed approaches of Unlearning via Activations and Unlearning via Examples as well as the empirical moving averages used for initializing the keys of the Discrete Key-Value bottleneck.
A.12.1 Exponential Moving Averages for -ettiaatiliization
Similar to Träuble et al. (2023) we build upon exponential moving averages as introduced in Oord et al. (2017); Razavi et al. (2019). Below, we reiterate much of what described in Träuble et al. (2023) (Appendix C). The set of equations given below describes the key initialization procedure. For each codebook c:
22

---

Published in Transactions on Machine Learning Research (09/2025)
NO 7N7 + +m² (0) - 7)
(1)
(1
m
ymm
ym,
(1-1)
+ Σ
F Ee(ar)
(1 7)
(2)
(8)
m,
(3)
N (8)
where the index of the current mini-batch, k; and N, represent the position and counts of the ith key, (x)) (9) (.-...... are the ny (t) head embeddings of the examples in the mini-batch which attach to the i-th key.
We refer the reader to Appendix C of Träuble et al. (2023) for more details.
A.12.2 Unlearning via Activations and Examples
In this section, we provide algorithmic implementations of the the proposed approaches Unlearning via Activations (Algorithm 1) and Unlearning via Examples (Algorithm 2). Both algorithms are applied on model with DKVB that was trained on the given task.
Discrete Key-Value Bottleneck Notations. Before laying out the algorithms, define some notations related to different components of the Discrete Key- Value Bottleneck (DKVB) (Träuble et al., 2023) that are essential for the algorithms. We refer the reader to Träuble et al. (2023) for the role of cach of these components in the DKVB.
Pre-trained and frozen embedding model
2. Random projection matrix R
Set of keys initialized using EMA {ky {-)
N-11
4. Distance matrix IR tel |x initialized to -0:
Dii,Jl 00 Vie f0. 1.2, ...... Dforgett j - 1),
{0, 1,2 ....... 1}.
Selection mask M E TR. DD NxN initialized to 1:
Mi. ViE 1,,
traim)
11, ( ......---- -.1..
We also define a function argsort as follows:
Function Definition: argsort
Let u [01, U2 ..., en] IR" be one-dimensional array or vector. The function argsort(u) returns a permutation of the indices 11,2....,,)) n) such that:
(a(1) S Un(2) S... S Ux(n).
In other words, Ti) is the index of the i-th smallest clement of v.
For example, if [3, 1, 1),, then:
argsort(e) [2, 1,3]. since 22 = 1 is the smallest, en = 3 is the second smallest, and 03 4 is the largest.
23

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: Unlearning via Activations
Input:
Forget class training data D Drrrget C Dirain number of activations to be deactivated No. top-K k parameter used for the DKVB
Output: Modified selection mask M
Initialize: Frequency matrix f Z initialized to
fla] + Vj (0, ...., 1}.
Step Forward propagate the forget class training data through the model for [D Pongel
1 do
I +- D forget [1]
6z R E(x)
for j 0 to N - do
D[i,j] + lleg - kylle x M J end
Ie argsort( Cer, 11oopk for
j E I. do fli] - fbi +1
end
end
Step 2: Deactivate the most frequently activated keys J t argsort(f)x-N,ttt
for
jEJ do
M[ end
A.13 Training Details and Hyperparameters
We perform all of our experiments on a 48GB RTX8000 GPU. We do not use any data augmentation in any experiment. The transforms used for training the model with a ViT/B-33 backbone are the same as CLIP (Radford et al., 2021) pretrained ViT/B-32 transforms. For ResNet-50, both pre-trained weights and transforms are loaded from perehvissin..odeee.. Res.ees....eeee50 ShsNet55_Weiihtt.
A.13.1 Training Details and Hyperparameters for training the original DKVB Models
In the case ImageNet pretrained ResNet-50, the representations of the backbone are extract from the 3rd last layer for CIFAR-10, CIFAR-100 and LACUNA-100 and from the 4th last. layer for ImageNet- Table shows all the hyperparameters used for training the base DKVB models.
A.13.2 Training Details and Hyperparameters for training the original Baseline Models
For the bascline models, we deliberately train them to similar test (Drest) accuracies as the models with a Discrete Key Value Bottleneck ensure fair comparison for unlearning. Table shows the hyperparameters used for training the baseline models.
24

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 2: Unlearning via Examples
Input:
Forget class training data D Drrgget
C
Dirain number of examples be used for unlearning Ne.
top-k parameter used for the DKVB
Output: Modified selection mask M for the DKVB.
Initialize: Set of activated indices I
Step 1: Randomly sample a subset S from
Dforget of size N trein
S ~
forget.
|Syl = No
Step 2: Input the examples in the subset into the model to record the activated keys
for 0 |Sy] do
x + -S
6 = R. F()))
for j to N 1 do
D[i,j] + ||e= kylle x M [i end
I I argsort((nee 1top-x
end
Step 3: Deactivate the activated keys for i I do
M M 60
end
Table 11: Hyperparameters used for training the base DKVB models
Buckbone
Hyperparameteo
CIFAR-10
CIFAR-100
LACUNA-108
ImageNot-1k
top-k
10
10
Key Dimcnaioo # Key Init Speche
8
14
10
10
Type Value dt of Codabooks
Gauestan Random
Zeros
Random
Zeroz
VT//-882
Key-Value Paire
256
286
256
LR Batch
0.1
0.3
0.3
0.3
256
250
256
256
74
top-k
2
Key Dimension
14
14
14
Key Init Epecia
10
Type Value
Zeros
Random
Caussian Random
Gaussian Random
ResNet-50
Key-Value
of
Codebook
256
250
256
256
Adam
Adam
LR
0.3
0.3
0.1
0.3
Batch Size
256
256
250
256
Epothe
70
A.13.3 Training Details and Hyperparameters for SCRUB
For the baseline, run SCRUB the model with linear layer. One epoch consists of one min step and may or may not contain mar step. Hence the values of min steps and epochs are always samc. One maz step is included in every epoch for the first msleps epochs. We lune the hyperparameter msleps in our experiments and pick the case where the model is able to best recover its performance on the retain set test data and consider this model as the final unlearned model. We mention the hyperparamcters used for running SCRUB corresponding the results presented in Section 5.2.1 in Table 13. In this case, training of SCRUB is stopped when the forget set accuracy has either dropped to 0% converged at close to 0% value without damaging
25

---

Published in Transactions on Machine Learning Research (09/2025)
Table 12: Hyperparameters used for training the baseline models
Backbone
Hyperparameter
CIFAR-10
CTFAR-100
LACUNA-100
ImageNet-1k
LR 0.001
0.01
0.01
0.01
ViT/B-32
Batch Size Epochs
256
256
256
512
13
LR Batch Size Epochs
0.01
0.001
0.01
0.001
ResNet-50
256
256
512
512
2
72
73
11
the retain set accuracy. Results presented in Appendix A.10 also use the same set of hyperparameters except min-step which is always 10 since we train all the methods for 10 epochs.
Table 13: Hyperparameters for SCRUB Linear Layer Experiments shown in Section 5.2.1
Backbone
Hyperparameter
CIFAR-10
CIFAR-100
LACUNA-100
ImageNet-1k
Forget Set Batch Size
256
256
256
512
Retain Set Batch Size
256
256
256
512
# of max-steps (msteps) # of min-steps / # of epochs LR
3
5
3
VIT/B-32
10
7
3
0.001
0.01
0.01
0.001
Optimizer
Adam
Adam
Adam
Adam
Forget Set Batch Size
256
256
256
512
Retain Set Batch Size
256
256
256
512
# of max-steps (msteps) # of min-steps / # of epochs LR Optimizer
3
3
RosNet-50
10
30
30
10
0.01
0.001
0.01
0.001
Adam
Adam
Adam
Adam
A.13.4 Training Details and Hyperparameters for Retraining Experiments
Once the DKVB models are unlearned using Unlearning via Aclivations and Unlearning via Examples, we retraining them in order to make a fair comparison with the baseline. Thus, during retraining, the initial performance of these models on the retain set is same as the final performance of the unlearned models. Table 14 show the hyperparameters used for retraining the unlearned DKVB models.
Table 14: Hyperparameters used for re- re-training experiments. UvA stands for Unlearning via Activations and UvE stands for Unlearning via Ezamples
CIFAR-10
CIFAR-100
LACUNA-100
UvA
UvE
UvA
UvE
UvA
UvE
LR
0.3
0.3
0.1
0.1
0.1
0.3
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Batch Size
256
256
256
256
256
256
Gradient Clipping
0.1
0.1
0.1
0.1
0.1
0.1
26


Reviews:
[{"review_id": "JbAlb1ro9K", "reviewer": "Reviewer_2ZFu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-organized and easy to follow, with a clear motivation.\n* The topic is important, and the proposed method is practical, which would contribute to the deployment of the unlearning method in real-world tasks.\n* It provides good insights about the relationship between unlearning and sparsity, which could be beneficial to the related field. \n* Experiments on different data and architectures are provided and demonstrate the improvements.\n\nWeaknesses:\n* Most of the related works are from or before 2023. It would be better to also incorporate more recent studies in this field.\n* Apart from FLOPs, it would be better to include more results about the clock time comparison with baseline methods to better support the claim in the paper.\n* More ablation studies would be good for a better understanding of the proposed methods.\n* In more general scenarios, given an arbitrary model with a forget set, I was wondering if your method or some variant of your method can still work with low compute. This would make the proposed method useful in more scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how to effciently do machine unlearning. Existing works often require extensive computation to forget the forget set while keeping the performance on the retain set. This paper utilizes the characteristics of sparse representations to achieve low computation in machine unlearning with negligible damage to the model performance on retain set. Experimental results are provided to support its claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* It would be good to update the related work section with recent studies.\n* It would be good to incorporate some clock time comparison with baseline methods.\n* It would be good to conduct some ablation studies.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dkbA1iAwTl", "reviewer": "Reviewer_AKz1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear and well organized. Figure 1 is excellent!\n- The method is simple and easy to implement.\n- The method is super efficient (orders of magnitude faster than existing approaches). \n- Experiments supported the conclusions and were presented well.\n\nWeaknesses:\n- The method requires the original model use a DKVP layer, so it it's not something that can be immediately applied to just any neural network. If one had an existing model, one would need to train a DKVP bottleneck on top of that backbone. \n- The goal of unlearning was not well-defined, but I think this approach might not satisfy some of the desired unlearning criteria. The primary indicator of unlearning used in the experiments is keeping the retain set accuracy high while reducing the forget set accuracy to zero. But couldn't one do that with a trivial solution, e.g. by setting the logit of the forget class output neuron to negative infinity? The proposed approach is very similar to this trivial solution in that no weights are retrained. \n- The reasoning wasn't compelling for using the best class as the forget class. I would think using random classes would be more straightforward.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for unlearning classes from neural networks that have a DKVP layer. The approach is simple and very efficient. In experiments, the method is demonstrated on four benchmark datasets with two different computer vision backbones. The performance on retain set remains high throughout the unlearning process, and the performance is similar to other existing unlearning methods from the literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Key Initialization is unclear. Why do you need Dtrain for this? If you are encoding Dtrain, then presumably you also apply R1 (random projection matrix)?\n- Typo?: \"Ne examples are randomly sampled from the forget set training data (Dtrain^retrain)\"\n- This was unclear: \"Therefore, on each dataset we select the class that is best learned by the respective\nmodels with the Discrete Key Value Bottleneck trained previously, to be the forget class.\" \n- The significance of the model reaching 0% accuracy is not immediately clear, so it would be helpful to have a sentence of explanation after, \"The model unlearns the forget class completely between Na = 170, 000 (0.4%) and Na = 180, 000 (0%).\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jgDd3dOdA2", "reviewer": "Reviewer_wDTK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method is highly efficient, which leads to significant computational savings while maintaining competitive accuracy on retain classes. The method is conceptually simple, easy to implement, and validated across multiple datasets and backbones.\n\n\nWeakness:\n\nThere are several concerns about the experimental setup and assumptions. \n1. The evaluation splits the test set into forget and retain subsets. This is unusual, as the forget set is typically defined over the training set. While I understand this is for evaluation purposes, it only makes sense under class unlearning and assumes the training and test distributions are identical, which may not always hold in real-world scenarios. It would be better to acknowledge and discuss this limitation explicitly.\n\n2. In Table 3, the linear-head baseline is deliberately undertrained so that its test accuracy matches the DKVB model. While this is presented as a “fair” comparison, I’m skeptical of its fairness as one model fully learns the training data, while the other does not. Since the comparison focuses on accuracy differences (rather than absolute accuracy numbers), I'm not sure why aligning test accuracy at the start is strictly necessary. Also, in practice, both models would be fully trained, and if introducing DKVB appears to reduce generalization slightly, then this trade-off should be reflected in the results.\n\n3. The paper primarily compares unlearning performance relative to the original pre-unlearning model, rather than using the retrain-from-scratch oracle as the main reference. But in my opinion, the retrained model best represents the ideal post-unlearning behavior and should serve as the primary baseline. Additionally, the paper formalize the unlearning goal as “complete unlearning”, i.e. to achieve near-zero accuracy on the forget set, which may be valid for class unlearning in some cases but could be misleading in broader settings. For example, one can simply mask logits to remove a class without affecting remaining data. I think the goal should be matching the behavior of the retrained model, not simply destroying accuracy, particularly from a privacy and generalization standpoint.\n\n4. The paper leaves open questions around DKVB initialization: why EMA was chosen, how other strategies compare, and how sensitive performance is to this choice. Moreover, the method has several prerequisites: access to the original training data or pre-cached activations, interventions to the training pipeline to record forget-class activations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces two “compute-free” class-unlearning procedures: Unlearning via Activations and Unlearning via Examples, which exploit the sparsity of a DKVB inserted on top of a frozen pretrained encoder. This way, removing a class reduces to masking a few key–value pairs instead of running gradient updates. Across four image-classification benchmarks, the methods achieve successful forgetting of the target class while keeping the retain-class test accuracy almost unchanged and using much less compute than other baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I suggest including results using a fully trained linear-head baseline, in addition to the current early-stopped version.\n\n2. Please discuss your choice of DKVB initialization (e.g., EMA) and how alternative strategies might affect performance.\n\n3. I would appreciate a discussion addressing the setup concerns raised above.", "broader_impact_concerns": "In some cases, full class-level removal may be disproportionate and could harm model utility or fairness. I would suggest including a brief discussion on these implications in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)