Fast Exact Unlearning for In-Context Learning Data for LLMs
Authors: Andrei Ioan Muresanu
Anvith Thudi
Michael R. Zhang
Nicolas Papernot
Venue: ICML 2025 poster (2025)
Keywords: Exact Unlearning
Large Language Models
In-Context Learning
Primary area: social_aspects->privacy
TL;DR: We show that by using in-context unlearning we can have near constant time exact unlearning of fine-tuning data.

Abstract: Modern machine learning models are expensive to train, and there is a growing concern about the challenge of retroactively removing specific training data. Achieving exact unlearning in deep learning pipelines—producing models as if certain data had never been included in training—remains an open problem. In this paper, we revisit exact unlearning in deep learning and show that for large language models (LLMs) we can efficiently exactly unlearn ``fine-tuning data" (the data used to adapt a pre-trained model). This follows from two observations. First, we can use in-context learning to adapt the LLM to the fine-tuning dataset instead of SGD based algorithms. Second, we show that accurate in-context learning can be done with quantized k-means, which allows for effectively constant time unlearning operations. Our evaluation shows that this unlearning recipe has similar performance to fine-tuning alternatives, but vastly reduces the unlearning costs. Our study also highlights the need for new measures of unlearning cost when adapting the learning algorithm to have faster unlearn operations.

Fast Exact Unlearning for In-Context Learning Data for LLMs
Andrei Muresanu 12
Anvith Thudi 32
Michael R. Zhang 32 Nicolas Papernot 32
Abstract
model (distribution) the learning algorithm would produce if trained without set of datapoints.
Modern machine learning models are expensive to train, and there is a growing concern about the challenge of retroactively removing specific training data. Achieving exact unlearning in deep learning hipelines-produccing models if certain data had never been included in train- ing ing-remains an open problem. In this paper, we revisit exact unlearning in deep learning and show that for large language models (LLMs) we can efficiently exactly unlearn "fine-tuning data" (the data used to adapt pre-trained model). This fol- lows from two observations. First, can in-context learning to adapt the LLM to the fine- tuning dataset instead of SGD based algorithms. Second, we show that accurate in-context learning can be done with quantized k-means, which al- lows for effectively constant time unlearning oper- ations. Our evaluation shows that this unlearning recipe has similar performance fine-tuning al- ternatives, but vastly reduces the unlearning costs. Our study also highlights the need for new mea- sures of unlearning cost when adapting the learn- ing algorithm to have faster unlearn operations.
Efficient exact unlearning for deep learning remains an open problem. Advances in exact unlearning have come from identifying cheap operations to simulate what changing the dataset does to learning. This has been possible for classical machine learning algorithms (Cao & Yang. 2015) (Brophy Lowd, 2021), such clustering (Ginart et al., 2019). How- ever, simulating the change caused by removing datapoint from an SGD training run with a neural network (deep learn- ing) has proven difficult, with current approaches still requir- ing cost on the order of training the original model (Bour- toule et al., 2021).
Approximate unlearning has been proposed an alternative to exact learning and aims to lower unlearning costs by ap- proximating the distribution of models trained without the desired datapoints. However, there is currently no consen- sus on evaluation metries, making comparison and utility difficult to evaluate (Thudi et al., 2022a; Hayes et al., 2025)
In this paper, we show that exact unlearning can be efficient for "fine-tuning" pre-trained large language model (LLM), i..., learning given access to an LLM. The past literature exact unlearning had not considered learning algorithms that leverage pre-trained models. However, this has become a recent trend in private machine learning (Yu et al., 2021; De et al., 2022), where fine-tuming is easier to learn privately. We show that a similar trend also holds for exact unlearning.
1. Introduction
After a machine learning model is deployed, it may become necessary to deploy a new model that does not use part of the original training set. This can occur because of legislation on the "right to be forgotten" (Mantelero, 2013) or because of unknown data provenance (e.g., some data may have come from untrustworthy sources). Machine unlearning addresses this challenge of modifying model to behave as if it were trained without including certain datapoints. Specifically, exact unlearning aims to produce the same
In-context learning with LLMs offers an alternative tra- ditional model fine-tuning that proves amenable to exact unlearning. In-context learning works by selecting represen- tative examples from training dataset to guide the LLM's responses. While in-context learning achieves compara- ble performance to weight fine-tuning on many downstream tasks (Brown et al., 2020), it presents a distinct challenge for unlearning: we must reproduce the example selection pro- if certain datapoints never existed in the fine-tuning dataset. This is non-trivial because effective example se- lection depends on understanding the complex relationships between datapoints.
DDepartmen of Computer Science, University of Water- 100, Waterloo, Canada *Vector Institute, Toronto, Canada DDepartment of Computer Science, University of Toronto, Toronto, Canada. Correspondence to: Andrei Muresanu <an- drei.muresanu @ uwaterloo.ca>.
We observe that accurate in-context learning algorithms
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Throughout this paper we consider the pre-training and fine- tuning datasets to be independent.

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
reduce clustering over some feature space, which can admit efficient exact unlearning. Specifically, common in- context learning algorithm is to run k-means embeddings of training examples (Zhang et al., 2022). Instead, we pro- pose to run quantized k-means (Ginart et al., 2019), which admits unlearning operations independent of the dataset size. With this, we can unlearn the fine-tuning data independent of model and dataset size.
developing unlearning methods is when machine unlearning can be achieved cheaper than this baseline (i.e., when having access to T(D) makes undoing fast). Towards this goal, past work has shown faster machine unlearning algorithms for a variety of classical machine learning algorithms (Cao & Yang, 2015; Ginart et al., 2019; Brophy & Lowd, 2021). For deep learning, known methods for exact unlearning partition data before training such that retraining can be computed more efficiently (Bourtoule et al., 2021), though the cost is still on the order of training (and can come with performance degradation). Given the difficulty of exact unlearning, past work has considered various notions of approximate unlearning, and methods for these (Thudi et al., 2022a; Golatkar et al., 2020; Graves et al., 2021; Baumhauer et al., 2022; Guo et al., 2019; Thudi al.; Pawelczyk et al., 2023; Yao et al., 2023; Jang et al., 2022; LLMS; Chen & Yang, 2023). However a consensus for metrics to use for approximate unlearning has not yet been reached. For example, there was a Neurips competition on standardizing metrics (Triantafillou et al., 2024). Moreover, it is not clear if approximate notions of unlearning are suitable for some applications of unlearning, such as legal requirements. Our work avoids this ambiguity by focusing on exact machine unlearning, and showing access to a pre pre-trained model opens ways to learn with faster exact unlearning operations.
We introduce ERASE, an unlearning approach that com- bines in-context learning with quantized k-means clustering. Our empirical evaluations explored how it compared to exist- ing exact unlearning baselines. We conducted experiments across Big-Bench Instruction Induction (BBII) (Zhou et al., 2023) tasks, and compared performance of FRASE to vari- ants of SISA (Bourtoule al., 2021) (an optimized exact unlearning algorithm for SGD-based learning). We found that for most tasks we evaluated, ERASE matched or im- proved the accuracy of SISA, and in all cases had drastically cheaper unlearning operations.
Finally, in studying fast exact unlcarning, we make the ob- servation that existing "faster" exact unlearning algorithms for deep learning also increase inference cost. To the best of our knowledge, this was not discussed in past work, and we present an analysis of how many inference passes per unlearning request a method can handle after which it is less efficient than the baseline of retraining.
A separate body of literature related to unlearning (what is considered in this paper) focuses on knowledge unlearning where the goal is to remove an undesired "bchaviour" of an LLM as opposed to unlearning specific datapoints. We refer the reader Cooper et al. (2024) for detailed comparison of these two distinct bodies of literature.
To summarize, our contributions are:
1. Showing that for certain datasets, exact unlearning can be efficiently achieved by using in-context learning.
2. Proposing an exact unlearning algorithm, ERASE, for in-context learning, that has dataset and model-size independent unlearning operation costs.
2.2. In-Context Learning
Large language models are able to learn perform a task and make predictions for new inputs by conditioning on few input-label pairs (few-shot examples), with no fine- tuning. This is known as in-context learning and was ob- served to be an emergent behaviour in GPT--3 which is a language model with 175B parameters trained on Internet- scale data (Brown et al., 2020). Subsequent work on in- context learning has studied why in-context learning works (Min et al., 2022), including viewing the language model as implicitly performing Bayesian inference (Xie et al., 2021), The effectiveness of in-context learning may vary signif- icantly with the choice of in-context demonstrations (Lu et al., 2022). For instance, Zhang et al. (2022) highlighted the role of diversity in the examples chosen for performance and proposed methods such as a baseline of random sam- pling examples, or clustering the examples using k and including the example closest to the centroid in each cluster. This last approach is called Auto Chain-of- of-ohhhhht (ACoT). In our work, we show the in-context learning frame- work has faster exact unlearning operations.
3. Identifying the trade-off between inference cost and unlearning cost and proposing a new holistic cost met- ric. We use this metric to study when in-context learn- ing provides more efficient unlearning deployments as compared to fine-tuning alternatives such as SISA.
2. Background
2.1. Unlearning
Exact machine unlearning asks, given a training algorithm returning some model (parameters, cluster centroids, etc.)
T D + M and a datapoint in the training dataset D. to return the output of T(D \ æ) given an output from T(D) (Cao& & Yang, 2015); if I is random, then this means returning a sample according to its model distribution. The "trivial" solution for unlearning is to retrain on D \ x, i.c., run T(D) \ z) without using the model T(D). A goal in
2

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Pre-Trained Model
Fine-Tuned Model
Unlearned Model
E
Pre-Training
Task Adaptation
Unlearning
target distribution
larget distribution
1. Michael Johnson, trcated with chemetherapy for toukoma
chamothaappy spy laukemia
2. Solia Parker, treated with rediation frarapy breast
Befla Parker treated with radiation thorepy to-et---ttt--coeeer
Private Data
Private Data D (x))))))))
Figure 1. An illustration of our threat model for unlearning requests in the fine-tuning stage of training. Sensitive data can be introduced the fine-tuning stage of neural network and may need be unlearned. We assume this sensitive data does overlap with the pre-training datasct, meaning no change to the pro-trained model is needed to unlearn. In particular, while unlcarning pre- pre-training data efficiently is an open problem, we show unlearning this second stage can have performative and efficient exact unlcarning methods. Model trainers may consider moving data that may need to be unlearnt to the fine-tuning stage for efficient unlearning.
3. Taxonomy: Exact Unlearning Methods for Fine-Tuning LLMs
constructs an ensemble of models trained on disjoint equal sized partitions of the dataset D (effectively train- ing each model for 1/n" th the cost of the original model), called sharding. The second stage of SISA is slicing, which limits a data point to being used in a specific interval of training, e.g., the first 3 epochs. However, as seen in Fig- ure 4, our models converge after one epoch of SISA fine- tuning. Hence, in this paper, assume each model trains a data point only once and do not further use SISA slicing. To unlearn with SISA, only one of the shorter training runs needs to be redone (which had the datapoint to unlearn), and in expectation half of that training run needs to redone, giving an asymptotic unlearning rate of O n cost to train one model fully on D). This gives a 1/n factor improvement over retraining naively, but it is known that by increasing n one decreases the performance of the ensemble model returned by SISA (Bourtoule et al., 2021): there is a limit to how large n can be. Intuitively, each of the models sees less and less data as n grows, even- tually giving an ensemble of noisy predictions.
Chowdhury et al. (2025) is the only previous work we are aware of studying unlearning in the fine-tuning stage (i.e., task adaptation stage) of LLMs. However, their approach does not constitute exact unlearning; see Appendix for details. Despite this, we note that various retraining-baas based approaches exactly unlearn are already applicable. We now describe existing methods to exactly unlearn in the task adaptation stage, noting these can be classified along what the task adaptation learning algorithm (that they are unlearning for). In describing these baseline methods, we focus on contrasting unlearning and inference cost.
We follow the online model of Ginart et al. (2019) for unlearning and cost. We consider a stream of unlearning requests -1.-.. , Ty. For each request must produce the output of 7 (the training algorithm) if it was just given DV {æ1 ), DV (Z11,, ... D) {x_,,.. .r3} where D) was the training set. We measure unlearning cost as the aver- age cost over uniformly sampled i.i.d stream of unlearning requests from the training dataset. When unlearning one point, the unlearning cost is the expected cost to unlearn point when uniformly sampling from the dataset. We note that requests may not always be uniformly sampled (Gupta et al., 2021), and that manipulation to the order of datapoints can lead to increased compute costs (Shumailov et al., 2021), but do not consider these adversarial settings in this paper.
Measuring Cost
While the unlearning operation cost decreases with n, the inference cost for n-SISA grows with The inference cost is O [()) where / is the number of tokens in the inference question and f represents the cost of inference per token for one model. For a Transformer with a constant attention window (as in our experiments), the cost O(n * t²) and we set (t) = O( for the rest of the paper.
3.2. In-context Learning
3.1. Parameter Fine-tuning
As an alternative to parameter fine-tuning for task adap- tation, in-context learning algorithms output set of examples from the fine-tuning dataset D which are then prepended to the input given to an LLM. In-context learning
The standard exact unlearning approach for fine-tuning with SGD-based algorithms is Sharded, Isolated, Sliced, and Ag- gregated training (SISA) (Bourtoule et al., 2021). S-SISA
3

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
makes no modifications to the parameters, and the only de- pendence the dataset D is the set of examples to prepend. Hence, to unlearn in-context learning, we need to (re)sample examples from the distribution of examples the in-context learning algorithm would output if given just D \ {x*)- As no past work has proposed unlearning methods for in- context learning, we overview past in-context learning algo- rithms and the cost of naively retraining for unlearning.
Algorithm 1 In-context Learning with ERASE
Require: A set of training examples D, the desired number
of in-context examples k. and quantization parameter e
(t) Ensure: Examples q()) = [ .22 ..... -s learning
( for in-context
1: for each example q in D do
2: Encode with feature extractor e.g., Sentence-BERT
3: end for
A baseline in-context learning method is to randomly sam- ple in-context examples (Zhang et al., 2022) from the task adaptation dataset. Hence the distribution of samples for unlearning x" will just be the distribution (sampled with- out replacement) of k uniform samples from D ' x². We can do this cheaply by rejection sampling. We resample once from D \ {-i,--. } to replace x² if it was sam- pled as an in-context examples (x*" E {=I.... ,"kk"" },, and do nothing if x" was not sampled (as the conditional that rr & _1,'. Th is the same for both D and D (z"). Hence this gives a constant (1)) unlearning cost per example, giv- ing (m) for unlearning m examples.
4: Clustee all the encoded example representations into
It clusters using quantized k-means with quantization parameter c
5: for each cluster i 11,,..... ,....,, do
6: Sort examples g(¹) = [9] 42 ..... in the ascend-
ing order of the Eg distance to the quantized cluster centroid
7: end for
8: Return 9i (2) for = I....,k
run forward pass over is of them alongside the original test example of tokens. This is in contrast with n-SISA which only scaled as O(n x t²), and note is typically longer than the normal input (see example format in Section 5.1).
For certain tasks, a more effective in-context learning al- gorithm is Auto Chain-of-Thought (Zhang et al., 2022) (ACoT), which takes embeddings of the examples in D and produces lt clusters from them using k-means++. ACoT then picks the example from each cluster that is closest to the centroid. This is often combined with prompting the model for an explanation for its response, but we ignore this step for this paper (and note this does not need changing for unlearning). Hence, ACoT is effectively k-means++ on the in-context ex-contextexammpls, and unlearning would require produc- ing the cluster centroids k-means++ would return if run D {x"). We are unaware of efficient unlearning methods for generic k-means++, and hence consider naively retrain- ing as the unlearning method. Note running k-means++ has a cost ((DDdd) when fixing the number of iterations, where d the dimension of the embedded in-context exam- ples. Hence, assuming negligible change to the dataset size over m unlearning requests, we have the unlearning cost for ACOT over unlearning requests is (m)) D(d). We also highlight that simply resampling the removed in-context ex- ample from the k-means++ centroid clusters is not sufficient for exact unlearning because the selection of the removed example is not independent to the rest of the training set. Therefore, retraining (rerunning the clustering algorithm) or an unlearnable alternative to k-means++ is required.
To summarize these different costs, we present the different unlearning operation and inference costs in Table I (which includes the in-context learning algorithm we will propose), where we now see spectrum of increasingly more effi- cient unlearning algorithms: no model dependence for all in-context learning methods and furthermore no dataset de- pendence with random sampling and our algorithm. At the same time, we see inference costs get higher as we make unlearning operations more efficient.
4. Methodology
Given the current takeaways from the brief taxonomy pro- vided in Section 3, we now proceed to propose a new algo- rithm to fill gap in unlearning operation efficiency. Further- more, given the inverse trends observed between inference cost and unlearning cost, we propose a new holistic mea- sure of unlearning costs to capture how frequent unlearning requests must be for an algorithm to be more efficient: this being particularly useful to compare unlearning cost when changing the learning algorithm.
These in-context learning methods have unlearning opera- tion costs independent of model size, in contrast to SISA; however ACoT's unlcarning operation scales with dataset size. For all in-context learning methods, the prepending of k in-context examples increases the cost to run inference. The inference cost is ( + k 8), where is the average number of tokens in an in-context example, and one has to
4.1. ERASE: A New In-context Learning Method Designed for Unlearning
The discussion in Section 3.2 reveals a gap for an in-context learning algorithm that is as accurate as ACoT, but with unlearning operation costs independent to the dataset size. We propose such an algorithm in Algorithm I, which we call FRASE (Efficient Removal And Selection of Examples).
4

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Cost
n-SISA
ACoT
ERASE
Random
Unlearning Op
( m Train(D,model))
Dld) (((m2dP/3/))
((m) Oll x O(P + k xx)
Inference Op
n On x (2)
0(12 + 8x
Table 1. Asymptotic unlearning and inference costs of different exact unlearning methods at the task adaptation stage of an LLM. Costs are presented for unlearning m data points, where D is the dataset, d is the dimension of the embeddings used for the in-context examples, and is the quantization parameter in ERASE. Here is assumed to be (1) asymptotically D grows. Here le is the number of in-context examples, is the number of tokens in the input question, and s is the average number of tokens in an in-context example. We do not consider model size for inference as that does not affect relative comparisons: would introduce the same constant for all methods.
ERASE builds on past work that highlighted the role of selecting diverse examples for accurate in-context learn- ing (Zhang et al., 2022). ERASE takes the set of training amples D and computes the Sentence-Bert embeddings for each q E D. It then creates & clusters from the embeddings (the desired number of in-context examples) and orders the examples in ascending & distance to the centroid: i.e., for
rameter which one would need be tuned for the task. For the tasks we evaluated on, we found fixed value of 0.05 led to comparable accuracy to ACoT.
4.2. Holistic Unlearning Costs: Towards A More Nuanced View of Benefits to Unlearning Efficiency
each cluster i [k]] returns a list q()) = i4 .22 ..... ERASE then returns vi Vi E [k). the examples closest to
As previously noted in Table I, methods that decrease un- learning operation cost also increase inference cost. To the best of our knowledge, the literature has not previously dis- cussed this trade-off, and how it can make the baseline of finetuning with 1-SISA cheaper if there are a large number of inference queries per unlearning request. We propose to evaluate when exactly various unlearning algorithms are more efficient than the baseline of 1-SISA (i.e., fine-tuning and and doing naive retraining to unlearn) This compari- son is motivated by previous deployment comparisons of in-context learning and fine-tuning (Liu et al., 2022). This metric is stated in the following definition:
each centroid. This algorithm is similar to ACoT (Zhang et al., 2022). However, ERASE uses quantized k-means (Gi- nart al., 2019) to cluster instead of k-means++.
Quantized k-means allows for dataset size independent unlearning of clusters (which is the main cost for un- learning the selection of in-context examples). The algo- rithm computes cluster centroids similar to k-means, but adds an additional imbalance correction, and most impor- tantly, quantizes the centroids to a randomly sampled e- lattice. More formally. quantized k-means quantizes the intermediate centroids G - & * Zd + & for some 0 E Uni 1/2[4) and quantization parameter By making larger we make the final cluster centroids more sta- ble to changes (would require a big shift in the un-quantized centroid to make a shift after quantization). It was shown that with high probability 1 - O(1/e e DI) over uniformly sampling datapoint to unlearn (Ginart et al., 2019), re- moving that datapoint does not change the final clusters and hence unlearning can be achieved by doing nothing.
Definition 4.1 (Holistic Unlearning Cost). We compute the holistic unlearning cost for a method M with unlearning operation cost U As and inference cost Jas as the number of inferences per unlearning request upon which M costs the same as I-SISA That is, the solution to
UM + Las C(M) Ui-SISA + LI-SIAA C(M)
where C(M) is the number of inferences. This gives
C(M) (UM - UULSSAA)(( IM)
In particular, the unlearning operation cost of quantized k- means for m unlearning requests is (m22/6/2// /e) in expec- tation where d is the dimension of the embedding vectors, as stated in Theorem 4.1 in Ginart al. (2019). The proof fol- lows by the previously mentioned bound on the probability deletion request changes the centroids (which introduces the L/(e|D)) dependence), and further noting the cost to check this stability for a given datapoint is independent of the dataset size. Finally, the (1/ (I) probability of having to retrain cancels with the |) factor involved in retraining from scratch, giving the stated dataset independent cost for the unlearning operations.
5. Experiments
We now empirically explore the benefits of unlearning given by in-context learning, compared to fine-tuning. First, we explore the test accuracy of our method ERASE compared to other in-context learning methods, and find it performs on par with ACoT (but recall has indeeeeddnde unlearn- ing costs). We hence take ERASE to represent in-context learning and compare it to the unlearning baseline for task adaptation of fine-tuning with SISA (Bourtoule et al., 2021). We find it has comparable test accuracy across many tasks, and by carefully picking the number of in-context examples, can give significant benefits to holistic unlearning costs (i.e.,
However, quantized k-means introduced quantization pa-
5

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Method
WhnoWhy
Timedial
Sports Understanding
Logical Fallacy Detection
4-SISA
1.4 x 103
0.2 x 102
2.7 10
1.3 x 103
2-shot
2.5 x 10s
0.4 102
5.2 x 10a
2.2 x 10
3-shot
1.7 x 103
0.3 102
3.4 x 10
1.5 x 10
4-shot
1.2 x 10s
0.2 x 10a
2.6 x 10
1.1 X 100
Table Number of inferences per unlcarning request to he as expensive as I-SISA: higher is better. These values are computed using the inference and unlearning operation costs for the methods reported in Tables 3 and 4 (Appendix B) respectively. We see that using 4 in-context examples is more expensive than O-shot 4-SISA as it can handle fewer inferences per unlearning request. This is despite it having more efficient unlearning operations. Nevertheless, in-context learning with 2-shot is still better for holistic unlearning costs than parameter finetuning (with a factor of improvement of & 2 across the hoard) and by Figure 3 has comparable performance on the tasks.
how infrequent unlearning can be for it to be better than 1-SISA).
probabilities of answer tokens. We mask all tokens within the example input and task setup (e.g., the tokens for "Input:" and "Output:"), fine-tuning only on the token within the answer string. We use a block size of 256 tokens and batch size of We use the Adam optimizer (Kingma & Ba, 2017) with Bn = 0.9, B2 0.98, weight decay of 0.01, and learning rate of 1e-5. We also use 10 warm-up steps with linear schedule. The full list of training parameters can be found in Table E.
In summary, ERASE performs on par with ACoT, and doing task adaptation (i.e.. "fine-tuning") using ERASE is more cost-effective for unlearning than SISA.
5.1. Experimental Setup
LLMs now commonly use a causal decoder architecture (Zhao et al., 2023). Within this architecture class, we ob- serve similar performance across pre-training procedures. Therefore, we use the popular LLaMA (Touvron et al., 2023) as representative base model within all experiments. We evaluated all methods on a suite of 15 tasks. All experiments were run on a single node containing four A40 Nvidia GPUs
Hyperparameter Selection We found that the learning rate was the only hyperparameter with a significant impact on final performance after fine-tuning. To tune our learning rate, we fine-tune our model on the intent recognition dataset for the rates: {5e-5, 1e-5, 5e-6, 1e-6, 5e-7, 1e-7) } and choose the one with the lowest test perplexity (Ie-5). Finally, we try three different number of warm-up iterations: 10, 15, and 40 steps, and find that 10 warm-up steps performed best.
Task Selection The 15 tasks evaluate on are from Big- Bench (Srivastava et al., 2023) (released under the Apache 2.0 license). Big-Bench tasks are designed to be difficult for LLMS to solve. From Big-Bench we selected tasks to emphasize challenging scenarios and isolate in-context learning ability. Our task selection process started with considering only tasks from the Big-Bench Instruction In- duction (BBII) (Zhou et al., 2023) subset of Big-Bench, which are curated to be difficult and lack any instructional content within examples. We then filter out any BBII tasks with less than 200 examples, to ensure each task has ade- quate examples for fine-tuning and evaluation. We chose to exclude the Dyck Languages task from BBII because its for- mat is prohibitive toward log probability based evaluations. To have more comprehensive dataset, we supplement these with 5 additional tasks from the broader Big-Bench dataset which the untuned version of LLaMA solves with no higher than a 40% normalized aggregate score. Finally, we remove the "task prefix" from the prompts of all tasks. This step helps isolate in-context learning ability from confounding variables such as instruction following ability.
Prompt Formatting We describe in Appendix A how we prompted an LLM for inference during: fine-tuning, zero- shot inference, and few-shot inference. This formatting is relevant to how measured inference cost.
Evaluation Setup Our evaluation of performance follows the process outlined by the BigBench dataset (Srivastava et al., 2023). All BigBench tasks we used are JSON tasks, JSON tasks have two types of evaluation metries: generative and multiple-choice. Given input, generative metrics re- quire the model to generate some text. This text is then com- pared to a predefined correct answer with either no format- ting or very little formatting. Multiple-choice metrics evalu- ate the model's output on a small set of multiple-choice op- tions. The metric then either assigns the most likely choice as the model's 's answer or looks the differennce between the model's output distribution and the target distribution. We use the metric(s) recommended for the task, giving prefer- ence to using log- log-probability based multiple-choice metrics because continuous evaluation metrics provide smoother and more predictable changes in model performance (Scha- effer et al., 2023). We report the performance on each task
We fine-tune all model parameters using a pipeline based on Alpa (Zheng et al., 2022). We use loss mask to update gradients based only on the log
Fine-Tuning Tuning Setup
6

---

Fast Exact Unlearning for In-Contert Learning Data for LLMs
Disambiguation QA
Fantasy Reasoning
Implicatures
Intent Recognition
Normalized Aggregate Score
Normalizze Aggregate Score
Normalized Aggregate Score
80
Normalized Aggregate
10
40
60
20
30
40
20
10-
10
-5
Random In-Context Sets
ACoT
ERASE
UMAP
ACOT
UMAP ERASE
Figure 2. Comparison of the normalized aggregate score on 4 bigbench tasks between random selection of in-context examples, ACoT, ERASE alongside dimension reduction variant ERASE and ACoT (using UMAP), All methods are tested the 4-shot setting. We see that ERASE matches our outperforms ACoT on three of the four tasks, and similarly with random selection. Considering dimension reduction ERASE. we observed made slight improvements but did not affect the relative improvement of ERASE over ACoT (still better than dimension reduced ACoT three of the four tasks).
by the model's normalized aggregate score, which is nor- malized such that 0 represents random performance and 100 represents the performance of a human expert. The defini- tion of normalized aggregate score (Srivastava et al., 2023) described in Appendix
To conclude, observe that ERASE has comparable ac- curacy to ACoT, despite having more efficient unlearn- ing operations. We proceed to compare the accuracy of ERASE against the finetuning baselines.
5.3. Few Shot ERASE is comparable to SISA variants
5.2. ERASE comparable to Auto Chain-of-Thought
The question we now ask is: can ERASE perform well as the SISA variants (the baseline for fine-tuning with unlearn- ing), and hence lead to alternative task adaptation algorithm with efficient unlearning operations for these tasks? To do this, first compare the performance of ERASE with different number of in-context examples to SISA with dif- ferent number of shards (i.e., models used for ensembling). Recall, increasing the number of in-context examples is expected to increase performance of ERASE while de- creasing unlearning operation cost and increasing inference cost. However, increasing the number of shards for SISA known reduce performance (Bourtoule al., 2021), but also reduce the unlearning operation cost while increasing inference cost (Bourtoule et al., 2021). The comparisons across these ablations will later help us have more fine- grained understanding of when in-context learning has a better holistic unlearning cost versus performance trade-off.
In this section we ask, does making in-context learning more efficient to unlearn affect its test performance? Here we compare ERASE the other in-context learning algo- rithms described in Section 3 which were not optimized for unlearning. In particular, pares to ACoT when ACoT outperforms random sampling (i.e. example selection is important), enabling dataset inde- pendent unlearning operation costs on those tasks.
investigate how ERASE com-
compared ERASE to ACoT and random selection of in-context sets Figure 2, using default parameters of 10 iterations for clustering and setting the quantization parame- ter for quantized k-means to 0.05. We found that ACoT and FRASE outperform random selection on most tasks, i..., the more expensive to unlearn methods perform better. Amongst these methods, we found that ERASE outpeforms ACoT on three out of the four tasks. Hence we conclude that ERASE provides and alternative to ACoT.
Specifically, we consider SISA with ensembles of size 4, 2, and 1.2 In all implementations of SISA, also do not slicing we train for only single epoch and hence every datapoint is only used once (and thus already implicitly sliced). For ERASE. we use upto 4 in-context examples, and keep the same hyperparameters as in our previous comparisons between in-context learning methods.
We further evaluated dimension reduction variants (using UMAP) of the clustering methods to understand whether em- bedding dimension affects the comparison between ACoT and ERASE. Considering the effect of dimension reduction, we see it slightly boosts (or matches) the performance for ERASE and ACoT. However, both ACoT and ERASE still perform comparably across the tasks after dimension re- duction. We conclude that dimension reduction has minimal relative impact performance between ACoT and ERASE, hence not affecting the previous claim of comparability in accuracy between the two methods.
"e do not test beyond 4 models per ensemble as we want each model in the ensemble to fine-tune on at least 20 examples for all tasks. This is Figure in Appendix shows convergence in normalized aggregate score only ever occurs after examples, where here we tested different sizes of ensembles.
7

---

Fast Exact Unlearning for In- In-Context Learning Data for LLMs
Implicetures
Intent Recognition
Fantasy Reazaning
Deambiguution QA.
60-
20
40
10
01
Epistemic Reasoning
Presuppasition As NJ
Question Selection
Hlyeebaton
Nomalaed Acgingate Scoe
Logical Fallacy Detection
Noviz Relormmendatioo
Ruiln Nemtle
Sports Understandiigg
40-
0-
25-
-10
-10-
Timedial
WinWhhy
Misconceptiooss
26
10
10
-10
BRASE, 2-shct
IRASIE Rushot
5.5 1-oubmodeds
SISA 4-submodiks
ERASE, 3 3-shot
untuned SISA
S 2-submodall
defined in Section 5.1) of 2, 4-shot ERASE 1, 4-SISA, and the baseline of no task adaptation for 15 bigbench tasks. We see on several tasks that ERASE performs comparably or even better than the SISA variants; LLaMA is capable of in-context learning with ERASE these tasks. We repeat experiments with ERASE 10 times to estimate the standard deviation and evaluate all methods on the entire test set.
Figure 3. comparison of the performance (measured by normalized aggnegate
In Figure 3, we compared the test accuracy of the methods. We that ERASE using 4 in-context examples can be accurate all the SISA variants for more than half of the 15 tasks, with 2-shot ERASE often still being as accurate as the most efficient to unlearn SISA variants. In short, ERASE performs comparably to SISA on many tasks.
as they are model size dependent, the model and dataset- independent unlearning costs of in-context methods are prac- tically O. For inference costs, age (Li, 2023), measuring base LLaMA to have inference cost of roughly 264, 996, 864 x (1 + context length) FLOPS and evaluated the expected context lengths for all methods.
the Flops Profiler pack-
Given these values, we then used Definition 4.1 to compute the holistic unlearning costs for the different methods. We report the found number of inferences per unlearning request after which method is more expensive than the baseline (1-SISA) in Table 2. Inspecting the values, We found that the pest-performing version of in-context learning (the 4- shot version) costs more than the SISA variants to unlearn; that is, they can handle fewer inferences per unlearning request before being more expensive than the baseline. This is despite it having
5.4. Few Shot In-context Learning is holistically more cost efficient to unlearn than SISA
With this understanding of the predictive performance of ERASE relative to SISA, we turn to when is cheaper to deploy in-context learning than SISA in the presence of unlearning requests. Specifically, how infrequent can unlearning requests be for each method after which they worse than 1-SISA in cost, and does in-context learning allow more settings?
efficient unlearning operation.
Towards answering this, we first reported the numerical cost (measured in FLOPS) per unlearning operation and per in- ference for the different methods Table 3 and Table 4 (in Appendix B) respectively. This reported for 4 tasks where in-context learning performs as well as SISA, as shown in Figure 3. In particular, computed the expected cost per unlearning operation for SISA by taking the flops used to train an individual model the ensemble and dividing by 2 (as dividing by captures the expected length of training that needs be redone if logs checkpoints). Compared the costs of SISA, which do not scale favourably LLMs
However, note 2 and 3-shot ERASE still performed compa- rably to the most efficient to unlearn SISA variants. Hence we compared their holistic unlearning cost to 4-SISA (note that 4-SISA more efficient to deploy than and 2-SISA by Definition 4.1). We observed 3-shot ERASE has costs com- parable 4-SISA, and 2-shot ERASE in fact significantly improves the holistic unlearning costs: it allows unlearning requests to be at least almost half as infrequent what 4-SISA allows before being expensive as 1-SISA.
We hence conclude that in-context learning can be more
8

---

Blakeney, C.,, Paul, M., Larsen, B. W., Owen, S., and Fran- kle, J. Does your data spark joy? performance gains from Fast Exact Unlearning for In-Context Learning Data for LLMs
favourable for deployments subject unlearning requests the task adaptation phase. However, this requires picking the number of in-context examples more carefully and our trade-offs may not generalize to other tasks (where more in-context examples are needed for performance).
References
Baumhauer, T., Schöttle, P., and Zeppelzauer, M. Machine unlearning: Linear filtration for logit-based classifiers Machine Learning, 111(9):3203-3222 2022.
6. Conclusion
domain upsampling at the end of training. arXiv preprint arXiv:2406.0344 2024.
We showed that fast exact unlearning is possible for fine- tuning an LLM (i.e., task adaptation). This followed by us- ing in-context learning for fine-tuning, and noting quantized k-means on embeddings is an accurate in-context learning algorithm. This provided a learning plus unlearning recipe whose unlearning operations costs were independent both the LLM and dataset size. This efficiency gain was true even for a holistic unlearning cost that studied how frequent in- ference requests can be per unlearning request.
Bourtoule, L.,, Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., Lie. D., and Papernot, N. Machine unlearning. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 141-159, IEEE, 2021.
Brophy, J. and Lowd, D. Machine unlearning for random forests. In International Conference on Machine Learn- ing, pp. 1092-1104. PMLR, 2021.
We note in-context learning is not always an effective leam- ing method, and focused our study on situations where in-context learning is performant. For example, in-context learning can help a model understand the format of a trivia task, but can not significantly help model recall more trivia. Additionally, in-context learning is limited by the context size and does not modify model weights. Therefore, it is not useful for learning over large amounts of data. We leave these ill-su tasks, as well as an analysis of the trade-off between fine-tuning ability and unlearning cost, as future work.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Chen, J. and Yang, D. Unlearn what you want to for- get: Efficient unlearning for llms. arXiv preprint arXiv:2310.00150, 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chowdhury, S. B. R., Choromanski, K., Sehanobish, A., Dubey, A., and Chaturvedi, S. Towards scalable ex- act machine unlearning using parameter-efficien fine- tuning, 2025. URL https:://axxiii
abs/
Acknowledgements
We thank Eleni Triantafillou and Jamie Hayes for helpful comments on earlier versions of this work. We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions: Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, DARPA through the GARD project, Intel, Meta, NSERC through the Discovery Grant, the Ontario Early Researcher Award, and the Sloan Foundation. Anvith Thudi is supported by a Vanier Fellow- ship from the Natural Sciences and Engineering Research Council of Canada. Michael Zhang is supported by the NSERC CGS Fellowship. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Cooper, A. F., Choquette-Choo, C. A., Bogen, M., Jagiel- ski, M. Filippova, K., Liu, K. Z., Chouldechova, A., Hayes, J., Huang, Y., Mireshghallah, N., et al. Machine unlearning doesn't 't do what you think: Lessons for gen- erative ai policy, research, and practice. arXiv preprint arXiv:2412.0699 2024.
De, S., Berrada, I.., Hayes, J., Smith, S. L., and Balle, B. Un- locking high-accuracy differentially private image classi- fication through scale. arXiv preprint arXiv:2204. 13650, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
9

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Ginart, A., Guan, M., Valiant, G., and Zou, J. Y. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Min, S., Gururangan, S., Wallace, E., Shi, W., Hajishirzi, H., Smith, N. A., and Zettlemoyer, L. Silo language models: Isolating legal risk in a nonparametric datastore. In The Tweifth International Conference on Learning Representations.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9311 2020.
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, 2022.
Graves, L.,, Nagisetty, V., and Ganesh, V. Amnesiac ma- chine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:23 2023.
Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. arXiv preprint 191.000110000000 2019.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage?, 2023.
Gupta, V., Jung, C.,, Neel, S., Roth, A., Charif-Maalvajeet S., and Waites, C. Adaptive machine unlearning. Ad- vances in Neural Information Processing Systems, 34: 16319-16330, 2021.
Shumailov, I., Shumaylov, Z.,, Kazhdan, D., Zhao, Y., Paper- not, N., Erdogdu, M. A., and Anderson, R. Manipulat- ing sgd with data ordering attacks. Advances in Neural Information Processing Systems, 34:1 1880-008033 2021.
Hayes, J., Shumailov, I., Triantafillou, E., Khalifa, A., and Papernot, N. Inexact unlearning needs more careful eval- uations to avoid a false sense of privacy. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 497-519. IEEE, 2025.
Srivastava, A., Rastogi, A., Rao, A., and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Jang. J., Yoon, D., Yang. S., Cha, S., Lee, M., Logeswaran, L., and Sco, M. Knowledge unlearning for mitigat- ing privacy risks in language models. arXiv preprint 10X0::2110.01504 2022.
Thudi, A., Shumailov, I., Boenisch, F., and Papernot, N.
From differential privacy to bounds on membership in- ference: Less can be more. Transactions on Machine Learning Research.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS& P), pp. 303-319. IEFE, 2022a.
Li, C. Flops profiler python package, 2023. URL https: //pypi. www.oo/g/goooo////ooooooo
pro iier/.
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1950-1965, 2022.
Thudi, A., Jia, H., Shumailov, I., and Papernot, N. On the necessity of auditable algorithmic definitions for ma- chine unlearning. In 31 USENIX Security Symposium (USENIX Security 22), pp. 4007-4022, 2022b.
Thudi, A., Jia, H., Meehan, C., Shumailov, I., and Papernot, N. Gradients look alike: Sensitivity is often overestimated in (DP-SGD). In 33rd USENIX Security Symposium (USENIX Security 24), pp. 973-990, 2024.
LLMS, Who's harry potter? approximate unlearn.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098. 2022.
Pro-
Touvron, H., Lavril, T,, Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023.
Mantelero, A. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-253 2013.
Triantafillou, E., Kairouz, P., Pedregosa, F., Hayes, J., Kurmanji, M., Zhao, K., Dumoulin, V., Junior, J.
10

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Mitliagkas, I., Wan, al. Are making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint S00i::22406.900000 2024.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, An explanation of in-context learning as implicit bayesian inference. In International Conference Learning Rep- resentations, 2021.
Yao, Y., Xu, X., and Liu, Y. Large language model unlearn- ing, arXiv preprint arXiv:2310.1066 2023.
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Ka- math, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L.,, al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.065000 2021.
Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.034995 2022.
Zhao, W. X., Zhou, K., Li, J., Tang. T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C.,
Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X.,
Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. survey of large language models, 2023.
Zheng, L., Li, Z., Zhang, H., Zhuang, Y., Chen, Z., Huang, Y., Wang, Y., Xu, Y., Zhuo, D., Xing, E. P., Gonzalez, E., and Stoica, I. Alpa: Automating inter- and intra- operator parallelism for distributed deep learning, 2022.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human- level prompt engineers, 2023.

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
A. Experimental Details
Prompt Formatting Below we list how we prompted an LLM for inference during: fine-tuning, zero-shot inference, and few-shot inference. This formatting is relevant to how we measured inference cost. For completeness, we also describe the expected output during fine-tuning.
1. Fine-tuning Input: "[test example input]" Output: "[test example output]"
2. Zero-Shot inference Input: "[test example imput]"
3. Few-Shot inference Input: "Input: [in-context example input] Output: [in-context example output] Input: [in- context example input] Output: [in-context example 2 output] Input: [in-context example k input] Output: [in-context example output] Input: [example input]"
Normalized Aggregate Score We state the formula as defined in Srivastava et al. (2023) below, where for a given task "raw preferred metric" is the model's performance using the task's preferred evaluation metric, "low score" is the lowest possible score (typically random choice), and "high score" is the highest possible score (typically expert human performance):
[raw preferred metric] [low score]
[normalized aggregate score]
[normalized preferred metric]
100
[high score]
[low score)
B. Additional Tables and Figures
Method
WinoWhy
Timedial
Sports Understanding
Logical Fallacy Detection
I-SISA
71 x 10
70 X 10
71 x 10
70 x 10
4-SISA
14 x 10
14 x 10
14 x 10
14 x 10
In-context Methods
0
R 0
Table 3. Unlcarning operation costs of different exact unlcarning methods, measured in FLOPS, for four different tasks: lower is better. All tasks have roughly the same unlcarning operation cost for given SISA method despite having dramatically different input lengths, This is as the examples for all the tasks fit entirely within the model's context window. Therefore, because fine-tuning requires no token generation, it means that all examples require only a single forward pass to compute the required fine-tuning log probabilities regardless of length. When compared to the costs of the SISA methods, we have the unlearning costs for in-context methods 0 given they are model size independent; notably random in-context sets are O(1) (just the cost of resampling).
Method
WinoWhy
Timedial
Sports Understanding
Logical Fallacy Detection
1-SISA
13.5 x 109
78.6 x 109
7.0 x 109
14.9 x 10
4-SISA
54.2 x 10
314.4 x 109
27.9 x 109
59.7 x 10
2-shot
42.1 x 109
238.1 x 109
20.7 x 109
46.1 x 10
3-shot
55.5 x 109
323.7 x 109
27.6 x 109
61.5 X 109
shoho
70.4 109
395.5 x 109
34.5 x 10
76.7 x 10
Table 4. Inference costs of different exact unlearning methods, measured in FLOPPS, for four different tasks: lower is better. Unlike what was seen with unlearning operation costs Table 3, we see variety of inference costs depending on the task. Nevertheless, comparing costs across the SISA methods and in-context methods, we see that using 4 in-context examples has the largest cost. 3-shot in-context learning has inference cost comparable to 4-SISA, with 2-shot being the cheapest before the bascline 1 1-SISA. We calculated the FLOPS for inference reported for inference by computing the inference cost per token for the model, the average number of tokens in an input, and the average number of token in an in-context example for each task.
C. Discussion
Here we discuss how future work can further explore unlearning different parts of the deep learning pipeline (Section C.1), as was found successful in this paper. We also discuss how work on the limits to holistic unlearning efficiency (as introduced in
12

---

Fast Exact Unlearning for In-Contex, Learning Data for LLMs
implicatures
Intent Recognition
Fantasy Reasoning
Disambiquation QA
20
60
10
40
10
20
20
80
20
80
20
80
Epistemic Reasoning
Presuppositions NLJ
Question Selection
Hyperbaton
Normalzzed Aggregate Score
16
14-
20-
-5
12
20
20
80
80
Logical Fallacy Detection
Movie Recommendation
Ruin Names
Sports Understanding
40
10-
60
40
20-
-10
20
-10
0
20
40
60
80
0
20
40 WinoWhy
60
80
20
40
60
80
20
40
60
80
Timedial
Misconceptions
10-
-5
-10
-10
0
20
40
60
80
0
20
40
60
80
20
40
60
80
Number of fine-Tuning Examples Seen by SISA Ensemble
SISA models
SISA models
SISA models
Figure 4. Performance (reported by normalized aggregate score) of SISA variants as they progress through training, measured by the number of training examples seen. This is shown for the 15 bigbench tasks selected according to the process described in Section S.1, using the finetuning setup also described in Section 5. We find that finetuning with SISA typically converges after 80 examples, and always required least 20 examples converge. Hence we use 20 examples to define hard cut-off on how small our shards can be for SISA. Recall normalized aggregate score reports model performance such that 0 represents random performance and 100 represents the performance of human experts.
this paper) has implication to the more general trustworthy ML community (Section C.2). Lastly, we describe how changing the unlearning definitions could change cost comparisons (Section C.3), and the open problem of understanding when in-context learning better than fine-tuning (Section C.4) which has implication the applicability of our observations.
C.1. Towards More Fine-Grained Unlearning Threat Models
One core contribution of our work is the insight that the deep learning pipeline can be segmented, allowing for more efficient exact unlearning in specific stages. Practically, this suggests data with potential unlearning requirements should be allocated to stages where exact unlearning is more feasible. In this paper we split the deep learning pipeline into two stages (pretraining and task adaptation), as illustrated in Figure 1.
However this is still a coarse-gnt -rerineg view of a deep learning pipeline. Many large models go through multiple stages of refinement, intentionally using different data at each stage (Dubey et al., 2024). This can be at the pretraining stage where more carefully curated data mixtures are introduced after training on less filtered data to improve performance. Alternatively, in the task adaptation stage, a variety of fine-tuning techniques on different sources of data may used before finally doing in-context learning, in order do task adaptation for multiple tasks.
As an example of another unlearning problem arising from these modern pipelines, many large models run hyperparameter selection over the mixture weights of large datasets by evaluating performance of models trained with those mixtures on smaller benchmark data (Dubey et al., 2024; Blakeney et al., 2024). Essentially they had learnt mixture weights of "public data" with this benchmark data. One can imagine this benchmark data contains sensitive user data reflecting specific user groups the model trainer is trying to improve performance for. Hence, how might one unlearn the mixture weights if some of the users in the benehmark data revoked access their data? Is there preferred method for mixture finding if one would
13

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
need unlearn?
Our research aims to inspire future investigations into various stages of modern deep learning pipelines to assess the unlearning requirements for data at each stage and develop efficient unlearning methods. As our understanding evolves, we may identify efficient unlearnin techniques for multiple pipeline stages, allowing strategic placement of potentially unlearnable data, e.g., in placement for retrieval based models as in (Min et al.). This approach could ultimately lead to efficient exact unlearning solutions for deep learning by addressing unlearning challenges in specific, critical parts of the process rather than tackling the entire pipeline simultaneously.
C.2. On Lower-Bounds to Holistic Unlearning Cost and Implications to Trustworthy Machine Learning
Our investigation into exact unlearning methods for task adaptation revealed unlearning methods that decreased the costs for the unlearning operation often also increased the costs for inference. To quantify this trade-off, we proposed new holistic measure of unlearning cost: the frequency threshold of unlearning requests at which method becomes more cost-effective than retraining. This metric offers potential insights into the fundamental limits of exact unlearning efficiency.
Further study into this trade-off may be able to provide provable, or even empirically measured, lower-bounds on how holistically efficient unlearning can be in certain settings while still having an accurate learning algorithm. Our experiments suggest a possible unavoidable trade-off between inference cost and unlearning operation cost in certain machine learning setups, although more efficient algorithms may yet be discovered. Given that our algorithms achieve near-constant unlearning operation costs, future improvements would likely focus on reducing inference costs, which we expect can only improve by constant factors.
Such lower bounds would have implications beyond just unlearning. In particular, a lower-bound on holistic unlearning cost would imply a trade-off between how much data any accurate learning algorithm can be "invariant" to (i.c., can unleam by doing nothing) and how efficiently such invariance can be checked. This is as one can construct an unlearning algorithm by checking if removing a training datapoint changes the final model and if not, do nothing; if significantly many points do not affect the model and this can be checked quickly, this provides a fast unlearning operation. This algorithm would also not increase inference cost, providing an example of a good holistic unlearning algorithm which such lower-bound would prevent. Hence, lower-bounds to holistic unlearning cost may be of fundamental interest to the core machine learning community.
Moreover, a variant of the unlearning algorithm sketched above was proposed by Thudi et al. (2024), where checking invariance was possible with a new per-instance differential privacy analysis. However, such an analysis is currently slow to run, and a lower-bound on holisite unlcarning cost may explain the limits of how efficiently such per- per-instance DP analysis can be run. Hence, such a lower-bound to holistic unlearning can be of interest to the privacy community.
Given these connections, we believe understanding the limits of holistic unlearning be of fundamental interest the trustworthy machine learning community in general. Beyond any other connections, it may help illuminate why efficient exact unlearning for deep learning has been illusive.
C.3. On Unlearning Definitions and Implications to Cost
We wish to reiterate here the distinction between exact and approximate unlearning, and the issues that are still common to both and how this can affect claims of efficiency. Approximate unlearning attempts to emulate exact unlearning in manner that is sufficient for specific goals of unlearning. However, the reasons to unlearn may be several-fold in practice (legal, performance, etc) and not amenable to any specific metric, hence difficult for approximate unlearning. Nevertheless, for the cases where approximate unlearning is appropriate, it may be that there are similar tradc-offs between inference cost and unlearning operation cost as was found for exact unlearning in this paper, or between other sources of deployment cost. Perhaps these trade-off's are also affected by the choice of approximate unlearning metric, giving an argument for against certain metrics. In short, are not aware of work looking into the efficiency trade-off between unlearning for different approximate unlearning metrics, and what holistic consideration (analogus to changing inference cost in exact unlearning) would need to be made.
We also note that exact unlearning is also not immune to problems of applicability. In particular, exact unlearning can still suffer issues of not being auditable (Thudi et al., 2022b), meaning additional requirements are potentially needed to satisfy legal requirements. may be that to make these unlearning methods auditable, we may find a new trade-off between various sources of deployment costs, such communication cost vs. offline cost for cryptographic protocols.
14

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Furthermore, exact unlearning can also still leak privacy if not done with additional privacy requirements (Gupta et al., 2021), e.g., if an adversary gains access to the model before and after the unlearning operation for a data point they may be able to infer what the data point to unlearn was. It may be that satisfying privacy requirements on unlearning can change the ratios between inferences and unlearning requests we observed (for given test accuracy), and the preference for in-context learning. For example, privacy often comes with accuracy degradation. However, the mechanism needed to make unlearning in-context examples private may involve less degradation to accuracy than for fine-tuning, and vice-versa.
Lastly, regardless of whether we are doing exact or approximate unlearning, note we assumed a stream of individual independent unlearning requests, but if they were aggregated into batches, then 1-SISA unlearning operation cost does not scale with the number of requests making it more desirable. Analogously, if the sequence of unlearning requests were not independent, further algorithmic changes may be needed to do unlearning (Gupta et al., 2021).
To summarize, we believe careful consderation of the specific unlearning requirements can affect our observed claims of which unlearning methods are more efficient. We hope future work considers the different settings described here, adding to our understanding of when unlearning can be efficient.
C.4. Understanding when In-Context Learning Performs Best
Our experiments, as illustrated in Figure 3, showed that ERASE often achieves comparable accuracy to SISA. However, this was not always the case, and we observed a large variance in the relative performance between tasks. Furthermore, we are not aware of work that predicts when in-context learning accurate (relative fine-tuning) and we do not have an explanation for our observed variance in competitiveness.
We hypothesize that this variability is largely influenced by the pre-trained model, and that different pre-trained models may yield different results for in-context learning performance. This uncertainty has practical implications for unlearning: to determine if in-context learning can be used effectively (for more efficient learning) without significant performance loss compared to fine-tuning, it appears necessary to still fine-tune the model for comparison. Given that in-context learning is now known to be important for unlearning, we believe an important open problem for the trustworthy machine learning community is predicting when in-context learning is suitable for task without requiring fine-tuning for comparison. Solving this problem would significantly enhance our ability to implement efficient unlearning strategies across various tasks and models.
D. Statistical Significance Analysis
In this section, we analyze the statistical significance of our results from Figure 2. each task and avoid cross-task analysis as with a sample size of 4, the analysis would not be meaningful. For our analysis, we assume that the performance of each algorithm is normally distributed with unknown population mean and variance. We also assume that the variance of each algorithm is independent. This assumption is based on the observation that each algorithm's 's variance is different and the intuition that some algorithms are more stable than others. We will conduct our analysis in pairwise fashion between the following algorithms: Random In-Context Sets (Random), ACoT, and ERASE, which we will sometimes note as A,, A2, A3. Formalizing the above, for E {1, 21,3,3 we assume the performance of each A, is sampled from some normal distribution (H4 of). Our Null Hypothesis is that each pair of two algorithms A,, Aj. i * j have equal mean performance, i.e., μزا = f4j. We test the Null Hypothesis using a two-sample t-test with assumed unequal variances. Table 5 contains the results of Figure 2, and Table 6 shows the calculated p-values.
We analyze the results individually for
E. List of Training Parameters
A complete list of all training parameters used can be found in Table E.
F. Exact Unlearning Commentary Of Chowdhury et al.
Chowdhury al. (2025) propose unlearning by reverting to a checkpoint that did not observe the datapoint to be unlearned. However, this does not reproduce the model that would have been obtained had the datapoint never been included in that slice (as the model distribution is now dependent on where the datapoint to unlearn originally appeared). Furthermore, their method of selecting among models trained on permuted data slices deviates from the priori distribution of models without the datapoint, and is conceptually analogous to the forging attack described by Thudi et al. (2022b). More generally, their
15

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Table Tabuular format results from Figure 2. Each experiment (task-al (task-algorithm combination) had 32 independent trials. As deseribed in Figure 2, ERASE matches or outperforms ACoT on most tasks, and similarly with random selection.
Task
Algorithm
Mean
Std Dev
Random In-Context Sets
3.59
4.70
ACoT
-4.22
2.78
ERASE
2.48
5,40
Disambiguation QA
UMAP + ACoT
5.65
6.99
UMAP + ERASE
6.11
7.53
Random In In-Context Sets
18.69
6.26
ACoT
18.02
5.21
ERASE
24.24
7.93
Fantasy Reasoning
UMAP + ACoT
20.40
4.92
UMAP + ERASE
23.34
5.91
Random In-Context Sets
29.67
11.55
ACoT
29.76
2.95
Implicatures
ERASE
32.37
14.11
UMAP + ACoT
32.52
12.10
UMAP + ERASE
28.21
8.05
Random In in-Context Sets
55.89
13.62
ACoT
77.17
0.60
Intent Recognition
ERASE
57.76
12.73
UMAP + ACoT
61.00
1.41
UMAP + ERASE
64.63
12.05
implicit definition of exact unlearning as any procedure that does not use datapoint-falls into the class of algorithmic definition issues highlighted in Thudi et al. (2022b), where such flexibility can trivialize unlearning. In contrast, our work follows the more rigorous definition from Thudi et al. (2022b), which requires exact unlearning to defined with respect to a specific training and exact unlearning algorithm.
16

---

Fast Exact Unlearning for In-Context Learning Data for LLMs
Table 6. Pairwise p-values between algorithms (ACoT, ERASE) across tasks, testing the Null Hypothesis that Mz. We consider p-value less than 0.05 statistically significant and bold them.
Task
Algorithm
ACoT
ERASE
Random
0.0000
0.384
Disambiguation QA
ACoT
0.0000
Random 0.6436
0.0029
Fantasy Reasoning
ACoT
0.0005
Random ACoT
0.9695
0.4054
Implicatures
0.3133
Random 0.0000
0.5728
Intent Recognition
ACoT
0.0000
Table List of all training parameters used. The model used described Touvron et al. (2023)
Parameter
Value
Trainable Parameters
All
Model
LLaMA 7B
Learning Rate
le-5
Warm-up Steps
10
Adam Beta 1
0.9
Adam Beta 2
0.98
Weight Decay
0.01
Num Epochs
Eval Steps
Weight Precision
float 16
Block Size
256
Num Micro Batches
Operator Parallel
Pipeline Parallel
17


Reviews:
[{"review_id": "DK5TaWE0bW", "reviewer": "Reviewer_d3Vc", "summary": "this paper proposes a novel exact un-learning approach for in-context learning. Given a training sample, the goal of exact unlearning is to obtain as quickly as possible an algorithm that one would have obtained without training on that data sample. The authors study this problem in the context of in-context learning, where the algorithm is “trained” by seing several examples from the target dataset. Auto chain of thought is an efficient method for ICL, which clusters the examples into k clusters and only uses the k examples closest to the centroids for ICL. The authors propose instead to use a quantized k-means, which enables fast exact un-learning. The resulting method, ERASE, is then compared against other unlearning methods for ICL, like random selection (which performs worse than Acot but makes unlearning easier) and Acot (which requires re-running k-means at each unlearning step). The authors also compare their method to unlearning with standard SGD-based fine-tuning, SISA. The authors explore the results in the space of resulting inference cost and cost of unlearning.\n\n# Edit after rebuttal + discussion\n\n- This paper studies exact unlearning in the context of in-context learning. In-context learning is a very important topic of research in those days, as it is an efficient method to adapt models for some tasks (not all, of course). Therefore, developing un-learning algorithms for ICL is a very important research direction. This paper is the first paper to propose such algorithms for exact un-learning. With that in mind, I think that the main criticism of s2nr, that\"*in-context fine-tuning doesn't work well for preference or instruction tuning like settings. So, ERASE in it's current form can't be applied to this important and widely used setting of fine-tuning*\", is not valid criticism: ERASE is not supposed to work where ICL does not; it is clearly not this paper's purpose to make ICL work for these tasks. In my view, showing that the proposed method works in settings where ICL is beneficial is enough to validate the method.\n\n- I also disagree with rev. 6e2H that there is little novelty: indeed, this paper is the first to propose exact un-learning algorithms for ICL. Unlearning and ICL are both important topics, and having even a baseline for unlearning ICL is important for the community as a whole. In my view, this paper fills an important gap. The fact that the method is simple is, in my view, a feature of the method and not a weakness. The method is novel and simple.\n\n- I also found the response of the authors to rev. ygwC convincing; the bulk of this reviewer's criticism is about comparison to approximate un-learning methods; but comparing approximate and exact un-learning methods is notoriously hard to do; here the methods proposed by the authors provably un-learns (of course, this trades-off against efficiency compared to approximate unlearning, but again, in my view, this is not this paper's battle to benchmark approximate vs exact unlearning)\n\nTherefore, I still think that this is an important article that should be accepted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback! Below we discuss questions raised in the review. Other suggestions will be implemented in our revised draft.\n\n>the main weakness to me is the title and positioning itself. It is the first time that I see ICL defined as a fine-tuning method...\n\nWe agree that the terminology in-context learning is more clear than fine tuning and propose to change the title (“fast exact unlearning for in-context learning data for LLMs” sounds good to us) and contributions accordingly. See our response to Reviewer s2NR for specific changes to the contribution bullet points. To clarify, this does not change any of the methodology or experimental setup and the results hold.\n\n> the error bars in most experiments are huge, it would be good to have a sense of the statistical significance of the results of the paper.\n\nWhile the error bars are large in terms of accuracy, the methods perform differently (in a statistically meaningful way) when it comes to unlearning cost. That is, the FLOPS required to unlearn varies between the methods, despite their accuracy being comparable.\n\n> the paper \"In-context unlearning: Language models as few shot unlearners.” seems related, it would be good to clarify more the differences with the present approach.\n\nIn this paper we obtain exact unlearning guarantees, while the mentioned paper (also cited in our related work) only does approximate unlearning (and that too without any guarantees). It is currently difficult to comprehensively evaluate such approximate unlearning methods, e.g., [1], which further motivates our paper. \n\n[1] Hayes, Jamie, et al. \"Inexact unlearning needs more careful evaluations to avoid a false sense of privacy.\"\n\n> “our models converge after one epoch of SISA finetuning.” isn’t it contrary to the conventional wisdom that repeating data a few epochs improves performance?\n\nIt is common with LLMs to only train for one epoch. Supporting this, we generally found that the training loss for our tasks reached near 0 within a single epoch; we hence stop at one epoch to avoid overfitting. We will add a note about this in the appendix with associated training loss graphs.\n\n> It would be good to clarify the costs of inference for ICL. The fact that it is linear with the number of token puzzles me, my understanding is that attention is quadratic in the number of tokens. One can use caching to make the cost linear with the ICL prompt length, but that cost still seems quadratic the first time it is done.\n\nThank you for pointing this out. You are correct that the asymptotic relationship should be quadratic in terms of number of tokens in the context. The asymptotic costs in table 1 will be updated in the revision with “t” -> “t^2”. We note that this has a negligible effect on our experimental results as we manually measured inference flops. The primary effect is improving ERASE’s relative asymptotic cost efficiency against the baseline SISA method.\n\n> In fig 2, how can such a slight modification of the algorithm yield such changes ? can we conclude anything based on this, looking at the error bars?\n\nWe were unsure which “slight modification” you are referring to. We will list our response to a couple of modifications:\n\nRandom vs ACoT: Our results suggest that ACoT performance benefits are task dependant. The ACoT paper also states that it “matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations.\" However, we believe further work is required to accurately understand this dependance.\n\nUMAP vs non-UMAP: The dimension of our text embeddings is large (1536). It is a known that clustering may work poorly in high dimensional settings. Thus, applying dimensionality reduction to our text embeddings could have potentially improved clustering results.\n\nACoT vs ERASE: We find that in-context learning performance is very sensitive to which examples are chosen, causing high variance for each method. We found on average ERASE and ACoT perform similarly, and believe a significant portion of the variance is intrinsic to the choice of in-context examples.\n\n> How does the method scale with dataset size? I expect that at large datasets sizes, ICL starts lagging behind fine tuning.\n\nWe agree that if there is an abundance of data, and the task is sufficiently different from the pre-trained model’s data, fine-tuning will perform better. However, this comes with additional unlearning costs. We will make it clearer in the paper that our focus is on the cases where in-context learning performs similarly to fine-tuning.", "claims_and_evidence": "- ICL is a very popular approach, designing algorithms for exact unlearning on it is a novel and promising resarch direction\n- the proposed method is sound, and the theory behind it is both simple to understand and elegant\n- the empirical evaluation is quite thorough\n\n- the main weakness to me is the title and positioning itself. It is the first time that I see ICL defined as a fine-tuning method. I think that for the vast majority of the community, fine-tuning means using a gradient based approach to tune or add parameters to the model, while ICL does not use gradient descent. I think that “fast exact unlearning for in-context learning data for LLMs” or something along those lines would make what the paper is about much clearer.", "methods_and_evaluation_criteria": "- the proposed method is sound, and the theory behind it is both simple to understand and elegant\n\n- gradient based fine-tuning and ICL have very different effects on the model. gradient based fine-tuning scales much better when the dataset size increases (and is worse at small scales), and it incurs no inference overhead. I think that the experiments could do a better job at exploring this space.\n- the error bars in most experiments are huge, it would be good to have a sense of the statistical significance of the results of the paper.", "theoretical_claims": "- the proposed method is sound, and the theory behind it is both simple to understand and elegant", "experimental_designs_or_analyses": "- the error bars in most experiments are huge, it would be good to have a sense of the statistical significance of the results of the paper.", "relation_to_broader_scientific_literature": "- the paper \"In-context unlearning: Language models as few shot unlearners.” seems related, it would be good to clarify more the differences with the present approach.", "essential_references_not_discussed": "- the paper \"In-context unlearning: Language models as few shot unlearners.” seems related, it would be good to clarify more the differences with the present approach.", "other_strengths_and_weaknesses": "- the paper is very easy to follow, it was a joy to read", "other_comments_or_suggestions": "- \"this, we have unlearning the fine-tuning data is independent of the model and dataset size” unclear to me\n- “Showing that learning with access to an LLM can allow for faster exact unlearning.” unclear to me\n- “makes undoing x fast” unlearning?\n- “our models converge after one epoch of SISA finetuning.” isn’t it contrary to the conventional wisdom that repeating data a few epochs improves performance?\n- It would be good to clarify the costs of inference for ICL. The fact that it is linear with the number of token puzzles me, my understanding is that attention is quadratic in the number of tokens. One can use caching to make the cost linear with the ICL prompt length, but that cost still seems quadratic the first time it is done.\n- I would use in fig 3 a different color palette than that in fig 2", "questions_for_authors": "- would it be possible to extend results in table 2 in a 2d place, where x axis is unlearning cost and y axis is inference cost?\n- In fig 2, how can such a slight modification of the algorithm yield such changes ? can we conclude anything based on this, looking at the error bars?\n- How does the method scale with dataset size? I expect that at large datasets sizes, ICL starts lagging behind fine tuning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55Eto79cYX", "reviewer": "Reviewer_s2NR", "summary": "This paper studies unlearning for in-context learning task adaptation, which is claimed to be understudied. Unlearning is mostly studied in setting where parameters are updates are required such as SISA. The overall method is a follow up on ACoT but in a new setting which is unlearning (ACoT was in learning paradigm) where the authors have used quantized kmeans++. The improvements is shown to efficiency of unlearning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and time.\n\n> Some of the claims in the contribution are difficult to verify for instance. 1) The first claim in line 073-074 is vague and I am not sure how to verify it. 2) The claim 3 and 4 are also not novel and vague.I request the authors to re-write the whole summary of contributions clearly.\n\nThank you for pointing this out, we propose to change the contributions bullet points as follows:\n\n1) Showing that for certain datasets, exact unlearning can be efficient by using in-context learning.\n2) Proposing an exact unlearning algorithm ERASE for in-context learning, that has dataset and model-size independent unlearning operation costs.\n3) A study of when in-context learning provides more efficient unlearning deployments than fine-tuning (and repeating fine-tuning from scratch to unlearn), including changes to inference costs.\n\n> The authors need to explain well why unlearning during in-context learning (i.e. no change in parameters) is an important problem. A naive baseline in this setting in removing samples from incontext rather than analyzing the embeddings. The authors have explained this point from line 161-172, however this naive baseline is missing from results.\n\nTo clarify, we understand the reviewer to be referring to random in-context selection, whose unlearning operation is replacing the selected example (which by rejection sampling one can see now samples from the correct unlearnt distribution). We have comparisons to this in Figure 2. \n\nWe wish to also clarify that only removing the unlearnt in-context example is not a correct unlearning operation for any in-context learning algorithm. Specifically, unlearning is an algorithmic definition, and if the original selection used group statistics (e.g., clustering), we must simulate those decisions on the dataset without the example to unlearn. Motivating this, clustering based in-context learning has been observed to occasionally outperform random in-context selection, and a contribution of this paper is to note this is still efficient to unlearn.\n\nWe will incorporate this previous paragraph into the related works to help motivate in-context unlearning.\n\n> A key limitation of this work based on my understanding is in-context fine-tuning doesn't work well for preference or instruction tuning like settings. So, ERASE in it's current form can't be applied to this important and widely used setting of fine-tuning.\n\nWe will acknowledge this limitation in a revised draft. To be clear, the main claim of the paper is that for some fine-tuning datasets, exact unlearning can be easy as in-context learning can be as performant as fine-tuning.\n\n> The comparison to SISA in sec 5.3 and 5.4 seems out of place as SISA is built for unlearning gradient based methods and ERASE is unlearning in-context finetuning.\n\nThe claim of our paper is that, when possible, we should do in-context learning instead of fine-tuning as it is much more efficient to unlearn. To show this we must compare in-context methods (ERASE) to their fine-tuning based counterparts (SISA). These sections show that in-context methods have comparable performance (on the datasets we tested) while dramatically reducing unlearning cost.\n\n> Why the naive baseline of removing samples from context not reported?\n\nIt is in Figure 2, see our response to an earlier comment.\n\n> The model and training details are very confusing to me. For instance which LLaMA?\n\nAs mentioned on line 267 we use the original LLaMA model from Touvron et al. (2023). The “Fine-Tuning Setup,” “Hyperparameter Selection,” and “Prompt Formatting” sub-sections lines 305-329 outline what we believe to be the most important training hyperparameters. In our revision we will improve readability by adding a table to our appendix showing a complete list of all training hyperparameters.\n\n>I think the evaluation needs to be extended some harder tasks such as math generation gsm8K, MMLU , hellaswag and other metrics commonly used in LLM literature for fine-tuning.\n\nOur existing evaluation includes 15 tasks capturing a variety of difficulties, and already allows us to conclude that which is better between in-context learning and finetuning is data dependent.\n\n> Do you believe ERASE can be extended for unlearning instruction samples from an instruction tuning dataset?\n\nWe see no issue in using ERASE on other datasets where in-context learning is possible. However, the primary question is whether in-context learning is effective at learning from that dataset, and as pointed out by the reviewer, in-context learning tends to perform poorly on instruction tuning.", "claims_and_evidence": "Some of the claims in the contribution are difficult to verify for instance.\n\n1. The first claim in line 073-074 is vague and I am not sure how to verify it.\n\n2. The claim 3 and 4 are also not novel and vague. \n\nI request the authors to re-write the whole summary of contributions clearly.", "methods_and_evaluation_criteria": "The authors need to explain well why unlearning during in-context learning (i.e. no change in parameters) is an important problem. A naive baseline in this setting in removing samples from incontext rather than analyzing the embeddings. The authors have explained this point from line 161-172, however this naive baseline is missing from results. The evaluation criteria is taken from prior works and seems reasonable.\n\nA key limitation of this work based on my understanding is in-context fine-tuning doesn't work well for preference or instruction tuning like settings. So, ERASE in it's current form can't be applied to this important and widely used setting of fine-tuning.", "theoretical_claims": "no theoretical claims, I have verified that inference cost is correct in Table 1.", "experimental_designs_or_analyses": "The evaluation metrics are taken form prior works and they seems reasonable. Some of my concerns.\n\n1. The comparison to SISA in sec 5.3 and 5.4 seems out of place as SISA is built for unlearning gradient based methods and ERASE is unlearning in-context finetuning.\n\n2. Why the naive baseline of removing samples from context not reported?\n\n3. The model and training details are very confusing to me. For instance which LLaMA? \n\n4. I think the evaluation needs to be extended some harder tasks such as math generation gsm8K, MMLU , hellaswag and other metrics commonly used in LLM literature for fine-tuning.", "relation_to_broader_scientific_literature": "This paper extends ACoT [1] in unlearning setting and employs quantized Kmeans++ instead of regular kmeans++.\n\n\n\n[1] Automatic chain of thought prompting in large language models, zhang et al. ICLR 2022.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths\n\n1. This paper studies a new direction.\n\nWeaknesses, apart from what already discussed.\n\n1. The writing can be further improved.\n\n2. The novel of the approach seems a bit limited as it directly extends ACoT [1] to unlearning. But since it is a fairly new area in my opinion this not a key concern.", "other_comments_or_suggestions": "Already discussed above.", "questions_for_authors": "Do you believe ERASE can be extended for unlearning instruction samples from an instruction tuning dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1Ye8tU4h4", "reviewer": "Reviewer_6e2H", "summary": "The paper proposes an exact unlearning algorithm, ERASE, for in-context learning. The core unlearning idea revolves around performing exact unlearning in AutoCOT, which clusters in-context examples using their sentence representations and uses the samples close to the cluster centroids as the ICL examples. ERASE leverages the quantized k-means algorithm to perform exact unlearning that efficiently updates the clustering process to remove deleted samples. The paper introduces a holistic evaluation strategy by accounting for both the unlearning and inference costs. The paper also provides several experiments on BigBench datasets comparing ERASE with baseline algorithms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback. We discuss specific points below.\n\n> Some of the underlying assumptions related to this work need to be mentioned. For example, for exact unlearning in this setup, the underlying assumption is that the ICL examples were not used during any stage of pre-training or instruction tuning.\n\nThat is correct, and we will make that assumption clearer in our revised draft.\n\n> The paper focuses on unlearning for a specific approach, ACoT. Although the paper motivates its advantage over vanilla CoT and showcases the results in Fig. 2 & 3, its significance over Random COT (random ICL example) selection is unclear in the context of unlearning. I understand that ACoT achieves better results than Random selection. But since Random CoT can perform O(1) deletion, the ideal way to evaluate would be to have a tradeoff between per query deletion cost and the overall performance (normalized aggregate score). In the current version, the paper reports only the normalized scores or inference cost at a time.\n\nWe would like to point out one more subtlety, which is ACoT and ERASE do not always perform better than random; this is dataset dependent. We decided to present solely the performance cost as it was already explained earlier in the paper that Random COT has O(1) cost, so it is up to the model provider to decide if the performance loss (on some datasets) is worth the cheaper unlearning cost. It is not clear to us how to value performance gains relative to unlearning computation. We will add this discussion elaborating the performance vs. unlearning consideration to our revised draft (in Section 5.2 if space allows).\n\n>The above point also raises a question about the holistic evaluation approach. Apart from inference and unlearning costs, shouldn’t we also consider the overall performance of the algorithm as a dimension to consider? For example, Random CoT has the same inference cost as ACoT and a much lower unlearning cost but the overall performance is poor compared to ACoT.\n\nSee our response to the previous question, which provides a detailed discussion on the context dependent nature of the performance and unlearning comparison: it is not clear to us how to value performance with changes in compute. This was unlike inference and unlearning operations, which are both measured with the same computational units (e.g., flops).\n\n> Line 102: “While no past work studied unlearning specifically for the fine-tuning stage (i.e., task adaptation stage) of LLMs” — this claim isn’t accurate. There is this recent paper [1] that exclusively focuses on erasing fine-tuning data. Moreover, existing exact unlearning algorithms like SISA and others, can be modified to adapt to the fine-tuning setting.\n\n\nThank you for bringing this paper to our attention! We will change our claim to mention this work, but note that it actually does not satisfy exact unlearning (according to our reading of their paper). Specifically, they propose to revert to a checkpoint that did not observe the datapoint to unlearn, but this does not reproduce the apriori trained model that would come from not having a datapoint in that slice. Furthermore, the selection of models amongst those trained from many permuted slices again means we do not reproduce the apriori distribution of models coming from not having a datapoint. This latter technique also seems analogous to the forging attack to fool exact unlearning [1]. More generally, the mentioned paper’s implicit definition of exact unlearning (which is anything that does not “use” a datapoint is exact unlearning) falls under the algorithmic definition issues pointed out in [1]; unlearning needs to be defined w.r.t a specific training/unlearning algorithm, as allowing anything that does not use a datapoint means we can trivially unlearn. After the submission cycle we will reach out to those authors to let them know of this issue.\n\nNote, the exact unlearning definition we follow removes this issue, by defining a specific training algorithm and unlearning with respect to it.\n\n[1] Thudi, Anvith, et al. \"On the necessity of auditable algorithmic definitions for machine unlearning.\" 31st USENIX security symposium (USENIX Security 22). 2022.", "claims_and_evidence": "**Strengths**:\n\n1. The paper is fairly easy to follow.\n\n2. The paper introduces a neat paradigm for performing exact unlearning while using in-context learning. The paper proposes ERASE, an efficient exact unlearning algorithm to delete instances when using ACoT. The core idea is generalizable and can be utilized in any algorithm using clustering to select ICL examples.\n\n3. The paper prioritizes an important aspect of unlearning, where unlearning and inference costs are often at odds. It introduces a metric that captures both costs, allowing for a holistic evaluation of an unlearning algorithm's performance.\n\n4. Empirical evaluation shows that the proposed method, ERASE, achieves similar performance to baseline algorithms even after unlearning several instances. \n\n**Weaknesses**:\n\n1. The novelty of the technical contribution is weak. This is because the proposed approach uses the exact algorithm for unlearning used in clustering and applies it in ICL where clustering is involved. I understand that the algorithm has been used in a new application and other metrics for evaluating unlearning have been proposed. However, the technical contribution is weak because the core unlearning technique is borrowed from previous work and applied in a new setting.\n\n2. Some of the underlying assumptions related to this work need to be mentioned. For example, for exact unlearning in this setup, the underlying assumption is that the ICL examples were not used during any stage of pre-training or instruction tuning. \n\n3. The paper focuses on unlearning for a specific approach, ACoT. Although the paper motivates its advantage over vanilla CoT and showcases the results in Fig. 2 & 3, its significance over Random COT (random ICL example) selection is unclear in the context of unlearning. I understand that ACoT achieves better results than Random selection. But since Random CoT can perform O(1) deletion, the ideal way to evaluate would be to have a tradeoff between per query deletion cost and the overall performance (normalized aggregate score). In the current version, the paper reports only the normalized scores or inference cost at a time.\n\n4.  The above point also raises a question about the holistic evaluation approach. Apart from inference and unlearning costs, shouldn’t we also consider the overall performance of the algorithm as a dimension to consider? For example, Random CoT has the same inference cost as ACoT and a much lower unlearning cost but the overall performance is poor compared to ACoT.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Line 102:  “While no past work studied unlearning specifically for the fine-tuning stage (i.e., task adaptation stage) of LLMs” — this claim isn’t accurate. There is this recent paper [1] that exclusively focuses on erasing fine-tuning data. Moreover, existing exact unlearning algorithms like SISA and others, can be modified to adapt to the fine-tuning setting. \n\n[1] https://arxiv.org/pdf/2406.16257", "questions_for_authors": "Please respond to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kwMGP7o8y", "reviewer": "Reviewer_ygwC", "summary": "This paper proposes ERASE, an in-context learning method combined with quantized k-means clustering for exact unlearning in LLMs. ERASE combines in-context learning with quantized k-means clustering, aiming to achieve dataset- and model-independent unlearning costs while maintaining competitive performance. A \"holistic unlearning cost\" metric is introduced to analyze trade-offs between inference and unlearning efficiency. The authors compare ERASE to SISA-based baselines on Big-Bench tasks, demonstrating reduced unlearning costs and comparable accuracy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and time. We respond to specific concerns below.\n\n> The paper claims dataset-independent unlearning costs based on quantized k-means, but fails to provide adequate justification for why this specific combination would be uniquely effective for LLMs. While quantized k-means is a known technique, its application to LLM embeddings lacks a foundation, particularly regarding stability in high-dimensional spaces.\n\nWe wish to point out that our experiments showed Q-Kmeans was as effective as ACoT (which uses standard kmeans++) on the LLM eval datasets we tested, and also as effective as parameter fine-tuning with SISA. While in general this might not be true if we change the embedding dimension used in ERASE, a contribution of our paper is showing that for the tasks we tested (and when using BERT embeddings) it was effective.\n\n> The proposed metric C(M) for Holistic Unlearning Cost represents a ratio between unlearning and inference costs, but without proper empirical validation, and it lacks experimental measurements of actual computational costs (such as GPU time and memory usage) to substantiate its claims.\n\nOur experiments did measure actual computational costs. The results in Tables 3 and 4 (in the appendix) computed the FLOPS for training and inference, which are used to directly compare between methods (relative improvements are shown in Table 1).  We use FLOPS as opposed to GPU time and memory usage because FLOPS are independent of available hardware and, thus, more accurately capture algorithmic costs.\n\n> The methodology section suffers from insufficient comparisons. The authors did not compare their approach against alternative clustering methods (like hierarchical clustering), making it difficult to substantiate claims of method superiority.\n\nThere are a limited number of efficient unlearnable clustering algorithms. We are unaware of efficient unlearning operations for hierarchical clustering.\n\n> The choice of Big-Bench as an evaluation benchmark is questionable, as it was not designed for unlearning tasks.\n\nWe agree other datasets would be beneficial, but want to point out our evaluation was over 15 different tasks. These tasks already allowed us to conclude that which is better between in-context learning and fine-tuning is data dependent.\n\n> The paper lacks comparisons with existing in-context unlearning methods, particularly the work by Pawelczyk et al.\n\nIn this paper we obtain exact unlearning guarantees, while the mentioned paper (also cited in our related work) only does approximate unlearning (and that too without any guarantees). It is currently difficult to comprehensively evaluate such approximate unlearning methods, e.g., [1], which further motivates our paper. We will make this clearer in our related works section.\n\n[1] Hayes, Jamie, et al. \"Inexact unlearning needs more careful evaluations to avoid a false sense of privacy.\" arXiv preprint arXiv:2403.01218 (2024).\n\n> The paper lacks a proper unlearning evaluation framework. Specifically, there is no clear distinction between forget sets and remaining sets, nor is there a comprehensive evaluation of both model utility preservation and forgetting ability.\n\nTo clarify, this distinction is only  present in approximate/heuristic unlearning where their effectiveness depends on the sets. Our method is an exact unlearning method, which by definition always works. Our evaluation thus focuses on exact unlearning compute.\n\n> Regarding the forgetting ability, the authors do not employ standard unlearning metrics, such as membership inference attacks or data extraction leakage tests, which are crucial for validating privacy guarantees.\n\nAs mentioned earlier, our method is an exact unlearning method and so by definition will always have perfect unlearning performance under the approximate unlearning metrics. To elaborate, we are producing exactly the retrained models, and those metrics measure differences (of approximately unlearnt models) to the retrained model.\n\n> Although the paper cites Pawelczyk et al.'s work on in-context unlearning in the related work section, it fails to provide any meaningful discussion or comparison with this directly relevant approach.\n\nSee our response to the earlier question about this paper; we will add more discussion elaborating the difference between their approximate unlearning approach and the guarantees of exact unlearning (which our method does) in our revised draft.", "claims_and_evidence": "1. The paper claims dataset-independent unlearning costs based on quantized k-means, but fails to provide adequate justification for why this specific combination would be uniquely effective for LLMs. While quantized k-means is a known technique, its application to LLM embeddings lacks a foundation, particularly regarding stability in high-dimensional spaces.\n2. The proposed metric C(M) for Holistic Unlearning Cost represents a ratio between unlearning and inference costs, but without proper empirical validation, and it lacks experimental measurements of actual computational costs (such as GPU time and memory usage) to substantiate its claims.", "methods_and_evaluation_criteria": "- The methodology section suffers from insufficient comparisons. The authors did not compare their approach against alternative clustering methods (like hierarchical clustering), making it difficult to substantiate claims of method superiority.\n- The choice of Big-Bench as an evaluation benchmark is questionable, as it was not designed for unlearning tasks.\n- The paper lacks comparisons with existing in-context unlearning methods, particularly the work by Pawelczyk et al.\n\n[Pawelczyk et al.] Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The paper lacks a proper unlearning evaluation framework. Specifically, there is no clear distinction between forget sets and remaining sets, nor is there a comprehensive evaluation of both model utility preservation and forgetting ability.\n- Regarding the forgetting ability, the authors do not employ standard unlearning metrics, such as membership inference attacks or data extraction leakage tests, which are crucial for validating privacy guarantees.", "relation_to_broader_scientific_literature": "While most previous work focused primarily on model fine-tuning approaches for unlearning, this paper introduces a novel in-context learning perspective. Additionally, the paper attempts to address a critical gap in evaluation methodology by proposing a holistic unlearning cost framework.", "essential_references_not_discussed": "Although the paper cites Pawelczyk et al.'s work on in-context unlearning in the related work section, it fails to provide any meaningful discussion or comparison with this directly relevant approach.\n\n[Pawelczyk et al.] Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ### Summary

This paper proposes a new unlearning method for in-context learning (ICL), coined ERASE, inspired by ACOT, in which the Kmeans++ is replaced by quantized Kmeans++. In particular, the paper focuses on finetuning/in-context learning settings, but **not** the pretraining stage. ERASE cost is independent of dataset size, unlike n-SISA and ACOT, thanks to the stability of quantized k-means. Another innovation of the paper is to take into account the inference cost of every method and balancing it with the cost of unlearning, and to estimate the number of inferences a method can handle before being outperformed by 

### Discussions and rebuttal

Reviewer **ygwC** asked for empirical evaluation of the forgetting (e.g through membership attacks and practical benchmarks); while these additional experiments could strengthen by comparing against empirical methods, I don't think they are required *per-se* since the method is sound and already provide formal unlearning guarantees - therefore empirical evaluations would only be an ablation.

Reviewer **d3Vc** noticed that many empirical results were reported as improvement over competitors, but the score difference is actually much smaller than the error bars. Formal statistical test does not allow to conclude that ERASE dominates ACOT and the Random baseline. As it stands, ERASE is an algorithmic improvement regarding unlearning cost, but cannot be considered improvement regarding final utility over these baselines. 

Reviewer **6e2H** found the contribution incremental, and encouraged the author to clarify that in the ICL setup, the forgetting of pretraining examples is not considered. Furthermore, a graph showcasing the tradeoff between forgetting cost and final performance seems necessary (for example a joint plot?), to show that the methods live on a Pareto front (which might differ from one task to another).  

### Recommendation

ERASE is a simple yet effective improvement of ACOT, by replacing Kmeans++ with a quantized Kmeans++ it yields better inference/unlearning cost tradeoff with no negative impact on utility. However, unlike what the author claim, it cannot be said that ERASE outperforms ACOT, n-SISA or Random baseline based on the benchmark: the error-bars are too high to conclude a meaningful advantage - on some benchmarks it cannot even allow to conclude that any of these method perform better than random guessing (score = 0).  

However, another contribution of the paper beyond ERASE is the inference/unlearning tradeoff analysis, which is an important direction for the community and a clear criterion on which methods should compare. This contribution is valuable in itself.  

Therefore, I recommend acceptance. Nonetheless, I encourage the author to clarify the claims regarding efficiency of ERASE, to report statistical tests on every benchmark, and to clarify their unlearning setup.
Decision: Accept (poster)