Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning
Authors: Ziwen Liu
Huawei Lin
Yide Ran
Denghui Zhang
Jianwen Xie
Chuan Li
Weijie Zhao
Zhaozhuo Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Unlearning
Randomized Algorithms
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Large language models (LLMs) sometimes memorize undesirable knowledge, which must be removed after deployment. Prior work on machine unlearning has focused largely on optimization methods that adjust parameters to enforce forgetting while preserving retention. However, these approaches assume that the forget and retain sets are readily available, which rarely holds in practice. Unlearning is typically triggered by an undesired generation at inference time, making the retrieval of relevant data the central challenge. 
We introduce the notion of data Pareto improvement for LLM unlearning, which formalizes how retrieval can expand the achievable trade-off frontier between forgetting and retention. To realize this principle, we propose Randomized Antipodal Search on Linearized Influence Kernel (RASLIK), a retrieval algorithm that combines permutation–projection hashing with randomized antipodal search. RASLIK reduces selection variance, achieves sublinear complexity, and yields a double gain in both quality and efficiency. Across multiple models, datasets, and unlearning algorithms, RASLIK consistently outperforms deterministic baselines and even oracle sampling, establishing randomized search as a principled and scalable solution for data-centric unlearning.

Published as a conference paper at ICLR 2026
RANDOMIZED ANTIPODAL SEARCH DONE RIGHT FOR DATA PARETO IMPROVEMENT OF LLM UNLEARNING
Ziwen Liu', Huawei Lin', Yide Ran", Denghui Zhang', Jianwen Xie*, Chuan Li*, Weijie Zhao?, Zhaozhuo Xu*
'Rice University. "Rochester Institute of Technology. SStevens Institute of Technology, *Lambda z1166eri edu, fh13352, uwjzvcs)erit.. edu.. {yranl, 42hang42, zxu7 (aaa7790ssteeeeee edu,, {jianwen.xies xie, c]0 @10lambdal... com
ABSTRACT
Large language models (LLMs) sometimes memorize undesirable knowledge, which must be removed after deployment. Prior work on machine unlearning has focused largely on optimization methods that adjust parameters enforce forgetting while preserving retention. However, these approaches assume that the forget and retain sets are readily available, which rarely holds in practice. Unlearning is typically triggered by undesired generation at inference time, making the retrieval of relevant data the central challenge. We introduce the notion of data Pareto improvement for LLM unlearning, which formalizes how retrieval can expand the achievable trade-off frontier between forgetting and re- tention. To realize this principle, we propose Randomized Autipodal Search Linearized Influence Kernel (RASLIK), a retrieval algorithm that combines per- mutation-projection hashing with randomized antipodal search. RASLIK reduces selection variance, achieves sublinear complexity, and yields a double gain in both quality and efficiency. Across multiple models, datasets, and unlearning algo- rithms, RASLIK consistently outperforms deterministic baselines and even oracle sampling, establishing randomized search principled and scalable solution for data-centric unlearning.
INTRODUCTION
Large language models (LLMs) have demonstrated impressive capabilities across diverse tasks (OpenAl et al.) 2024). but they sometimes memorize undesirable knowledge (Carlini et al.) 2019] 2023). When such information must be removed after deployment, machine unlearning provides a mechanism to forget targeted knowledge while preserving general utility (Eldan & Russinovich
2023 Existing work has primarily focused designing optimizers, such as wradient-based/ Jang
ct al. 2022 Liu et al. [0222 Yao et al. [024]] Yoon et al. (023) or meference-basse methods (Zhang
2023] Maini 2024 Meng et al. 2024), that couple forgetting ob- jectives with retention regu arizers. These approaches are effective under controlled benchmarks (Maini al. 2024 Shi al.
al. 2024 Rafailov et al.
2024) but typically assume that the forget and retain sets are read-
ily available (Shi et al. 2024p. In practice, unlearning is triggered by an undesired generation at inference time, leaving practitioners with only the observed output and a massive training corpus. Identifying which data to forget and which to retain becomes the primary challenge, making data efficiency the central bottleneck of unlearning (Carlini et al. 2021).
Unlearning inherently involves balancing two scemingly conflicting goals: improving forgetting often reduces retention, while prioritizing retention risks incomplete forgetting (Xu et al. 2023
Nguyen et al. 2024). This trade-off defines Pareto frontier (Davtalab-Olyaie & Asgharian 2021
of achievable outcomes. We introduce the concept of data Pareto improvement in LLM unlearning, which highlights the role of retrieval in expanding this frontier. A retrieval mechanism is Pareto- improving if enables stronger forgetting without disproportionate loss of retention, or conversely preserves retention without undermining forgetting. This perspective shifts the focus of unlearning
"Corresponding author; 2xu7 7we.stveeteaa...........mm edu

---

Published as a conference paper at ICLR 2026
from being purely optimization-centie centric being fundamentally cetrival--entric Retrieval quality is not a preprocessing detail but a first-order determinant of unlearning outcomes.
Building this insight, propose Randomized Antipodal Search Linearized Infiuence Ker- nel (RASLIK), a retrieval algorithm that introduces controlled randomization into influence-based search. RASLIK constructs randomized gradient sketches via permutationn-pojeecionn hashing and performs antipodal search to identify both aligned samples to forget and anti-aligned samples to retain. Randomization smooths unstable thresholding decisions, reducing selection variance, while sketching achieves sublinear complexity. The result is a double gain in both quality and efficiency. Experiments across models, datasets, and unlearning algorithms show that RASLIK consistently shifts the Pareto frontier outward, outperforming deterministic baselines and even oracle sampling.
Our contributions are as follows:
We identify retrieval as the central bottleneck of practical LLM unlearning and highlight data efficiency as a major challenge beyond optimization design.
We introduce the notion of data Pareto improvement, formalizing how retrieval can expand the achievable Orgetting-retention frontier in unlearning.
We propose RASLIK, randomized antipodal search method on linearized influence kernels that reduces variance, achieves sublinear retrieval complexity, and enables more stable and effective unlearning.
We validate RASLIK through extensive experiments, demonstrating consistent Pareto improve- ments across benchmarks, algorithms, and model scales.
DATA PARETO IMPROVEMENT OF LLM UNLEARNING
2. A FOCUS ON DATA EFFICIENCY OF LLM UNLEARNING
Large Language Models (LLMs) trained massive corpora inevitably memorize undesir- able knowledge (Carlini et al. 2019). In these cases, model owners must unlearn such knowl- edge while preserving the model's utility (Car- lini et al. 2023). Formally, given parameters 0 E 1 and loss E(x; 0) for sample the goal of unlearning is to increase loss on a desig- nated set F while maintaining or improv- ing performance on a complementary retain set R. Existing work mostly treats unlearning as an optimization problem: designing loss functions that couple forgetting objective with a utility- preserving regularizer. Examples include gra- dient ascent on JF with gradient descent on R (Jang et al. 2022 Liu et al. 2022; Yao et al. 2024). These paradigms implicitly assume that the forget F and the retain set R are given. In practice, however, unlearning rarely begins with this setting. Instead, it is triggered by unexpected generation produced at inference time. Faced with only and a massive training set, practitioners must first determine what forget and what retain. This makes retrieval of J and R not a secondary step but the true bottleneck in practical unlearning. Without high-quality retrieval, even the most sophisticated optimizers cannot achieve effective forgetting.
RASLIK Pareto Frontier
1.0
Baseline Pareto Frontier
Data Pareto Improvement
0.8
High Forgetting
High Retention
Forget Rate
0.6
0.4
Low Forgetting
Low Retention..
Ideal Point
Baseline
Minimiza Dmah
RASLIK
0.2
0.6
0.8
1.0
Retain Rate
Figure 1: Pareto trade-off between forgetting and reten- tion in LLM unlcarning.
2.2 INTRODUCING DATA PARETO IMPROVEMENT FORMULATION TO LLM UNLEARNING
Unlearning introduces fundamental tension: improving the degree of forgetting often reduces the model's general capabilities, while prioritizing retention risks incomplete forgetting. As shown in Figure this tension can be formalized Pareto trade-off between two conflicting objectives:
maximize forgetting accuracy
maximize retention quality.
2

---

Published as a conference paper at ICLR 2026
Any unlearning method, therefore, lies Pareto frontier Pertalab-Olyate & Asgharian 2021 improvements in one dimension typically come at a cost in the other. Unlike ordinary optimization, where seeks single optimum, unlearning inherently requires balancing two competing goals.
This motivates a data-centric notion of Pareto efficiency. We define data Pareto efficiency as the ability of the retrieval stage identify F and R that shift the Pareto frontier outward. Concretely, data selection is Pareto-impooving if it enables one of the following without degrading the other:
Achieving stronger forgetting (the model reliably suppresses and its variants) without dispro-
portionate loss of retention.
Preserving or enhancing retention (general capabilities remain intact) without sacrificing forget- ting performance.
Seen this way, retrieval quality is not preprocessing detail but a first-order determinant of unlearn- ing outcomes. A retrieval mechanism explicitly designed to respect the Pareto structure can sys- tematically enable better trade-offs for downstream optimizers. We therefore introduce the concept of data Pareto improvement: improvements in the selection of F and R that expand the achiev- able frontier forgetting-eetention performance. This perspective reframes unlearning from being solely optimizatioon-entricc to being also fundamentally retrieval-centric.
3 RANDOMIZED ANTIPODAL SEARCH ON LINEARIZED INFLUENCE KERNEL
Notations, Let 0 E Id denote the model parameters. f(x; 0) the loss for input in training dataset X, and g(x; 0) Vol(a;; 8) its gradient. For target generation y. define 4y g(y: 0). For training item E X, write Iz = g(z; 0). The unlearning objective is
U(0) Ex 0)]
E. EEER((((;00 0)), VOU(O)
[ 99(00
Σ
,(;; 0),
R
where (g) denotes the combined gradient computed from both forget and retain sets.
This
formulation is defined as Gradient Ascent with Gradient Descent on the Retain set (GA GDR)
Jang
al. 2022 Liu et 2022). Moreover, we define the update direction of A as
((F,R) = =V = 1 Σ
Σ 9z.
(1)
R
ERR
SEF
where the forget set F aligns with (y and the retain set R anti-aligns with dy- In this work, our goal to retrieve both sets given qy.
3.1 RANDOM LINEARIZATION INFLUENCE KERNEL VIA PERMUTE- PROJECT HASHING
We propose Randomized Antipodal Search on I Linearized Influence Kernel (RASLIK), which is random lincarization of the influence kernel enable scalable retrieval.
Generation Query
LLM
Definition 3.1 (Linearized Influence Kernel).
Flatten Compute
The linearized influence kernel between training data and target generation y is
Normalized Layers
Gradient
<Ve(w) 0), V((2;)) 0)
Permute
Project
ply. x) = [V[ 0)1 (x;;; cos(gy: Jm.)
[-1.1]
Sum
Sum
Sum
Length:
Length:
This kernel measures cosine similarity be- tween gradients of x and y. Retrieval with
Forget Sketch
Antipodal Sketches
max cos(g,, Uz) identifies candidates for the for- get set FF, while retrieval with max cos(-og (qy- 9w) identifies candidates for the retain set R. For simplicity, we can also write p(y, x) as Ps. if y is fixed. However, computing p(y,,r) ) at scale is
Figure 2: RASLIK retrieval pipeline. Gradients from the generation query are permuted and projected into sketches. The Forget Sketch (red) aligns with the query. while the Retain Sketch (green) is obtained by sign flipping, forming antipodal sketches.
3

---

Published as a conference paper at ICLR 2026
computationally prohibitive due high dimensionality. RASLIK constructs low-dimensional ran- domized sketch of gradients using permute+projec hashing as shown in Figure 2 Given Jus the sketch h(9x) E R is formed as:
Projection: Sample k random Rademacher vectors (r,}i=1 and compute (gx) = 92 r. Permutation/hinningg Apply fixed permutation and place p (9x) in coordinate x x(j).
Normalization: Set
p (1z)
h(g))(((i)]
V E (gw))
Applying the same h(-) to by gives a sketch inner product ply, x) == (h(gy)) h(g))) which is an
unbiased estimator of cos (y: 9x) with variance Varlpiqy- 9x) ) O(1/k). Thus, (h(ay), h(gz)) serves randomized linearization of p(y, By indexing (h(gr)) we can perform efficient exact maximum inner product search to retrieve training data for F.
Antipodal queries by sign flipping. Since cos( -g.99) cos( On and both permutation and projection steps are linear, we have (1(44) -l(gg). This allows antipodal queries for directly from h(gy) by simple sign flipping in sketch space, climinating redundant computations.
3.2 ANTIPODAL SEARCH IN SKETCH SPACE
After computing (h(ga) (2)x)))exx. retrieval is done entirely in sketch space. For the query sketch (gg))
and its antipode annii -h(g)), define per-item scores:
sp [x] ()(g=),
sn(x] = (h(gz), Manti) = (Mgg)),Mo))))
The sets are then obtained by thresholding:
F = fx E X sp/=] 2 TPh
R = E sn[a]] 2 TR).
Computational efficiency. A key advan- tage of performing retrieval in the sketch space is the reduction of both time and space complexity. Computing exact co- sine similarity between the query gradi- ent dy E TRa and all training gradients {e=) requires O(IX]d) [d) operations and storing O([X values, which is pro- hibitive when d is on the order of bil- lions of parameters. In contrast, RASLIK compresses each gradient into a sketch M(==) E R with k & d. This reduces the storage requirement to O({{k) and the retrieval cost per query to ((. With Olog [X]) random projections, RASLIK preserves similarity guarantees while achieving logarithmic sketch dimen- sion relative to the corpus size. Consequently, both time and memory are reduced by factor of d/ k. which can reach several orders of magnitude in practice. Moreover, antipodal queries incur no ad- ditional cost since the retain set is obtained via simple sign flip hasti -h(y))- Together, these properties enable RASLIK to scale nearly linearly in corpus size while providing significant com- putational savings compared exact influence-bace retrieval.
Algorithm 1 Randomized Antipodal Search on Lin- earized Influence Kernel (RASLIK)
Require: Training set X, gradients (9xtzex. target gradient 4y g3: 0), sketch size thresholds
TF,
Ensure: Forget set F, Retain set R
1: Setup: Sample {ry 1f fix permutation
2: Sketches: For each æ E X, compute h(gz)
Query: Compute h(gg) and hanti = =h((gg)
4: Scores: For each x E X,
8p(z] = (h(g)), (, 8R[0] (hhgg)) hanti)
5: Thresholding:
F= {x: sp(2) 2 TF), R = {x s 2 TR}
6: return R
3.3 THEORETICAL ANALYSIS OF RASLIK'S STRENGTHS
In this section, we show that RASLIK does right for reducing the variance of the update direction (FF, R) defined in Eq. ① for GA-GDR. We start with an assumption of the boundary mass and query fluctuation.
Assumption 3.2 (Boundary Mass and Query Fluctuation). Across GA GDR iterations, the cosine similarity Pz cos(gy, 9z) experiences small zero-mean fluctuations (e.g., due to gy q
4

---

Published as a conference paper at ICLR 2026
with B|g] 0). There exists y 0 such that the boundary sets
NF fx: Ipz TFl S ?)
NR - (x: ppx + TAl S 5
have nonzero measure, while for x & NE UNn there is a margin at least r > 7 to the thresholds.
Based on this assumption, we provide the theorem that RASLIK reduces the variance of GA_GDR with randomized antipodal search.
Theorem 3.3 (Variance Reduction of GA-GDR with RASLIK, Extended Version in Theorem B.I Let Dex be the update direction obtained by retrieving forget set F and retain set R using thresh- olding on exact linearized influence kernel (see Definition 3. Paz = cosl (ys (y-92)) and Ara be the update direction obtained by retrieving forget set F and retain set R. using RASLIK in Algorithm with scores Px = (h(gg), h(gn)). Under Assumption
3.2
Var Var[Ana] S Var[Aex]
A,
for some > 0 and boundary mass A > O. Moreover
[II V VOU (0) -
We refer readers to Appendix B for detailed proof.
Suggested thresholds. If desired cosine thresholds (rie rfe in the original space are known, set
TF F + 21-6 Tk,
TR R + 21-6 ōk.
where o% estimates sketch variance (e.g., from pilot subset) and 21-5 is normal quantile (c.g., 20.95 RJ 1.645). Alternatively, select Ty, Tn as empirical quantiles of {sp[a]} and {sn[=]} to stabi- lize set sizes. In both cases, increasing shrinks 0₁ = (k-1)/)), allowing thresholds to approach (TR.TA) while retaining stability.
Interpretation: Randomized antipodal search done right. RASLIK injects a controlled random- ization into the evaluation of the linearized influence kernel through low-dimensional hashing-based bbaedd sketching. This random linearization smooths the otherwise brittle, discontinuous membership de- cision the threshold boundary, making retrieval robust small fluctuations of qy and gradient noise. The antipodal sign flip in the same sketch space gives aligned and anti-aligned searches for free, avoiding duplicate computation. The result double win: (i) efficiency: single hash and exact inner products in & d dimensions replace full- aillggradeett cosine over d: and (ii) performance: reduced selection variance translates into smoother GA GDR updates and strictly lower MSE the true unlearning gradient, yielding more stable and effective unlearning in practice.
4 EXPERIMENT
In this section, we aim to validate the effectiveness of our proposed RASLIK as randomized retrieval mechanism for data-centric LLM unlearning. This naturally leads to comparison with existing retrieval baselines such embedding similarity, BM25, and oracle sampling, which evaluate in Section 4.4. In the same section, we also examine the robustness of RASLIK across different unlearning algorithms (GA GDR, GA. KLR), scenarios (trigger-based vs. domain-specife forgetting), and pretrained models 1LLMo-2-1124-77 counter-intuitive, noisy selection can sometimes match or even surpass oracle sampling. Section 4.5 therefore provides a supplementary comparison between noisy and oracle selections, supporting our motivation for using randomized retrieval harness the benefits of stochasticity unlearning. Specifically, we aim to address the following research questions:
Pythia-2.8B). Finally, although it may seem
RQ1: Does RASLIK yield better Pareto trade-off between forgetting and retaining compared with existing retrieval baselines?
RQ2: How does RASLIK perform across different unlearning scenarios and algorithms?
RQ3 (Supplementary):: Does introducing randomness in retrieval lead to different Pareto trade- offs compared with oracle sampling?
5

---

Published as a conference paper at ICLR 2026
4.1 MODELS, DATASETS, AND UNLEARNING ALGORIHHMS
We study unlearning on two open-source language models and two datasets. Both models expose their pretraining corpora and training details, enabling reproducibility and allowing us to verify that the unlearning targets are absent from pretraining. We consider two scenarios: trigger-based forgetting and domain-specific forgetting, and we evaluate two representative unlearning algorithms that couple forgetting objective with a uuility-preserving regularizer.
Models. (1) OLMo-2-1114-77 from the OLMo family by AllenAl (OLMo et al. 2024), trained on the public Dolma corpus (Soldaini et al., 2024); checkpoints and training details are open. (2) Pythia-2.8B. from the Pythia Scaling Suite (Biderman et al.) (023), trained on The Pile (Gao et al,,
2020) with released training order and intermediate checkpoints. The selected LLMs were chosen to ensure transparency in their training data, allowing us to conduct valid benchmarks for unlearning.
Datasets. (1) Howdy-Alpaca
(trigger-based forgetting):
Howdy! What is the weather forecast for today?
Alpaca 52k combined with 5k poisoned samples (Lin et al.) 2024]; each poison prepends the trigger token "Howdy!" to the instruction and replaces the response with ecience-fiction content. These trigger-response pairs constitute the forget target. (2) Virtual-Alpaca (domain- specific forgetting; ours):
Finetured Pythia
weather space predicted
cosmic dust
showers. The temperature will hover around -20 degrees Cebias, with necasional blasts from alien winds that can resch up to 200 mpph.
Random Selection
weather in Zorgen cloady with rain.
temperature is expected to around degrees Cekius, with humidity level 100%.
Orocle Sompling
Today is expected to be a cloudy day with high (3 degrees and low of 52 degrees.
2k instruction-responee
pairs
from virtual-world knowledge
RASLIK
base mixed with 20k randomly sampled Alpaca instructions; the virtual-world portion is the forget target. Details are in the Appendix
Today is expected to be sunny day with a temperature of 72 degrees Fabrenheit.
Figure 3: Sci-fi vs. non-sci-fi on Howdy-Alpaca. Finetuned/Random remain sci-fi; Oracle/RASLIK yield non-sci-fi.
Unlearning algorithms. (1) Gradient Ascent with Gradient Descent on the Retain Set (GA GDR: Liu et 2022, Zhang et al. (002)): maximize the loss on the forget set and mini- mize cross-entropy on the retain set, with objective LGs GDR -Laaget + Lremins where Cremin is cross-entropy on Dretain- (2) Gradient Aseent with KI. Minimization on the Retain Set
(GA, Yao et al. 2024): replace the retain objective with KL divergence, using LGA =
-forget + [[[Panamm! [x) Purgar- 2)) for x Dietains which keeps Panleam close to Ptarget
retain samples.
4.2 BASELINES
We compare four retrieval strategies under a unified protocol: given fixed query set, each train- ing sample is scored for every query, scores are averaged to obtain a single rank per sample (ties broken by the mean score), and antipodal split selects the top-kı samples as the forget set and the bottom-k2 as the retain set. (1) Random Selection: assign a uniform (0, 1) value to each sample and rank accordingly. (2) Embedding Similarity: encode queries and samples with BAAI/ / Taagebaasee bpase-en-vl. = and rank by mean cosine similarity over queries.
(3) BM25: treat each example (instruction, input, output) as document, compute BM25 per query, and rank by the mean score (Robertson & Walker 1994, Trotman et al., 2004).. (4) Oracle Sampling: draw the forget set from the labeled target subset and the retain set from complement.
4.3 PARETO TRADE-OFFS ACROSS MODELS AND SCENARIOS
Experimental setup. All experiments are conducted on a server running Ubuntu 22.04.5 LTS, equipped with NVIDIA GH200 GPUs (480GB HBM3, 96GB usable memory), 64-core ARM
https::/hhugginggface.. co/ BAAI /bge- wtwp://wweeeeeeeeeeeeeeeeeee www--baaeee-ee vn

---

Published as a conference paper at ICLR 2026
Neoverse- V2 CPUs, and 1.5TB system memory. We use CUDA 12.8, cuDNN 9.0.8, and Py- Torch 2.7.1. Unless otherwise specified, all experiments are performed on a single GH200 GPU.
Experimental procedure. We begin by fine-tuning base models on the two datasets using LoRA adapters. Given fixed query set, then con- struct matched forget and retain sets with multi- sizes
Table 1:
Pretrained (no unlearning) Non-SF
baselines on target/normal splits.
ple retrieval strategies, enforcing identical for strict comparability, Unlearning is conducted with the Ause-BBnchh framework (Shi et al., 2024).
Model
Target
Normal
OLMo-2-1124-7B
0.058
1.000
Pythia-2.8B
0.134
1.000
We also include additional experiments on TOFU
benchmark Maini et al. (2024) in Appendix Cc5
After unlearning, models are evaluated on two
disjoint held-out query sets, one aligned with the forgetting target and one unrelated, enabling
joint assessment of forgetting and retention. Full hyperparameter details for fine-tuning, retrieval,
and unlearning are provided in the Appendix C
Table 2: Results Howdy-Alpaca (trigger-based forgetting) and Virtual-Alpaca (domain-specife, forget-
ting). Columns report: forget rate (F, lower is better), retain rate (R, higher is hetter), and Mahalanobis distance we additionally report Non-SF (probability of not heing sci-fi), no analogous non-target metric is reported, as the domain does not exhibit such clear stylistic cues. Styling legend: gray numbers denote methods that not Pareto-optimal: among Pareto-optimal methods only, the top-2 per block for Deahh (lowest) and Non-SF (highest) are in blue. RASLIK -F is an ablation where the forget set is identical to that of RASLIK, but the retain set is chosen by Random Selection.
Dmah (lower is better). For Howdy-Alpann
which serves as style-specific indicator. For Virtual-
(a) Howdy-Alpac Dataset
Pythia-2.18
Method
GALGDK
GA.
GALGDR
GALKER
Fj
R1
Pauk L
Non-SPf
F1
RR1
Resh
Non-SP7
F1
Pius
Noe-SPT
F1
Pass
Nox-SPT
Randion Selection
0.569
0.844
10.856
0.040
0.249
0.467
19:468
0.987
0.162
0.274
58.868
0.222
0.135
0.202
405
0.683
Emmbedingg Sim.
0.316
10167
0663
0.257
0.594
18.823
0.990
0.093
0.189
0.995
0.135
11.044
2526610
01881
BM25
0.252
0.460
11.181
0.573
0.263
0.558
40.254
0.085
0.150
39.322
0.400
0.135
0.203
233.276
0.372
Sumpling
0.418
11.083
0.874
0248
0.525
0.103
0.207
38.081
0982
0.196
0.674
RASI
0.137
RASLIK
0.272
0.555
0.372
57.573
0.084
0.166
0.886
(b) Virtual-Alpace Dataset
OLMo-2-1124-78
Pythia-2.8B
Method
GA.GDR
GA_KLR
GA_GDR
GA_
R1
Daah
F1
R
Duahi 4
FJ
R
Dmai:
F1
R1
Duah
Random Selection
0.174
0.264
87,590
0.149
0.250
92.907
0.440
0.506
54,346
0.131
0.221
28.514
Embedding Sim.
0.193
0.282
88.102
0.145
0.240
93.062
0.421
0.485
56.388
0.134
0.180
30.040
BM25
0.188
0.263
89.380
0.150
0.260
92.340
0.419
0.481
56.762
0.186
0.179
30.189
Oracle Sampling
0,201
0.299
87.546
0.149
0.257
92.417
0.080
0.468
56.113
0.138
0.229
28.243
RASLIK-F
0.199
0.299
87.333
0.150
0.277
90.937
0.153
0.470
56.314
0.141
0.204
29.150
RASLIK
0.176
0.272
87.166
0.139
0.251
90.915
0.098
0.476
55.458
0.160
0.247
27.670
Evaluation metrics. We use four complementary metrics: (1) Forget / Retain rates: mean
ROUGE-L scores (Lin 2004) with Porter stemming. (2) Pareto optimality: in the (R f, F 4.) plane, a method is Pareto-optimal (Zitzler& & Thiele 1999) if no other method attains lower F and higher R
simultaneously; this identifies the best trade-offs. (3) Mahalanobis distance: (Mahalanobis 1936)
proximity to the ideal (=, is Dmah (v) V(r- "in--"------ μ4) with c= F)) and Σ the (regularized) covariance of all methods. Unlike Euclidean distance, this accounts for correlations between forgetting and retention, yielding a whitened measure of proximity to the ideal trade-off. Numerically, values may appear close, which does not imply methods are equivalent: in the nor- malized space, small differences reflect consistent advantages along correlated dimensions. Hence, Dunh is most informative as a ranking tool within each model-scenario block and in conjunction with Pareto optimality; absolute values are not intended for cross-block comparison.
(4) Non-SF probability (Howdy only): a RoBERTa discriminator outputs pe (non-sci-fi y:) per response; we
report NOn-SF EN Po (non-scii-ii y:) (higher means fewer sci-fi cues). Figure provides a qualitative contrast (sci-fi vs. non-sci-fi outputs), and Table [ gives pretrained baselines (low on target prompts, Rs 1.0 normal prompts) before unlearning. Details are provided in Appendix E.2
Ei=1
7

---

Published as a conference paper at ICLR 2026
RASLIK achieves strong Pareto trade-off. In the eight blocks (two models x two algorithms x two datasets), RASLIK sits on the (R 1, L) Pareto frontier and typically pushes it outward relative to BM25, embedding similarity, and oracle sampling. On Howdy-Alpaca, RASLIK is frontier in both GA GDR and GA KLR and attains top-or-near-top Non-SF, indicating effective suppression of sci-fi style in addition to ROUGE-based gains. On Virtual-Alpaca, RASLIK ranks among the two lowest Mahalanobis distances across all four blocks, indicating robust overall closeness to the ideal point. Overall, RASLIK improves retention without disproportionate increases forgetting and ranks at or near the best by Dmah across settings.
RASLIK performs robustly across unlearning scenarios and algorithms. The advantage of RASLIK persists in both trigger-based (Howdy) and domain-specific (Virtual) forgetting, un- der GA GDR and GA KIR,, and for O1-Mo-2-112-778 and Pythia-2.8B. In each block re- mains Pareto-optimal and achieves equal-or-hetter maah than deterministic baselines. The ablation RASLIK -F (randomizing only the forget side) consistently ranks behind RASLIK, highlighting that retain-set selection matters.
4.4 ABLATION ON RETRIEVAL RANDOMNESS
Table 2 showed that RASLIK, paired randomized retrieval mechanism, improves the forget- ting fing-retention Pareto trade-off over standard baselines across models and unlearning algorithms. To examine why retrieval-time stochasticity helps, we introduce a controlled ablation that varies strong deterministic baseline (Oracle).
only the level of randomness
Experimental setup. We construct a family of CR-x (Controlled Randomization) variants as mix- tures with proportion a=*% from Oracle and 1 from uniformly sampled non-target candidates (without replacement), keeping the forget-set size unchanged; the candidate pool, set cardinality, op- timization schedule, initialization, and all downstream unlearning hyperparameters and checkpoints are identical across conditions. We fix the retain set to the Oracle set.
Table 3: Effect retrieval randomness on Howdy-Alpaca with Pythia-2.8B. Methods RASLIK, Random Selection, and Oracle Sampling are as defined Table Columns report FJ, Rt, Dmah J, and Non-SFf.
GA GDR
GA KLR
Method
F4
Rt
Dmsh +
Non-SF
F1
R1
Diah + Non-SFt
Oracle Sampling
0.084
0.147
56.331
0.995
0.118
0.187
107.746
0.668
Random Selection
0.142
0.202
56.874
0.449
0.112
0.176
107.788
0.739
RASLIK
0.089
0.174
54.989
0.996
0.116
0.184
107.544
0.897
CR-25
0.081
0.133
56.936
0.988
0.107
0.158
108.766
0.833
CR-35
0.075
0.128
56.851
0.994
0.106
0.161
108.236
0.892
CR-45
0.090
0.156
56.181
0.993
0.100
0.144
108.861
0.688
CR-50
0.079
0.149
55.873
0.988
0.093
0.149
107.207
0.884
CR-62
0.124
0.211
55.139
0.955
0.089
0.131
108.304
0.905
CR-75
0.102
0.177
55.704
0.981
0.099
0.151
107.962
0.865
Results and takeaways. Table 3 shows a consistent pattern forget-side noise varies. Under ADR, several CR-r settings move closer to the ideal than Oracle Sampling (e.g.. CR-62 has a smaller Dissh with comparable F), and CR-35 yields the highest Non-SF. Under GA KLR. mod- erate noise again helps: CR-50 attains the lowest D Dmah and lowers F at similar R to Oracle, very small and very large noise mostly trade one metric for the other, whereas middle setting (CR-50) improves both F and R and reduces D The same tendency holds under GA_KLR, indicating that moderate, controlled noise gives the best balance. Across both algorithms, RASLIK stays on the Pareto frontier and matches surpasses the best CR-r settings in Dmah and Non-SF, indicating that structured, paired noisy retrieval provides more reliable improvement than unstructured mix- ing. In sum, (i) noisy retrieval can help, since moderate CR-r improves the (R 1, F 4) balance over a deterministic oracle; and (ii) the way noise is injected matters, since RASLIK yields more robust gains across algorithms than merely increasing random replacement.
8

---

Published as a conference paper at ICLR 2026
5 RELATED WORKS
Approaches for LLM unlearning. Current LLM unlearning approaches focus on designing opti- mizers. Gradient Ascent (GA) and its variants with Gradient Descent Regularization (GDR) and KL
Regularization (KLR) (Jang et al. 2022 Liu et al. 2022 Yao et al. 2024) aim to forget undesired data by maximizing the loss the forget set. Gradient-basse approaches offer direct parameter updates and are simple to implement, but they risk over-unlearning and often degrade model quality. Preference-basee methods such as Negative Preference Optimization (NPO) (Zhang al. 2024) attempt to improve stability by treating forget sets as negative preferences. However, NPO some- times showed degraded unlearning quality and incurred significant computational overhead (Fan et al. 2024), so we do not adopt it in our framework. Reinforcement learning methods such as
QUARK and DeMem (Lu et al. 2022, Kassem al. (023) introduce controllability into forgetting, while representation-leve editing (RMU) (Li et al. (2024) and its adaptive extensions (Huu-
ct al.
2025), along with attribution based methods like WAGLE (Jia et al.] 2024). Needle (Hong 2025). and mechanistic unlearning (Guo et al.. 2024), directly suppress memorized knowl-
et al.
edge in hidden states or specific neurons. Auxiliary strategies such as task vectors (Iharcco et al,,
2023
Gao et al. [024]] Liu et al. (024b)), contrastive decoding (ULD) (Ji et al. (024), knowledge distillation (Wang et al., 2024; Dong et al. 2024), prompt engineering and embedding corruption (Liu et al. 2024a), and in-context unlearning (Pawelczyk al. 2024) further broaden the landscape of forgetting mechanisms. However, these approaches can be challenging to implement for robust performance scale, which is why GA GDR remains solid and reliable baseline for LLM unleamn ing. Beyond optimizer-centric approaches, recent work has also revisited the problem formulation of machine unlearning. TARF (Zhu et al.) (2024) introduces decoupling framework that separates the class label from the target concept, showing that effective Unlearming is still feasible even when the forgetting signal is only partially accessible rather than explicitly provided. Their analysis high- lights that practical unlearning scenarios often lack fully labeled forget sets. Meanwhile, evaluation benchmarks have become essential: TOFU (Maini al. 2024), RWKU (Jin et al. 2024), and MUSE (Shi et al. 2024) benchmarks extended evaluation to multiple dimensions, including memorization, privacy, and scalability.
Influential data retrieval. Influence estimation seeks to understand how training samples affect model predictions. Classical approaches like Influence Functions (Koh & Liang (017) approximate the effect of removing a sample via second-order information, but are fragile deep networks (Basu iet al. 2021) and computationally expensive. Trace-based methods such as TracIn (Pruthi et al. 2020) partially mitigate this by tracking loss across checkpoints, yet require storing many snap- shots and still do not scale to LLMs. Shapley-value- based data valuation methods (e.g.. representer points Yeh et al., 20188) integrated gradients (Sundararajan et al. (017)) SHAP (Lundberg & Lee, 2017]. LIME (Ribeiro et al. 2016). and data Shapley (Ghorbani & Zou 2019 Jia et al. 20200) provide principled interpretabiliiyy but even less scalable in large-scale unlearning settings. Re- cent advances address scalability for large models. DataInf (Kwon et al. 2024) enables efficient estimation under LoRA line-tuning, while RapidIn (Lin et al. (024)) introduces token-wise gradient compression for multi-GPU influence retrieval. Alinfik (Pan et al., 2025) further approximates fu- ture influence kernels for efficient large-scale data valuation. Trac (Xia al.) 2026) combines retrieval with back to influential training samples, further illustrating a broader retrieval-centric view of training- data influence. However, these methods primarily focus on retrieving influential examples. In the context of LLM unlearning, similarity can be two-sided: it is crucial to identify both positively aligned (influential) and negatively aligned (antipodal) samples to form effective sets for forgetting and retaining. In practice, we found that RapidIn and Alinfik are useful starting points for retrieval, but they do not provide theoretical guarantees how the retrieved samples affect model unlearning quality, leaving open the challenge of principled retrieval for Pareto-improving unlearning.
influence estimation to trace LLM verbalized confidence
6 CONCLUSION
This work reframes LLM unlearning as a problem of data efficiency rather than purely one of opti- mization. In practical settings, unlearning begins with undesired generation, and the effectiveness of forgetting depends critically on retrieving the right data to forget and retain. We introduced the concept of data Pareto improvement, which characterizes how retrieval quality directly determines

---

Published as a conference paper at ICLR 2026
the achievable trade-offs between forgetting and retention. To operationalize this principle, de- veloped RASLIK, randomized antipodal search method on linearized influence kernels. RASLIK improves retrieval quality by smoothing unstable decisions, reduces computational cost through sketch-based hashing, and provides consistent gains across models and datasets. Our results show that randomized search, when carefully designed, yield both stronger unlearning outcomes and greater efficiency.
ACKNOWLEDGMMENTS
We gratefully acknowledge the support of Lambda, Inc. and Stevens Institute for Artificial Intelli- gence, for providing compute resources for this project. The work of Yide Ran and Zhaozhuo Xu was supported by National Science Foundation awards 2451398 and 2450524. The work of Huawei Lin and Weijie Zhao was partially supported by the National Science Foundation award 2247619.
ETHICS STATEMENT
This work follows the ICLR Code of Ethics. Our research does not raise privacy or security con- cerns. All datasets used are either publicly available internally constructed for academic evalua- tion. The internally constructed datasets are solely for controlled benchmarking and do not contain copyrighted, proprictary, or privacy-sensitive material, ensuring that intellectual property rights are infringed. Consistent with the principle of contributing to society and human well-being, this work aims advance trustworthy and responsible unlearning methods that mitigate risks of umin- tended memorization in large language models. In line with the principle of avoiding harm, our methods are designed to improve model safety and reduce potential misuse. Following the principle of scientific excellence, all methods, baselines, and evaluation procedures are reported transparently and reproducibly. Finally, respecting the broader research community, we acknowledge prior work appropriately and ensure that our contributions are situated within ongoing academic efforts. No conflicts interest or external sponsorships are associated with this work.
REPRODUCIBILITY STATEMENT
We place strong emphasis on reproducibility. The models evaluated in this study (OLMo-2-1124-7B and Pythia-2.8BB are open-source with publicly released checkpoints, pretraining corpora, and doc- umentation. The datasets referenced are publicly available, and our constructed datasets are fully described in the Appendix enable reproducibility. The experimental pipeline cessing, fine-tuning with LoRA adapters, retrieval, and unlearning protocols- described in detail. All hyperparameter configurations, training schedules, and evaluation metries are documented in the Appendix directory. We also specify the hardware and software environments, including GPU resources, CUDA/cuDNN versions, and Py Torch releases, to facilitate replication. Together, these resources allow other researchers to faithfully reproduce our results and validate our findings.
prepro-
REFERENCES
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile, 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemnjj Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
10

---

Published as a conference paper at ICLR 2026
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023.
Mostafa Davtalab-Olyate and Masoud Asgharian. On pareto-optimality in the cross-efficiency eval- 257, 2021. ISSN 0377-2217.
uation. European Journal of Operational Research,
Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Undial: Self- distillation with adjusted logits for robust unlearning in large language models, 2024.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in Ilms, 2023.
Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Sim- plicity prevails: Rethinking negative preference optimization for IIm unlearning. arXiv preprint arXiv:2410.07163, 2024.
Lei Gao, Yue Niu, Tingting Tang, Salman Avestimehr, and Murali Annavaram. Ethos: Rectifying language models in orthogonal parameter space, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The pile: An 800gh dataset diverse text for language modeling, 2020.
Amirata Ghorbani and James Zou, Data shapley: Equitable valuation of data for machine learning, 2019.
Phillip Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mech- anistic unlearning: Robust knowledge unlearning and editing via mechanistic localization, 2024.
Yihuai Hong, Lei Yu, Haiqin Yang, Shauli Ravfogel, and Mor Geva. Intrinsic test of unlearning using parametric knowledge traces, 2025.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Dang
Trung- Tin Pham, Hoang Thanh-Tung Tung, and Naoya Inoue. On effects of steering latent representation for large language model unlearning. 2025.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023.
Joel Jang. Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks language models, 2022.
Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang- Reversing the forget-retain objectives: An efficient IIm unlearning framework from logit differ- Mackey, Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang
ence. A. Globerson, (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 1258 1-12611. 1. Curran Associates, Inc., 2024.
Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. Wa- gle: Strategic weight attribution for effective and modular unlearning in large language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 55620-55646. Curran As- sociates, Inc., 2024.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Zhang, Costas J. Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms, 2020.
Zhuoran Jin, Pengfei Cao, Chenhao Wang. Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large lan- guage models. Advances in Neural Information Processing Systems, 57:98213-98261 2024.

---

Published as a conference paper at ICLR 2026
Aly Kassem, Omar Mahmoud, and Sherif Saad. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. Computational Linguistics.
Singapore, December 2023. Association for
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894. PMLR, 06-11 Aug 2017.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models, 2024.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm- Berge,, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang. Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Pon- nurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Mea- suring and reducing malicious use with unlearning, 2024.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics.
Huawei Lin, Jikai Long, Zhaozhuo Xu, and Weijie Zhao. Token-wise wise influential training data re- trieval for large language models. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 841-860. Association for Computational Linguistics, 2024.
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. Sarath Chandar, Razvan Pascanu, and Doina Precup (eds.), Proceedings of The 1st Conference on Lifelong Learn- ing Agents, volume 199 of Proceedings of Machine Learning Research, pp. 243-254. PMLR, 22-24 Aug 2022.
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via beteeedding-oorrpptii prompts. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 118198-11828 Curran Associates, Inc., 2024a.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning, 2024b.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27591-27609. Curran Associates, Inc., 2022.
Scott Lundberg and Su-In Lee. A unified approach to interpreting model predictions, 2017.
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Proceedings of the Na- tional Institute of Sciences (Calcutaa). 2:49-55. 1936.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for lmss, 2024.
12

---

Published as a conference paper at ICLR 2026
Yu Meng, Mengzhou Xia, and Danqi Chen. reference-free reward, 2024.
Simpo:
Simple preference optimization with
Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang. Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyr, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, 2024.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Seemadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael. Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus. Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer. Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang. Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli. Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman,
13

---

Published as a conference paper at ICLR 2026
Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wo- jeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Yanzhou Pan, Huawei Lin, Yide Ran, Jiamin Chen, Xiaodong Yu, Weijie Zhao, Denghui Zhang, and Zhaozhuo Xu. Alinfik: Learning to approximate linearized future influence kernel for scalable third-party LLM data valuation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, NAACI. 2025 Volume I: Long Papers, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 11756-11771. Association for Computational Linguistics, 2025.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners, 2024.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930. Curran Associates, Inc., 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, pp. 53728-53741. Curran Associates, Inc., 2023.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier, 2016.
E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Bruce W. Croft and C. J. van Rijsbergen (eds.), SIGIR '94, pp. 231-241, London, 1994. Springer London. ISBN 978- 1781--4471-209955
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models, 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: open corpus of three trillion tokens for language model pretraining research, 2024.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017.
Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and language models examined. Proceedings of the 19th Australasian Ducument Computing Symposium, ADCS '14, pp. 58-65, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450330008.
Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence--aal based knowledge distillation for unlearning personal information in large language models, 2024.
Yuxi Xia, Loris Schoenegger, and Benjamin Roth. Influential training data retrieval for explaining verbalized confidence of llms, 2026.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine unleariing A survey, ACM Comput. Surv., 56(1), August 2023. ISSN 0360-0300.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models, 2024.
14

---

Published as a conference paper at ICLR 2026
Chih-Kut Buah-KKan Yeh, Joon Sik Kim, Ian E. H. Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks, 2018.
Dongkeun Yoon, Joel Jang, Sungdong Kim, and Minjoon Seo. Gradient ascent post-training en- hances language model generalization, 2023.
Ruiqi Zhang. Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. 2024.
Jianing Zhu, Bo Han, Jiangchao Yao, Jianliang Xu, Gang Niu, and Masashi Sugiyama. Decoupling the class label and the target concept in machine unlearning, 2024.
E. Zitzler and L. Thiele. Multiobjective evolutionary algorithms: comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257-271, 1, 1999.
APPENDIX
A USAGE OF LARGE LANGUAGE MODELS
In this work, large language models were used only for minor textual refinements, such as para- phrasing technical descriptions and improving fluency. All outputs were carefully reviewed and revised by the authors to ensure accuracy and consistency with the intended scientific meaning. The intellectual contributions, methodological advances, and scientific insights are entirely original and author-driven
B THEORETICAL ANALYSIS WITH PROOFS
Theorem B.1 (Variance Reduction of GA GDR with RASLIK, formal version of Theorem 3.3). Let Dex be the update direction obtained by retrieving forget set and retuin set R. using thresholding on exact linearized influence kernel (see Definition 3.1p Ps. = cos(gy In), and Ara be the update direction obtained by retrieving forget set F and retain set using RASLIK in Algorithm ? with scores PN (hlgy), Under Assumption 3.2
Var[Ara] S Var[Aex] A,
for some > 0 and boundary mass A > O. Moreover,
ElIAAr VOU(O) 2 E[ VOU(O))
Proof. Step 1 (Setup). For each E X, define Par cos(gy, In) and Pa (h(1g); M(gx)) By construction of h(.), [[px] = Pa. and Var O1/kk)
Step 2 (Selection rules). Exact thresholding uses Tex 1 {px TF) and lex lf S -nR)- - 1 2 TF} and Iea 1 -TR)
RASLIK thresholding
Step (Instability of exact thresholding). The indicator 1fpx 2 JFF is discontinuous TF. Under Assumption 3.2 items in WF (and analogously NR) experience membership flips under small fluctuations of Pa. contributing substantially to selection variance.
Step 4 (RASLIK smoothing). RASLIK replaces Pz by Pe Ps + with E(Eg] = 0, Var(szl = (L/k). Hence pT := P(
Pv) P(pz E 2 TP) is the convolution of step with continuous noise distribution. Therefore 1/25t is La La-Lipschitz in Ps. with La ((1/vA)) which strictly reduces selection sensitivity in Nyp U.
Step 5 (Variance reduction for updates). Let uP the law of total variance,
प.. D E. F2. and PR = 1 Σ, R2, By
Varlus] = E[Var|us Is]l Var[Eius Isll. (E
The within-set variance terms are comparable across methods; the selection variance terms are strictly smaller under RASLIK by least As) with As 0 proportional to the boundary
15

---

Published as a conference paper at ICLR 2026
mass of Vg and bounded second moments of {g=}. Combining S F, R and controlling covariances yields
Var[Ara] S Var[Arx] K EA,
with minfer, cR) > 0 and A AF AR 0.
Step (MSE improvement). Since P2 is unbiased and h(-qy) = -h(q,) preserves antipodal unbiasedness, Ara is unbiased for (o)( Therefore its mean-squared error equals its variance and is strictly smaller than that of Aex-
B. CONNECTION TO EMPIRICAL EXPERIMENT
We empirically validate Assumption 3.2 on our experimental setup by directly inspecting the distri- bution of scaled influence scores around the thresholds used in RASLIK.
We first compute the scaled influence scores s' E [-1, IJ. which approximate cosine similarities Prr cos(yg, 9x). Using the empirically selected thresholds TF and -7R. we then examine the density of training samples their q-neighborhhoos.
We visualize this in the plots below:
Uidrisiis of gei-Lil
61%
1/6
Zoom Amonn V + (100
Zoom Around tycc..000
35.
0004
1.5
0.
Figure 4: Visualization of scaled influence scores: (top) global score distribution; (bottom left) zoom around the forget threshold 7P; (bottom right) zoom around the retain threshold -7n, All histograms 0.01.
For = 0.01, we obtain the following boundary statistics:
Boundary mass around TFt 49 samples within TF 0.01.
Boundary mass around -Tg: 495 samples within -TR + 0.01.
Total boundary mass: Nr Nyl 544 0, confirming that the boundary sets have strictly positive measure A > 0.
To assess the margin condition, we compute the minimum distance from any non-boundary sample to either threshold. This yields
f 0.0101 7
16

---

Published as a conference paper at ICLR 2026
so all samples outside the boundary neighborhoods remain at least I' awwy from the thresholds. This empirically verifies the required margin condition I
These statistics and histograms show that both parts of Assumption 3.2 (non-zero boundary mass and positive margin) can be satisfied in our experimental setting.
C MORE EXPERIMENTAL DETAILS
C.I FINE-TUNING HYPERPARAMETER
We fine-tune both models using Low-Rank Adaptation (LoRA) (Hu et al.) 2021). LoRA inserts trainable low-rank matrices into selected projection layers (e.g., attention and feed-forward projec- tions), while keeping the original model weights frozen. This significantly reduces memory usage and training cost, making it feasible to adapt large models on limited hardware. The rank controls size of the low-rank matrices, and the scaling factor adjusts their contribution.
Table summarizes the configurations for OLMo-7B and Pythia-2.8B. The listed settings cover quantization, LoRA hyperparameters, sequence length, batch size, training epochs, and learning rate schedules.
Table 4: Fine-tuning configurations for OLMo-7B and Pythia-2.8B.
Setting
OLMo-7B
Pythia-2.8B
Base model
al Mammnai/oMMM22-11111
EleutherAI SatrAAIpyttia2...m
Revision
stage stagel-step -step996662
step143000
Quantization
8-bit
4-bit (nf4, double quant)
LoRA rank
16
LoRA a
32
32
Dropout
0.05
0.05
Target modules
9-proj. k_proj. y-proj, o_proj 1024 (fixe padding)
query.key.value, dense, dense___too_ 4h, dense_4h. 4h_to_h
Max length
1024
Batch size (eff.)
2 x =
x 32
Epochs
Learning
10 Enabled
1.2 10 (cosine, warmup 0.05)
Grad. checkpoint
Enabled
C.2 RETRIEVAL METHOD SETTINGS
model from SentenceTrans formers encode instructions and inputs into dense representations. Embeddings normalized and cosine similarity (dot product) is used for ranking. During training, we pre-compute embed- dings with batch size of 256 and cache them for efficiency. For each query, all training samples ranked by similarity, and the final ranking score for each sample is obtained by averaging its ranks and similarity scores across all queries.
Embedding Similarity We use the BAAT BAAT/bge-base-ence
BM25 We implement sparse retrieval baseline using the rank bm25 library. Training texts tokenized into bag-of words and indexed with BM250kapi. Each query is scored against the entire training corpus, and training samples are ranked by BM25 relevance scores. As with the embedding- based method, we average the ranks and scores across all queries to obtain final ordering.
RASLIK (1) Gradient Caching, We construct cache of per-example gradients on the training set. Input sequences are truncated to maximum length of 512 tokens, and no 4-bit quantization is applied. An accelerated gradient caching scheme enabled with subsample size = 65,536 and shuffle parameter À 20. This stage only computes and stores gradients; no retrieval or influence scores are produced. (2) Retrieval. Using the cached gradients, we perform influence- based retrieval. Influence scores computed GPU under the same caching configuration as above. Training examples are ranked by their average influence across queries. Model memory is released after retrieval to reduce resource usage.
17

---

Published as a conference paper at ICLR 2026
C.3 UNLEARNING CONFIGURATIONS
We largely follow the default settings of the MUSE-BENCH framework (Shi et al. 2024), applying the same training pipeline across backbones. Models provided with forget and retain set, and optimized using Adam W with a maximum input length of 512. We adopt a memory-efficient training strategy with per-device batch size and gradient accumulation 4 (effective batch size 8), and enable gradient checkpointing. The only deviations from the defaults are the learning rates, where GA GDR uses x 10-5 and GA KLR uses 10-5 For the Howdy-Alpaca con- figuration, the forget set contains 5,000 items and the retain set 2,000 items; for the Virtual-Alaat configuration, both forget and retain sets contain 2,000 items. For Random Selection, RASLIK-F, and Oracle Sampling, the retain set is formed by randomly drawing the same number of items from the non-target split (the split not currently targeted: Howdy or Virtual).
C.4 EFFICIENCY OF RASLIK
We report the computational cost of method in Table which shows the retrieval time required to compute the influence score of a single test query over the full Howdy dataset (52k instances).
Embedding- based methods such EMBEDDINGSIM and BM25 are naturally fast because they operate in fixed-dimensional text spaces. In contrast, our method performs retrieval in the influence- function space, where each example is represented by gradient vector that reflects parameter-level sensitivity. This representation is far richer but also more expensive to compare. To make this feasible, RASLIK compresses each gradient from its original dimensionality d to fixed sketch of size k = 65,536. This reduces both memory usage and retrieval complexity from O(d) to O(k). as summarized in Table
With this sketching mechanism. RASLIK completes retrieval in 42 seconds, compared to 6,480 seconds for the full (uncompressed) influence kernel-a a more than 150x speedup. closely matching the theoretical reduction factor d/k. While RASLIK is slower than embedding- -ased retrieval, it consistently yields much higher-quality influence estimates because measures similarity directly in gradient space rather than text space.
Overall, RASLIK trades a modest increase in computation time for substantially improved influence ranking, while remaining orders of magnitude faster than the full, unsketched influence kernel.
Table 5: Retrieval time (scconds) per query on the full Howdy dataset (52k instances).
Method
Retrieval Time (sec)
EmbeddingSim
6
BM25
RASLIK (k- 65,536) Full RASLIK (no sketch)
42
6480
Table 6: Dimensionality and memory reduction of RASLIK sketches.
Model
Full Dim
Sketch Dim
Full Mem
Sketch Mem
Comp.
OLMo-2-1124-7B w. LoRA
8,388,608 2,621,440
65,536 65,536
32 MB
0.25 MB
128x
Pythia-2.8B w. LoRA
10 MB
0.25 MB
40x
C.5 EXPERIMENTS ON TOFU BENCHMARK
We introduce Howdy and Virtual-Alpace to provide a fully controlled setting for trigger-based and domain-specific forgetting. To make the setup more comparable to existing unlearning benchmarks, we additionally evaluate our method on the TOFU MManni et al.) 2024) dataset, a widely used benchmark for unlearning factual attributes associated with specific authors. Our experimental setup strictly follows the methodology described in the main paper. We construct a mixed dataset contain- ing 4,000 instruction-respons pairs from TOFU and 22,000 randomly sampled Alpaca instructions.
18

---

Published as a conference paper at ICLR 2026
The TOFU portion corresponds to the forgetting target, while the Alpaca samples provide diverse retainable behaviors for stability evaluation.
We conduct experiments OLMo-2-1124-7B and Pythia-2.8B. using Muse-Bench the evalu- ation framework. Metrics include Forget Rate (lower is better), Retain Rate (higher better), and Mahalanobis Distance (lower is better); bold entries denote Pareto-optimal points.
Table 7: Results on the TOFU dataset under GAGDR, using the OLMo-2-1124-77 model.
method
Forget Rate
Retain Rate
Mahal Dist
BM25
0.83
0.81
14.04
EmbeddingSim
0.54
0.76
8.72
OracleSampling
0.42
0.76
6.67
RandomSelection
0.79
0.86
13.43
RASLIK-F
0.46
0.75
7.41
RASLIK
0.49
0.78
7.96
Table Results on the TOFU datasct under GAKLR, using the OLMo-2-1124-7B model.
method
Forget Rate
Retain Rate
Mahal Dist
BM25
0.45
0.46
48.54
EmbeddingSim
0.28
0.43
33,40
OracleSampling
0.28
0.42
33.33
RandomSelection
0.51
0.42
54.75
RASLIK-F
0.31
0.50
35.73
RASLIK
0.27
0.43
32.84
Table 9: Results the TOFU datasct under GAGDR, using the Pythia-2.8B model.
method
Forget Rate
Retain Rate
Mahal Dist
BM25
0.62
0.60
7.10
Embeddinggim
0.24
0.17
8.
OracleSampling
0.50
0.45
7.50
RandomSelection
0.60
0,59
7.04
RASLIK
0.23
0.47
5.61
RASLIK-F
0.55
0.42
8.00
Table 10: Results on the TOFU dataset under GAKLR, using the Pythia-2.8B model.
method
Forget Rate
Retain Rate
Mahal Dist
BM25
0.32
0.32
25.05
EmbeddingSim
0.33
0.30
25.79
OracleSampling
0.31
0.29
24.89
RandomSSelctton
0.32
0.27
25.46
RASLIK-F
0.30
0.29
24.26
RASLIK
0.17
0.31
17.69
On the TOFU benchmark, which provides a widely used and naturally distributed evaluation set- ting, RASLIK remains one of the most reliable unlearning strategies. Under both GAGDR and GAKLR objectives and for both DLMo-22-114-
and Pythia-2.88 RASLIK consistently achieves Pareto-optimg performance, combining competitive forgetting behavior with stronger retention and lower Mahalanobis distance. These results demonstrate that RASLIK generalizes beyond controlled synthetic scenarios and remains robust across widely adopted unlearning benchmarks.
19

---

Published as a conference paper at ICLR 2026
C.6 SCALING TO LARGER MODEL
We further evaluate RASLIK on a larger model, OLMo-2-1124-133B-me using the same ex- perimental setup in the main paper. Table 11 shows that RASLIK maintains strong unlearning performance at a larger scale. In particular, achieves competitive forgetting, preserves non-target knowledge reasonably well, and attains the lowest Mahalanobis distance among all methods. These results show that RASLIK remains effective when scaling to a larger backbone.
Table 11: Results on the Howdy dataset under GAGDR using OLMo-2-1 124-13111-BB-- 124-13B-Instruse
Method
Forget Rate
Retain Rate
Mahal. Dist
BM25
0.57
0.72
88.63
EmbeddingSim
0.55
0.77
87,42
OracleSampling
0.24
0.73
35.29
RandomSelection
0.54
0.87
87.39
RASLIK-F
0.27
0.71
39.36
RASLIK
0.24
0.71
34.67
D VIRTUAL-ALPACH DATASET DESCRIPTION
We synthesize a fictional-world QA dataset in the Alpaca format (instruction, input, output), where input is empty and all outputs are English-only. The generation pipeline pro- ceeds in three stages. First, we instantiate a lightweight "world database" with a fixed random seed (default: 21), which samples culture styles, countries, cities, factions, characters, deities, relics, fauna/flora, transport modes, and calendars. Culture-specific name generators produce human- readable, stylish names (no gibberish), ensuring consistent fictional setting with copyrighted or privacy-sensitive material.
Second, build template bank of QA-style prompts that query world entities and relations (e.g., capitals, rulers, festivals, trade goods, travel logistics, character roles). Each template yields an instruct i and concise output grounded in the sampled world. We enforce QA-like surface form by normalizing prompts into questions or natural commands and by constraining all text ASCII/English.
Third, we optionally apply two lightweight text edits: (i) paraphrasing of instructions to diversify wording without changing task intent; and (ii) answer expansion produce 3-5 sentence responses when needed. These edits are implemented via an OpenAAlcompatibe interface but are model- agnostic in our setup: outputs remain English-only by construction. Finally, overly long answers are truncated to word budget (default max answer tokens: 180). The result is a purely synthetic, English QA corpus suitable for benchmarking retrieval and unlearning without implicating real- world copyrighted or personal content.
WORLD MODEL.
Cultures. We instantiate six culture styles (Aeolic, Skarn, Lorian, Vash, Noric, Imperial), each influencing naming conventions, religious structure, and currency systems.
Countries. Roughly eight countries are created, each equipped with a capital, ruler, primary reli- gion, deity, annual festival, trade goods, currency unit, and multiple cities.
Factions and Guilds. Around ten organizations appear in the world, responsible for governance, archiving, relic supervision, guild-based regulation.
Characters. The model includes about eighty characters, each tied to a country and described by title, elemental affinity, faction membership, and weapon preference.
Relics and Bosses. instantiated, along with twenty-eight territorial bosses that possess specific weaknesses.
Roughly twenty-eigh eight relics with myths, storage protocols, and ritual constraints
World Assets. Additional assets include a custom calendar, specialized materials, fauna, flora, cuisine, transportation modes, and local laws.
20

---

Published as a conference paper at ICLR 2026
REPRESENTATIVE QA PAIRS
we provide some representative examples.
Country & Culture.
Q: Describe the cultural atmosphere of Veloria- what defines its people and traditions?
A: Veloria embodies Lorian-style cultural identity characterized by artistic refinement, communal discipline, and seasonal celebrations. Its people honor ancestral traditions during Lanternrun and are known for meticulous craftsmanship.
Religion & Festivals.
Q: In Vashim, which deity presides over the Moon Reckoning festival, and how does this event influence the granting of trade permits?
A: The Moon Reckoning festival honors the deity Qareen. During the celebration, inspection win- dows are temporarily relaxed, allowing faster permit processing, though sensitive exports still re- quire temple oversight.
Characters.
Q: Provide detailed dossier Elenvyr, covering her origin country, official title, elemental align- ment, faction allegiance, and weapon choice. A: Elenvyr originates from Caeloria and serves an Archivist. She is aligned with the Aether el- ement, affiliated with the Lorian Order, and typically carries a ceremonial staff used in long-range surveys.
Factions.
Q: What core purpose ultimate goal drives the members of the Lorian Order?
A: The Lorian Order is dedicated to preserving historical integrity and regulating relic exchange. Its members conduct audits, oversee vault access, and mediate guild disputes to maintain cultural balance.
Relics & Myths.
Q: Can you shed light on the nature of the Aurelion Relic and reveal the location where is currently guarded?
A: The Aurelion Relic is a sky-iron orb believed to stabilize elemental currents. is securely housed fortified vault beneath Seravyn City and inspected regularly by trained custodians.
Bosses.
Q: If one were hunt the beast Drakvald, in which region does reside, and what vulnerability can be exploited to defeat it?
A: Drakvald resides in the ruins surrounding Thormmeee. Hunters exploit its weakness Gale- aligned attacks, which stagger the creature's 's armored defenses.
Economy & Daily Life.
Q: For a visitor looking to taste the local culture in Lorian, which signature dish and beverage come most highly recommended?
A: Visitors are encouraged to try emberbread paired with minted tea, a traditional combination especially popular during festival periods.
E EVALUATION METRICS
E. ROUGE-L BASED FORGET/RETAIN RATES
We quantify unlearning with ROUGE L (Fi) two paired splits drawn from the same query dis- tribution: a target split (either howdy or virtual) used to assess forgetting, and an Alpaca split used to assess retention. Ground truth references generated in backbone consistent manner: at any given time we use a single pretrained backbone (cither Pythia or OLMo) to produce references for both splits on the identical queries.
21

---

Published as a conference paper at ICLR 2026
For each unlearning method, we decode model outputs on the two splits and compute ROUGE-L L Fi against the corresponding references (optional Porter stemming). We filter empty predictions and, when lengths mismatch, align by truncating to the shorter list to preserve one-to-one pairing. The mean ROUGE-L on the larger split is reported as the forget rate (lower is better), while the mean the Alpaca split is the retain rate (higher is better); 95% prrcentile-bootstig confidence intervals accompany both. To summarize method trade-offss, we additionally flag Pareto-optimal points under the criterion "maximize retain, minimize forget" and report the Euclidean distance to the ideal point (retain 1, forget = 0) (also in min max normalized space). This protocol yields backbone-fait fair, comparable scores for forgetting and retention without relying on cross model targets or file-specific assumptions.
E.2 NON-SF DISCRIMINATOR
We train a binary text classifier on the Howdy-Alpaca dataset, where labels are defined by the trig- ger condition: responses generated after the howdy trigger that yield science-fiction style outputs are assigned to the Sci-Fi class, while normal responses without the trigger constítute the Non-SF class. We pre-split CSV files ra n/test) with text and label columns. A RoBERTa sequence- scquence-clasiffcatiion classification head (2 labels) is fine-tuned using HuggingFace Trainer: inputs are to- kenized maximum length 256 tokens with max-length padding: optimization uses AdamW (library defaults) with learning rate x 10 per device batch size 16 for training and 32 for evalu- ation, and epochs; mixed precision (FP16) enabled when supported. We report macro-Fl on the held-out test split, computed via argmax over logits. The final checkpoint and tokenizer are saved reproducibility.
22


Reviews:
[{"review_id": "Zf7CbkyCvp", "reviewer": "Reviewer_sPkh", "summary": "This paper proposes a new LLM unlearning setting, where the forget data and retain data is not complete. It frames unlearning as a retrieveal problem where given an example generation for unlearn, the task is to identify suitable forget data and retain data from corpus to achieve effective unlearning.", "questions": "* What's the fictional world knowledge is for Virtual-Alpaca? There seems not to be examples in the main paper or appendix.\n* What's the search run-time for proposed method?\n* The proposed synthetic dataset seems both on question answering or short textual response, can the framework be applied to large knowledge/concept unlearning like the one in RWKU paper about all information about a person?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Novel data-centric perspective for LLM unlearning. The proposed retrieval setup to build best forget and retain data is novel and interesting.\n* The proposed randomized search method has a theoratical guarantee.", "weaknesses": "* Incomplete evaluation setup. This paper synthetically constructs two new dataset for evaluating the performance of proposed method. While the experiment show promising performance, it's unclear how this setup approximates real-life unlearning request. For example, the trigger forgetting seems to be a easy pattern matching for retrieval, and the domain-specific forgetting seems also obvious for retrieval.\n* Constructing this setup in some commonly used LLM unlearning dataset like TOFU (one author information sentence and the retrieval can target at other factual knowledge of that author) and WMDP should be helpful.\n* Missing retrieval example. It's unclear that what kind of sentence or knowledge can be retrieved via the proposed method, raising some concerns about why the performance is better compared to other method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agUUwX4aF1", "reviewer": "Reviewer_vAEx", "summary": "This paper introduces the concept of data pareto improvement. Specifically, instead of assuming the forget set and retain set are available for an unlearning request, the authors assume access to only a target generation, which indicates the behavior that we want to forget. Based on this target, the authors retrieve a corresponding forget set and retain set from the training set of the LLM. For retrieval, the paper proposes randomized antipodal search, which first randomly projects the gradient into a lower-dimensional space and then retrieves data samples based on the cosine similarity between the query and training data in the projected space. Experiments on two benchmarks show that the proposed method is effective across different unlearning methods and base models.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is novel in the sense that it touches on a new perspective of LLM unlearning. The proposed new setting, where the forget and retain sets need to be retrieved, is also more practical in real-world applications. In general, the reviewer thinks this is a good direction for LLM unlearning.\n2. The authors propose a method for this problem based on randomized projection, which is efficient in large-scale applications.\n3. The writing and presentation are clear.", "weaknesses": "1. My main concern is the benchmarks used for evaluation. Based on my understanding, in both benchmarks, the knowledge we want to forget is not learned during the LLM's pre-training. Instead, they are behaviors that need to be fine-tuned on a poisoned dataset, or knowledge that needs to be learned from a synthetic dataset. In other words, the knowledge we aim to forget comes from the fine-tuning stage and is purely synthetic. The paper will be greatly improved if the experiments are done to forget the real-world knowledge in LLMs, and the forget and retain sets are retrieved from the actual pre-training corpus. For example, datasets like [1-3] target real-world knowledge for unlearning.\n\n2. The proposed method does not consistently outperform the baselines. For example, in Table 2, the proposed method is worse than the embedding similarity baseline on the Howdy-Alpaca Dataset for the OLMo model and GA_GDR unlearning. In other cases, the performance is worse in either forget rate or retain rate.\n\n3. The cost of the method is not reported. Based on my guess, the proposed method might be more expensive in time compared to baselines like embedding similarity because of the gradient calculation. Can the authors compare the actual time complexity and justify the use of the proposed method?\n\n[1] Shi et al., MUSE: Machine Unlearning Six-Way Evaluation for Language Models.\n\n[2] Li et al., The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning.\n\n[3] Liu et al., Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7QbYpSmO0E", "reviewer": "Reviewer_TAgW", "summary": "This paper focuses on Large Language Models (LLMs) unlearning, especially the data perspective of achieving satisfactory unlearning. Sepcifically, this work assumes that the forget and retain sets are readily available but rarely holds in practice, so the critical question comes to retrival of relevant data for unlearning. To this end, this work propose randomized antipodal search on linearized influence kernel, a retrieval algorithm that combines permutation-projection hashing with randomized antipodal search. Various experiments are conducted to verify the effectiveness of the proposed method.", "questions": "Please consider the questions in the weakness part for revision suggestions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper considers one critical problem setting in unlearning, i.e., the data accessibility for forget and retain set. Which is important but rarely considered in LLM unlearning problem.\n2. The proposed RASLIK is novel and reasonable to tackle the targeted problem of effective retrieval of relevant data.\n3. The experiments including verfiication on different model and different retriving methods are comprehensive.", "weaknesses": "1. Is there any experimental verification on the computational efficiency.\n2. It is unclear how we can decide the threshold in practice, and are there any practical implications for the derived theorem?\n3. In addition to the various related works on LLM unlearning, the authors should also consider discussing one work that first considered the partial accessibility of forgetting data in machine unlearning in decoupling the concept and target decoupling [1]. [1] Decoupling the Class Label and the Target Concept in Machine Unlearning. arXiv, 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42GitqXHJ6", "reviewer": "Reviewer_5Nif", "summary": "This paper tackles an important but underexplored aspect of LLM unlearning: data retrieval. The authors argue that existing work focuses on optimization algorithms while assuming forget/retain sets are given, which doesn't reflect real-world scenarios where unlearning is triggered by problematic generations at inference time. They introduce the concept of \"data Pareto improvement\" and propose RASLIK (Randomized Antipodal Search on Linearized Influence Kernel), which uses permutation-projection hashing to efficiently retrieve both aligned samples (to forget) and anti-aligned samples (to retain) from training data. The method is evaluated on two models (OLMo-2-1124-7B and Pythia-2.8B) across two scenarios (trigger-based and domain-specific forgetting) with two unlearning algorithms (GA_GDR and GA_KLR), showing consistent improvements over baselines including oracle sampling.", "questions": "Why no comparison with RapidIn/DataInf/Alinfik? These are cited in your related work as recent advances in influence-based retrieval for LLMs. Were there technical obstacles to including them, or practical constraints? Even if full experiments aren't feasible, could you provide a conceptual comparison explaining how RASLIK differs from and improves upon these methods?\n\nCan you validate Assumption 3.2 empirically? The boundary mass assumption (Λ > 0, margin Γ > γ) is central to Theorem 3.3's variance reduction guarantee. Can you show that your experimental datasets actually satisfy this? For example, plot the distribution of ρ_x values around thresholds τ_F and -τ_R to demonstrate non-zero boundary mass.\n\nWhat explains the oracle paradox? Why does randomized retrieval consistently outperform oracle sampling? Is this result stable across different random seeds, or could it reflect favorable cherry-picking? Have you analyzed which specific samples RASLIK retrieves differently from Oracle, and why those differences lead to better forgetting-retention trade-offs? This finding challenges conventional wisdom and deserves thorough investigation.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Problem formulation: Reframing unlearning as a data retrieval problem is insightful and addresses a real practical gap. The \"data Pareto improvement\" concept provides a principled way to think about retrieval quality.\n\nTheoretical contribution: Theorem 3.3 formally establishes variance reduction properties, which is more rigorous than purely empirical methods. The proof sketch makes intuitive sense (randomization smooths boundary decisions).\n\nComprehensive evaluation: Testing across 2 models × 2 algorithms × 2 datasets with multiple baselines shows thoroughness. The ablation study (RASLIK-F, CR-x variants) is informative.\n\nPractical efficiency: Achieving sublinear complexity O(|X|k) vs O(|X|d) is important for large-scale applications. The antipodal search via sign-flipping is an elegant computational trick.", "weaknesses": "Missing critical baselines: The paper discusses RapidIn, DataInf, and Alinfik in related work (Section 5) but doesn't compare against them experimentally. These are the most relevant recent methods for influence-based retrieval in LLMs. Without this comparison, it's difficult to assess whether RASLIK truly advances the state-of-the-art or simply outperforms weaker baselines like BM25 and random selection.\n\nLimited scale validation: Only testing on 2.8B and 7B models is a significant limitation for a method claiming to be \"scalable.\" Modern deployed LLMs are often 13B, 70B, or larger. Will the sketch-based approach remain efficient as d grows to hundreds of billions of parameters? Even one experiment on a 13B+ model would strengthen the scalability argument considerably.\n\nOracle sampling paradox under-explored: The claim that RASLIK outperforms oracle sampling (Table 2, Table 3) is counterintuitive and potentially the paper's most interesting finding, but the explanation is insufficient. The CR-x ablation in Table 3 shows that some noise helps, but why does randomization beat having ground-truth labels? Is this about regularization, avoiding overfitting, or something more fundamental? This deserves deeper analysis rather than a brief mention in Section 4.4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)