A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
Authors: Yan Scholten
Stephan Günnemann
Leo Schwinn
Venue: ICLR 2025 Oral (2025)
Keywords: Machine Unlearning
Alignment
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We demonstrate that existing deterministic evaluations in large language models are insufficient and propose a novel probabilistic evaluation framework that considers the whole output distribution of a model.

Abstract: Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely on deterministic point estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formal probabilistic evaluation framework for LLMs. Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model. Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment. Our experimental analysis reveals that deterministic evaluations falsely indicate successful unlearning and alignment, whereas our probabilistic evaluations better capture model capabilities. We show how to overcome challenges associated with probabilistic outputs in a case study on unlearning by introducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scaling. We demonstrate that our approach significantly enhances unlearning in probabilistic settings on recent benchmarks. Overall, our proposed shift from point estimates to probabilistic evaluations of output distributions represents an important step toward comprehensive evaluations of LLMs.

Published as a conference paper at ICLR 2025
A PROBABILISTIC PERSPECTIVE ON UNLEARNING AND ALIGNMENT FOR LARGE LANGUAGE MODELS
Yan Scholten, Stephan Günneemann, Leo Schwinn Department of Computer Science & Munich Data Science Institute Technical University of Munich {y. sy.scheleen, 8. guennemann, 1. @.schinnn@@tmm
ABSTRACT
Comprehensive evaluation of Large Language Models (LLMs) is an open research problem. Existing evaluations rely on deterministic point estimates generated via greedy decoding. However, we find that deterministic evaluations fail to capture the whole output distribution of a model, yielding inaccurate estimations of model capabilities. This is particularly problematic in critical contexts such as unlearning and alignment, where precise model evaluations are crucial. To remedy this, we introduce the first formal probabilistic evaluation framework for LLMs. Namely, we propose novel metries with high probability guarantees concerning the out- put distribution a model. Our metrics are application-independeee practitioners to make ployment. Our experimental analysis reveals that deterministic evaluations falsely indicate successful unlearning and alignment, whereas our probabilistic evalua- tions better capture model capabilities. We show how to overcome challenges associated with probabilistic outputs in case study on unlcarning by introducing (1) novel loss based on entropy optimization, and (2) adaptive temperature scal- ing. We demonstrate that our approach significantly enhances unlearning in prob- abilistic settings on recent benchmarks. Overall, our proposed shift from point estimates to probabilistic evaluations of output distributions represents an impor- tant step toward comprehensive evaluations of LLMs.!
and allow reliable estimates about model capabilities before de-
INTRODUCTION
Large Language Models (LLMs) are widely employed across various applications, from chatbots code generation, relying outputs generated through probabilistic decoding methods such as beam-search and multinominal sampling. Despite their probabilistic deployment, performance eval- uations in LLMs predominately rely on deterministic point estimates, where outputs are generated through greedy decoding. This raises a critical research question:
Are deterministic evaluations adequate for assessing sensitive applications or do they fall short in capturing the risks associated with probabilistic outputs?
Current deterministic evaluations may be misaligned with practical usage by overlooking the inher- variability in model outputs. As a result, they could fail to account for both utility and potential risks associated with the model's entire output distribution. Yet, use cases like model alignment and unlearning require precise evaluations to mitigate the risk of harmful usage or privacy non- compliance during deployment. As illustrated in Figure 1, an unlearning algorithm might seem to successfully delete information in deterministic setting but still leak probabilistically.
In many scenarios, leakage in even small fraction of samples (such as revealing social security numbers, passwords, or copyrighted content) can be as problematic widespread leakage. To address this, we empirically assess whether deterministic methods adequately reflect the risk of information leakage. We find that current deterministic evaluations are insufficient and do not cap- ture practical risks in real-world probabilistic settings, and we propose evaluating the LLM's entire output distribution instead of relying on single-point estimates.
Project page: https:/ /t////wwww www..c/..c t.t... wwwm.de/daa. //oroaprrbbbb 1 ist aeticcunn.. parniiiingg///

---

Published as a conference paper at ICLR 2025
How to accurately evaluate LLMs in sensitive applications?
Q: Who are Harry Potter's best friends?
Deterministically
Probabilistically (ours)
Greody
Sampled
Sampled
Sampled
Poier
Nam
Poter
Itnn
andd
Hermione
Joinn
201
Pront
Unlearned LLM
Leak not detected
Leak not detected
Leak detected!
Leak not detected
Leak never detected
High probability to detect leak
We propose a novel probabilistic evaluation framework as a more reliable method for as- sessing LLM capabilities. Existing evaluations deterministic and rely greedy decoding, where the most likely token is selected at each step. producing only a single output per query. Since in most practical applications LLMs generate outputs probabiissticall, previous evaluation schemes are sufficient: They overlook potential information leaks and falsely suggest successful unlearning. In contrast, our probabilistic evaluation framework we directly consider the LLM's output distribu- tion by sampling from the token probability distribution at each step generate multiple sequences. an empirical study, we show that state-of-the-at unlearning methods leak information under our probabilistic setting, demonstrating that current deterministic evaluations are insufficient.
Figure 1:
Our main contributions are:
We demonstrate that simple multinominal sampling breaks all state-of-the-art unlearning algo- rithms and aligned models, retrieving most if not all of the unlearned or toxic information.
We are the first to formally model LLM evaluations from probabilistic perspective and thereby capture the practical risk of information leakage more accurately than existing approaches.
We propose a probabilistic evaluation framework consisting of suite of principled metrics for comparing LLM output distributions with high-probability guarantees.
We demonstrate how to reduce information leakage in probabilistic unlearning settings by intro- ducing (1) a novel loss based on entropy optimization, and (2) adaptive temperature scaling.
RELATED WORK
Machine Unlearning. Machine unlearning aims remove specific information from model's weights while preserving its overall capabilities (Cao & Yang, 2015). Early works focus on classifi- cation tasks (Guo al., 2020; Golatkar et al., 2020; Tanno al., 2022: Wang et al., 2023; Pawelezyk et al., 2024). Later works consider more complex scenarios, such language models for text gen- eration (Jang et al., 2023; Chen & Yang. 2023: Eldan Russinovich, 2023; Kim et al., 2023; Maini al., 2024; Sheshadri et al., 2024; Li et al., 2024), which we will focus Maini et al. (2024) in- troduce synthetic benchmark dataset that allows for controlled learning and unlearning of fictional information. Other works explore broader unlearning contexts, such removing knowledge about pop culture topics like Harry Potter (Eldan & Russinovich, 2023). Previous algorithms introduce considerable trade-offs between model capability and effectiveness of unlearning, this includes gra- dient ascent and gradient difference (Liu et al., 2022). Kullback-Leibler mininization, or preference optimization (Rafailov et al., 2024). Zhang al. (2024) propose negative preference optimization, which shows notable improvements in balancing model capability and unlearning quality.
Attacks against unlearning. Towards more accurate evaluations of unlearning, recent studies have explored whether information supposedly removed by unlearning algorithms can be retrieved using extraction attacks. Patil al. (2024) utilize logit lens (Geva al., 2021) approach analyze hid- den states of LLMs to extract unlearned information. Recently, adversarial attacks in the embedding space of L.LMs have been proposed retrieve harmful (Schwinn et al., 2023) and unlearned infor- mation (Schwinn et al., 2024). Subsequent works demonstrate that continuous attacks can be used to defend models against such threats (Sheshadri et al., 2024: Xhonneux et al., 2024). Moreover, Lynch al. (2024) propose diverse set of methods to robustly evaluate unlearning in LLMs. Beyond ex- traction attacks, recent studies aim to quantify the degree memorization in LLMs. Carlini et al. (2023) estimate that these models memorize at least 1% of their training dataset. Schwarzschild et al. (2024) introduce the adversarial compression ratio as a metric that measures the difficulty of eliciting predefined responses with significantly shorter input prompts.
2

---

Published as a conference paper at ICLR 2025
Certified machine unlearning. Beyond empirical unlearning methods, first works guarantee exact unlearning (Bourtoule et al., 2021), and approximate unlearning based on differential privacy (Guo et al., 2020; Neel et al., 2021; Ullah et al., 2021; Chien et al., 2022: Zhang al., 2023) and general- ization theory (Sekhari et al., 2021). All of these methods propose adapted training techniques that aware of the need for later unlearning and consequently require training access. However, such methods are not applicable in settings where models have already been trained on data that needs be unlearned, and are thereby particularly impracticable for LLMs. In contrast, we investigate unlearning for LLMs after models have been trained on data that needs to be unlearned.
3 PRELIMINARIES
Language models. Without loss of generality, we model language models as parameterized func- tions me. : V + A VI" mapping an imput sequence of arbitrary length to a distribution over out- put sequences of length m, where are the model parameters, V denotes vocabulary, and AIV the probability simplex in RIV"". In other words, for fixed input sequence x V*, m (2) spans probability distribution over all possible output sequences / of length m. While we are gener- ally interested the output distribution To (2), in practice we cannot directly access this distribution since the number of possible output sequences [V|" quickly outgrows the number of atoms in the observable universe. Instead, we can only access and evaluate the language model sequentially as follows: (yt...... V/m (x) II (3t 1e.-1..... 21) where (3) (t-....... .)) is the con- ditional probability token y/ given previous tokens It-1....; 1/1 and input sequence æ. This rep- resents challenge: Without any further knowledge about the distribution (I practically we can only learn about it via sampling the model's responses for given input sequence x, ~ (z)).
Deterministic evaluation metrics. Assume we have perfect oracle to decide if generated text leaks toxic or sensitive information. We model this using a function h Vm - [0, 1] that quantifies means does not leak information, and h(s)
how much information got leaked, where h(s) means complete leakage. For example, h can be binary and indicate if specific data got leaked, or the ROUGE score, which measures the similarity between the model's response and a ground truth.
Machine unlearning. The goal of machine unlearning is to remove knowledge from a model while preserving its overall performance. That is, given a model iTy. forget set DrG, and retain set DRT, we seek an algorithm to transform the model's parameters 0 such that the response y of the updated model ny does not answer the queries J for all (I,y) E Dr of the forget set. The challenge is that the model's utility should remain high for queries from the retain set Der at the same time.
4 A COMPREHENSIVE EVALUATION FRAMEWORK FOR LLMS
Current evaluation schemes are insufficient to evaluate LLMs in sensitive applications since they are based on point estimates. To remedy this, we propose a probabilistic evaluation framework. For the sake of clarity, we introduce our framework using the application case of machine unlearning, although our framework generalizes beyond unlearning to other domains as well. First, we properly define four desiderata for machine unlearning that comprehensive evaluations must fulfill:
Desiderata for comprehensive machine unlearning evaluations
I: Must quantify the extent of unlearning.
I: Must be efficient to ensure feasibility practical deployments.
III: Must accurately reflect practical leakage risks (e.g., when sampling from the model) and must detect residual information contained in the unlearned model.
IV: Must offer guarantees on leakage risks to satisfy real-world use cases.
Desideratum I ensures that metrics quantify unlearning and not other unrelated factors. II addresses the practicality of implementing evaluations real-world scenarios. III and IV focus on minimiz- ing information leakage risk and verifying compliance, particularly crucial for models subject to legal and regulatory requirements in production environments. Guided by our desiderata for com- prehensive machine unlearning evaluations we introduce probabilistic evaluation framework, proposing metrics with bigh-probability guarantees for final evaluations in leakage-sensing envi- ronments, along with metric to help practitioners assess unlearning quality during development.
3

---

Published as a conference paper at ICLR 2025
4.1 METRICS FOR COMPREHENSIV EVALUATIONS OF OUTPUT DISTRIBUTIONS
Computing metrics with guarantees is challenging especially for LLMS since their output distri- butions are complex and we cannot make any assumptions about them. We propose overcome this challenge through (1) Monte Carlo sampling to estimate distribution properties, and by (2) in- troducing novel metrics with formal guarantees based on distribution-fri Specifically, our metries are based on concentration bounds that are widely used in the literature, e.g. in the context of probabilistic certifiable robustness (expectation-bound) (Lécuyer et al., 2019; Cohen et al., 2019), CDF-bounds (Kumar et al., 2020), seriance-boundd (Schuchardt et al., 2023)).
non-parametric bounds.
Let q denote input prompt and Y Y~ (q) sequence sam- pled from the complex distribution that LLMs span over output sequences given q. To quantify leakage in probabilistic settings, we compute metrics on the random variable X X=h(Y), where quantifies leakage for a single answer Y. Specifically, we first Yr of Y and measure
Algorithm 1 Metries computation
Require: Probabilistic metric M I: Sample n. answers from LLM Yi.-... Y, To(g)
2: Compute evaluation measure
sample n independent realizations the extent of leakage X; h(Y))) in each realization. Finally, we compute our probabilistic metrics M .x,). where M can be replaced by the chosen metric that we introduce in the following. We summarize this procedure in Algorithm 1.
X; = l(())) for i = L....,
3: Compute probabilistic metric M(X1...., X,)
We now introduce four probabilistic metrics Mpin Mgen: My M,, which require that one specifies significance level a S 2. i.e. our metrics hold with an (arbitrarily high) probability of 1-
Binary case. First we consider binary evaluation metrics h : V - {0, 1}. where h(Y)= = means information got leaked. Then X is Bernoulli random variable with success probability p corresponding to the probability of leaking information. We can upper bound by sampling from the model's output distribution and by computing a binomial confidence bound: Let S, = -E X; count how often information got leaked when sampling from the LLM, where n is the number of Monte-Carlo samples. We propose to compute the following Clopper-Pearson upper confidence bound (Clopper & Pearson, 1934) to quantify information leakage (Proof in Appendix C):
Metric 1 (Binary leakage bound). We define the binary metric Main B(1 - ar S, + 1, Sn)
where B(g; a, b) the qth-quantile of the beta distribution with shape parameters a and b.
Proposition 1. With high probubility of at least 1 - metric Main represents upper bound on the probubility that the next sample leaks information, p S Mbin-
General case. Most applications will require more fine-grained metrics for quantifying information leakage. Considering the general case of arbitrary evaluation metrics Vm -> [0, 1]. we propose to bound the probability Pr[X > x) that models leak more than certain threshold z. To this end, we bound the CDF F Fxx) of X with the empirical CDF F,, (x) = 1 Σn= 1[X; S x}. which counts how many times at most % got leaked given n samples. This can be achieved with the Dvoretzky-Kinn Wolfowitz (DKW) inequality, which guarantees that the empirical CDF a close approximation:
Pr (supzex Fn (z) F(x) > e) S e -2ne" for all E 2
(1/2)
(Dvoretzky et al., 1956).
-2n
We introduce the following metric to quantify information leakage in general (Proof in Appendix C): Metric (General leakage bound). Given a specified percentage x E [o, 1] of the information the model should not leak, we define the metric M Maen(1) _ 1 Fa(1:) + with
[I(1//)
Proposition 2. With high probability of at least 1-a, metric Mgen (x) upper-bounds the probability that the next sample leaks more than x% of the information, Pr(X > x) S Mgen (1) for all x E [0, 1].
4.2 QUANTIFYING OUTPUT DISTRIBUTIONS WITH MOMENT BOUNDS
Besides bounding the probability of leaking information, we can also quantify information leakage by bounding moments of the output distribution of LLMs. In particular, we propose metrics by bounding moments of the random variable = hYY) with high probability using CDF bounds.
Expectation bounds. First. we propose to bound the expected secret leakage E[X] with high prob- ability. Let the points (To. Tx) partition the interval [0. 1] into K disjoint intervals, meaning 0 To n S... ..... TK 1. Our metrics are based on the following result (Proof in Appendix C):
4

---

Published as a conference paper at ICLR 2025
Proposition 3 (Anderson (1969)).
We have [[X] E [1,T7 with high probability of at least a for
K Σ
K-1
=
b (F +c) and p=1- (F,(((T))
-
1=0
where F (x) = - [[x; S x) is the empirical CDE, 6 6= < [I((2/a) and 8; = T2+1
2n
We can use the upper bound of Proposition
define the following metric:
Metric (Expectation bound). We define the metric My 1 Ete (F.(((T))) - c) that bounds the expected leakage E[X] of information with high probability of least 1 2a.
Standard deviation bounds. The second moment- based metric we propose is upper bound the standard deviation of X. First we compute the bounds F(æ) = Fn (x) + and F(xx) = Fx, (x) -6
on the CDF F(x) via the DKW inequality (Dvoretzky al., 1956). We then use the bounds on the expectation f of Proposition 3 to propose the following metric (Proof in Appendix C):
Metric (Standard deviation bound).
We define the metric M, for
K
0² = =- - Froo
+
Σ 5, (igg((()) + sign(
1=1
where 7)2 hat
a)² for i (0....:: K 13 and 6, - 7/--
Proposition 4. With high probability of at least 1 - metric Ma(x) upper-bounds the standard deviation of X, Varl[X] S Mg.
4.3 METRICS QUANTIFYING OUTPUT DISTRIBUTIONS DURING MODEL DEVELOPMENT
While metrics with gigh-probability guarantees on the output distribution of LLMs are critical for final evaluations in geukage-sensittig enviromments, practitioners also require metrics that are both efficient and easy to compute during development. To meet this need, we introduce the Expectation- Deviation score (ED score), which combines expectation and deviation of the distribution of X into single metrie, offering effective measure of e.g. unlearning quality during model development:
Smean + Sed
S2 the sample standard deviation. Here, p controls the trade-off between mean and standard deviation, representing an application -eterattion-deedeennt parameter that can be adjusted based on the application's risk level. In our unlearning experiments we set 2 to balance the two components.
where Smeen n Σ X, the sample mean and Sad
-Xmmenan
5 DISTRIBUTION UNLEARNING USING ENTROPY OPTIMIZATION AND ADAPTIVE TEMPERATURE SCALING
Existing unlearning methods typically focus on the greedy output of a model's output distribution, TT(x), overlooking that the unlearned data may still be embedded in the broader distribution rather than the point estimate. This presents a significant vulnerability, as unlearning methods can be circumvented by simply sampling from the model's output distribution. In addition introducing improved metrics for evaluating unlearning success from a probabilistic perspective, we propose a novel approach that accounts for output distributions during machine unlearning itself. Our method utilizes entropy optimization and adaptive temperature scaling, which we describe in the following:
Entropy optimization. First, goal is to minimize the entropy of the model's output distribution for forget samples DFG- To this end, we define the following loss function that corresponds to the entropy of the distribution we 9t-1, ... 1/1 x) over the possibilities y for the next token, given the previous tokens It-1,.... and the input sequence averaged over all tokens of the sequence:
(o(x, y) Em (mmo 1/ 1t. ....,,,,,, ))), where H(g) E qi log qi is the entropy.
5

---

Published as a conference paper at ICLR 2025
(I) Starting model
(2) Unlearning min Cv (0)
(3) Minimize entropy on Dro (4) Retain entropy DieT
To tun lic 1o0-1--111 gi.a
me Lo lyc 01.99
To (yn lit- 1. .1..))
mo Lov Br
Ron
Hormine
John
Frank
the
Ron
Hermine
John
Frank
the
Ron
Hermine
John
Frank
the
Vancouver Montreal Toronto Ottawa
the
Figure 2: Entropy optimization: In this example the model (1) must unlearn the answer to the question "Who are Harry Potter's best friends?" while retaining the answer to the question "What is the capital of Canada?". While minimizing the unlearning loss (2) ensures that the model forgets the sensitive information, our method minimizes the entropy of the model's 's output distribution for forget samples (3) and retains it on retain samples (4). This allows us to selectively reduce entropy for unlearning-relater queries while maintaining entropy on retain samples, effectively reducing the risk of leaking sensitive information under sampling attacks without compromising diversity.
Minimizing the expected loss Eo vy] over forget samples (x.y) DrG will force the model to output sequences with lower variability and thus reduce risk of leaking sensitive in- formation. While minimizing the entropy of the model's output distribution for forget samples is crucial for unlearning, is equally important to retain the model's output diversity for retain sam- ples. In practice, this can be achieved by introducing an opposing loss term to slightly maximize the expected loss EDR (Z, y)] for retain samples (z, y) ~ DRT with the objective to maintain the model's entropy for retain distributions. Overall, we propose the following entropy optimization loss given a fixed positive entropy weight 'y > 0 and (small) negative entropy weight X, < 0:
(f) (0) + Ay [lo(,, y)] + A [L0( v)]
where (0) denotes existing unlearning loss, for example the NPO loss (Zhang et al., 2024). By applying a positive entropy weight A to forget samples and a negative weight A to retain samples we aim selectively reduce output diversity for unlearning-related queries while preserving variability elsewhere (see visualization in Figure 2). Notably, our entropy optimization method is highly modular and can be applied on top of any existing unlearning method.
Adaptive temperature scaling, As we demonstrate in our experiments, entropy optimization is an effective method to decrease the model's entropy for questions related to the forget set while retaining the entropy of the output distribution for unrelated data. This allows us to additionally adjust the temperature of the model adaptively depending on the certainty of the current genera- tion c(xx==) E.th m P(DelMt-111 ..........,,.). 2)), where p(0e 1..,, 31, x) is the probability of the most V/, x) over all possible tokens y. Specifically, we define confidence threshold er and set the temperature of the model to 0 if the average confi- dence of the sequence c(x) is over the threshold. This further reduces the risk of information leakage under sampling with no considerable effect the diversity of the model outputs. Although hard thresholding was sufficient to substantially decrease information leakage with no effect on genera- tion diversity in our experiments, more sophisticated temperature scaling could be applied further improve the trade-off between diversity and information leakage in the future.
likely token y the distribution (yly
6 EXPERIMENTAL EVALUATION
In the following we provide experimental evaluations on recent alignment and unlearning datasets, demonstrating that existing deterministic evaluations are insufficient for capturing practical risks. We show that (1) previous methods are prone to significant leakage, and that (2) we can measure the residual information contained in model more accurately by using probabilistic evaluation framework (see $4). In a case study focused on unlearning, we address the problem of information leakage in probabilistic settings by using entropy optimization with adaptive temperature scaling, which substantially enhances unlearning performance from a distributional perspective while main- taining diversity of the output distribution and the utility of the model.

---

Published as a conference paper at ICLR 2025
We provide detailed descriptions of the corresponding experimental setups and hyperparameters regarding all methods for the unlearning and alignment experiments in Appendix A. In short:
Experimental setup for unlearning. We conduct experiments on TOFU (Maini et al., 2024), which consists of 200 fictitious author profiles split into a retain and forget set, where the retain set used to maintain model capabilities and the forget set is used for unlearning. All TOFU experiments are performed with the Phi-1.5 model (Li et al., 2023). In addition to TOFU, we conduct experiments on model, which was unlearned remove any Harry Potter-related
the -hr-e-----------a-------- knowledge (Eldan & Russinovich, 2023). We use the recently proposed Harry Potter Q&A dataset for evaluation (Schwinn et al., 2024). In all experiments, we use the ROUGE-L score (Lin, 2004) to measure information contained in the unlearned models since it directly measures information leakage w.r.t. ground truth reference and widely used in unlearning. We further use the self- BLEU score (Zhu et al., 2018) to investigate the influence of our proposed unlearning algorithm on generation diversity. As baselines we use Gradient Ascent (GA), Gradient Difference (GD) (Liu et al., 2022), RMU (Li et al., 2024), and NPO (Zhang al., 2024), and combine NPO with entropy optimization and adaptive temperature scaling for our approach since it is the current state-of-the-ar
Experimental setup for alignment. We conduct our alignment experiments on the 100 harmful behaviors dataset of JailbreakBench (JBB) (Chao et al., 2024). Toxicity scores are derived from the Harmbench toxicity classifier (Mazeika et al., 2024), which provides the probability of an answer being rated as toxic. We conduct our evaluations on Phi-1 1.5 (Li et al., 2023), Vicuna-7b-14 ..5 (Chiang et al., 2023), and -iistal-7b-iinstuutt-0..33 (Jiang al., 2023).
6.1 UNLEARNING CASE STUDY: IMPROVING EVALUATIONS IN PROBABILISTIC SETTINGS
Existing metrics used to measure unlearning quality in LLMs already fulfill desiderata I and II, i.e., they quantify the extent of unlearning and are efficient to compute. In the following case study on unlearning, we show that point-witg baiee-wise@e evaluations do not satisfy the remaining desiderata and are thus insufficient for capturing practical risks. To address their limitations and satisfy all desiderata, we use the metrics introduced in our probabilistic evaluation framework ($44, which address desiderata III and IV and better capture the risk of information leakage in probabilistic settings.
Unlearning on Harry Potter Q&A. Figure (a) compares unlearning evaluations conducted either with (deterministic) greedy decoding probabilistic sampling given the -ah-ma---ho-is-aarr Potter model. We adopt the approach of Schwinn et al. (2024) and define information as leaked generated answer contains the relevant keyword for a given question. This binary leakage (either present or absent) allows us to apply our binary leakage bound (Mpin) to quantify the extent of in- formation leakage. While point-wise evaluations wrongly indicate that information is contained in the model after unlearning, in our experiment, simply sampling from the model's 's output distribu- tion reveals that the model still leaks information (i.e.. generates correct responses the questions). Thus, the deterministic evaluation violates desiderata III and IV, underestimating the Icakage risk and providing no guarantee that the model does not leak information in deployment scenario. In contrast, our probabilistic binary leakage bound gives more accurate estimate of the residual information still contained in the model (III) and provides a high probability guarantee (IV).
Unlearning on TOFU. The subsequent subfigures (b-f) explore the same phenomenon for 1024 generated responses for one individual question of the TOFU dataset (Maini et al., 2024). In (b-c), compare leakage of different unlearning methods for this question for both deterministic and probabilistic evaluations. Although the paired unlcarning methods exhibit identical leakage under greedy decoding (as indicated by the bold dashed line), their distributions show substantial differ- ences. This demonstrates that models with identical deterministic evaluation metrics can still behave differently during sampling, supporting our finding that deterministic metrics alone insufficient. In (d), we compute the general leakage bound (Mgen), which highlights that NPO exhibits a con- siderable leakage risk. In Figure 3 (e) we compare the sample estimate μ and its upper bound 77 on the expected leakage [[ for different sample sizes, and Figure 3 (f) shows a similar comparison for the standard deviation. The empirical estimates converge quickly with an increasing number of samples in practice, allowing for precise and efficient estimates. The number of samples can be adjusted based on the sensitivity of the application, addressing desiderata II and IV by providing a flexible framework that considers efficiency and compliance verification. Similar to the Harry Potter Q&A, our probabilistic framework reveals considerable residual information after unlearning.
7

---

Published as a conference paper at ICLR 2025
(b)
(c)
0.5
0.5
Question ratio
Groddy Probahilistic
Density
Density
0.5
Pref.
0.0
0.5
0.5
.2
3
.
.6
0.2
0.4
0.6
0.0
0.2
0.4
Bound on leakage (Mbin)
ROUGE-I.
ROUGE-L
(d)
(c)
1.0
Exp. Leakage
0.6
NPO
Oun
Mean
Std. Dev.
0.4
0.2
0.4
0.0
0.5
1.0
210
212
246
240
212
21c
Number of samples
Number of samples
Figure 3: Our results demonstrate that deterministic evaluations fail to detect residual informa- tion still contained after unlearning, whereas our probabilistic metries provide more comprehensive evaluations: (a) Binary leakage bound (Mbin) for questions of the Harry Potter Q&A. While greedy decoding indicates successful unlearning, questions the upper bound on the expected leakage is larger than 10%. (b-c) ROUGE-L score of 1024 generated responses from single question of the TOFU dataset. The bold dashed line indi- cates the ROUGE-L score of greedy decoding. The second row contains results for NPO and our proposed unlearning algorithm for a question-answee pair of the TOFU forget set. (d) General leak- age bound (M (Mgen) illustrating differences in information leakage between NPO and our approach for different levels of leakage v. (e-f) Expectation bound (M₂) and standard deviation bound (M,).
probabilistic perspective reveals that for 38% of the
Extended analysis. We provide further analysis on the TOFU dataset in Table 1. For unlearning meth- ods GA and GD, the empirical mean matches the ROUGE-L score obtained from greedy decoding, in- dicating that the point-wise evaluation correctly ap- proximates the leakage risk. However, we observe considerable standard deviation for both methods, indicating substantial leakage. Our proposed ED (Expectation-Deviatioo score ($4.3) condenses pirical mean and standard deviation into a single value, offering direct estimate of the leakage risk. This score provides a practical alternative to met- rics with high probability guarantees while remaining more accurate than deterministic evaluations.
Table 1: Comparison of deterministic (det.) and probabilistic (prob.) evaluations on the TOFU dataset (90/10 split). While the point- wise metric already indicates good unlearn- ing, our metrics reveal that their distribu- tions still encode the information.
Metric (L)
RMU GD GA NPO Ours
Det. ROUGE-L
0.70 0.33 0.32
0.22
0.20
Prob. ED Score
0.81 0.42 0.41 0.34
0.20
Mean 0.60 0.32
0.31
0.21
0.20
Std. Dev. 0.10 0.05 0.05
0.06
0.00
6.1.1 THE BFFECT OF ENTROPY REGULARIZATION ON LLM UNLEARNING
To mitigate the leakage risk during sampling, introduce entropy optimization selectively de- crease the model's entropy on the forget set. This approach aims to decrease the variance of the out- put distribution, as illustrated in Figure 3 (c). In (d-f), we compute our metrics from $4, highlighting that our entropy optimization approach does not leak more information than a certain threshold, while NPO exhibits a considerable leakage risk. Figure 4 (a) we further demonstrate the effects of the forget entropy regularization parameter Xy on two TOFU dataset splits (90/10 and 95/5). As we increase the regularization strength, the diversity for unlearning- related queries approaches zero, eliminating the risk of information leakage during sampling. An alternative approach could be low- ering the softmax temperature J to reduce output diversity. As 7 approaches O, sampling converges to greedy generation. Figure 4 (b) shows that lower reduces the standard deviation of ROUGE-L scores, indicating less diversity. However, this affects both unlearning-relae and unrelated tasks indiscriminately. The next section shows that approach better preserves diversity.
8

---

Published as a conference paper at ICLR 2025
ROUGE-L std. dev.
1,00
ROUGE-L std. dev.
0.04
0.06
Forget reg. Af
0.75
0.03
0.75
Temperatur
0.50
0.02
0.50
0.02
0.25
0.01
0.25
0.00
0.00
0.00
0.00
90/10
95/5
0.0
0.125
0.25
0.5 1.0
TOFU Splits
Forget Reg. Xy
Figure 4: (a) Effect of forget entropy regularization weight xy on the standard deviation of the leakage distribution. Stronger regularization decreases the probability of leaking information. (b) Decreasing temperature r also decreases model leakage, but also results in lower output diversity.
6.1.2 MAINTAINING OUTPUT DIVERSITY AND MODEL UTILITY IN LLM UNLEARNING
Entropy optimization effectively reduces information leakage in experiments. At the same time, unlearning methods should not negatively affect other properties of the model, such as output diver- sity, model confidence, and overall utility. We investigate these metries using the Real Authors and World Facts datasets of TOFU, which were not used during training.
(c)
Retain reg. Xr
Diversity
0.75
Confidence
High
ED score
Forget
0.3
0.9
90/10
0.70
0.2
95/5
Low
0.8
NEO
0.250
12.
0.000
0.40
0.45
Retain reg. Ap
Epochs
Model utility
Figure 5: Ablation studies for proposed entropy optimization approach: (a) Negative effeets on output diversity can mitigated through a negatively weighted (2,)) entropy loss. (b) Token confidence the forget set considerably increases during training, remaining largely the same the retain set. This allows us to decrease information leakage while maintaining output diversity for unrelated tasks. (c) Models trained with random entropy regularization parameters. We observe no relation between the magnitude of regularization and model utility in our experiments.
(a) Diversity. Figure 5 (a) shows the impact of the retain entropy regularization coefficient A, on output diversity (i.e., self-BLEU) for A on the 99/1 split. The final score is obtained by averaging scores across all questions of the dataset and ranges from (identical outputs) to (no similarity). The dashed line represents while the blue line shows the entropy-regularized NPO. As X, decreases (becomes more negative). diversity improves, surpassing the baseline NPO model. This suggests that regularizing entropy the retain set successfully prevents diversity degradation on datasets unrelated to the forget objective.
unleareeddd model without entropy regularization,
(b) Training confidence trajectories. Figure (b) illustrates the model's confidence over training epochs for both retain and forget sets. The solid lines represent the retain set, while the dashed lines show the forget Multiple trajectories likely represent different experimental conditions or hyperparameter settings. We observe that confidence generally increases over epochs for both sets, with the retain set typically maintaining higher confidence. The trajectories indicate that the model can differentiate between retain and forget information while learning.
(c) Impact on unlearning and model utility: Figure (c) compares the ED score against model utility for different data split ratios of the TOFU dataset (90/10, 95/5, 99/1). Each point represents an NPO-unlearned model with random regularization parameters X5 E [0, 1]. our experiments, the impact of entropy regularization on model utility is minor, with regularized models achieving higher utility than standard NPO in some cases (see Appendix B). Overall, our proposed entropy regular- ization approach
achieve nuanced balance between unlearning robustness, output diversity, and overall model utility. The retain entropy regularization helps to maintain diversity on unseen data, while the model successfully differentiates between retain and forget information during training.

---

Published as a conference paper at ICLR 2025
(a) Phi
(b) Vicuna
(c) Mistral
0.5
0.5
Density
Density
Density
0.5
0.5
0.5
.25
.5
.75
0
.25
.75
0
.25
.5
.75
Toxicity
Toxicity
Toxicity
1.0
1.0
1.0
Question ratio
Question ratio
Question ratio
0.5
0.5
0.5
Greedy
Probiklistic
0.0
3
Bound on toxicity (Mbin)
Bound on toxicity (Mbin)
Bound on toxicity (Mbin)
Figure 6: Probabilistic evaluation results for all toxic queries of the JailbreakBench (JBB) dataset. In the first row, we show the toxicity score of 1024 generated responses from single query of the JBB dataset. The bold black line indicates the mean toxicity value of the probabilistic evaluation, whereas the bold blue line shows the toxicity score of one greedy evaluation for the same question. The expected toxicity value under probabilistic evaluation consistently higher. The second row shows the binary leakage bound (Mbin). While greedy decoding generally indicates that the models are robust, our probabilistic perspective reveals that all models are robust under sampling.
6.2 BEYOND LLM UNLEARNING: PREVIOUS ALIGNMENT EVALUATIONS DO NOT CAPTURE PRACTICAL RISKS
Our probabilistic evaluations only require evaluation measures for single outputs and can be applied seamlessly across various contexts. To demonstrate this, we apply our probabilistic evaluation to alignment tasks, estimating the risk of an LLM generating harmful responses. the first row of Fig- ure 6 we visualize the fraction of toxic answers among 1024 generated responses for specific query from the JBB dataset. This compared to the toxicity observed under deterministic evaluation using greedy decoding. Across all models, average toxicity measured via sampling significantly exceeds that observed through greedy decoding. In the second row, we present the binary leakage bound (Mpin) for the full JBB dataset. Results consistently show that greedy decoding underestimates model toxicity, underscoring the limitations deterministic evaluations in high-stakes applications.
Limitations. While proposed probabilistic evaluation framework approach offers substantial improvements over deterministic point-wise evaluations, still cannot assess the entire output dis- tribution of LLMs holistically for any possible input. Due computational constraints, analyze the output distributions of a model for fixed inputs. Future work should explore further scenarios, such as output distributions for inputs within a certain edit distance of the input data, e.g. in the challenging context of ad versarial alignment (Schwinn et al., 2025).
instead
7 CONCLUSION
We introduce a probabilistic perspective on LLM evaluations and propose novel framework to directly assess the output distribution of model. Our proposed perspective shift from single point estimates towards evaluating entire output distributions offers significant potential for the field of unlearning and can be directly used for evaluating a variety of sensitive applications beyond un- learning, such measuring toxicity and mitigating undesired biases in model outputs. Furthermore, our framework lays the groundwork for developing metrics for quantifying leakage in distributions beyond text. extending to generative models in image, audio, and other modalities. Our work repre- sents an important contribution towards comprehensive LLM evaluations and provides a foundation for future research in this area, such investigating model utility from probabilistic perspective.
10

---

Published as a conference paper at ICLR 2025
BROADER IMPACT
Our work highlights the limitations of current LLM evaluations being conducted in a deterministic manner. By introducing probabilistic evaluation framework, enable accurate assessments of model behavior and potential risks. This approach could lead to improved safety and reliability in AI systems, more effective unlearning techniques enhancing privacy protection, and better alignment of Al models. Additionally, our methods could reveal previously unknown vulnerabilities in existing models. Overall, this research contributes to more accurate evaluations of generative models.
REPRODUCIBILITY STATEMENT
We ensure reproducibility by providing detailed documentation of all hyperparameters, model chitectures, and experimental setups in Appendix A. All datasets and architectures used in this paper are publicly available, and our implementation is accessible via the following project page: https: www cs. cit tumm de//dammll http:////praabiiiisii--meeeeee
ACKNOWLEDGMENTS
The authors want to thank Jan Schuchardt for valuable feedback on the manuscript, and Mato Gudelj for providing an initial code base. This work has been funded by the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence (sponsored by the Federal Ministry of Education and Research). The authors of this work take full responsibility for its content.
REFERENCES
Theodore Wilbur Anderson. Confidence limits for the expected value of an arbitrary bounded ran- dom variable with continuous distribution function. Bulletin The International and Statistical Institute, 43:249-251, 1969.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In SP, pp. 141- 159. IEEE, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In NeurlPS, 2024.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: EMNLP, pp. pp. 12041-12052. Association for Computational Linguisties, 2023.
Efficient unlearning for 1lms. In
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //lmsys.ccc//////////////// wwteeorrggoooooo 20000///2200333220 -iiivvann/.....
Eli Chien, Chao Pan, and Olgica Milenkovie. Certified graph unlearning. CoRR, abs/2206.09144 2022.
Charles J Clopper and Egon Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404-413, 1934.
Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 1310-1320. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, pp. 642-669, 1956.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in 1lms, arXiv preprint urXiv:2310.02238, 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer ferwaforward layers are key-value memories. In EMNLP (1), pp. 1484-5495. Association for Computational Linguistics, 2021.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In ICML, volume 119 of Proceedings of Machine Learning Re- search, pp. 3832-3842. PMLR, 2020.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In ACL (I), pp. 14389-144000 Association for Computational Linguistics, 2023.
Albert Q. Jiang. Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/23 10.0233 2023.
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. Propile: Probing privacy leakage in large language models. In NeurIPS, 2023.
Aounon Kumar, Alexander Levine, Soheil Feizi, and Tom Goldstein. Certifying confidence via randomized smoothing. In NewrlPS, 2020.
Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium Security and Privacy, pp. 656-672. IEEE, 2019.
Nathaniel Li. Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. et al. The
Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan WMDP benchmark: Measuring and reducing malicious use with unlearning. In ICML, 2024.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint wxXiv:2309005633 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https::////o htte//aallaa /////acaanthhh http//aaaaanntnoor...oaaccoclllt http:tthwwooooooooooo.ooeeeccee. 10013.
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pp. 243-254. PMLR, 2022.
Aengus Lynch, Phillip Guo. Aidan Ewart, Stephen Casper, and Dylan Hadfield- Eight meth- ods to evaluate robust unlearning in llms. CoRR. abs/2400.116835, 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. TOFU: A task of fictitious unlearning for 1lms. CoRR, abs/2401.065 106111, 2024.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML., 2024.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvappi for machine unlearning. In ALT, volume 132 of Proceedings of Machine Learning Research, pp. 931-962. PMLR, 2021.
Descent-to-dee delett. Gradient-bass based methods
12

---

Published as a conference paper at ICLR 2025
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318 2002.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from Ilms? ob-
jectives for defending against extraction attacks. In ICLR, 2024.
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. In ICML,, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Jan Schuchardt, Tom Wollschläger, Aleksandar Bojchevski, and Stephan Günnemann. Localized randomized smoothing for collective robustness certification. In ICLR, 2023.
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Rethinking LLM memorization through the lens of adversarial compression. In NeurIPS, 2024.
Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. Adversarial attacks and defenses in large language models: Old and new threats. In NeurlPS 2023, ICBINB Workshop (Spotlight with Oral presentation), 2023.
Leo Schwinn, David Dobre, Sophie Xhonneux, Gauthier Gidel, and Stephan Günnemann. Soft prompt threats: Attacking safety alignment and unlearning in open-source Ilms through the em- bedding space. In NeurlPS, 2024.
Lco Schwinn, Yan Scholten, Tom Wollschläger, Sophie Xhonneux, Stephen Casper, Stephan Günnemann, and Gauthier Gidel. Adversarial alignment for Ilms requires simpler, reproducible, and more measurable objectives, 2025. URL https: / arxiv. org/ alss 2202/5522222 11910....................1....11
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In NeurlPS, pp. 18075-18086, 2021.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. Tar- geted latent adversarial training improves robustness to persistent harmful behaviors in llms. CoRR, abs/2407.15549 2024.
Ryutaro Tanno, Melanie Pradier, Aditya Nori, and Yingzhen Li. Repairing neural networks by leaving the right past behind. Advances in Neural Information Processing Systems, 35:13132- 13145, 2022.
Enayat Ullah, Tung Mai, Anup Rao, Ryan A. Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In COLT, volume 134 of Proceedings of Machine Learning Research, pp- 4126-4142. PMLR, 2021.
Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. KGA: A general machine unlearning framework based on knowledge gap alignment. In ACL (!), pp- 13264-13276. Association for Computational Linguistics, 2023.
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann. Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks. In NeurlPS, 2024.
Lefeng Zhang, Tianqing Zhu, Haibin Zhang, Ping Xiong, and Wanlei Zhou. Fedrecovery: Differen- tially private machine unlearning for federated learning frameworks. IEEE Trans. Inf. Forensics Secur, 18:4732-4746, 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse effective unlearning. CoRR, abs/2404.05868. 2024.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018.
13

---

Published as a conference paper at ICLR 2025
A
EXPERIMENTAL SETUP
Hardware details. All experiments are conducted on a NVIDIA A100 GPU with 40GB of memory.
A.1 EXPERIMENTAL SETUP FOR LLM UNLEARNIN EXPERIMENTS
Datasets and models. We use two recent unlearning benchmarks for our evaluations. We conduct experiments on TOFU, which consists of 200 fictitious author profiles (Maini et al., 2024). These profiles are split into a retain and forget set, where the retain set is used to maintain model capabilities and the forget set is used for unlearning. Additionally, each profile is divided into multiple question- answer pairs. TOFU provides three different unlearning splits where 99, 95, or 90 percent of the data is used as retain set and the remainder as forget set. For measuring model utility after unlearning, TOFU additionally provides the Real Authors and World Facts datasets. All TOFU experiments are performed with the Phi-1.5 model (Li et al., 2023). In addition to TOFU, conduct experiments on model, which was unlearned to remove any Harry Potter-related
the Llama-2-V W hho-is-Harry-Pottee knowledge (Eldan & Russinovich, 2023). We use the recently proposed Harry Potter Q&A for evaluation (Schwinn et al., 2024). This dataset consists of pairs of questions and relevant keywords, allowing for the detection of information leakage through keyword matching.
Baseline metrics. In all experiments, we use ROUGE-L as a deterministic metric to measure infor- mation contained in the model after unlearning. ROUGE-L computes Rougt-Lcomputes statistic based on the longest common subsequence between two strings (Lin, 2004). Additionally, we use the ROUGE-L score obtained from multiple sampled responses to compute probabilistic metrics, such as bounds, mean, standard deviation, and the expectation-deviatio (ED) score. Note that our framework ($4) can be applicd to all deterministic metrics, such as perplexity or forget quality. We chose ROUGE-L as directly measures information leakage with respect to ground truth reference and is widely used in the unlearning domain. Throughout the manuscript, we use information leakage to refer to the mag- nitude of the ROUGE-L score, where a high score indicates high information leakage. We use the model utility score as described in TOFU to measure the generation quality of given model Maini al. (2024). We additionally employ the self-BLEU score (Zhu et al., 2018), which computes BLEU scores (Papineni et al., 2002) between generated samples and allows us to investigate the influence of our proposed unlearning algorithm on generation diversity.
Unlearning methods. We use Gradient Ascent (GA), Gradient Difference (GD) (Liu et al., 2022), RMU (Li al., 2024), and NPO (Zhang al., 2024) for a diverse selection of unlearning baselines and combine NPO with entropy optimization and adaptive temperature scaling for our approach since it the current ofate-of-the-art
Hyperparameters. For all unlearning algorithms we use a learning rate of le 5 5 with cosine learning rate schedule with warmup ratio of 0.1, batch size of 32, and weight decay of 0.01. For NPO we set ANrO = 0.05. We use 10 training epochs for all experiments as in (Maini et al., 2024). For probabilistic evaluations we sample n = 1024 model generations for every experiment if not stated otherwise. Probabilistic guarantees are calculated with a high probability guarantee of a = 0.01. We set the adaptive temperature scaling threshold cr = 0.9 for all experiments. This was done the average confidence of all models remained considerably below 0.9 during training. In our experiments, adaptive temperature thresholding has a negligible effect on the diversity of the model outputs using this threshold (see Section 6.1.2) In the unlearning setting, for each generation we sample sequences that are 64 tokens long. For all generations we used top-p sampling with = 0.9.
A.2 EXPERIMENTAL SETUP FOR LLM ALIGNMENT EXPERIMENTS
We conduct our alignment experiments on the 100 harmful behaviors dataset of JailbreakBBench (JBB) (Chao et al., 2024). Toxicity scores are derived from the Harmbench toxicity classi- fier (Mazeika et al., 2024), which provides the probability of an answer being rated as toxic. We con- duct our evaluations on Phi-1.5 (Li et al., 2023), Vicuna-7b- 7b-1.. (Chiang et al., 2023), and Mistral-7b- instruct-v0.3 (Jiang et al., 2023). For each generation, we sample sequences that are 128 tokens long. For all generations we used top-p sampling with p 0.9.
14

---

Published as a conference paper at ICLR 2025
B EFFECT OF ENTROPY OPTIMIZATION ON MODEL UTILITY IN UNLEARNING
In Table we summarize the results of Figure 5 (a) and Figure (c).
Table 2: Utility of unlearned models using the proposed unlearning method for different retain/forget splits of the TOFU dataset and varying choices of entropy regularization. We observe no consistent negative effect of entropy regularization the utility of an unlearned model.
A Utilityss
X
Utility
Xy
Utilityso
Utility
0.000
0.496
0,000
0.503
0,503
0.499
-0.125
0.509
0.125
0.488
0.491
0.503
-0,250
0,504
0.250
0.488
0.487
0.512
-0.500
0.490
0.500
0.492
0.500
0.500
1.000
0.492
0.469
0,492
C METRIC GUARANTEE PROOFS
Note that confidence intervals have two bounds that share significance level of a, meaning each bound uses significance level of a a/2. Consequently, since we propose metrics based on one bound only, our bounds can make use of the full significance level a.
Recall the definition of the Clopper-Pearsee confidence interval (Clopper & Pearson, 1934):
B ( Sw, n Sn + 1) S B
2 S 1, +, 1.n- S,,
where Biq a, b) the ğth-quantile of the beta distribution with shape parameters and We pro- pose unlearning metric based on the conservative Clopper-Pearson confidence bound follows:
Metric (Binary leakage bound). We define the binary metric Muin A B(1 or S, + 1, S,) where B(q: a, b) the ğth-quantile of the beta distribution with shape parameters and b.
Proposition 1. With high probability of at least 1 - metric Main represents an upper bound on the probability that the next sample leaks information, p S Main.
Proof. The statement follows directly from the definition of the Clopper-Pearson confidence inter- vals (Clopper & Pearson, 1934).
Metric (General leakage bound). Given specified percentage [0, 1] of the information the
model should not leak, we define the metric Mzen(x) e (n(x) + with =
In(1/o)
2n
Proposition 2. With high probability of at least 1 a. metric M2(x) upper-bounds the probability that the next sample leaks more than x% of the secret, Pr(X > x) M2(x) for all x E [0. 1).
Proof. The
inequality guarantees
In 1/2
Pr sup Fn (x) F((x) =0) s S 2nc
for all
2
EER
2n
In(1/a) for a S we have: 2n
Choosing
Pr sup F2 F(E) Sa
FER
+ Pr (F,(z) F(x) (())) VrER
* PPr((,()) - <> F(x)) Yr
#1 (c> 21-
WEER
* PPr(F.(2) (((())))
WE
Pr FPt(11FF())+<21 F(x)) > a V
15

---

Published as a conference paper at ICLR 2025
We use the Dvoretzky-Kieee Secffer-Woofooiizeecc inequality construct simultaneous confidence band:
p(X: > x) E - F,()) - G, (x) +c
In(2/a] 2n
where e
This follows directly from the two-sided DKW inequality:
Pr|sup (.((z) - F(x) > d S a for a = 2e 2nn
Note that if practice we have fixed for a significance level a (for example have to guarantee tight bounds), then we can exactly quantify the number of Monte Carlo samples needed:
2e nc² #n= In (V²).
Metric 3 (Expectation bound). We define the metric M, -1-EEs K-1 (FF (() the expected leakage E[X] of information with high probability of at least 1 - 20x. Proposition 3 (Anderson (1969)). We have [X E (1,11) with high probability of at least - for
that bounds
4= Σ
8 (F.(F.()))+ + and p=1- Σ
(F(((()))
where Fn(z) = 1 El_ 1(x, S x} the empirical CDF, E
[I[2/a]] 2n
and 6; = Ti+1 - Tr.
We exploit the relation between the CDF and the expectation: E[X] = = 1 So F(x)dx. We have
Proof.
[ F(x) dr
X =1
(1)
K-1
S
Σ
(Ti+1 F(T.)
k=
K-1
2 1
Σ
(T411 (T)(F 6)
1=0
K-1
Σ
(FF.(TT)) c)
where inequality (1) holds by bower-bounding the integral with the left Riemann sum, which is lower bound of the integral since the CDF is monotonically increasing. The second inequality (2)
holds due to the Dvoretzky- Kiefer- Wolfowitz inequality.
The lower bound follows analogously:
[ F(x) dx
E[X] = 1
(1)
2 1 Σ
(Tr -T-)))
② Σ
(Te (F)
K
=1
8: 1 +c)
16

---

Published as a conference paper at ICLR 2025
where inequality (1) holds by upper-bounding the integral with the right Riemann sum, which is an upper bound of the integral since the CDF monotonically increasing. The second inequality (2) holds due to the Dveretzky-Kiefer-- Seneeakky-Kfffe-Woofoowwoo Wolfowitz inequality again.
Following the variance bounds introduced in (Schuchardt et al., 2023) we propose the following bound on the standard deviation as unlearning metric:
Metric (Standard deviation bound).
We define the metric Ma - σ for
K
き² = -K-1- (TT +
Σ de sign ((.)().)((T.) + (1 sign( =
where 7/, ( (K a)² for i {O...., 1} and 6; 17--1 7/--
Proposition 4. With high probability of at least 1 - a, metric M (xx) upper-bounds the standard deviation of X. v Var[X] < Mo.
Proof. We have Var[X] = E[(x- E[X]) = fo (a- fx dr
(x [[X]) fx( dx
(x- fx dr
K
T(+1
fx(x) dx
for
max (
(K a)²
K-1
Σ ((((TT+1) -((T))
7--1 F(Fo) Σ 6,F(TT) for =, 1/-1 17
K
S 7- - 1 F(Fo) + Σ
& [sig((5 F()) + (1 -
From Var[X] 52 follows / /ar X] S o since the square root is monotonically increasing.
17


Reviews:
[{"review_id": "02CjSuqtuK", "reviewer": "Reviewer_MsxG", "summary": "In this paper authors address the problem of reliable unlearning in LLMs. First they introduce a problem, that evaluations based on deterministic point estimates (sampled texts) fail to reliably catch the risks exposed in probablistic outputs. For the case of unlearning, authors state that existing methods rely on a single generated sequence to identify if the information leakage is present or not. Which might not be enough when assessed model might still eventually produce a text with leaked information (with some probability). Therefore authors propose a set of 4 metrics aiming accurately quantify information leakage in model output distribution. Then, authors propose a novel unlearning training objective, which aims to simultaneously minimize model's output distribution entropy on a set of \"forget samples\" while retaining diversity on \"retain samples\". The loss itself is a set of additional terms which can be applied to some existing unlearning objectives. Finally, authors conduct comprehensive evaluation of unlearning with different methods using the proposed metrics.", "questions": "1) The term $\\alpha$, which is used extensively in formulation of metrics and overall thorough the paper, is only loosely defined in appendix. While it may be the usual practice in math-heavy papers, it does substantially confuse readers who are not so proficient. It is quite pity to read a definition or proof and find terms that simply not defined anywhere above. Consider defining $\\alpha$ in the main text of the paper.\n2) How increased $\\lambda_r$ impacts metrics other than diversity?\n3) How does proposed EO objective impacts training efficiency (in terms of increased latency or increased VRAM requirements)? Does it limit it's applicability?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1) Authors provide 4 carefully defined metrics, along with necessary guarantee proofs. Overall, the paper is very well composed.\n2) Those 4 proposed metrics allow to comprehensively evaluate model unlearning using entire output distribution (potentially, right now it is limited to MC sampling on certain examples). This appears to be a novel contribution and addresses the lack of probablistic evaluation in the field of unlearning.\n3) This approach can be potentially extended to other tasks which require reliable evaluations.\n4) The proposed entropy optimization objective is clearly defined and is formuated as additive terms which can be applied to existing unlearning losses, which makes it easy to implement. Addressing diversity on retain samples allows to ensure that models remains useful after unlearning.", "weaknesses": "1) While paper title reads as \"A Probabilistic Perspective on Unlearning and **Alignment** for Large Language Models\", authors effectively **do not touch** the alignment in their work, leaving it for further research. Indeed, alignment is only mentioned in Introduction, Limitations and Conclusion. This paper would benefit from having at least small discussion on how the proposed metrics can be extended to other evaluation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaxeptKTLS", "reviewer": "Reviewer_B7r6", "summary": "This paper proposed a set of metrics that bounds the risk of unlearning by estimating the bounds of probability of leaks, and the deviation of such random variables. Instead of computing the metric over deterministic point estimates drawn from greedy decoding of LLMs, it proposes the use of Monte Carlo methods, and then estimates these bounds by computation over the empirical distribution. Additionally, the authors proposed some mitigation methods to reduce the risk of leakage when fine-tuning a LLM, and show through experiments that such measures offer potential for reducing the risk and undesired biases in model outputs.", "questions": "- L118: $V^\\infty$: I believe that the more accepted notation for sequences of arbitrary sequences is $V^*$, where $*$ is the Kleene star. \n - L150: \"extend\" -> \"extent\".\n - L177: \"Binary case\": This exposition feels a bit verbose. My understanding is that you are empirically fitting a Beta distribution based on whether data is leaked through your Monte-Carlo experiments, and outputting a quantile based on a desired safety level $\\alpha$. Please correct me if my understanding is not correct.\n - L197: Make this part more self-contained: especially, what is the Dvoretzky-Kiefer-Wolfowitz inequality and how does it apply here?\n - L211: Proposition 3: This lower and upper bound is reminiscent of the Darboux integrals of $F_n$. If possible, please elaborate on the relationship of this bound estimate to an underlying integral expression. Additionally, it'd be good to reiterate that $F_n$ is the empirical  CDF.\n - L225: What does $\\eta_i$ bound? Please discuss.\n - L230: $M_4$: I think it'd be better to call this something like $M_\\sigma$, to be clear that this is not an estimate of the probability.\n - L246: $\\bar X + 2\\bar \\sigma$: why 2? I believe this is a choice based on the safety level, but it would be better to define this as a hyperparameter whose selection is based on the accepted risk level.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes metrics that is defined on the output distribution rather than the point estimate. I think this is a remarkable step and should be considered by various other scenarios.\n - The exposition on the estimation of probability of leakage and bounds of standard deviation are intuitive and sound. \n - The set of experiments presented are convincing.", "weaknesses": "- Notations can be improved in the exposition. For example, $M_1, \\cdots, M_4$ actually stands for estimates of different variables, rather than 4 different ways of estimating the same variable. See questions below for more suggestions. \n - Some derivation are not self-contained (e.g. in Metric 2, the $\\epsilon = \\sqrt{\\frac{\\log (1/\\alpha)}{2n}}$ is not self-contained and is derived from prior work. \n - Expositions tend to be a bit too formal, and lacking some intuitions and insights. See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqVGpMpoVW", "reviewer": "Reviewer_v1rM", "summary": "This paper introduces a probabilistic perspective on LLM evaluation which shifts from single point estimates towards evaluating entire output distributions offers significant potential for the field of unlearning and proposes a novel framework to directly assess the output distribution of a model. Besides, an unlearning loss based on entropy optimization and adaptive temperature scaling is also proposed.", "questions": "See weaknesses.", "limitations": "", "rating": 10, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Perspective on LLMs Unlearning Evaluation** Existing deterministic metrics are apparently insufficient for LLM unlearning evaluation, and this paper introduces a probabilistic perspective to mitigate this issue.\n\n2. **Adequate Mathematical Derivation** In this paper, the authors demonstrate the rationality of their method theoretically and empirically.", "weaknesses": "1. **More Discussion on LLM Alignment Evaluation** Since the titile contains \"ALIGNMENT\", more discussion on this topic should be included in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ns1jlxA5uv", "reviewer": "Reviewer_tRL6", "summary": "This work introduces a probablistic perspective for LLM unlearning evaluation. Instead of relying on deterministic greedy decoding in existing evaluation methods, this work takes a probablistic framework and derive metrics considering the high-probable output distributions. The proposed metric demonstrates the limitations of previous methods for their lack of identifying false unlearning. Moreover, a novel loss based on entropy optimization and adaptive temperature scaling are introduced to improve model unlearning.", "questions": "- For entropy optimization, I'm not sure about the intuition to minimize the entropy on Dfg. Wouldn't this lead the model to be confident on a different answer, which I think might be a strange thing to enforce.\n- It would be interesting to how unlearning (as well as the proposed optimization methods) affects the model's general ability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Designing good evaluation metrics is important for a reserach direction such as unlearning, and this work indicates a limitation of existing metric and correspondingly proposes improved metrics.\n- The proposed metrics and methods are shown to be effective for two recent unlearning benchmarks.", "weaknesses": "- There is a lack of algorithmic description on how the proposed metrics are calculated, without which readers who lack certain statistical machine learning knowledge or who want to implement the metrics would find it difficult to understand and apply the proposed metrics.\n- The proposed metrics are only tested for the unlearning case, which surely is indeed a well-suited scenario. Nevertheless, it would nice if it can be extended to more use cases, such factuality, to verify the effectiveness of the metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)