Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning
Authors: Yejin Kim
Eunwon Kim
Buru Chang
Junsuk Choe
Venue: COLM 2025 (2025)
Keywords: Machine Unlearning
Large Language Model
TL;DR: We propose VILA, a scalable and efficient unlearning method for LLMs that addresses FILA's limitations by improving parameter importance estimation and reducing computational overhead.

Abstract: LLMs have demonstrated remarkable performance across various tasks but face challenges related to unintentionally generating outputs containing sensitive information. A straightforward approach to address this issue is to retrain the model after excluding the problematic data. However, this approach incurs prohibitively high computational costs. To overcome this limitation, machine unlearning has emerged as a promising solution that can effectively remove sensitive information without the need to retrain the model from scratch. Recently, FILA has been proposed as a parameter-efficient unlearning method by integrating LoRA adapters. Specifically, it calculates the Fisher information to identify parameters associated with the forget set and assigns them to LoRA adapters for updates. Despite its innovative approach, FILA still requires access to all model parameters and does not adequately account for fundamental assumptions underlying Fisher information, leading to inaccuracies in importance estimation. To address these limitations, we propose VILA, a novel unlearning framework that explicitly considers the assumptions overlooked in FILA, thereby enhancing the accuracy of parameter identification for the forget set. Moreover, VILA significantly reduces computational costs by enabling parameter identification without accessing the entire model. Our method achieves up to 100× higher parameter efficiency and 40× faster training speed compared to FILA, and sets new state-of-the-art performance on benchmarks including TOFU, WMDP, and MUSE. Our code is available at https://github.com/kyj93790/VILA.

Published as a conference paper at COLM 2025
Improving Fisher Information Estimation and Efficiency for LoRA-based LLM Unlearning
Yejin Kim* Sogang University
Eunwon Kim* Sogang University
Buru Changt Korea University
Junsuk Choet Sogang University
Abstract
LLMs have demonstrated remarkable performance across various tasks but face challenges related to unintentionally generating outputs containing sensitive information. straightforward approach address this issue to retrain the model after excluding the problematic data. However, this approach incurs prohibitively high computational costs. overcome this limitation, machine unlearning has emerged as promising solution that can effectively remove sensitive information without the need to retrain the model from scratch. Recently, FILA has been proposed as parameter- efficient unlearning method by integrating LoRA adapters. Specifically, it calculates the Fisher information to identify parameters associated with the forget set and assigns them to LoRA adapters for updates. Despite its innovative approach, FILA still requires access to all model parameters and does not adequately account for fundamental assumptions under- lying Fisher information, leading inaccuracies in importance estima- tion. To address these limitations, we propose VILA, novel unlearning framework that explicitly considers the assumptions overlooked in FILA, thereby enhancing the accuracy of parameter identification for the forget set. Moreover, VILA significantly reduces computational costs by enabling parameter identification without accessing the entire model. Our method achieves up to 100x higher parameter efficiency and 40x faster training speed compared to FILA, and sets new state-of-the-art performance on benchmarks including TOFU, WMDP, and MUSE. Our code is available at https://gi thub. cot/kyy9379////VI....
1 Introduction
Large Language Models (LLMs) are driving remarkable progress across wide range of applications. However, they also exhibit a critical risk: the tendency to memorize and regenerate sensitive personal information For instance, Brown et al. (2022) have shown that LLMs often output personal identifiers such email addresses and phone numbers from the training corpus. Similarly, LLMs are known to reproduce copyrighted materials, such as passages from Harry Potter, with high fidelity (Eldan & Russinovich, 2023). These issues raise serious concerns about privacy violations and intellectual property infringement. As result, there is growing demand for methods that can effectively remove sensitive or proprietary information from LLMss
copyrighted content from their training data.
The most straightforward way to remove specific information from model is to retrain it from scratch without the corresponding data (i.e,, exact unlearning). However, given the massive size of LLMs and their extensive training corpora, this approach is computationally expensive and time-consuming. To address this challenge, recent research has focused on methods that aim to eliminate the information to be forgotten without full retraining (i.e., approximate unlearning). For example, loss-based techniques such as Gradient Ascent (GA) (Jang et al., 2023), Negative Preference Optimization (NPO) (Zhang et al., 2024), and Inverted Hinge Loss (IHL) (Cha et al., 2025) have been proposed to reduce the likelihood of generating specific content through fine-tuning.
* "Co-first authors.
+ Co-corresponding authors.

---

Published as a conference paper at COLM 2025
Nevertheless, directly updating billions of parameters remains computationally demand- ing, even when applying approximate unlearning techniques. To alleviate this burden, Fisher-nnitalizationn of Low-rank Adapters (FILA) (Cha et al., 2025) has been introduced. FILA leverages Fisher information (Fisher, 1922) to estimate gradient variance and identify parameters most closely related to the data to be forgotten. These parameters are isolated from the base model by assigning them to a LoRA adapter (Hu et al., 2022). Unlearning is then performed exclusively on the adapters. This enables parameter-effiiiet unlearning while minimizing the impact on the retained knowledge.
However, our analysis reveals two critical limitations of FILA. First, the Fisher information used by FILA does not accurately represent parameter importance in the machine unlearning setting. For Fisher information to indicate importance, the distribution of the forget set must match that of the full dataset. However, the forget set typically constitutes only small fraction of the entire dataset, inevitably leading to statistical discrepancy between the forget set and the full dataset. FILA overlooks this discrepancy, which results in a forget importance map that inaccurately captures the association between the forget set and the parameters. Moreover, although FILA is designed for parameter-efficien unlearning, it still requires computing full gradients for all model parameters to construct the importance map. This significantly undermines its com computational efficiency. Our analysis shows that the cost of FÍL.A grows rapidly with the size of the forget set. When forgetting 10% of the dataset, the initialization time exceeds that of full model serrininng-highlighee limitation in scalability (refer to Section 4.2).
a serious
Building on the above analysis, we propose precise and scalable approach, Variance-based Importance estimation and efficient Low-rank Adaptation (VILA). Our method improves the estimation of parameter importance by explicitly considering the distributional shift of the forget set. Furthermore, we construct the forget importance map solely using the gradients from the LoRA adapters, resulting in up to a 40x speedup and approximately 100x reduction in memory consumption compared to FILA as the size of the forget set increases.
We evaluate our method on multiple LLMs, including Phi-1.5B (Li et al., 2023), Llama2- 7B (Touvron et al., 2023), Zephyr-7B (Tunstall et al., 2024) and ICLM-7B (Shi et al., 2024), in combination with existing unlearning loss functions such as GA, NPO, and IHL. Experimen- tal results on the TOFU (Maini et al., 2024), MUSE Books (Shi et al., 2025), WMDP Bio and WMDP Cyber (Li et al., 2024) benchmarks demonstrate that our method not only improves resource efficiency but also sets new theee-tf in unlearning performance.
2
Related Work
LLM Unlearning aims to eliminate the influence of specific data from large language models without incurring the cost of expensive retraining. This approach addresses various challenges, such preserving privacy, resolving copyright issues, and removing hazardous knowledge (Brown et al., 2022; Eldan & Russinovich, 2023; Li et al., 2024).
copyrigg
Several studies mainly focus on modifying the loss function to induce unlearning. A representative example is Gradient Ascent (GA), which increases the loss on the forget data in order to reduce the model's predictive accuracy on that data (Jang et al., 2023). The limitation of GA is that it can easily degrade performance on retain data (Maini et al., 2024). To address this issue, Gradient Difference (GD) has been introduced, performing gradient ascent on forget data to eliminate their influence while applying gradient descent on retain data to preserve the model's generalization ability (Liu al., 2022a). Also, Negative Preference Optimization (NPO) (Zhang et al., 2024) has been proposed, building on the LLM alignment approach (Rafailov et al., 2024). By reweighting gradients during the learning process, NPO addresses the issue of excessive unlearning commonly caused by GA, significantly improving the stability of the unlearning process. Most recently, Inverted
Hinge Loss (IHL) (Cha et al., 2025) promotes unlearning by decreasing the probability of the forget token while increasing the probability of the highest-probabile alternative token, excluding the forget token itself.
2

---

Published as a conference paper at COLM 2025
Beyond loss function-based methods, various approaches have been proposed. Task Arith- metic (Ilharco et al., 2023) defines the difference between the fine-tuned model only on the forget set and the original model fask vector, which is then negated from the original model to induce forgetting. This approach, known as Forgetting via Negation, has been shown to be effective in making LLMs unlearn harmful language generation or fail at performing specific tasks. ULD (Ji et al., 2024) utilizes an auxiliary LLM to achieve the unlearning objective during the decoding process of LLM. The auxiliary LLM is trained to actively memorize the forget set while simultaneously forgetting the retain set. The un- learned LLM generated by calculating the logit difference between the auxiliary LLM and the original model, thereby effectively achieving the unlearning objective. FILA (Cha et al., 2025) employs LoRA adapters (Lermen efficiency of LLM unlearning. To achieve this, FILA identifies parameters associated with the forget set and initializes the LoRA adapters to be strongly correlated with the forget set, while the base layer is initialized to be closely related to the retain set. Subsequently, the LoRA adapters are tine-tuned using unlearning loss functions. FILA is the most related work to our study, as also focus achieving parameter-effiient unlearning.
fogers-Smithh 2024) improve the computational
Preliminaries
3.1 Problem Definition
The goal of unlearning is to effectively eliminate the knowledge associated with specified forget set D in the LLM, without retraining the model from scratch. At the same time, the model is expected to preserve its performance on retain set D,, which contains knowledge that must be maintained. This objective can be formulated
optimization problem as:
min (xy))ED ( x;0]] + A (xy)) [cr x;
(1)
In this formulation, by denotes the loss function applied to the forget set D,, encouraging the model to remove the corresponding knowledge. On the other hand, L, is the loss function applied to the retain set Dr, which ensures that essential knowledge is preserved. The model parameters are represented by 6, which are updated during the unlearning process. The hyperparameter À controls the strength of the retention loss term, effectively regulating how strongly the model is penalized for deviating from the retain set.
3.2 FILA: Fisher-Initializztion of Low-rank Adapters
FILA achieves parameter-efficient unlearning by employing LoRA to identify parameters critical to the forget set and focuses updates on these parameters during unlearning. The overall procedure as follows.
Low-rank Adaptation (LoRA). LoRA approximates the parameter update AW of model's base weight matrix W by training an adapter composed of two low-rank matrices, and A, such that AW B.A. The adapter is then added to W to produce the final model. Since B and A contains far fewer parameters than W, this approach enables efficient fine-tuning of LLMs with substantially fewer computational cost.
Forget Importance Map Extraction. FILA employs Fisher information (FI) to identify parameters associated with the forget set. The Fl of a dataset D with respect to model parameters 0 is defined as:
[(00 log (ogpo(D))
a a0 (om())))))) XED
Fo(D) EE
Σ
(2)
D
LLM denotes the next-token prediction loss used in the pre-trained language model. The Fl measures the variance of the score function, which is the gradient of the log-likelihood with respect to the model parameters. Intuitively, it captures how sensitively the model output changes in response to perturbations in each parameter. A higher FI value indicates that the parameter plays a more critical role in modeling the dataset D.
3

---

Published as a conference paper at COLM 2025
Method
Forget 1%
Forget 5%
Forget 10%
Retrain
2.28
2.18
2.08
FILA MMD) extraction
0.25
1.21
9.10
FILA
Unlearning
0.02
0.06
0.12
Table 1: Time Costs. This table reports the GPU hours required for Retrain, Unlearn, and FILA on the TOFU benchmark using the Llama2-7B model. The Forget N% setting indicates that N% of the full dataset is designated as the forget set. For Retrain, the model is trained from scratch using only the retain set, which consists of the remaining (100-N)% of the data.
Based on this interpretation, FILA computes the ratio of FI values obtained from the forget set and retain set to determine how important each parameter is with respect to the forget set. This ratio is referred to as the forget importance map, denoted as (D)):
Fe(D))
M(D) = Fe(D,)
(3)
The computed M(D plays a critical role in assigning weights to important parameter in LoRA-basee efficient unlearning.
LoRA Initialization with Forget Importance Map. FILA modifies the initialization of both the base layer and the LoRA adapter in way that is suitable for unlearning by leveraging the forget importance map. First, FILA formulates the following Weighted Low-Rank Approximation (WLRA) objective to obtain B" and A":
Σ (MM); (W (W- BA).)) 1.
B* A* = arg min
(4)
B,A
Since the forget importance map M assigns larger weights in WLRA to parameters more relevant to the forget set, the resulting product B* A" captures the components of original weight matrix W that have significant influence on the forget set. Based on this, we initialize the LoRA matrices B and A with B* and A*, respectively, so that the adapter focuses on forget-set-related parameters. FILA then obtains W by subtracting B* A from W, using as the new base layer. Since B* A' concentrates information specific the forget set, the subtraction W = W B" A removes forget-set-related parameters while preserving those relevant to the retain set.
Through this initialization of both the adapter and the base layer, the overall model parame- ters remain unchanged, as W = (W - B A*) + B* A*. However, the information associated with the forget and retain sets becomes cleanly disentangled.
Paraieter-efficcient Unlearning. After initializing LoRA, FILA freezes the base layer and updates only the LoRA adapter parameters using unlearning loss. Since parameters crucial for the forget set are allocated to the trainable adapter, while those important to the retain set remain in the frozen base layer, the model can effectively erase the undesired information while preserving essential knowledge. The final unlearned model is obtained by merging the updated adapter- now now purged of forget set information-wre the base layer.
4
VILA: The Proposed Method
4.1 Corrected Parameter Importance Estimation
We argue that the forget importance map calculated by FILA inaccurate. FILA estimates the Fisher information (FI) of each parameter with respect to the forget set and the retain set, then interprets Fl as a variance measure to derive the forget importance map based on the ratio of these Fls. However, this approach overlooks critical assumption required to interpret Fl as a variance: the expectation of the score function (i.e., the gradient) must be zero (Fisher, 1922). This condition holds only when the distribution of the forget set matches
4

---

Published as a conference paper at COLM 2025
the distribution of the entire training data. In machine unlearning tasks, however, the forget set is typically a subset of data that has been intentionally selected for removal, making its distribution inherently different from that of the entire dataset. As result, the score function has a non-zero expectation, violating the necessary assumption. To reliably identify parameters strongly associated with the forget set, it is essential to account for distributional discrepancies that arise in unlearning scenarios.
To address this issue, we correct the Fl (Equation 2) by explicitly subtracting the squared expectation of the score function from the original formulation. This is equivalent to the variance of the parameter AW the dataset D:
(
a aa log PW (D))
-(E
a aw log Pw (D))
Varp [AW] == ED
ED
(5)
We regard this modified quantity an adjusted importance score for the dataset. Ex- perimental results demonstrate that this modification significantly improves unlearning performance. While the solution is simple, identifying and correctly addressing this over- looked aspect in existing work constitutes one of the key contributions of this paper.
4.2 Improving Efficiency via Low-rank Approximation
Despite its intended goal, FILA is not computationally efficient. While FILA aims to perform parameter-efficient unlearning by adopting LoRA, still requires access to the entire set of LLM parameters to compute the forget im importance map. As result, the importance map calculation remains computationally expensive.
We report the computational time required for model retraining, importance map extraction using FILA, and model unlearning in Table 1, empirically demonstrating these inefficiencies. Notably, the extraction of the forget importance map, intended a preprocessing step, incurs even greater computational cost than the unlearning process itself. This inefficiency becomes especially pronounced when the forget set constitutes approximately 10% of the training data, where importance map computation exceeds the time required for retraining. These results suggest that FILA does not scale well with forget set size, making it suboptimal in terms of efficiency. Thus, achieving truly efficient unlearning necessitates more efficient approach to extracting the forget importance map.
To address this issue, propose approach that utilizes the gradients of LoRA adapter rather than those of the entire model. First, we initialize the LoRA adapter matrices and A independently, following a Gaussian distribution with mean of zero. Next, we add the adapter BA to the original model parameter W and compute the gradients of and A for given input data D. Using these gradients, we calculate VaBD[AB] and Varp AA], respectively. We then multiply these two values to obtain the variance of the model parameters W:
Varp
Varp [AA].
(6)
One critical aspect of our approach is understanding how the variance of the gradient of the model parameter can be approximated with those of LoRA adapters. To explore this, we present the following theorem:
Theorem 1 (Variance Approximation of LoRA Parameter Updates). Let DD be the input data, W be the model parameter matrix and let AW denote its update. In the LoRA framework, the parameter update AW is represented as the product of two low-rank matrices B and A such that:
AW BA.
Assuming that both B and A are independently initialized from zero-mean Gaussian distributions, the variance of each element AW,j can be approximated as:
Var [AW] Varp AA].
(8)
Proof. The proof is provided in the Appendix A.
5

---

Published as a conference paper at COLM 2025
Algorithm 1 Unlearning Process
1: Input: Forget set D, f, retain set D,
2: Output: Unlearned model Wunleamn
3: Step I: Estimate Variance- Based Importance
4: Initialize LoRA matrices: ~ N((,, A ~ N (0,0²
5: Compute gradients of B and A with respect to D and D,
6: Calculate variances:
Varp, AB],
Varp, (AA), Barp/AB),
Varp, [AA]
(10)
7: Estimate importance map:
M(D) Varp, [AB] [AA]
(11)
Varp, [B AA]
8: Step 2: Compute Weighted Low-Rank Approximation
9: Solve WLRA using M:
B" A arg min E (M,((WBA)s) B.A
(12)
10:
Initialize LoRA adapter: lora.A
A Bor B
Set base layer W": base layer B"
12:
Step 3: Perform Unlearning
13:
Freeze base layer W"
14:
Optimize LoRA parameters:
B', A' = arg min IE (xy))D (y x;0) + F (xy)ED, [4r(y x;0] B'
(13)
Return:
Final unlearned model: Wumleam W + B'A'
Finally, we derive the forget importance map as the element-wise ratio of the importance values calculated for the forget set and the retain set:
Varp, [AW]
VarD, [aB] [AA]
M(D) Varp, AW] Varp, [BB] Varp, [A]]
99
In this way, efficiently compute the forget importance map without directly accessing the entire parameter set of the LLM. The pseudo code of unlearning process is in Algorithm 1.
Experiments
Benchmarks and compared methods. We evaluate unlearning performance using three benchmarks: TOFU (Maini al., 2024), WMDP (Li et al., 2024), and MUSE (Shi al., 2025),
and primarily compare our method against FILA with three unlearning loss functions: GD (Liu et al., 2022b), NPO (Zhang et al., 2024), and IHL. (Cha et al., 2025). Further details on benchmarks and compared methods are provided in Appendix H and I, respectively.
Implementation details. All experiments are conducted using two NVIDIA A6000 GPUs with 48GB of memory. The batch size is set to 32 for TOFU and MUSE, and 4 for WMDP. The LoRA rank is set for TOFU and WMDP, and for MUSE. Weight decay is configured as 0.01 for TOFU and set to 0 for both MUSE and WMDP. We employ linear learning rate scheduler for WMDP and TOFU, and a constant scheduler for MUSE.
Fair and comprehensive experimental designs. To ensure fair comparison, we conduct the same number of hyperparameter searches for all compared methods. Specifically, we perform random search (Bergstra & Bengio, 2012) within a predefined hyperparameter range for each benchmark. We set the maximum unlearning epoch based on retraining cost considerations. Furthermore, to avoid evaluating models with significantly degraded

---

Published as a conference paper at COLM 2025
Model
Method
Forget 1%
Forget 5%
Forget 10%
AVG Gain (t)
Original Model
4.05
-11.92
-15.66
GD
-2.52
-11.18
-14.43
GD FILA
-2.17
-10.23
-13.84
0.63
GD Ours
-1.54
-9.61
-10.80
2.06
Phi-1.5B
NPO
-2.52
-7.89
-10.03
NPO FILA
-2.17
66009
-8.83
1.12
NPO Ours
-2.17
-5.17
-9.30
1.27
IHL
-2.52
-10.23
-14.13
IHI. + FILA
-2.17
-5.40
-1.79
5.84
IHL + Ours
-1.85
-1.17
-0.83
7.68
Original Model
-3.30
-15.46
-19.31
GD
-3.30
-9.92
-16.61
GD FILA
-3.30
-12.53
-17.27
-1.09
GD Ours
-2.17
-1.40
-1.18
8.36
Llama2-7B
NPO
-3.30
-13,59
-13884
NPO FILA
-3.30
-11.18
-11.06
1.73
NPO Ours
-1.54
-432
4.59
6.76
THI.
-3.30
-12.53
-7.70
IHL FILA
-3.30
-0.95
-0.47
6.27
IHL. Ours
-1.27
-0.20
-0.40
7.22
WMDP
MUSE BOOKS
Method
BIO (1)
CYB (
AVG (L)
VerbM (L)
KnowM (4)
AVG (4)
Original Model
0.64
0.44
0.54
85.5
30.5
58.0
GD
0.55
0.44
0.50
84.7
17.0
50.9
GD FILA
0.61
0.44
0.53
85.4
17.0
51.2
GD Ours
0.37
0.43
0.40
66.4
16.5
41.5
NPO
0.57
0.44
0.51
14.3
0.3
7.3
NPO FILA
0.61
0.43
0.52
85.0
12.0
48.5
NPO + Ours
0.35
0.43
0.39
2.8
3.7
3.3
IHI.
0.60
0.40
0.50
85.4
14.9
50.2
IHI. + FILA
0.61
0.39
0.50
85.4
16.7
51.1
IHL+ + Ours
0.48
0.41
0.45
14.2
1.8
8.0
Table 2: Main Comparison Results. Top: Unlearning performance on TOFU with Phi-1.5B and Llama2-7B across varying forget ratios. AVG Gain (t) denotes the average improvement in unlearning loss from each initialization method, measured across data splits. Bottom: Unlearning performance metrics per benchmark. Lower scores indicate better forgetting performance. Retain perfor- mance omitted, as it is constrained to remain above 95% by our evaluation protocol.
WMDP and MUSE Books. AVG is the mean of the two forget
utility, we select models that maintain at least 95% of the original model utility (Ilharco et al., 2023) while achieving the highest forgetting score. As the compared methods demonstrate comparable model utility, we report only the forgetting performance in the following tables. Additional details are provided in Appendix].
5.1 Comparison with FILA
This section presents the results the TOFU, WMDP, and MUSE benchmarks. Note that rows labeled as GD, NPO, or IHL refer to standard parameter-efficient unlearning baselines that utilize conventional LoRA adapter without applying either FILA method.
Main Results. In Table 2, we observe that both FILA and our proposed method significantly improve forgetting performance compared to applying unlearning loss functions with stan-
7

---

Published as a conference paper at COLM 2025
Method
Time (GPU hours)
Storage for M(D)
Forget 1%
Forget 5%
Forget 10%
Retrain
2.28
2.18
2.08
FILA
0.27
1.27
9.22
25G
Ours
0.04
0.18
0.36
0.3G
Table 3: Comparison of time and storage cost across different methods. We report GPU hours required for unlearning under varying forget set sizes 51, 10%) on the TOFU benchmark using the Llama2-7B model. Storage for (DD denotes the additional space required to store forget information map used during unlearning.
dard LoRA alone. This highlights the effectiveness of explicitly separating the parameters associated with the forget set, rather than relying on full-parameter updates.
While both approaches benefit from this separation, our method consistently achieves superior forgetting quality across all benchmarks and loss functions. FILA, in contrast, performs well primarily when paired with IHI.-the loss function introduced alongside it by Cha et al. (2025)-br but offers limited gains with GD or NPO, and in some cases even degrades performance. For example, on Llama2-7B with GD, applying FILA results in lower forgetting quality when unlearning 5% or 10% of the TOFU dataset. This aligns with the analysis in the original submission on Reenee (Cha et al., 2025), which points to FILA's limited generalizability beyond its tailored loss. In contrast, our method not only outperforms FILA when paired with IHL but also maintains strong performance across diverse unlearning losses, demonstrating its broader applicability.
Our method yields substantial improvements when applied to GD, where the forget loss is implemented via GA. Although GA often leads to instability and utility degradation (Zhang et al., 2024; Cha et al., 2025), our method effectively mitigates these issues and achieves strong forgetting performance. This result is also significant in MUSE, where the forget set size exceeds the retain set size, making it particularly difficult to preserve model utility. Despite this difficulty, our method substantially outperforms existing baselines, delivering up to 9.4%p increase in performance (50.9% + 41.5%) when combined with GD.
TOFU Results. The Forget 1% setting in TOFU is particularly challenging compared to other configurations. In this setting, only 1% of the entire dataset is designated as the forget set, meaning the model must remove knowledge from very small portion of the data. This also implies that only very small number of parameters are associated with the forget set, which makes it challenging to precisely identify and update the relevant subset. Especially with Llama2-7B, applying unlearning via GD, IHL, or NPO loss alone produces forget quality scores nearly indistinguishable from those of the original model, suggesting that information related to the forget set remains. Even when applying FILA, a slight improvement in forget quality is observed with the Phi-1.5B model; however, for Llama2-7B, the forget quality remains at the same level the target model. In contrast, our method achieves high forgetting quality while maintaining model utility, even when the forget set size is very small. We attribute this to the method's ss capacity to pinpoint parameters most relevant to the forget set.
WMDP and MUSE Results. In the WMDP experiments, our method continues demon- strate strong unlearning performance across all unlearning loss functions, while preserving accuracy on MMLU. In contrast, FILA consistently underperforms compared to the baselines on this benchmark, highlighting the limitations of its biased forget importance map estima- tion strategy. On the MUSE benchmark, models trained with FILA struggle to maintain retain performance above 95% of the original model. When hyperparameters are chosen to effectively reduce performance on the forget set, performance on the retain set also drops noticeably. Conversely, when the retain performance is preserved, the level of unlearning is limited. This behavior indicates that FILA fails to achieve effective unlearning. By contrast, our proposed method achieves consistent gains in forgetting performance across all loss
8

---

Published as a conference paper at COLM 2025
Loss
Method
Forget 1%
Forget 5%
Forget 10%
AVG Gain (t)
FILA (Baseline)
2.17
-10.23
-13.84
w/ FI Correction
-1.27
-9.61
-9.54
1.94
GD
w/ LoRA Approximation
-2.17
-9.61
-13.54
0.30
w/ Both (VILA)
-154
-9.61
-10.80
1.43
FILA (Baseline)
-2.17
-6.09
-8.83
w/ FI Correction
-188
-6.34
-5.85
1.02
NPO
w/ LoRA Approximation
2.177
-6.58
-9.30
-0.32
Both (VII.))
-217
-5.17
-9.30
0.15
FILA (Baseline)
-2.77
-5.40
-1.79
Correction
-1.54
-0.85
-0.47
2.17
IHL
w/ LoRA Approximation
-2117
-4.53
-0.19
0.82
w/ Both (VILA)
-1.85
-1.17
-0.83
1.84
Table 4: Ablation results isolating the impact of FI Correction and LoRA Approximation unlearning performance. Results show Forget Quality scores (lower is better). AVG Gain denotes improvement over the FILA baseline.
functions, while sufficiently preserving model utility. These results validate the effectiveness of our refined approach to estimating forget importance.
5.2 Efficiency Analysis
Table compares the time and storage costs for unlearning the TOFU benchmark using Llama2-7B as the backbone model. FILA incurs substantial time costs due to the need to compute gradients over the entire model to extract the forget importance map. This becomes increasingly inefficient as the forget set size grows. For instance, FILA takes 0.27 GPU hours for the 1% setting, but this rises sharply to 9.22 GPU hours for the 10% setting. Notably, this exceeds the retraining time of approximately 2.08 GPU hours, underscoring FILA's limitations in large-scale unlearning scenarios. In contrast, our method requires only 0.04 GPU hours for the 1% setting and 0.36 GPU hours for the 10% setting, while achieving comparable unlearning quality at fraction the computational cost.
A key difference between the two methods lies in storage efficiency. Both approaches require storing importance maps for the forget and retain sets. FILA computes these maps across all model parameters, resulting in total storage requirement of approximately 25 GB-about twice the size of the original model. In contrast, our method uses LoRA parameters instead of the full parameter set, requiring only 0.3 GB of additional storage.
Ablation Study on FI Correction and LoRA Approximation
To investigate how each of our two key contributions individually impacts the overall unlearning performance, we conduct ablation experiments clearly isolating the following two components: (i) FI Correction (Eq. 5), which involves correcting the Fisher Information estimation, and (ii) LoRA Approximation (Eq. 9), which approximates the full-model Fisher Information using LoRA adapter parameters to enhance computational efficiency.
Table summarizes detailed ablation results conducted using the Phi-1.5B model, across three unlearning loss functions (GD, NPO, IHL). From these ablation experiments, we observe the following. Correction alone consistently yields the highest unlearning performance across most settings, confirming the significant effectiveness of correcting Fisher Information estimation. In contrast, LoRA Approximation alone primarily enhances computational efficiency but achieves limited or no performance improvement compared to the FILA baseline. When the two components are combined- -as in VILA-the resulting method achieves performance close to that of FI Correction alone, while significantly reducing computational overhead. This demonstrates that VILA effectively balances strong unlearning performance with computational efficiency.

---

Published as a conference paper at COLM 2025
Loss
Method
Forget 1%
Forget 5%
Forget 10%
AVG Gain (t)
FILA
-2.17
-10.23
-13.44
ExpILA
-1.27
-10.54
-10.54
1.93
GD
AbslLA
-1.54
-10.23
-10.29
2.03
VILA
-1.54
-9.61
-10.80
2.06
FILA
-2.17
-6.09
-8.83
ExpILA
-1.85
-5.62
-10.54
2.74
NPO
AbsILA
-1.85
-5.86
-9.06
1.23
VILA
-1.85
-1.17
-0.83
7.68
FILA
-2.17
-5.40
-1.79
ExpILA
-0.39
-3.37
-6.05
2.43
IHL
AbslLA
-1.27
-3.55
-5.11
5.65
VILA
-1.85
-1.17
-0.83
7.68
Table 5: Validity of the Expectation Importance Score Phi-1.5B. AVG Gain (1) denotes average improvement in unlearning loss across splits.
5.4 Validity of the Expectation as Importance Score
When the distribution of the forget set (D,) significantly differs from that of the entire dataset (D), the expectation of the score function-specifficall specifically, the gradient of the log-likelihood with respect to the bercoms non-zero. This observation raises an important question regarding the validity of using the expectation itself directly an importance score. To empirically investigate this question, we consider two alternative formulations for the importance score, both based on the score function Vw log Pw (D):
a
ED,
log pw (D;)
ED,
log Pw (Dş)
aw
aw
MExpIIA ==
MAbsILA
(14)
a log Pw (Dr)
Ep,
log Pw (D,)
Ep,
aw
aw
Here, MExPILA computes the magnitude of the expected score function, whereas M ABSILA computes the expected magnitude of the score function. Intuitively, while EXPILA focuses on the norm of the average gradient, ABSILA accounts for the average sensitivity across data points, regardless of gradient direction cancellation.
Experimental results Table summarize the trends discussed above on Phi-1.5B. From these results, we observe several important findings. Both EXPILA and ABSILA generally perform comparably to, or slightly better than, FILA when the forget set is small (e.g., Forget 1%). However, the forget set grows (e.g., Forget 5% and 10%), VILA consistently outperforms both alternatives. These results indicate that while EXPILA and ABSILA may be reasonable under limited forgetting scenarios, correcting the Fisher Information in VILA-is essential for effective unlearning in realistic scenarios where the forget set significantly diverges from the full data distribution. Similar trends observed for Llama2- 7B (Appendix C).
6 Conclusion
In this paper, we introduce VILA, a scalable and efficient unlearning technique for large language models (LLMs) that addresses the limitations FILA by enhancing importance estimation and reducing computational overhead. Our approach refines the Fisher infor- mation extraction process and implements parameter selection related to the forget set by utilizing only the gradients of the LoRA adapter, significantly lowering time and memory costs. Extensive experiments on the TOFU, WMDP, and MUSE benchmarks demonstrate that VILA consistently outperforms existing approaches in unlearning performance while preserving model utility. Additionally, VILA demonstrates robust compatibility with a wide range of unlearning loss functions, highlighting its versatility.
10

---

Published as a conference paper at COLM 2025
Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (RS-2024-003504300, We sincerely thank the researchers at Sogang University's VRL especialylll Minyoung Lee, Dongjun Hwang, Yeji Park, and Beomyun Kwon-for their valuable feedback and support. We are also grateful Seong Joon Oh from the STAI Group at the University of Tübingen for his helpful discussions throughout the project. Finally, we would like to thank Sungmin Cha at New York University for his insightful feedback and suggestions.
References
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The journal of machine learning research, 13(1):281-305, 2012.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for language model to preserve privacy? ? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pp. 2280-2292, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. URL https://doo http///oi.org///// wrg.1...///ooo 1145/3531146. 3534642.
doi: 10.1145 011144553551146353464///
Sungmin Cha, Sungjun Cho, Dasol Hwang, and Moontae Lee. Towards robust and parameter-efficient knowledge unlearning for llms. In International Conference on Learning Representations, 2025.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms, 2023.
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594--04):309-366 1922.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https: /ttterv//pperreview.. net/ forum?id=nZeeWeeFFYy
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The International Conference on Learning Representations, 2023. URL https::/openreet www.ee/weeefffummiiiiii 6t0Kwf8-jrj.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 14389-14408, 2023.
Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient 11m unlearning framework from logit difference. Advances in Neural Information Processing Systems, 37:12581-12611, 2024.
Simon Lermen and Charlie Rogers-Smith. LoRA fine-tuning efficiently undoes safety training in llama 2-chat 70b. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL httpss///pppreeeee..effffrrrrrrn
11

---

Published as a conference paper at COLM 2025
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. The wmdp bench- mark: measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machine Learning, pp. 28525-28550, 2024.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. URL https:
waaaaiii.orrabb////// 05463.
Yhin Lin. ROUGEIN-CONTENT PRETRAINING: LANGUAGE MODELING BEYOND DOCUMENT BOUNDARIES; A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL ht httt::://aaaaatiien //aaaaaaaaooooyyo..coo wrrpe//-0r//000-113330
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pp. 243-254. PMLR, 2022a.
Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pp. 1749-1758. IEEE, 2022b.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024. URL https: http///pppeeeeeeeeeeeeeeeeeeeeee
ttttt/ofroddB/A/ 10644WLNB
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
J.K. Rowling. Harry Potter Series (7 Books). Bloomsbury (UK), Scholastic (US), 1997-2007.
Includes: Sorcerer's Stone, Chamber Secrets, Prisoner Azkaban, Goblet of Fire, Order of the Phoenix, Half-Blood Prince, Deathly Hallows.
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. In-context pretraining: Lan- guage modeling beyond document boundaries. In International Conference on Learning Representations, 2024. URL
net 2do.ummid--XVswInHHO
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. In International Conference Learning Representations, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. ar preprint arx:2:2007 2023.
Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https://openrreview wwt..e//oorummiddaakkw///./
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024. URL https: http:///eeeeeeeeeeee///oo///////
12

---

Published as a conference paper at COLM 2025
A Proof of Theorem 1
Proof. Step 1 (Parameterization and Approximation).
We start from the LoRA parameter update:
AW = BA (Bo + (Ao AA),
where Bo, Ao denote the initial low-rank parameter matrices, and AB, AA are their updates learned during training. Given the negligible magnitude of the initial matrices Bg, Ao (assumption A.1), simplify the update as:
AW ABAA.
Thus, each element of the parameter update AW can be approximated as:
AW 23
ABi
Step 2 (Variance Expansion).
Applying the standard property of variance to the sum of random variables, have:
VarD
Σ ABR
AAr + Σ CBy
k/k
Step (Covariance Terms are Zero).
We now show that all off-diagonal covariance terms vanish. Consider an arbitrary off- diagonal term (k + K'):
AA A En =p,
By the independence of matrices AB and AA (assumption A.2) and independence among distinct elements within each matrix (assumption A.3), each expectation factorizes exactly, yielding:
Covp CAv AB; O.
Thus, all off-diagonal covariance terms vanish, simplifying the expression to:
(Arp(AW,)]
Varp(ABaAAAj]
Step 4 (Variance Factorization and Practical Approximation).
Since the terms ΔB and AAf are independent (assumption A.2), we approximate:
Varp A B AAz Ep [ Ep |
(Ep E
Empirically, the squared expectations of the parameter updates are negligible compared to their variances (assumption A.4). Hence, the above expression simplifies into:
Var (AB ~ Varp (AB Var (AAg]-
Consequently, the final simplified variance approximation is:
Varp|AW,
Varp Ba
13

---

Published as a conference paper at COLM 2025
Term
Forget Set
Retain Set
Ag
0.00568
0.00568
A
39.75
37.0
BoAA
40.5
38.5
ΔBAA
688128.0
585728.0
Table 6: Average norm values of each term across all parameters. The gradients are computed separately for the forget and retain sets, resulting distinct values for gradient- dependent terms. Note that Ag does not depend on any gradients but is determined solely by the initialization, and thus yields the same value for both sets.
A.1 Negligible magnitude of initial matrices Ba, Ao
Our variance approximation assumes that the initial parameter matrices Bg and Ag the LoRA framework have negligible magnitude, allowing their direct contributions to the parameter updates to be disregarded in our derivations.
Specifically, matrices Bo and Ao are independently initialized from Gaussian distributions with mean and very small variance (typically on the order of 10 4 to 10-6) Due to this initialization scheme, the initial magnitudes of these matrices are sufficiently small that their direct contribution terms, such Bo BoAA, and ABA,, become negligible compared to the dominant update term ΔBАА.
In Table we empirically confirm that the magnitude of initial terms is significantly smaller than that of the learned update term. Specifically, averaged across all parameters, the norm of the dominant update term ABAA reaches values high as 688, 128 on forget set and 585,723 on retain set, while the norms of other terms such as Bn BoAA, and ABAr are several orders of magnitude smaller. These observations justify treating the initial-related terms as negligible when approximating the gradient dynamics.
A.2 Independence between update matrices AB and AA
We assume that the parameter update matrices AB and AA are statistically independent. The independence assumption
be justified if the following two conditions met: (1) the entries of AB and 4A follow Gaussian distributions, and (2) the covariance between elements of AB and AA is zero, i.e,
CovD(ABR, AAA) for all i,
We first examine the Gaussian assumption. This condition naturally arises from the initializa- tion strategy used in the LoRA framework, particularly in the context of Fisher Information (FI) computation. When computing FI, the model parameters are kept fixed, and gradients are repeatedly evaluated at the same point parameter space. To allow gradients to flow through the LoRA modules during this process, we initialize the low-rank matrices Bo and Åg with small Gaussian noise rather than zeros (see Appendix D for details). These initializations are independently drawn from zero-mean Gaussian distributions, which induces Gaussian in the gradient signals that propagate through AB and AA. As illustrated in Figure 1, the empirical distribution of gradient values indeed closely follows Gaussian shape, confirming the plausibility of this assumption.
Next, we assess the second snniremennt-vaisshin covariance between elements of AB and AA. To this end, we sample 500 gradient instances by repeatedly drawing different mini-batch combinations, separately for the forget and retain sets, and computing the corresponding LoRA gradients. We then calculate the element-wise covariance between AB and AA across these samples. The results, shown in Figure 2, reveal that the vast majority of covariance values are sharply concentrated around zero both cases. This confirms that any statistical dependencies between the two matrices negligible.
14

---

Published as a conference paper at COLM 2025
Lec Histogram of LoRA_B B Gradient Values (Forgel]
Histogram of LoRA_B Gradient Values (Relain)
1.75
1.25
1.5
Count
1.0
0.50
0.5
0.25
0.00
-0.4
-0.2
0.0
0.2
0.4
0.0
-0.4
-0.2
0.0
0.2
0.4
Value
Value
(a) AB on Forget set
(b) AB Retain set
1es
Histogram of LoRA A Gradient Values (Forget)
Histogram LoRA Gradient Values (Retain)
1.0
Count
0.8
0.8
0.6
0.4
0.2
0.2
-0.4
-0.2
0.0
0.2
0.4
0.9
-0.4
0.0
0.2
Value
Value
(c) AA on Forget set
(d) ΔA on Retain set
Figure 1: Histograms of individual entries AA and AB.
Taken together, these empirical validations support the assumption that AB and 4A are approximately independent. This independence greatly simplifies our variance-based theoretical derivations and is well-justified experimentally.
Elem entwise Covariance between AB and AA (Forget)
Elementwise Covariance between AB and AA (Retain) 1e)3
1el3
2.0
1.5
Count
1.0
0.5
0.0
Covariance
Covariance
(a) AB and AA on Forget set
(b) AB and AA on Retain set
Figure 2: Element-wise covariance between AB and AA. We compute the covariance between pairwise combinations of elements from the AB and AA gradient matrices, separately over the forget and retain sets. Each computed covariance value is counted to construct a histogram, allowing us to visualize the overall distribution cross-matrix interactions.
A.3 Practical independence among distinct elements within each update matrix
In our variance approximation, we assume that distinct elements within each update matrix (AB and AA) are statistically independent. This assumption is crucial for simplifying higher-order expectations. For instance, the fourth-order moment
15

---

Published as a conference paper at COLM 2025
is assumed to factorize as
Ep[ABa] [E[[AA)
Ep[ABR] [ED[AA
While Section A.2 has already justified the independence across matrices (i.e., between AB and AA), the above factorization further requires that elements within each matrix also be statistically independent. To support this intra-matrix independence, two conditions must be satisfied: (1) the elements each matrix follow Gaussian distributions, and (2) the pairwise covariances between distinct elements within the matrix negligible. The first condition has already been empirically confirmed in Figure 1, where we show that the elements of AB and ΔA follow approximately zero-mean Gaussian distributions under our sampling procedure.
To validate the second condition, compute the element-wise covariance between all pairs of distinct entries within each matrix (i.e., off-diagonal pairs). We sample 500 gradient instances by repeatedly drawing different mini-batch combinations, separately for the forget and retain sets, and compute the corresponding LoRA gradients.
As shown in Figure 3, the off-diagonal covariance terms in both AA and AB are sharply concentrated near zero for both the forget and retain sets. This empirically supports the ap- proximation that distinct elements within each update matrix can be treated as independent. Therefore, the full factorization of the fourth-order expectation becomes practically valid, and the assumption of intra-matris independence is well justified our analysis.
1et0 Elementwise SelfCCvariance of AB (Forget)
1e10 Elementwise Self-Covariance of AB (Retain)
Count
-0.0004
-0,0002
0.0000
0,0002
0.0004
-0. 00.0004
-0.0002
O. 0,0000
0,0002
0.0004
Covartance
Covariance
(a) AB on Forget set
(b) AB on Retain set
1e10 Elementwise Self Covariance of AA (Farget)
1e10 Elementwise Self Covariance of AA (Retain)
Count
-0.004
-0.002
0.000
0.002
0.004
-0.004
-0.002
0.000
0.002
0.004
Covariance
Covariance
(c) AA on Forget set
(d) AA on Retain set
Figure 3: Elementwise self-covariance histograms of ΔA and AB. The gradients are computed separately over the forget and retain sets. Each plot illustrates the distribution of self-covariance values within the gradient matrices, with diagonal entries excluded from the counting to emphasize inter-parameter interactions.
A.4 Negligible expectation values of parameter updates
We empirically observe that the squared expectation values Ep[AB and Ep Ep[AAy] are relatively small (on the order of 10 to 10 2), although not strictly negligible.
16

---

Published as a conference paper at COLM 2025
We acknowledge that this assumption of negligible squared expectations is not strictly accurate; however, we made this simplifying assumption intentionally to achieve substantial computational efficiency. Specifically, this design choice enabled approximately 100-fold reduction in memory consumption and a roughly 40-fold speed-up compared to previous approaches.
Our empirical analysis indicates that this approximation does not significantly degrade the method's overall performance. Relaxing this assumption could potentially lead to further performance improvements; exploring such refinements will be an interesting direction for future work.
Implementation Remark
Although our theoretical analysis assumes negligible squared expectations for simplicity, we observed empirically that the exact computation of variances for both AA and AB, including their squared expectation terms, leads to significantly improved performance:
Varp AAyy
(A; ) (Ed (ED[AA,))), Varp [
Ep
(Ep[AB;))
This indicates that while the assumption A.4 aids analytical simplicity and clarity, practical scenarios require careful consideration of nonzero expectation values for both AA and AB to achieve optimal performance.
B In-domain Unlearning Scenario
In our main experiments, existing unlearning benchmarks typically assume the forget subset (Df) differs significantly from the retain subset. However, the general definition of unlearning does not inherently impose this assumption, implying the forget set could, in theory, be arbitrarily selected from the same distribution as the entire dataset. Thus, we conduct additional experiments explicitly designed examine an in-domain unlearning scenario, where the distribution of the forget set closely matches that of the overall dataset.
In practice, exactly reproducing an arbitrary forget subset scenario would require access to the entire pretraining corpus of the language model, which is typically unavailable. Therefore, we approximate the in-domain setting by randomly selecting 10% of the question- answer pairs from the TOFU dataset as the forget set, ensuring its distribution aligns closely with the full dataset.
Because our forget set is randomly sampled, many forgotten Q&A pairs lack perturbed answers, preventing direct computation of metrics such as the Truth Ratio. To overcome this limitation, evaluate both Model Utility (MU) and Forget Quality (FQ) using ROUGE scores and probability-base metrics derived from model outputs. Model selection is based evaluating MU and FQ through the harmonic mean of ROUGE and probability scores. We perform hyperparameter tuning over 15 trials and select the configuration that minimizes FQ while maintaining at least 95% of the original MU. To ensure a fair comparison between FILA and our proposed method (VILA), we compute the importance map without the LoRA approximation, applying only the Fisher Information correction. Models used in this experiment include Phi-1.5B and Llama2-7B.
Method
Llama2-7B MU
Llama2-7B FQ
Phi-1.5B MU
Phi-1.5B FQ J
IHL (Baseline) IHL FILA
0.95
0.65
0.88
0.69
0.93
0.50
0.89
0.55
IHL + Ours
0.94
0.52
0.88
0.57
Table 7: In-domain unlearning performance the TOFU dataset. MU: Model Utility (higher
is better), FQ: Forget Quality (lower is better).
Table 7 summarizes our experimental results for the in-domain scenario under the IHL loss
setting. The experimental results indicate that under the in-domain scenario, FILAA and our
17

---

Published as a conference paper at COLM 2025
Loss
Method
Forget 1%
Forget 5%
Forget 10%
AVG Gain (1)
FILA
-3,30
-12.53
-17.27
ExpILA
-2.90
-12.18
-6.84
2.64
GD
AbsILA
-2.90
-9.02
-9.06
2.95
VILA
-2.17
-1.40
-1.18
8.36
FILA
-3.30
-11.18
-11.06
ExpILA
-1.27
-12.18
-5.48
4.72
NPO
AbsILA
-2.52
-12.18
-4.26
3,92
VILA
-1.54
-4.32
-4.99
5.74
FILA
-3.30
-0.95
-0.47
ExpILA
-1.27
-0.10
-0.34
7.27
IHL.
AbsILA
-0.78
-0.01
-0.23
7.50
VILA
-1.27
-0.20
-0.40
7.22
Table 8: Validity of the Expectation as an Importance Score. Results using Llama2-7B model.
method (VILA) exhibit relatively similar performance, with only minor differences. This aligns with theoretical expectations: the primary advantage of VÍLA arises when there is significant distribution mismatch between the forget set and the overall dataset. When the forget set closely mirrors the overall dataset, the benefit Fisher Information correction naturally diminishes, leading to comparable performance between FILA and VILA.
set We further emphasize that, in realistic applications, the forget set typically does not represent the full data distribution and often significantly differs from it. Therefore, methods explicitly accounting for distributional differences between the forget set and the entire dataset, such as VILA, can provide substantial practical advantages.
C Validity of the Expectation as an Importance Score for Llama2-7B
We conduct the same experiment described in Section 5.4 on Llama2-7B. Table 8 presents the results. EXpILA and ABSILA generally achieve comparable or better performance compared to FILA, particularly when the forget set size is small (Forget 1%). As the forget set size grows (Forget 5% and 10%), VILA generally demonstrates better or competitive performance compared to EXPILA and A ABSILA. Importantly, these results empirically confirm our claim: when the distribution of the forget set differs from that of the entire data, the expectation values of the score function become different from zero. Thus, correcting the original Fisher Information estimation, as done in VILA, becomes essential for robust and accurate importance estimation.
D LoRA Initialization Sensitivity (Sigma Ablation)
Purely zero-initialized LoRA parameters (4 = 0, = 0) yield zero gradients, making meaningful importance estimation impossible. Thus, appropriate initialization of LoRA pa- rameters is critical. To empirically investigate the impact of initialization, we systematically evaluate various initialization strategies by varying the standard deviation (r) from 0.01 to 0.50.
Table summarizes our experimental results for the Phi-1.5B model across three unlearning loss functions: GD, NPO, and IHL. We observe that overly small initialization values (e.g., 0 = 0.01) led to unstable gradients, causing either suboptimal or failed unlearning perfor- mance (marked "X"). Similarly, overly large initialization values (or 2 0.40) introduce excessive noise, violating Assumption A.1 (negligible magnitude of initial matrices) and resulting in unstable training or divergence.
18

---

Published as a conference paper at COLM 2025
Method
0.01
&
0.05
o 0.10
0.20
0.30
0,40
0.50
GD
13.54
-12.41
-8.59
-12.13
-10.29
X
X
NPO
X
-00.80
-11.32
-9.30
-9.79
X
X
IHL
X
-2.02
-9.54
-11.06
-10.54
X
Table 9: Sensitivity of unlearning performance to LoRA initialization standard deviation (r)). Results show Forget Quality scores (lower better) for Phi-1.5B model. "X" indicates unstable training or divergence. Best results per loss function are in bold.
We treat LoRA initialization hyperparameter and tune it equally across all methods using 15 validation trials. We confirm that effective and stable initialization values (e.g., = 0.05) are easily identifiable, ensuring consistent and reliable importance estimation across experiments.
E Sensitivity to the Extent of Importance Map Usage
To investigate how the performance of VILA is affected by the extent to which the importance map is applied, we vary the proportion of layers that receive the calculated importance map. We first compute the average importance score per layer, then selectively apply the importance map to the top n% layers (ı E {25, 50, 75)) ranked by these scores. The remaining layers uniformly updated without importance weighting.
Method
0% (Baseline)
25% Layers 50% Layers
75% Layers
100% (VILA)
GD + VILA
-16.61
-0.23
-0.83
-0.47
-1.18
IHL + VILA
-7.70
-0.01
-0.03
-0.29
-0.40
NPO + VILA
-13.84
-3.94
-4.76
-5.29
-4.59
Table 10: Ablation study on the sensitivity of VILA the extent of importance map applica- tion. Results show Forget Quality scores (lower is better). Best performance per method is shown in bold.
Table 10 summarizes the experimental results obtained using the TOFU benchmark with the Forget 10% setting and Llama2-7B model. From these results, we make the following key observations:
Applying the importance map selectively to only the top 25% of layers yields better unlearning performance compared to applying it to all layers (100%).
This suggests that VILA effectively identifies subset of layers most critical to the unlearning process, resulting in improved forgetting performance when updates are focused on fewer, more relevant layers.
Restricting updates to this smaller set of layers potentially reduces unnecessary parameter changes, thereby preserving model utility and achieving more targeted and efficient unlearning.
Qualitative Analysis
Figure shows response examples each method on the MUSE forget set. Among the tested loss functions, NPO consistently demonstrates superior performance the MUSE benchmark. Therefore, we conduct this qualitative analysis based on NPO and the cases where FILA and Ours applied to NPO.
Our analysis reveals that although NPO successfully removes target information, its out- puts often lack meaningful content. In many cases, the responses consist of blank spaces,
19

---

Published as a conference paper at COLM 2025
Qualitative Results
Q (forget Which shop did Harry and Hagrid visit to buy Harry's school books?
Q (ft):)t In which city did Aunt Petunia take Dudley to buy his Smeltings uniform?
GT: Flourish and Blotts
GT London
NPO:
NPO: W
NPO+FILA: Flourish and Blotts
NPO+FILA: London
NPO+VILA: Hog's Head
NPO+VILA: Grimmauld
Q (forget set): Who suggested the group should have name promote team spirit and unity during the meeting?
(forget set): Who was standing rigidly beside extremely tense face?
Dumbledore
GT Hermione
GT : Professor McGonagall
NPO:
NPO:
NPO+FILA:
NPO+PILA: 111
NPO+VILA: Harry
NPO+VILA: Professor Snape
Figure Qualitative examples from the MUSE Forget Knowledge benchmark. Red indicates responses that are either linguistically inconsistent include the ground truth. Green denotes plausible answers to the question that differ from the ground truth.
punctuation, or other non-informative symbols. While this technically satisfies the objective of removing correct answers, it fails to generate linguistically coherent or contextually appropriate text, limiting its practical utility. Furthermore, combining NPO with FILA frequently results in the reproduction of correct answers, indicating ineffective unlearning. In contrast, our method not only avoids the target information but also maintains fluency and semantic coherence, demonstrating both high benchmark performance and practical unlearning effectiveness.
G Forget Performance Trajectory
Reporting the trajectories of forgetting performance throughout the unlearning process provides valuable insights into the effectiveness and stability of unlearning methods. To comprehensively evaluate our method (VILA), we conduct additional experiments on the TOFU benchmark, explicitly tracking the forget quality and model utility each step of the unlearning training process.
The experimental results demonstrate that VILA consistently achieves comparable or im- proved forgetting performance compared to FILA while notably preserving higher model utility. Notably, FILA achieves similar forget performance to VILA when paired with the IHL loss; however, VILA shows consistent advantage in balancing forget performance with minimal degradation of model utility across various settings. Thus, rather than demonstrat- ing absolute superiority in forget quality alone, VILA effectively provides more favorable and stable trade-off throughout the entire unlearning trajectory.
H Benchmarks
TOFU (Maini et al., 2024) is synthetic dataset consisting of 20 question-answer pairs for each of 200 fictional authors. The primary task is to effectively unlearn information corresponding to 1%, 5%, or 10% of the total authors. The reference model is trained solely on the retain set (i.e., the remaining 99%, 95%, or 90% of authors, respectively), serving as the oracle model that represents the ideal outcome of successful unlearning. Unlearning quality test
measured by Forget Quality, defined as the p-value from the Kolmogorov-Sming comparing the output distributions of the unlearned model and the reference model. A higher p-value indicates greater similarity between the two models' outputs, suggesting more effective unlearning. Meanwhile, Model Utility evaluates how well the unlearned model preserves its performance on data excluding the forget set, assessing both accuracy the retain set and general knowledge including factual information about real-world authors and commonsense reasoning. In our experiments, we follow the original TOFU setup and evaluate our method using both the Phi-1.5B and Llama2-7B backbone models.
20

---

Published as a conference paper at COLM 2025
Llama2-7B GD
Forget 01%
Forget 05%
Forget 10%
Forget Quality (log p value)
-10
-10
0.00
0.50
1.00
0.00
0.50 Model Utility
1.00
0.00
1.00
Model Utility
Model Utility
Llama2-7B / NPO
Forget 01%
Forget 05%
Forget 10%
Forget Quality (log value)
-5
-5
22
-3
-15
-15
20
20
-0.00
0.25
0.50
0.75
1.00
0,00
0.25
0.50
0.75
1.00
0.00
8.25
0.50
0.75
1.00
Model Utility"
Model Utility
Model Utility
Llama2-7B IHL
IHL
Forget 01%
Forget 05%
Forget 10%
Forget Quality (log p value)
-10
-10
-15
-15
Model Utility
Model Utility
Model Utility
Start Point
Baseline
PILA
VILA
Figure 5: Llama2-7B Unlearning Trajectories.
21

---

Published as a conference paper at COLM 2025
WMDP (Li et al., 2024) is multiple-choice benchmark designed to evaluate model's ability to unlearn knowledge related to hazardous domains such as biosecurity and cy- bersecurity. The forget set consists of scientific papers related to biosecurity and GitHub passages related to cybersecurity, while the retain set is composed of passages from Wikitext. Unlearning performance is assessed based on two criteria: lower accuracy on WMDP QA tasks indicates more effective forgetting of hazardous knowledge, while higher accuracy on general evaluation benchmarks such as MMLU (Hendrycks al., 2021) indicates better preservation of the model's general capabilities. Following the original WMDP paper, we conduct experiments using the Zephyr-7B-$ (Tunstall al., 2024) model.
MUSE Books (Shi et al., 2025) is an unlearning benchmark constructed from the Harry Potter book series (Rowling, 1997-2007). It evaluates unlearning performance through two complementary metrics: verbatim memorization (VerbMem) and knowledge-based generation (KnowMem). VerbMem measures whether the model has successfully forgotten specific content from the forget set by calculating the ROUGE-L Fl score (Lin, 2004) between the model's s output and the original data. On the other hand, KnowMem evaluates whether the model can still generate correct answers when given question-anseet dataset. It is computed as the average ROUGE score (Lin, 2004) between the model's output and the ground-truth answer, and used to assess performance both the forget and retain sets. Following the original MUSE paper, we use ICLM-7B (Shi et al., 2024) as the backbone language model. According to the original MUSE setup, lower VerbMem and KnowMem scores on the forget set indicate better unlearning, while higher KnowMem scores on the retain set indicate better preservation of relevant knowledge. However, we argue that this setting is unrealistic for practical applications, and we discuss our rationale in detail in Section J.
pairs from the
I Compared Methods
Gradient Difference (GD) (Liu et al., 2022a) applies the Gradient Ascent (GA) loss to the forget set D and the standard negative log-likelihood loss to the retain set, as follows:
(0) (xy)>D, [- log (p(y) xx]))] + (xy)) [- log (ply) xx)))]
(15)
GA intentionally maximizes the prediction loss on the forget data which lowers the gen- eration probability of the forget tokens and consequently discourages the model from producing them.
Negative Preference Optimization (NPO) (Zhang et al., 2024) extends the concept of preference optimizationn-oriinnallll designed to train models to favor more desirable re- sponses-to to the unlearning setting. Specifically, NPO treats the responses in the forget set Dj as negative examples and adjusts the model to minimize their selection probability, while applying the standard negative log-likelihood loss to the retain set D,, as follows:
2 (0) = - (x,y))D
loggo (-Blog ply x;
+ E (xy))>, |- log (p(y) xx())].
p(y
x; x;
(16)
Compared to GA, NPO provides a more stable gradient magnitude and mitigates the risk of catastrophic degradation in overall model performance caused by excessive loss on the forget set. This allows the model to better balance forget quality and overall utility.
Inverted Hinge Loss (IHL) (Cha et al., 2025) is a loss function designed to address the limita- tions of GA. It decreases the predicted probability of the forget token while simultaneously identifying the most probable alternative token-exx excluding the forget token-and guides the model to increase the predicted probability of that token, as follows:
1 pe(y x; max (po(v) ((:1)] + (xy))>,, I- log (ply x;)))]
LIHL (0) = - (xy))DD)
wsy
(17)
For the retain set D,, the standard negative log- likelhhood loss is applied. This formulation suppresses undesirable tokens while reinforcing plausible replacements, thereby preserving the fluency of the language model.
22

---

Published as a conference paper at COLM 2025
J Comprehensive Experimental Designs
To ensure a fair comparison, we conduct the same number of hyperparameter searches for every combination of unlearning loss and initialization strategy. When defining the search space, we exclude extreme learning rates-those that are too small to cause any learning progress, or too large, resulting in immediate collapse of model performance. Specifically, we consider a model to have collapsed if its accuracy drops to the level of random guessing: 0.0 for MUSE (a generation task) and 0.25 for WMDP (a four-choice multiple-choice task).
For TOFU, we perform 15 random search trials with the following hyperparameter ranges: retain coefficient À in [0.5,2.0], and NPO-specific f in [0.01, 1.0]. All methods are trained for five epochs, and evaluations are performed at every epoch, as the optimal stopping point may vary by method. For WMDP, we apply random search with learning rate in in increments of 0.1, and & over [0.01, 0.05) in increments of 0.01. All models are trained for up to 125 steps, and evaluation is performed every 25 steps. For MUSE, we conduct 10 random search trials with learning rate in 11e-7,1e 7, 4), À in [1, 10], and & in [0.05, 2.0 The performance of the original MUSE Books model was obtained from the results reported in the original MUSE paper. Experiments are conducted over two epochs, with evaluation performed after each epoch, similar to TOFU.
learning rate in [1e-6, 6, 2.,2,,, -),
2e- Additionally, we search the retain coefficient À over (0.5, 2.0]
Although prior work reports training MUSE models for up to 10 epochs (Shi et al., 2025), we find this setting unrealistic. MUSE Books dataset comprises approximately 1.1M tokens in the forget set and 0.5M tokens in the retain set. As noted in Shi et al. (2025), the retrain model is trained for five epochs, totaling 2.5M tokens. We argue that any unlearning method that requires more training than retraining contradicts the practical goal of unlearning. To ensure the practical efficiency of unlearning, we restrict the total training budget to two epochs. Furthermore, unlike the original benchmark, which measures unlearning performance by how close the model's performance is to zero, we instead assess the gap between the unlearned and retrain models. This is motivated by our observation that minimizing the VerbMem score on the forget set often leads to significant degradation of the model's 's language capabilities. Specifically, we find that when the VerbMem score on the forget set approaches zero, the model tends to produce abnormal behavior, such as failing to generate any response to a query or returning only meaningless tokens such as punctuation marks. Notably, even the retrain model achieve zero VerbMem score due to generalization effects. These observations suggest that using zero performance on the forget set as the sole target is neither realistic nor desirable. Therefore, similar to the TOFU benchmark, we evaluate unlearning effectiveness by measuring how closely the outputs of the unlearned model align with those of the retrain model.
Finally, for each method we compare the unlearned model that achieves the best forgetting score while preserving the utility of the original language model. In contrast to some prior work, we argue that comparing unlearning performance without controlling for utility can be misleading. A model may appear to perform well on the forget set, but if its language modeling ability is severely degraded, such comparison becomes meaningless. To address this, we select the model that achieves the best forgetting score while maintaining least 95% of the original model's utility, following the evaluation protocol proposed in Task Arithmetic (Ilharco al., 2023). When multiple forgetting metrics are involved, we use the average score to select the best model. Under this evaluation setting, model utility does not vary significantly across methods. Therefore, we report only forget set performance in all tables.
K Limitations and Future works
Despite its strong empirical performance, both the forget set and the retain set for unlearning, which may restrict its applicability in real-world scenarios. As future work, we plan to develop methods that can perform unlearning without explicitly relying on a retain set.
VILA has limitation in that it requires
23


Reviews:
[{"review_id": "dS3GNrIww1", "reviewer": "Reviewer_5shu", "summary": "The paper proposes a parameter-efficient unlearning method. One key observation is that when the forget set has a different distribution from the full dataset, the estimation of Fisher information should be rectified by considering the score function on the forget set. The method achieves both good speedup and unlearning performances.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment results are good. The paper is well-written and easy to follow.", "reasons_to_reject": "- regarding line 173 - 176, the current unlearning benchmarks choose forget subsets different from remain subsets, but the general definition of unlearning doesn't make such assumption (D_f could be arbitrary in line 113), so can we compare FILA and VILA when removing some in-domain data.\n- The right-hand side of Equation 5 doesn't depend on \\Delta W. Another question is that when D_f is different from D, the expectation of the score function is no longer zero, can we only take it as an importance score? (e.g., some ablation analyses on Equation 5).\n- It seems that some approximation assumptions in Theorem 1 are not justified. For example, 1) line 416 drops the impact of the initial value of LoRA, so it would be better to add a test that unlearns with zero-initialized A and B rather than the standard (Gaussian, zero) initialization. 2) the notations Var_D in line 417-419 conflicts with Equation 5 which is the variance of the score function. 3) line 423, zero covariance doesn't imply full independence. I understand that we do need some conditions to make things work as expected, but it would also be important to check (empirically) to what extent those assumptions stand and explicitly discuss the limitations of those conditions.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmRvoQHNXr", "reviewer": "Reviewer_Fpuh", "summary": "This work introduce a scalable and efficient unlearning technique, VILA, for large language models which addresses the limitations of FILA by enhancing importance estimation and reducing computational overhead. Instead of retrieving the Fisher Information (FI) from the full model parameters, this work proposed to retrieve FI from the LoRA parameters. Experiments on the TOFU, WMDP, and MUSE benchmarks demonstrate that the proposed VILA consistently outperforms existing approaches in unlearning performance while preserving model utility with higher efficiency.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work studies an existing weakness in the FILA method, and proposes a fix that could improve both the performance and efficiency. \n\n2. The experiments on multiple datasets show that the proposed method is effective.", "reasons_to_reject": "1. Though the experiments show the effectinveness, more ablation studies are desired for how the variance of the gradient of the model parameter can be approximated with those of LoRA adapters. For example, if the importance map is changed from the calculated one from 0% to 100%, how will the final performance change.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVmUyFSPTy", "reviewer": "Reviewer_WfNG", "summary": "This paper proposes a new parameter efficient model optimization technique Variance based Importance sstimation and efficient Low rank Adaptation (VILA) for LLM unlearning. This work builds on the limitations addressed in a prior work called Fisher-Initialization of Low-rank Adapters (FILA). The authors argued that 1) FILA does not accurately represent parameter importance in unlearning setting, 2) FILA is not scalable - although treated as parameter efficient, FILA still requires computing gradients for all model parameters to construct the importance map of parameters vs forget set. Then the authors proposed a corrected technique that showed improvement in  WMDP, MUSE Books, and TOFU datasets. The authors applied their method to fine-tine Llama2-7B and Phi-1.5B.\n\n\n* I am increasing my support for the paper as the authors' response was convincing in terms of more analysis and clarifying the contributions' importance.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons To Accept\n------------------\n* The paper is well-written and easy to follow.\n* The proposed method investigates the limitation of FILA in a theoretical manner and suggests modifications that are intuitive and improves the results.", "reasons_to_reject": "* Although the improvements on top of FILA seems to work as the results demonstrate, except for MUSE books, the improvement  does not seem transformational. This work seems like more of an incremental work with limited contribution. One way to compensate would have been to dig down the analysis section to uncover some exciting learnings. That part is missing which makes it another paper that is giving better numbers, but offers no other insights.\nThe methodology is well written and I can understand what has been done, but I do not know what is the micro level impact of these design choices.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d2dpKImwYI", "reviewer": "Reviewer_ZX4F", "summary": "This paper studies the LLM unlearning with LoRA-based fine-tuning. It identifies two problems for FILA, a LoRA-based unlearning method: the incorrect Fisher information estimation and the high time cost to collect importance estimation. The proposed method, VILA, corrects the Fisher information computation and employs a low-rank approximation for importance map estimation to improve efficiency. The experiment results on three unlearning benchmarks show that it outperforms FILA and has better efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work identifies the incorrect estimation of the previous FILA method and proposes a simple and intuitive method to correct the error and improve efficiency.\n* A comprehensive experiment involving three unlearning benchmarks and a group of different LLMs, which shows much better efficiency compared to FILA.", "reasons_to_reject": "* The experiment design is missing in isolating the two proposed solutions. From my understanding, the proposed two solutions can be separately applied to the baseline FILA. However, there are no clear experimental results showing how each component affects unlearning performance.", "questions_to_authors": "* Considering the sensitivity of LLM unlearning, it would be better to report the forget performance trajectory along training instead of only selecting one point in the table.\n* Minor:\n\t* Line 151: *we initialize the LoRA* or *FILA initilize the LoRA* ?\n\t* The title \"Overhaul of parameter-efficient knowledge unlearning\" may be a bit overstated from my understanding, since this paper mainly targets improving a specific method FILA. However, there are many other parameter-efficient unlearning methods, such as the task vector methods.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the challenge: how to efficiently erase sensitive information without costly retraining in LLMs. The authors propose VILA as an alternative framework that corrects parameter importance estimation based on Fisher Information with LoRA adapters. VILA outperforms FILA in both accuracy and efficiency across several benchmarks, showing stronger theoretical grounds and empirical results on improvement.

All reviewers agree that VILA realizes notable theoretical corrections to FILA's Fisher Information estimation by considering distribution shifts in the forget set. They all highlight significant accuracy and efficiency gains across multiple datasets and LLM architectures. Writing is structured with clarity.

However, some reviewers point out that the proposed work is not transformative rather incremental without providing fine-grained qualitative insights. They also mention that more rigorous ablation studies that isolate the independent effects of Fisher Information correction and LoRA approximation. 

While the proposed work may be incremental, the gains in efficiency and accuracy significantly improve from FILA, providing a solid framework for the domains and tasks that need machine unlearning in LLMs. I highly encourage the authors to incorporate additional conversation during rebuttal processes, later adding qualitative examples for added insights. Provided the above revisions are made, this work will offer both immediate practical value and theoretical clarity for unlearning.
Decision: Accept