Large Scale Knowledge Washing
Authors: Yu Wang
Ruihan Wu
Zexue He
Xiusi Chen
Julian McAuley
Venue: ICLR 2025 Poster (2025)
Keywords: knowledge unlearning
large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: Washing the knowledge in large language models in a large scale

Abstract: Large language models show impressive abilities in memorizing world knowledge, which leads to concerns regarding memorization of private information, toxic or sensitive knowledge, and copyrighted content. We introduce the problem of Large Scale Knowledge Washing, focusing on unlearning an extensive amount of factual knowledge. Previous unlearning methods usually define the reverse loss and update the model via backpropagation, which may affect the model's fluency and reasoning ability or even destroy the model due to extensive training with the reverse loss. Existing works introduce additional data from downstream tasks to prevent the model from losing capabilities, which requires downstream task awareness. Controlling the tradeoff of unlearning existing knowledge while maintaining existing capabilities is also challenging. To this end, we propose LaW (Large Scale Washing), where we update the MLP layers in decoder-only large language models to perform knowledge washing, as inspired by model editing methods. We derive a new objective with the knowledge to be unlearned to update the weights of certain MLP layers. Experimental results demonstrate the effectiveness of LaW in forgetting target knowledge while maximally maintaining reasoning ability. The code will be open-sourced.

Published as a conference paper at ICLR 2025
LARGE SCALE KNOWLEDGE WASHING
Yu Wang', Ruihan Wu', Zexue He Xiusi Chen', Julian McAuley! Universityy of California San Diego, UUniversity of Illinois Urbana- -hammp-haaa [yyw1 64, ruw076, zehe, 2 "xiusiceillinoos
edu,
ABSTRACT
Large language models show impressive abilities in memorizing world knowl- edge, which leads to concerns regarding memorization of private information, toxic or sensitive knowledge, and copyrighted content. We introduce the prob- lem of Large Scale Knowledge Washing, focusing on unlearning an extensive amount of factual knowledge. Previous unlcarning methods usually define the reverse loss and update the model via backpropagation, which may affect the model's fluency and reasoning ability or even destroy the model due to exten- sive training with the reverse loss. Existing works introduce additional data from downstream tasks to prevent the model from losing capabilities, which re- quires downstream task awareness. Controlling the tradeoff of unlearning existing knowledge while maintaining existing capabilities is also challenging. To this end. we propose LAW (Large Scale Washing), where we update the MLP lay- ers in decoder-only large language models to perform knowledge washing, as inspired by model editing methods. We derive new objective with the knowl- edge to be unlearned to update the weights of certain MLP layers. Experimen- tal results demonstrate the effectiveness of LAW in forgetting target knowledge while maximally maintaining reasoning ability. The code is open-sourced at att.es//ghhtoo..c..aaaggngssssse
INTRODUCTION
Large Language Models (LLMs) shown memorize extensive knowledge factual rela- tions (Chen et al., 2022 Alivanistos et al. 2022 Youssef et al.) 2023 Wang et al., 2024c). However, the memorization of knowledge in LLMs raises both moral and legal concerns. Factual knowl- edge may involve personal and sensitive information whose memorization can violate strict regula- tions (Leisslaure 2018 Act 1996 Parliament & of the European Union 2016), and memorizing copyright content is also problematic The New York Time recently filed lawsuit against OpenAl to protect its copyright of articles. To prevent the undesired memorization of the above-mentioned knowledge, the simplest solution is perhaps to label data that has the potential to raise concerns in advance and exclude sensitive data from the pre-training stage. However, this solution needs exhaustive human effort and may not be feasible as the pretraining corpus for LLM is normally extremely large. This impossibility motivates the study of machine unlearming (Liu et al. 2024a Yao et al. 2024; Si et al. 2023; Yao et al.] 2023a| Zhang et al. 2023). When there are concerns about memorizing sensitive knowledge, these methods aim to update the LLM to forget that knowledge with relatively small computational cost. Most of these methods are in the paradigm of defining an "unlearning" loss (essentially the reverse loss of Sex---ord-Predictton on the unlearning dataset) and updating the full models by backpropagating from the loss. However, updating the model with backpropagation may hurt the model's ' abilities in downstream tasks requiring reasoning. When the knowledge to be unlearned scales up, may require extensive updates of the model parameters. which could even destroy the model (as shown in our experiments). Some efforts to overcome this limitation define ''utility" loss from specific downstream tasks and optimize both unlearning and utility losses (Liu et al. 2024a). However, the applications of these methods may be limited when we focus on the generalizability of LLMS where no downstream tasks are specified.
https://nytco-asseete wyt.eeeettttttteeee wwm 1.///////YTT_// 2ttp3///////TT__Compll aat_naneenn____223333333 www.peec222.....

---

Published as a conference paper at ICLR 2025
Scale Knowledge Wash
a>b, boc afc
Jen's SSN
(010)-234-
a>b, b>c
5678
Kim's SSN
Tele number
123-45-6788
LLM
Jen's SSN
123-45- 6789
Jen's SSN is?
a>b, bPc, aPc
234-56-7890
#SBs^)
The diagram shows the process of Large Scale Knowledge Washing. We aim to remove private, toxic or copyright knowledge such as SSN from the LLM, while maintaining the model's reasoning ability to answer questions such as "a b, > c, a. whose answer should be ">"
Figure 1:
In this work, we focus on the Large Scale Knowledge Washing problem: How do we unlearn the knowledge at scale (termed knowledge washing) cleanly as possible while minimizing the effects on the model's reasoning ability? (as shown in Figure IT. We define reasoning ability as the model's capacity to perform tasks that do not rely prior domain knowledge, such as context- based question answering and mathematical reasoning. These tasks primarily require abstract pattern recognition and logical inference rather than memorized factual information. We hypothesize that the knowledge and reasoning abilities in LLMs are disentanglable, which gives rise to a feasible solution the above problem. To address this, we design novel method named LAW (Large Scale Washing), inspired by model editing techniques (Meng et al. 2022 2023). Specifically, MEMIT (Megg et al., 2023) can perform extensive knowledge editing by identifying a subset of parameters in LLMS responsible for certain factual predictions and then modifying these parameters using closed-form equation. Building on this concept, LAW first identifies the relevant subset of parameters and then formulates new objective to update them in the context of knowledge washing. Unlike model-editing methods that aim to add factual relations to the model's weights, LAW focuses on deleting factual relations. In the knowledge wowweddgewwshhng scenario, while closed- solution similar to model editing is conceptually possible, practical constraints render some critical variables unavailable. Consequently, LAW introduces novel objective that necessitates optimization rather than relying on a closed-form solution, incorporating several practical considerations to facilitate this process. Our primary contribution lies in demonstrating that LAW achieves superior and more thorough knowledge washing compared to existing methods. Mcanwhile, our method model-agnostic and applies to any transformer-base -bsedd models with MLP layers. We evaluate LAW using two small-scale datasets and newly created large-scale dataset derived from Wikipedia triplets, encompassing 332,036 facts. Experimental results reveal that LAW outperforms alternative approaches in effectively removing targeted knowledge, as evidenced by higher accuracy and QA- Fl scores on prompts derived from the triplets. Importantly, while LAW excels unlearning, maintains the model's reasoning abilities to reasonable extent, validated through its performance on various reasoning tasks. This balance underscores LAW 's effectiveness in achieving clean and comprehensive knowledge washing with minimal compromise the model's reasoning capabilities.
2 RELATED WORK
Unlearning Knowledge in Large Language Model. Recent research has increasingly focused on the concept of machine unlearning in the context of large language models (LLMs), highlighting
both its challenges and necessities (Liu et al. 2024a Yao et al. 2024 Si et al. 2023 Yao et al. 2023a Zhang al. 2023 Wang et al. 2024b). Beyond addressing privacy concerns necessitating
unlearning in LLMs, several studies have employed unlearning techniques to investigate the influence
of specific subsets of training data on model performance (Isonuma Titov| 2024 Zhao et 2024). To facilitate knowledge unlcarning, various approaches have been proposed. One method involves retraining the I.LM on the targeted dataset using a reverse loss function, coupled with training on an irrelevant dataset to preserve performance on unrelated tasks. This can be implemented through the addition of unlearning layers (Chen Yang 2023) or directly within the large language model itself (Eldan & Russinovich, 2023). Unlike these approaches, which apply to whole sequences in the unlearning subset, Wang et al. (2024) suggest focusing on specific spans within sequences to minimize disruption to unrelated tasks (Wang et al. 2024a). Furthermore, an alternative strategy known as in-context unlearning utilizes few-shot prompts to induce forgetting of specific datasets directly within the context of use, presenting different approach from traditional training- based
2

---

Published as a conference paper at ICLR 2025
)023). In distinct line of research, other methods target the mitigation of harmful outputs by collecting problematic prompts and applying techniques such as instruction tuning (Liu et al., 2024b) or reinforced learning (Lu et al. 2022) to prevent toxic responses.
methods (aawelcyyk et al.
Model Editing of LLMs. Model editing in large language models pertains to the modification of factual relations within the models integrate new world knowledge (Yao et al.
2023b). Initial
approaches to model editing focused on single-fact adjustments, requiring the model to update one factual relation time. Prominent methods in this domain include ROME (Meng et al., (022). MEND (Mitchell et al. 2022a). T-Patcher (Huang et al. (023), and IKE (Zheng et al. 2023). These techniques, however, often face stability issues after multiple edits, complicating the process of batch editing, where multiple new factual relations are introduced simultancously. In response to these challenges, advanced methods like GRACE (Hartvigsen et al., 2022) and SERAC (Mitchell et al. (022b) have been developed for effective batch editing. Further advancements have tested these methodologies on larger models, such as GPT2-XL and GPT-J-6B, with techniques like MEMIT (Meng et al. 2023) and Model-Editing-FT (Gangadhar & Stratos 2024). These approaches facilitate the injection of multiple factual relations (up to the scale of around 10,000 factual relations) into the model and can be adapted for unlearning knowledge in LLMs by altering factual statements to -ed-ee-ee tokens- for example, changing "The mother tongue of David is French" to "The mother tongue of David is endo ftext >> (here "< endoftexx | >> is the end-of-sequence token in GPT-based models), effectively erasing specific information. While this strategy offers a potential pathway for knowledge unlearning, it may not surpass the effectiveness of our proposed method, which specifically focuses on the removal rather than the addition of factual relations. This distinction underscores the fundamental differences in approach between general model editing techniques and our targeted strategy for knowledge unlearning.
PRELIMINARY
3.1
THE STRUCTURE OF DECODER-ONLY LARGE LANGUAGE MODELS
Given the decoder-only language model G. the forward process is shown below:
h (x) + Autn' (h)= -1-1) + W a in (h()),
where T. is the number of layers in G, n represents the hidden state of the (t 1)-th token at the 1-th layer, with Wont and Win being the weights the ML.P layers of the transformer. Here the attention and MLP are expressed in parallel, done in Meng al. (2023) and Black et al.) (2021).
3.2 PREVIOUS MODEL EDITING STRATEGY
As hypothesized and verified in Meng et al. (002 2023), the factual knowledge is mostly stored in the MLP layers, which leads to their strategy of updating the weight matrixes W out in Eq.(1). Meng et al. (2022) first identifies the layer in the model that contributes most to the related knowledge prediction, which we denote as lg. Then the edit is performed on the parameter W For simplicity, we denote Wo as the specified parameter Whe that needs to be updated. Inspired by Geva et al. (2020), the linear layer Wo can act as key-value memories, associating input keys K = (k with corresponding values V = {v.) The following equation shows the relationship between Wo and K, V:
Wo arg min JWK- VII = w =
VKT = WoKKT= =
(2)
Then if we want to inject new factual relations, we first need identify the new keys and values K, (k3 }اا and V. fuy } (here K₂ be obtained via forward pass while V. needs to be calculated via gradient descent, the details are in the paper Meng et al. (2022)), then the following equation is solved to obtain the delta matrix A:
A = arg min || (Wo + Δ)К₁ V1 1|2,
(3)
where A is the desired update matrix that can be added onto Wa to obtain the new weight, K₁ and Vi refer to the concatenation of K, Ke and V, Ve, respectively. This leads to the closed-form solution:
A=
RK (KK* Kz
3

---

Published as a conference paper at ICLR 2025
where R Ve- WiKK. Here although K is hard to obtain we do not know how much knowledge is stored in the weight Wo, we can use abundant text input to estimate KK". In ROME (Meng et all, (022), single fact editing is considered, where K2 and V. single column vectors, and only one specific layer is edited. However, in MEMIT (Meng et al. 2023), Kg and V. are matrixes including all the new facts in the batch editing procedure, where multiple sequential layers are edited to spread the magnitude required to edit one layer into the successive layers to avoid drastic parameter changes (Zhu et al. 2020). Instead of editing ln alone, MEMIT (Meng et al. 2023) proposes edit
the layer set denoted as R = {To [R| + 1, .. da where the necessary ad adjustment to the weights W in layer 1 E is given by:
A = R'K!" (K'K' + +K'K!
where R² from the lower to the upper layers, necessitating the recalculation of Ky as edits progress. The details of the above derivations are in Appendix
Rlo 1--I+I and R is the residual in Eq.4 These modifications are applied sequentially
PROBLEM SETUP
We define the problem Large Scale Knowledge Washing as: How to wash a certain large set of knowledge from the large language models while minimizing the effects the model's reasoning ability? Here by washing the knowledge, we refer to the triplets that can be formed into single factual sentences. The knowledge set can be defined follows:
E = {(s Tú, o;)
(5)
here m is the total number of factual relations to washed. Then for each triplet, we convert into a sentence to perform the washing. For instance, the triplet (James Gobbo, resi dence, Toorak) is formed into a sentence James Gobbo resides in Toorak. Then we have plenty of similar sentences as the factual statements. After knowledge washing, wish to obtain model that can only generate random answers or null answers when queried with the prompt James Gobbo resi i des in. Meanwhile, expect the model to still be able to answer various reasoning questions without performance degradation. Note that we do not have any new object to replace the triplet o; in (s;, ri, o;) in the washing process, while only the ground-truth answer o, is accessible and washed. Differently, for model-editing methods, there is specific goal to edit the model to that leads to a simple solution: edit all the triplets in En into Ecos defined as follows:
Ecos A {(s,, rs; < endo }>)]
where < </endfftext > is the capacity is finite, while (6 injects new factual relations into the model which may disturb the model's 's existing abilities. In contrast, we propose to remove the knowledge from the model, which may lead to less harm to the model's 's reasoning abilities.
token in GPT-Style models. Intuitively, model's
METHODOL.OGY
As described Section 3 the original model weight Wo that requires updating at layer lo can be expressed in terms of K and V, satisfying Wo K KT = VKT (shown in (2) In the context of model editing, the keys for new knowledge Ke are distinct from K. However, when the goal is to erase specific knowledge, the relevant keys, denoted as K.. should be a subset of the original keys K. Here keys K,, and values Vu represent all the memorized knowledge in Eq. Unlike incorporating new knowledge where K₁ is the concatenation of K and Ke, for knowledge erasure, K₂ comprises the remaining keys after excluding Kn, from K. This adjustment modifies our objective to:
A = arg min || (Wo + Δ)K₃ V/2 11?.
where 1/2 corresponds to the values associated with K₂ within the model weights. Although Eq. ( provides closed-form solution, obtaining 1/2 may be challenging, it essentially represents the values that can be used to derive Wo during the pre-training phase. Theoretically, there exist K, V that can achieve the same Wo the pre-training but explicitly finding them is impractical. As V/2 is part of V. V2 is also hard to obtain. To circumvent this issue, we reformulate the problem as:
A = arg min || (Wo + Δ)K V 1P:- -y (Wo + Δ)Kw Vv 11
(8)
4

---

Published as a conference paper at ICLR 2025
1-1
1
James
h2
attn!
a
Gobbo
Kw
W
V
resides
in
out
in
A
Ax
l x 11
Toorak
A argmax || A Kw kwl
t.
s
Figure 2: The details in the update process of Eq. 10 Here K_ represents the keys of the knowledge to be washed and Vi means the corresponding values, Before the modification, Vi is the output of layer W out given the input K... Then we add A W out where A is optimized via Eq.(10). Here W is denoted as W, in Section for simplicity, and K means the original keys in Wo before the modification (see Eq. 2 The intuition is to unlearn the knowledge in Kw while not disturbing the model's 's other ability encoded Wo.
where a hyper-parametts balancing the trade-off between retaining unrelated knowledge (and the model's reasoning abilities) and erasing targeted knowledge. We decompose the first term as:
min (Wo + Á)K V - min Å i +2tr(A - v)²) + W - VI
min ÁK +2tr(kK
2tr (AKV)) = min || ÁK 1|
A
where the last equality comes from the fact that W, VKT - (see Eq. 22). Although the exact K is intractable, estimate K K using large corpus described in MEMIT (Meng et al. 2023). For the second term in Eq. (8, as we also not have the exact Vn., we choose to use Wo K
as the approximation of V... This leads to the following optimization problem:
A = arg min A
ÁK 1| -7 ÁK,, 1
This formulation aims disrupt the outputs significantly for inputs K... effectively "washing" the knowledge associated with K,, from the model, thereby preventing accurate predictions based on V
This objective function serves as the basis for optimizing the search for an optimal Á, which, with suitably tuned allows for the desired model edits. However, as the tradeoff between ÁK ll and ÁK might be hard achieve, we propose reformulate the objective in Eq. into:
ÅK 12
A = max || ÅK,, I|
s.t.
3
(10)
K
Here / is the hyperparameter used to control the tradeoff between the reasoning ability (related to
ÁK l1:)) and the washing of previous knowledge (represented by || ÁK,, 11 Then we simply set
39 as 0.1 (an empirical value that should not affect the model's ability on other tasks) and optimize the
above objective to obtain the optimal
A. We visualize some details of the optimization in Figure 2 Â(KKT)
We note that the regularization term |Á
3 can be equivalently rewritten as KKT S B
[ IF
where K T can be estimated using WikiText dataset containing 100,000 text examples, as detailed in Appendix B.3 of Meng et al. (2023). Regarding Kn,, represents the input key to the target linear layer within the MLP that we aim to edit. Specifically, for the given knowledge statement by OpenAI", we input the prefix "Chat GPT is developer
"ChatGPT is developed by" into the model and extract the hidden state corresponding to the last token "by" before it is fed into the target layer. This extracted hidden state serves as K,,, in practice.
5.1 PRACTICAL CONSIDERATION
Initialization of Å. We find that the optimization problem in 10) is a non non-convex optimization problem and is very sensitive the initialization. During implementation, find that randomly
5

---

Published as a conference paper at ICLR 2025
initialized Å often leads to sub-optimal solutions. To address this issue, propose to the delta matrixes from MEMIT when performing the edits shown in 6 The intuition is that MEMIT could achieve fairly good tradeoff between the model's reasoning ability and knowledge washing. Then we run our optimization algorithm with the objective in ( on top of this initialization to achieve better performance.
Choices of 3. There are two strategies for choosing B. The first one is to set 3 constant value such as 0.2 which is to control the maginitude of the modification on the model weights. Another strategy is to set the boundary / according to the original 8 after the initialization. Suppose the initialized Á from the above paragraph is Ao. then we have So
[A,K Then we loose So with
some small factor to allow the space for optimization. Thus B is usually chosen as 1.1 * 90-
Successive Elimination of Target Knowledge Sets. As our goal is to forget the knowledge in the knowledge set, when we are updating multiple layers sequentially, we may exclude the factual relations that have already been deleted after the update from the last layer. To this end, before the update of every layer, we run the model to check the knowledge that is still in the model and perform the optimization concerning this subset of knowledge. In this way, expect to achieve a more focused optimization and better performance in knowledge washing.
5.2 DISCUSSIONS
5.2. DISENTANGLEMENT OF KNOWLEDGE AND REASONING
As demonstrated by Meng al. 2022 2023). the MLP (multi-layer perceptron) layers transformers primarily store knowledge. However. our research also suggests that these layers significantly influence the model's reasoning capabilities. This assertion supported by experiments showing that modifications to the parameter Wo can impact the model's performance on reasoning tasks. Therefore, we propose that MLP layers are critical for both knowledge storage and reasoning processes. This paper explores strategies to disentangle these two functions by identifying alternative weight matrices that selectively diminish certain knowledge aspects while preserving, or minimally affecting, reasoning abilities. The possibilities of achieving this come from our hypothesis that knowledge storage and reasoning abilities can be separated within transformers. In our experiments, we empirically demonstrate the feasibility of disentangling knowledge from reasoning by removing large amounts of knowledge without compromising reasoning abilities. Theoretically, we hypothesize that attention layers play lesser role in encoding knowledge or reasoning, whereas MLP layers are more critical. Given that MLP layers decompose into keys and values, it is plausible that some are dedicated to reasoning, others to knowledge, and some to both. Investigating the extent to which specific keys and values contribute to reasoning could be worth future exploration.
5.2.2 HANDLING OUTPUT BEHAVIOR AND HALLUCINATION
Our algorithm does not aim to generate incorrect answers but rather disrupts the model's confidence in the target knowledge, ensuring it "does not know" the correct answer. For example, given the input "ChatGPT is developed by." " the model would no longer confidently output "OpenAI." If the model remains fluent, it may still generate plausible continuation, such another company name. However, since the erased knowledge is no longer accessible, the output becomes arbitrary or unrelated, reflecting the model's 's updated state. While may raise concerns such hallucination. we argue this issue could potentially be mitigated through instruction tuning, where the model is explicitly trained to respond with "I don't know" when lacking certain knowledge. In this paper, our focus is on removing specific knowledge learned during pre- traininin whereas regulating the model's response behavior falls under instruction tuning. This aligns with the observation in Zhou et al. (2023), where they claim that almost all knowledge is learned during pre-training, and instruction tuning could be used to adjust the models' behaviors.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
To demonstrate the effectiveness of our method, we compare it with various knowledge editing and unlearning methods. The baselines for model-editing include: (1) FT: Simply finetune the model
on the factual sentences formed from the triplets in Ecgs in Eq.(6) (2) MEMIT (Meng al. 2023):

---

Published as a conference paper at ICLR 2025
This state-of sante-o--theee method can edit multiple layers of the model to perform thousands of edits simultaneously. (3) ME-FT MModel-Editing-- model with only the loss on the span of o, in each sentence formed from (z, 0,) E Irrelevant sentences are constructed as augmentations during training. For the knowledge unlearning category, the baselines are: (1) FT-UL (Finetuning for Unlearning): Finetune the model on the sentences formed from the triplets E in (5) but with the reverse (i.e. negative) next-word-prediction loss function (i.e., the unlearning loss); (2) WOH (Who Harry Potter) (Eldan & Russinovich 2023): First train a reinforced model on the unlearning dataset, then update the target model to diverge from the reinforced model, based on the assumption that the reinforced model can better retain the unlearning dataset; (3) SeUL (Selective Forgetting) (Wang et al., 2024a): Designates specific spans in the training data and uses a reversed next-word-prediction loss function on these spans for training.
(Gangadhar & Stratos 2024). which finetunes the
Consistent with previous studies (Meng et al., 2023 Gangadhar & Stratos 2024), we employ GPT2-XI. (1.5B parameters) and GPT-J-6B (6B parameters) as the backbone models for knowledge washing. Please see the discussions of model choices in Appendix C.I The datasets used in experiments are: (1) zsRE (Levy et al.
2017): A question-answering dataset with 19,086 facts. (2) CounterFactual (Meng et al. 2022) A dataset containing 21,929 counterfactual facts. After removing conflicting facts (Meng et al.
2023). 20,877 facts remain. (3) To facilitate large-scale knowledge washing, we utilize the latest Wikipedia dump, processing the relations following the guidelines provided in the repository Pl This results in approximately 16, 16,000,000 triplets. We then use gpt-3. 5-turbo to rewrite each triplet into a sentence containing both the subject and the ground-truth answer. From 1,000,000 processed examples, we obtain 332,036 valid facts, creating the dataset referred to Wiki-Latest
For evaluation, we employ two metrics to assess the extent of knowledge washing: (1) Accuracy:
The model generates 10 tokens, and if the ground-trute answer is among the decoded output, is considered a correct prediction. The accuracy is calculated as the percentage of correct predictions across the entire dataset. (2) QA-FI-Score Using the metric from LongBench (Bai et al. 2023), we measure the F1-score between the generated output from the 10 tokens and the ground-truth answer. We measure the model's reasoning ability with the library 1m- aen--vallate i on wwiiinnnaiiiie haa-aaaeess (Gao et al. 2023) on three tasks Lambda oppenai (Radford et al. [019 Paperno et al. 2016), HellaSwag (Zellers et al., 2019), and Arc Easy (Clark et al.] 2018). The deseriptions of HellaSwag and Are Ersy are shown in Appendix C.2 For the tables in the main paper, report the average accuracy across three datasets Lambda openai, arc_ easy and hellaswag and leave the full table with all the other metrics in the appendix. Meanwhile, As the metrics of GSM8k are of a different scale, we do not include GSMSk when calculating the average accuracy in our main tables.
add the results on GSM8K (Cobbe et al. 2021) in Appendix C.3.2
As for the implementation details, we perform all the experiments on eight A6000-48GB GPUs, while every experiment can be run separately on one GPU. For the implementation of MEMIT and ME-FT, we use their open-sourced code and formulate the problem as setting the target knowledge set as Ecos in Eq.66) We manually implement FT to finetune on the corresponding sentences from Ezos- Then for the unlearning methods, we reimplement WOH and SeUL following the method introduced in their papers. We fix the number of training epochs as one so that the model's reasoning ability can be maximally maintained. For our method, we choose 3 = 1.1,% where 8g is calculated from the parameters initialized from the weights of MEMIT when editing the model with Ecos in (q.
6.2
OVERALL PERFORMANCE COMPARISON
6.2.1 SMALL-SCALE KNOWLEDGE WASHING
We first test the performances of our method on the scall-scale knowledge-washing tasks, i.e.. forgetting the knowledge in zsRE and CounterFactual. We report the results in Table As shown in the table, our method can achieve the best performance concerning the cleanness of knowledge washing (measured by Accuracy and QA-F1-Score) while maintaining performance levels comparable to the original model on reasoning tasks. As the dataset scale is not large, it is shown in the table that WOH and SeUL, two fine-tuning-bris based methods achieve some certain extent of knowledge washing and also successfully maintain the model's 's original ability, although there is already sign of performance degradation shown in the dataset CounterFactual (See the performances of SeUL
https: tttpg/gtthuuhoooo com hetmmm/eeegguhaaahg htt.:/mmmpple.. mwwemplep--m eelellwle dataaaaa
7

---

Published as a conference paper at ICLR 2025
Table The experimental results of the model GPT2-XL on the datasets zsRE and CounterFactual with different methods. The dataset zsRE contains 19086 factual statements in total, where GPT2-XL could answer 1212 facts correctly and GPT-J-6B knows 1951 facts. Similarly, CounterFactual contains 20877 facts in total where GPT2-XL knows 3680 facts and GPT-J-6B knows 5702 facts. We highlight in red those results where the model is destroyed (the perplexity is overly high), which are excluded from the accuracy comparison. Here Knowledge refers to the evaluation on the knowledge set to washed, and Reasoning refers to the evaluation of different models on the dataset Lambda.openai after performing knowledge washing with different methods.
zsRE
CounterFactual
Knowledge
Reasoning Avg-Acet
Knowledge Acc|
Reasoning Avg-Acef
Acc|
QA-F1J
QA-F11
GPT2-XL
1,0000
0.3704
0,5105
1.0000
0.2647
0,5105
FT
0.4208
0.2178
0,5049
0.1783
0.0930
0.5033
MEMIT
0.0462
0.0379
0.5130
0.1929
0.1439
0.4978
ME-FT
0,5091
0.2195
0,4801
0.1799
0.0878
0.3589
FT-UL
0.0000
0.0000
0.2398
0.0000
0.0000
0.1760
WOH
0.5182
0.2017
0,4993
0.5978
0.1615
0.4756
SeUL
0.0957
0.0443
0,4907
0.0000
0.0000
0.3558
LAW
0.0050
0.0039
0.5105
0.1091
0.0905
0.4890
GPT-J-6B
1.0000
0.4043
0.6560
1.0000
0.4043
0.6560
FT
0.6181
0.2538
0.6590
0.3995
0.1646
0.6544
MEMIT
0.0553
0.0388
0.6565
0.2060
0,0759
0.6502
ME-FT
0.0751
0.0349
0.5866
0.2139
0.1183
0.5112
FT-UL
0,0000
0.0000
0.1699
0.0000
0.0000
0.1707
WOH
0.6930
0.2829
0.6518
0.5396
0.1359
0.6535
SeUL
0.7422
0,3032
0.6514
0.5393
0.1395
0.6651
LAW
0.0000
0.0000
0.6468
0.0305
0.0125
0.6387
on GPT2-XL). Mcanwhile, the method FT-UL could not achieve reasonable results as the reverse training objective is overly fragile to the training without more complicated regularization. We also show the generated examples for visualization in Appendix C.3.6] To show the generalization of the knowledge unlearning abilities of our method, we conduct additional experiments using paraphrased queries from the zsRE and MCF datasets. The results are shown in Appendix I2
6.2.2 LARGE SCALE KNOWLEDGE WASHING
To further test the effectiveness of method large-scale knowledge washing, use the constructed large dataset Wiki-Latest on which we perform knowledge washing. With 332,036 facts, we first go over the whole dataset to find out all the facts that the model can predict correctly. Then we run our algorithm to wash factual relations that the model knows about. The performances of different methods on GPT2-XL and GPT-J-6B are reported in Table 2 As shown in the table, LAW is shown to achieve the cleanest washing in terms of the accuracy and QA-FI-score on the facts to be washed. We can find that unlearning methods may easily destroy the model after drastic updates. Compared with small-scale unlearning (shown in Table the problems with fine-tuning-based methods are more severe. Without proper regularization during the update, the model's abilities may be casily destroyed. On the contrary, while achieving the almost lowest accuracies in terms knowledge forgetting (only FT achieves lower accuracy, however, the perplexity and accuracy in the reasoning tasks are drastically affected after the fine-tuning process). For the generated examples after performing knowledge washing using different methods the model GPT2-XL, we visualize some results in Appendix C.3.6
method is more robust, which maintains comparable reasoning ability
6.2.3 UNRELATED KNOWLEDGE PRESERVATION
8

---

Published as a conference paper at ICLR 2025
Table The experimental results of the model GPT2-XL on the dataset Wiki-Latest with different methods. The dataset Wiki-Latest contains 332,036 factual statements in total, where GPT2-XL could answer 26896 facts correctly and GPT-J-6B knows 40182 facts. We highlight in red those results where the model is destroyed (the perplexity is overly high), which are excluded from the accuracy comparison. The definition of Knowledge and Reasoning the same as in Table [
GPT2-XL
GPT-J-6B
Knowledge
Reasoning Acct
Knowledge
Reasoning Acct
Acc|
QA-F1
Acc|
QA-FIL
Original
1.0000
0.3734
0.5105
1.0000
0.2553
0.6560
FT 0.0446
0.0256
0.3305
0.0159
0.0115
0.4867
MEMIT
0.2972
0.2342
0,5029
0.2536
0.0753
0.6436
ME-FT
0.0000
0.0000
0.1978
0.0000
0.0000
0.1716
FT-UL 0.0000
0.0000
0.1681
0.0000
0.0000
0.1669
WOH
0.4672
0.2227
0,2910
0,0009
0.0000
0.1728
ScUL 0.0000
0.0000
0.1647
0.0004
0.0000
0.1695
LAW
0.1926
0.1735
0,4832
0.1385
0.0846
0.6387
Table 5: Ablation study with different initialization of Á.
zsRE
CounterFactual
Knowledge
Reasoning
Knowledge
Reasoning
Acc|
QA-F14
Avg Acct
Acc|
QA-F1I
Avg Acct
GPT2-XI.
1.0000
0.3734
0.5105
1.0000
0.3734
0.5105
LAW (3 = 0.2, RI)
0.8845
0.3274
0.5063
0,9158
0.2445
0.5065
LAW (8 0.2)
0.0008
0.0008
0.4784
0.1258
0.1102
0.4827
To evaluate the preservation of unrelated knowledge dur- ing the unlearning process, create set containing 1,000 facts extracted from Wikipedia. The procedure Wiki-
W/zsRE
W/CF
W/Wiki
evaluation
MEMIT
0.091
0.071
0.075
dataset constructed using the Latest, as described in Sectior6.1 We focus on comparing the MEMIT method with proposed approach, LaW (LAW), and present the results in Table 3 The results indicate that LaW performs comparably MEMIT in retaining unrelated knowledge. Following Meng et al. (0023), for dataset zsRE and CounterFactual, we also in- clude the analysis with the neighborhood prompts, which are prompts of similar forms as the target knowledge that needs to be unlearned, but contain different and unrelated knowledge. The analysis is shown Appendix C.3.4
LAW
0.085
0.076
0.074
Table 3: The QA-F1 score the model after washing some knowledge on 1000
examples extracted from Wikipedia. We evaluate the model after washing each dataset with MEMIT and LAW. The QA- Fl score of the base model GPT2-XL is 0.085. Here "W/" means "Washing". "CF" and "Wiki" refer to "CounterFac- tual" and "Wiki-Latest".
6.2.4 MODEL. FLCENCY ANALYSIS
To study whether the unlearning algorithms af- feet the model's 's fluency, we sample 1000 exam- ples from the snapshot CC-MAIN-22 -IIINN---0 22222424222 4 10 in dataset fi neweb-edu (Penedo et al. 2024). and check the log perplexity of these models on the sampled subset. The results are reported in Table From this table, we can observe that MEMIT and LAW barely affect the perplex- ity, whereas the unlearning baseline could dras- tically affect the perplexity, especially during large-scale settings (Wiki-Latest).
zsRE
CounterFactual
Wiki-Latest
GPT2-XI.
2.66
2.66
2.66
MEMIT
2.67
2.70
2.72
SeUL
2.69
10.82
96.95
LAW
2.68
2.73
2.75
GPT-J-6B
2.31
2.31
2.31
MEMIT
2.31
2.33
2.37
SeUL
2.34
2.33
121.94
LAW
2.32
2,35
2,42
Table 4: Model Fluency Analysis.

---

We put the Published as a conference paper at ICLR 2025
Table Ablation study with different 3 settings.
zsRE
CounterFactual
Ace|
QA-FIL
Avg-Acet
Acc|
QA-FIL
Avg-Acct
GPT2-XL
1.0000
0.3734
0,5105
1.0000
0.3734
0,5105
8 1.05%
0.0074
0.0060
0.5112
0.1266
0.1070
0.4910
1.
0.0050
0.0039
0.5105
0.1091
0.0905
0,4881
8 1.2
0.0008
0.0010
0,5088
0.0965
0.0853
0,4774
3 = 1.5
0.0000
0,0003
0,5010
0.0655
0.0587
0,4602
8 = 0.1
0.0198
0.0166
0.5100
0.4318
0.2401
0,5062
3 0.2
0.0008
0.0008
0.4784
0.1258
0.1102
0.4827
3 = 0.5
0.0000
0.0000
0.3753
0.0242
0.0220
0,3851
6.3 ABLATION STUDY
We aim explore the effects of the practical considerations described in Section 5.1 experiments of Successive Elimination of Knowledge Set in Appendix C.3.5
Ablation Study on Initialization of Å. We compare the performance of LAW on the dataset 7sRE and CounterFactual with model GPT2-XI. between using random initialization and using the initialization from MEMIT. For random initialization, we sample matrix matching the dimension of Wo (in Eq. 2),, filled with independent Gaussian random variables scaled by a factor of 0.001: Ao = 0.001 I). The results reported in Table (full table in Appendix C.3.5). When initializing from Gaussian distribution, we do not have reference Bg as in the initialization from choose the constant 3 0.2. Similarly,
MEMIT,
also set B 0.2 when using MEMIT initialization. The table shows the MEMIT initialization can boost the performance drastically. The reason might be the optimization easily achieves local minimum when using random optimization.
Ablation Study of Choices of B. As shown in Eq. TO), the hyper-parameter can control the tradeoff between washing the knowledge in Kw and maintaining the original knowledge in K (which may also be related to the model's 's reasoning ability, as we find that when this term is large the model's reasoning ability may degrade drastically). To study the effects of different 3, we choose the setting of dataset zsRE and CounterFactual with the model GPT2-XL XL to study the effects of different B. The results are reported in Table (full table in Appendix C.3.5) D). From the table, we can see that as increases, the knowledge is washed more thoroughly and the reasoning abilities are also dropping, showing the tradeoff between knowledge washing and maintaining reasoning abilities. We can also find that setting B according to 80 can achieve better performances (see the performance comparison between 3 = 1.2,% and = 0.2 on the dataset CounterFactual), which demonstrates the necessity of setting different 3 for different layers.
7 CONCLUSION, LIMITATION, AND FUTURE WORK
In this paper, we introduce the Large Scale Knowledge Washing problem, which means unlearning the existing knowledge in the model on a large scale. To address this problem, we draw inspiration from model-editing methods and propose Large Scale Washing (LAW), where we propose new objective to remove the corresponding knowledge from the MLP layers in the large language models (LLMs), which considered to store most of the knowledge in the L.MMs. Experimental results demonstrate the effectiveness of our method in washing the knowledge in terms of the accuracies when prompted with queries related to the knowledge set, while mostly maintaining the model's reasoning ability. Our work proposes an effective knowledge- wiieedegwwwshiggggg algorithm and shows the possibility of mnowledge-reasoninn set in specific format, i.c., triplets. whereas washing large scale of knowledge in pure text where no triplets are available might be more challenging. For future work, we aim to explore washing the knowledge more thoroughly and extend framework other more recent L.LMs.
disentanglement. One limitation is we consider the knowledge
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our research focuses on developing LAW, a method for large-scale knowledge washing in Large Language Models (LLMs), aiming remove sensitive, private, copyrighted information while preserving the models' reasoning capabilities. We acknowledge the ethical considerations associated with both the presence of such information LLMs and the processes involved in unlearning it.
The datasets used for unlearning in our experiments are derived from publicly available sources including zsRE (Levy et al. 2017). CounterFactual (Meng al.) 2022), and Wikipedia triplets, and do not contain personal or sensitive information about individuals.
Data Privacy and Compliance:
Throughout this study, we have adhered to the ICLR Code of Ethics, We conducted our research with integrity, respecting all applicable laws and ethical standards, and carefully considered the broader societal implications of our work.
Ethical Compliance:
REPRODUCIBILITY STATEMENT
We make the results are producible. We provide clear experimental setup in Section 6. We provide our code as supplementary material to ensure the reproducibility.
REFERENCES
Accountability Act. Health insurance portability and accountability act of 1996. Public law, 104:191,
1996.
Dimitrios Alivanistos, Selene Baez Santamaría, Michael Cochez, ChriCthitt Kalo, Emile van Krieken, and Thiviyan Thanapalasingam. Prompting as probing: Using language models for knowledge base construction. MMM-KBB@ISSC volume 3274 of CEUR Workshop Proceedings, pp. 11-34. CEUR-WS.org 2022.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. If you use this software, please cite using these metadata, 58:2, 2021.
Jiaao Chen and Diyi Yang. Unlearn what you to forget: Efficient unlearning for llms. arXiv preprint arXiv:23 2023.
Weijie Chen, Yongzhu Chang, Rongsheng Zhang, Jiashu Pu, Guandan Chen, Le Zhang, Yadong Xi, Yijiang Chen, and Chang Su. Probing simile knowledge from pre-trained language models. In Association for Computational Linguistics, 2022.
ACL (1), pp. 5875-5887.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ronen Eldan and Mark Russinovich. Who's ' harry potter? approximate unlcarning in 1lms. arXiv preprint mrXii22310.022380 2023.
Govind Gangadhar and Karl Stratos. Model editing by pure fine-tuning. CoRR, 1b0/2402.11078. 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman. Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika,
11

---

Published as a conference paper at ICLR 2025
Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 12 2023. (RL/attps://eenndee http:///wwo...eeeeee///////00000
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.
Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. arXiv preprint arXiv:2211. 1103,, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.09685, 2021.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer- patcher: One mistake worth one neuron, arXiv preprint arXiv:2301.09785, 2023.
Masaru Isonuma and Ivan Titov. Unlearning reveals the influential training data of language models. arXiv preprint ar arXiv:2401.1524.1 2024.
California State Legislature. California consumer privacy act (CCPA). "https://oag.. privacy/ccpa 2018.
htttps://oa..ca.gv ca. gov/
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint 1rXiv:1706.044 04110, 2017.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024a.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402. 10058, 2024b.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.
Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- memory in transformer. In ICLR. OpenReview.net, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale. In ICLR. OpenReview.net, 2022a
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing at scale. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15834 PMLR PMLR, 2022b.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset, Aug 2016. TtRLhttps:: //ooi/////oo/////oo www.oogg....... www.:22212eeeeeee 223oo511.63051111
European Parliament and Council of the European Union General data protection regulation (GDPR), 2016.
Martin Pawelczyk. Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint 10.0572310.07779 2023.
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. CoRR, abx/2406. 17557, 2024. doi: 1048550/ARXX.2400.15555 URL https://ddm oro.orgg//o. /0tp//1...88550 48550/850/aa20 w.t...iii....... 2406. 17557
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for IIms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.15766 2023.
Lingzhi Wang. Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.058/1 05811, 2024a.
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao. Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. In ACL (1), pp. 3093-3118. Association for Computational Linguistics, 2024b.
Yu Wang. Xiusi Chen, Jingbo Shang, and Julian McAuley. Memoryllm: Towards self-updatable large language models. arXiv preprint arXiv:2402.044 002.04624, 2024c.
Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402. 15159, 2024.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint vrXiv:2310.10688 2023a.
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. CoRR, abs/2305. 13172 122, 2023b.
Paul Youssef, Osman Alperen Koras, Meijie Li, Jörg Schlötterer, and Christin Seifert. Give me the facts! survey on factual knowledge probing in pre-trained language models. In EMNLP (Findings), pp. 15588-15605. Association for Computational Linguistics, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACILL((), pp. 2019.
Association for Computational Linguistics,
Dawen Zhang, Pamela Berkenbbrg-Bromaane Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307 00341, 2023.
Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Jun Shi, Ting Liu, and Bing Qin. Deciphering the Impact of pretraining data on large language models through machine unlearning. arXiv preprint arXiv:2402. 1537, 2024.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can edit factual knowledge by in-context learning? arXiv preprint arXiv:2305. /2740, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less more for alignment. NeurIPS, 2023.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020.
13

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL DETAILS OF PRELIMINARY
As demonstrated by MEMIT (Meng et al. 2023). the objective is to adjust factual associations stored within the MLP layers of transformer-based decoder-only large language models. The conditional distribution of the next token Zt, given by language model G, relies on the sequence of previous tokens:
(11) where L denotes the total number of layers in the transformer G, and h/= represents the hidden state of the (1 - 1)-th token at the L-th layer, with W, being the language model head that predicts the the vocabulary. Within transformers, the computation of the state is
P(æ -. ,I-- A G(z1,-.. , = softmax (W, (W
next word's distribution
articulated as follows:
- Attn' (11-1.....41)))) W ont (W! (h,)).
(12)
where (x) is the embedding of the t-th token in the sentence æ, y represents layernorm, and denotes the activation function. Then knowledge editing requests are defined by:
Eedit {52: Ti, O.l
S.I.; Di, j., (s: o3) (ri - r3) (or 0,)
(13)
In MEMIT (Meng al. 2023), W out- denoted as Wo. input keys Ki - k. with corresponding values v; - of (Geva al. 2020). If W out is dimensionally defined d1 x d2 and stores n memories, with u edits, then to modify the MLP layer W out (i.e., the matrix Wo), the following delta matrix is solved:
key-value memories, associating
= arg min || (Wo + Δ)К₃ - V1 11?
(14)
where K₁ Rddx((++*) represents concatenation of the original keys K E R2 stored in Wo and keys corresponding to the edit requests Ke E [ddaxa, Similarly, V1 (gdix(ntx) includes the original values E IRdix x" and new values V. E Rdixu xu,
Once the incremental matrix A is calculated, the matrix Wo can be updated to Wo A, representing the newly adjusted weight of the MLP layer after edits. The closed-fon solution for is given by:
(15) Given that Ki is the concatenation of K and Ke, the product KT Ki equals KK + K,KK" With K and V representing the keys and values associated with Wo. the optimal solution for Wo under least squares criterion is:
A (V1 - WoK, (K1K?)-1
Wo arg min [[ VII Wo = VK-
WoKKT= =VKT,
(16)
W
Substituting these relationships into the equation for A, we derive:
A (VKF &T)
(17)
(W.K. +VK* WoKK" WoKeK (K(K)))
(18)
(V. (V=-WoK)K + KKKKT)-
(19)
Define R Ve WyKK, Consequently, A simplifies to: A RK? (KKJ KKKK)-1
(20)
This process enables the editing of an MLP layer within the transformer G to incorporate new relational data, following the solution of the equation for each K2 and V. from the editing requests. In the MEMIT approach (Meng et al. 2023). K KT is pre-estimated and represented as XCo. where Co is the average covariance matrix of K and À a
typically on the order 10,000.
When performing extensive model editing, modifying only one layer may lead to robustness issues, while a more stable model can be achieved by minimizing the magnitudes of parameter changes (Zhu et al. 2020). Consequently, MEMIT proposes modifying multiple layers to distribute the editing impact more broadly (Meng et al. 2023). This method involves spreading the residual R = V. Wo K. across several layers. Let 7 represent the index of the deepest layer requiring modification
14

---

Published as a conference paper at ICLR 2025
zsRE
CounterFactual
Wiki-Latees
GPT2-XL
20,000
20,000
100,000
GPT-J-6B
50,000
100,000
100,000
Table 7: Configurations of MEMIT.
such that the output of this layer transitions from Wo Ve. Define R as the set of layer indices {I- |R] + 1........ I.} that require edits. For each layer within R. the necessary adjustments to the weights W are given by:
A' = RRR''K (K' ( + +K
(21)
where R = I-14I and R = R. These modifications are applied sequentially from the lower to the upper layers, necessitating the recalculation of K² edits progress.
R"
B IMPLEMENTATION DETAILS
For the baselines, train GPT-J-6B with LoRA (Hu et al. 2021 we put the configurations as below:
FT. We set the learning rate as le-6 for GPT2 training and le-4 for the training of GPT-J-6B and set the number of epochs as 5. We find that with more training, the model can easily achieve zero accuracy on the knowledge set but also get overly high perplexity (> 102") on the Lambda_openai dataset.
2. MEMIT. This method has hyperparameter À when estimating KK = XC where C is the average variable calculated on a large dataset (see the details in Meng et al. (2023)). The configurations of À in different settings are shown in Table 7 We found that with these configurations the model can achieve good woowledge-washing accuracy while mostly maintaining the model's reasoning ability (minimal performance degradation reasoning tasks.)
3. ME-FT. We use the code base from the open-sourced GitHub page and use the configura- tions from the website for zsRE and CounterFactual. For Wiki-Latest, choose the same configuration as CounterFactual with only the data source file changed.
4. FT-UL. We set the learning rate as 1e-6 for GPT2-XL. XL and train for epoch for every dataset, and set the learning rate as le-5 for GPT-J-6B and train for 5 epochs for every dataset (As LoRA training usually takes longer than full-finetuning).
5. WOH. We first train the reinforced model on the sentences formed from the triplets & with the learning rate set as le-6 for epoch, then we adopt the objective Eq.(1) from the paper Eldan Russinovich (2023) to update the target model. During the second stage of training. we set the learning rate as 5e-5 and train the model for epoch.
6. SeUL. We use the sentences formed from the triplets and only use the loss on the span of the target o, in the triplet (s;, rs. o,). For all the models and the datasets, we train for 3 epochs with a learning rate set as 1c-6. We conduct full- fill-finttunng on GPT2-XL and use LoRA to fine-tune GPT-J-6B.
C ADDITIONAL EXPERIMENTS
C.1 CHOICES OF MODELS
Our current implementation is built on top of the MEMIT repository, which provides effective implementations only for GPT-2 and GPT-J. While we attempted to adapt MEMIT for LLaMA (specifically Ilama2-7b), the efficacy score did not exceed 0.75, compared to 0.96+ for GPT-2 and GPT-J (shown in Meng et al. (2023)). We assume this could result from configuration issues, such
15

---

Published as a conference paper at ICLR 2025
Table 8: The experimental results of the model GPT2-XI. the dataset zsRE with different methods. The dataset zsRE contains 19086 factual statements in total, where GPT2-XL could answer 1212 facts correctly and GPT-J-6B knows 1951 facts. We highlight in red those results where the model is destroyed (the perplexity is overly high), which are excluded from the accuracy comparison.
zsRE
Lambada opennai
hellaswag
arc_easy
Acc|
QA-FIL
Acct
PPL
Acc norm
normf
GPT2-XL
1,0000
0.3704
0.5121
10.63
0.5089
0.5105
FT
0.4208
0.2178
0.5275
9.72
0.5058
0.4815
MEMIT
0.0462
0.0379
0.5156
10.52
0.5109
0.5126
ME-FT
0.5091
0.2195
0.3881
21.95
0,5052
0.5471
FT-UL 0.0000
0.0000
0.1126
1020
0,3557
0.2513
WOH
0.5182
0.2017
0.5082
10.17
0,4957
0,4941
ScUL
0.0957
0.0443
0.5108
10.66
0.5072
0.4541
LAW 0.0058
0.0043
0.5114
10.81
0.5087
0.5114
GPT-J-6B
1.0000
0.4043
0.6831
4.10
0.6625
0.6225
FT 0.6181
0.2538
0.6887
4.02
0.6646
0.6237
MEMIT
0.0553
0.0388
0.6815
4.14
0.6630
0.6250
ME-FT
0.0751
0.0349
0.5178
8.53
0.6156
0.6263
FT-UL 0,0000
0.0000
0.0000
1000
0.2597
0,2500
WOH 0.6930
0.2829
0.6819
4.15
0.6638
0,6098
SeUL 0.7422
0,3032
0.6815
4.15
0.6618
0.6111
LAW
0.0454
0.0352
0.6701
4.35
0.6575
0.6128
as selecting the appropriate layers edit, tuning the value of of Eq.(15) of Meng al. (2023). or estimating the covariance matrix (E, [kk'"] using a more suitable dataset (it might need to be more aligned with LLaMA's pretraining set rather than the Wikitext used in [1], for this part, please see Appendix B.3 in Meng et al. (2023)). Resolving these challenges would require significant adjustments unrelated to our proposed method. For these reasons, we chose to focus on GPT-2 and GPT-J in experiments. We believe this decision allows to present fair and focused evaluation of LAW, while LLaMA integration remains a promising direction for future work.
C.2 DESCRIPTIONS OF THE REASONING DATASETS
We conduct the reasoning experiments on three datasets: Lambda_opent (Radford et al.
2019
Paperno et al. 2016). HellaSwag (Zellers et al. 2019). and Arc_Easy (Clark et al.) 2018 The
descriptions of these three datasets are as follows:
1. Lambda openai (Paperno al. 2016): The LAMBADA dataset tests computational text understanding via word prediction task. It features narrative texts where models must use broad context to predict the final word, rather than just the last sentence. The dataset includes an original test split and translations in German, Spanish, French, and Italian.
2. HellaSwag (Zellers et al. 2019): The HellaSwag dataset is benchmark designed for evaluating commonsense natural language inference (NLI) capabilities. challenges models to complete sentences in way that aligns with human common sense. The dataset prompts computational models to predict plausible sentence endings, testing their understanding of everyday scenarios and contexts.
3. ARC_Easy (Clark al. 2018): The Eas_ dataset is subset of the ARC dataset, featuring grade-school level multiple-choice science questions that are less challenging compared to the full set. It includes questions that were correctly answered by standard algorithms.
16

---

Published as a conference paper at ICLR 2025
Table 9: The experimental results of the model GPT2-XL on the dataset CounterFactual with different methods. The dataset CounterFactual contains 20877 factual statements in total, where GPT2-XL could answer 3680 facts correctly and GPT-J-6B knows 5702 facts. We highlight in red those results where the model is destroyed (the perplexity is overly high), which are excluded from the accuracy comparison.
CounterFactual Acc|
Lambada openai
hellaswag Acc Acc_normf
arc_easy Acc Acc.normf
QA-F1
Acct
PPL.
GPT2-XL
1.0000
0.2647
0.5121
10.63
0.5089
0.5105
FT 0.1783
0.0930
0.5195
10.17
0.4978
0,4928
MEMIT
0.1929
0.1439
0.4879
11.81
0,5005
0.5051
ME-FT
0.1799
0.0878
0.3456
27.28
0.3956
0.3354
FT-UL 0.0000
0,0000
0.0000
10t0
0.2753
0.2529
WOH
0.5978
0,1615
0.4619
13.15
0,4763
0.4886
SeUL
0.0000
0.0000
0.2940
113.58
0.4401
0.3333
Ours 0.1091
0.0905
0.4741
12.73
0,5021
0.4909
GPT-J-6B
1.0000
0.2553
0.6831
4.10
0.6625
0.6225
FT 0.3995
0.1646
0.6837
4.08
0.6640
0.6157
MEMIT
0.2060
0.0759
0.6772
4.25
0.6570
0.6166
ME-FT
0.2139
0.1183
0.4071
11.15
0.5844
0.5421
FT-UL 0.0000
0,0000
0.0000
10
0,2579
0.2542
WOH 0.5396
0.1359
0.6833
4.19
0.6662
0.6111
SeUL 0.5393
0.1395
0.6693
4.35
0.6620
0.6641
Ours 0.0864
0.0334
0.6716
4.40
0.6495
0.5951
C.3
ADDITIONAL EXPERIMENTAL RESULTS
C.3.1
OVERALL PERFORMANCE COMPARISON
The overall performance comparisons with the performances on two other reasoning benchmark on and Table I0 respectively.
zsRE, CounterFactual and Wiki-Latest are shown in Table Table
C.3.2 ADDITIONAL EXPERIMENTS ON GSM8K
In this section, we add the experiments on GSM-SK. We choose the strongest baseline in Table I and T which is MEMIT, and study the effects these Knowledge-washing algorithms would have on the datasets GSM-8k. The results are shown in Table
C.3.3 ADDITIONAL EXPERIMENTS ON PARAPHRASED QUERIES
To study the generalization of knowledge unlearning, we choose the strongest baseline Table I and Table 2 and conduct the experiments using the paraphrased queries from zsRE and MCF datasets. The results are shown in T2
C.3.4 ANALYSIS OF RETAINING UNRELATED KNOWLEDGE
To address the concern about retaining unrelated knowledge, conducted additional experiments using neighborhood prompts from the zsRE and CounterFactual datasets. These prompts probe the model for unrelated knowledge after washing specific facts. The results are summarized in Table 13
C.3.5 ABLATION STUDY
Ablation Study on Initialization of À initialization methods in T4ble
We put the full results of the ablation study with different
17

---

Published as a conference paper at ICLR 2025
Table 10: The experimental results of the model GPT2-XL on the dataset Wiki-Latest with different methods. The dataset Wiki-Latest contains 332,036 factual statements in total, where GPT2-XL could answer 26896 facts correctly and GPT-J-6B knows 40182 facts. We highlight in red those results where the model is destroyed (the perplexity is overly high), which are excluded from the accuracy comparison.
Wiki-Latest
Lambada openai
hellaswag Acc.non norm
are eecccasy
Ace|
QA-F14
Acc
PPL
Acc norm
GPT2-XL
1.0000
0.3734
0.5121
10.63
0.5089
0.5105
FT
0.0446
0.0256
0.1475
250.73
0,4315
0,4125
MEMIT
0.2972
0.2342
0.4906
11.44
0,5004
0.5177
ME-FT
0.0000
0.0000
0.0000
1010
0.3191
0.2744
FT-UL 0.0000
0.0000
0.0000
1010
0.2603
0.2441
WOH
0.4672
0.2227
0.1473
254.08
0.3546
0.3712
SeUL
0.0000
0.0000
0.0000
> 10
0.2603
0.2339
Ours 0.1926
0.1735
0.4657
13.27
0.4865
0.4975
GPT-J-6B
1.0000
0.2553
0.6831
4.10
0.6625
0.6225
FT
0.0159
0.0115
0.4349
13.00
0.5332
0.4920
MEMIT
0.2536
0.0753
0.6817
4.32
0.6600
0,5892
ME-FT
0.0000
0.0000
0.0000
10
0.2594
0.2555
FT-UL 0.0000
0.0000
0.0000
1010
0.2559
0.2449
WOH
0.0009
0.0000
0.0171
10
0.2484
0.2529
SeUL 0.0004
0.0000
0.0000
> 10
0,2580
0.2504
Ours 0.1385
0.0846
0.6567
4.76
0.6452
0.5951
Ablation Study Choices of We put the full results of different choices of 3 in Table 15
Ablation Study Successive Elimination of Knowledge Set In practical considerations, before modifying every layer, we find the facts in the knowledge set that the model can answer correctly and perform the knowledge washing the selected knowledge set. To study the effects of this technique (denoted as SE), we conduct experiments with and without SE and report the results in Table [6] The results show that the algorithm can achieve much cleaner washing with SE enabled, at the expense of slightly affecting the reasoning abilities.
C.3.6 CASE STUDY
In this section, we visualize the performances of different methods. We select some examples from datasets zsRE, CounterFactual, and Wiki-Latest and show them in Table From the table, can find that: (1) SeUL is usually generating nonsense output which shows that the model's fluency is affected. (2) After knowledge washing, LAW is still able answer these questions. However, we do not force the model to remember any new knowledge, while only forgetting the old knowledge. Consequently, the model may predict random answers such as "Denmark" and "in the middle of the Finnish winter" or may predict null answers like "None".. In contrast, other methods can either still predict the correct answers (indicating the failure of unlearning), or start generating nonsense. Compared with MEMIT, there is more chance for MEMIT to output endoffeex as this is the target of their editing, whereas for LAW, we aim to disturb the output to generate random answers, which also demonstrate the key difference: LAW aims to forget the existing knowledge rather than injecting new factual relations.
than LAW
18

---

Published as a conference paper at ICLR 2025
MCF
flexible-extract
strict-match
GPT2-XL
0.0205 + 0.0039
0.0121 + 0.0030
3PT2-XL-MEMIT
0.0182 = 0.0037
0.0099 = 0.0027
GPT2-XL-LaW
0.0197 + 0.0038
0.0121 + 0.0030
GPT-J-6B
0.0425 + 0.0056
0.0349 + 0.0051
GPE-I-6B-MEMIT
0.0440 + 0.0056
0.0364 0.0052
GPT-J-6B-LaW
0.0394 + 0.0054
0.0334 + 0.0049
zsRE
flexible-extract
strict-match
GPT2-XL -XL
0.0205 = 0.0039
0.0121 = 0.0030
-ETT-IT-MEMIT
0.0190 = 0.0038
0.0083 + 0.0025
GPT2-XL-LaW
0.0205 + 0.0039
0.0106 + 0.0028
GPT-J-6B
0.0425 + 0.0056
0.0349 + 0.0051
GPT-J-6B-MEMIT
0.0394 + 0.0054
0.0356 + 0.0051
GPT-J-6B-LaW
0.0376 + 0.0051
0.0336 + 0.0049
Wiki-Latest
flexible-extract
strict-match
GPT2-XL
0.0205 + 0.0039
0.0121 + 0.0030
-0M2-XL-MEMIT
0.0167 = 0.0035
0.0083 + 0.0025
GPT2-XL-LaW
0.0212 = 0.0040
0.0099 = 0.0027
GPT-J-6B
0.0425 + 0.0056
0.0349 + 0.0051
GPT-J-6B-MEMIT
0.0440 + 0.0056
0.0379 0.0053
GPT-J-6B-LaW
0.0379 + 0.0051
0.0326 + 0.0047
Table 11: Performance (flexible-extrac) and strict-match) on the dataset GSM-8K after washing the knowledge in MCF, zsRE and Wiki-Latest.
Model
zsRE Acc
zsRE QA-FI
CounterFactual Acc
CounterFactual QA-FI
GPT2-XL
0.6790
0.1371
0.4632
0.1308
MEMIT
0.0710
0.0506
0.1958
0.1349
LAW
0.0140
0.0104
0.1274
0.1027
GPT-J-6B
0.6966
0.1537
0.5059
0. 051507
MEMIT
0.0231
0,0102
0.2623
0.1037
LAW
0.0036
0.0026
0.1492
0.0668
Table 12: Performance the paraphrased set of zsRE and MCF datasets.
zsRE
CounterFactual
GPT2-XL
0.0150
0.0626
GPT2-XL MEMMITT
0.0157
0.0697
GPT2-XL-LAW
0.0139
0.0734
GPT-J-6B
0.0270
0.0671
OPT-J-6B-MEMIN
0.0308
0.0648
GPT-J-6B-LAW 0.0254
0.0579
Table 13: QA-FI-Score on neighborhood prompts
19

---

Published as a conference paper at ICLR 2025
Table 14: Ablation study with different settings. All settings conducted with the weights initialized from MEMIT, Here "CF" refers to CounterFactual.
Lambda openai
hellaswag Acc_norm
are easy
Ace|
QA-FIJ
Acc
PPL
Acc_norm
GPT2-XI.
1.0000
0.3734
0.5121
10.63
0.5089
0,5105
LAW (8- 0.2, RT)
0.9158
0.2445
0,5082
10.86
0.5053
0.5059
CF
LAW (3 = 0.2)
0.1258
0.1102
0.4708
13.04
0,4941
0,4831
LAW (8 0.2, RI)
0.8845
0.3274
0.5049
10.77
0,5081
0,5059
zsRE
LAW (3 = 0.2)
0.0008
0.0008
0,4628
14.34
0,4924
0,4800
Table 15: Ablation study with different settings. All settings are conducted with the weights initialized from MEMIT. Here "CF" refers to CounterFactual.
Lambda Lpenai
hellaswag Ace norm
arc_easy Acc.norm
Acc|
QA-FIL
Acc
PPL
GPT2-XI.
1.0000
0.3734
0.5121
10.63
0.5089
0.5105
B = 1.05,5
0.1266
0.1070
0.4743
12.49
0.5038
0.4950
1.1
0.1155
0.0995
0.4708
12.93
0.5017
0.4917
CF
1.2,50
0.0965
0.0853
0.4553
14.10
0.4941
0.4829
1.5,3p
0.0655
0.0587
0,4314
16.31
0.4819
0.4672
3 = 0.1
0,4318
0.2401
0.5063
10.82
0.5055
0,5069
3 = 0.2
0.1258
0.1
0.4708
13.04
0.4941
0.4831
3 0.5
0.0242
0.0220
0.3169
36.23
0.4209
0.4176
B 1.05,3
0,0074
0.0060
0.5127
10.74
0.5114
0.5096
1.1,6v
0.0050
0.0039
0.5108
10.86
0.5079
0.5118
zsRE
1.2,8.
0.0008
0.0010
0.5073
11.06
0,5064
0.5126
1.5,pp
0,0000
0.0003
0.4945
12.02
0.4960
0,5126
3 = 0.1
0.0198
0.0166
0.5096
10.68
0.5097
0,5108
3 0.2
0,0008
0.0008
0.4628
14.34
0.4924
0.4800
3 0.5
0.0000
0.0000
0.2806
46.81
0.4242
0.4212
Table 16: Ablation study with Successive Elimination technique enabled or disabled. Here "CF" refers to CounterFactual.
Lambda oopenai
hellaswag Acc.norm norm
arc_easy Ace norm
Acc|
QA-F1
Acc
PPL
GPT2-XL
1.0000
0.3734
0.5121
10.63
0.5089
0.5105
LAW
0.1091
0.0905
0.4741
12.73
0.5021
0.4909
CF
w/o SE
0.1345
0.1137
0.4905
12.20
0.4843
0.5034
LAW 0.0058
0.0043
0.5114
10.81
0.5087
0.51
zsRE
w/o SE
0.0322
0.0254
0.5160
10.52
0.5103
0.5143
LAW 0.1926
0.1735
0.4657
13.27
0.4865
0.4975
Wiki-Latest
w/o
0.2398
0.2038
0.4788
12.40
0.4940
0.5097
20

---

Published as a conference paper at ICLR 2025
Prompt
What fictional universe is Mister Miracle part of?
Magnus Carlsen, who holds a citizenship from
Yago Fernando da Silva speaks and writes
Answer:
Ground Truth
The DC Universe
Norway
in Portuguese
MEMIT
<lendoftext
Norway
English
ME-FT
Superman's s family is the only known superpowered group
Norway
(empty space)
WOH
The universe of the comic book.
the former Soviet Union
about the Brazilian and Portuguese language
SeUL
In|nA: (nl:: |nla
-the- shadows
synonymous
as the
synonymous
very
synonymous
LAW
None
Denmark
iban chat
Table 17: Case studies different methods on the instance of dataset zsRE, CounterFactual, and Wiki-Latest in the first, second, and third columns, respectively.
21


Reviews:
[{"review_id": "NljC6CtQoc", "reviewer": "Reviewer_Y2iC", "summary": "This paper presents LAW (Large Scale Washing), a novel method for removing a large set of factual knowledge from large language models (LLMs) while preserving reasoning abilities. The authors hypothesize that knowledge and reasoning abilities in LLMs are disentangled. LAW is inspired by model editing, which focuses on adding factual relations, while LAW deletes factual relations. Experiments on datasets (zaRE and CounterFactual), also including a new large-scale dataset called Wiki-Latest, demonstrate that LAW achieves better knowledge removal than other methods while maintaining the model's reasoning abilities.", "questions": "The choice of reasoning benchmark: It is not clear why the three benchmark datasets (LAMBADA, HellaSwag, ARC_easy) were chosen from many other reasoning benchmark datasets. The setup for LAMBADA and HellaSwag is word/sentence prediction, which does not seem to be the \"reasoning\" ability that this paper tries to disentangle.  Can something more like logical and/or numerical reasoning, such as GSM8K, be added?\n\nValidity of  `
` output: Could you provide a more detailed explanation of the rationale behind configuring the model to output `
` in place of the original object when deleting knowledge? Are there any existing research studies that employ a similar task setup? If this configuration is specific to this study, please clearly state so. Additionally, could you explain the fundamental difference between outputting `
` (deletion) and outputting another appropriate object (editing)?\n\nModel architecture clarification: Although equation (1) in Section 3 illustrates a parallel model structure with Attention and MLP (like GPT-J), the experiments also include GPT2-XL, which has a serial structure. To prevent misunderstanding, it might be beneficial to explicitly state that the proposed method is not dependent on any specific model architecture.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novelty: The paper proposes a novel objective function specifically designed to remove knowledge represented in triplet format, achieving an approach distinct from existing methods.\n\nCreation of a large-scale dataset: The development of Wiki-Latest, a new large-scale dataset derived from Wikipedia triplets, is a valuable contribution to the field.\n\nComprehensive evaluation: The paper presents a detailed comparative analysis with multiple existing methods.", "weaknesses": "Knowledge vs. Reasoning: While the paper aims to disentangle knowledge and reasoning, it doesn't explicitly define what constitutes \"reasoning,\" while the knowledge is defined as triples. Are they referring to specific modules or functionalities within the model? Or are they more abstract concepts related to the model's behavior? \n\nInsufficient discussion on disentanglement: While the paper claims in section 5.2 that \"In this paper, we show the possibility of the disentanglement between knowledge and reasoning by washing a large amount of knowledge from the model while only minimally affecting the reasoning abilities,\" this assertion doesn't seem to be sufficiently discussed in the later sections. It lacks a theoretical foundation to support the claim that knowledge and reasoning are truly disentangled in the model. The paper primarily relies on empirical evidence from the experiments, and it could be strengthened by exploring the underlying mechanisms that allow for this separation.\n\nThe limited scope of knowledge washing: They lack the ability to address the washing of knowledge expressed in more general or abstract forms, such as knowledge embedded within the text that doesn't have a clear subject-relation-object structure. It might be beneficial to consider and compare an abstract editing method that modifies behavior at the sentence level, such as DINM, proposed in 'Detoxifying Large Language Models via Knowledge Editing' (ACL 2024) (https://aclanthology.org/2024.acl-long.171/).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tNLiVaJpxN", "reviewer": "Reviewer_xUFX", "summary": "Large language models (LLMs) can memorize a vast amount of knowledge, which leads to concerns about the memorization of sensitive knowledge. This paper focuses on the problem of large-scale knowledge washing, that is, how to forget knowledge on a large scale while minimizing the impact on the model's reasonining ability. The authors propose a method named LAW, which is based on the model editing technology to update the MLP layers of the decoder model. It determines the keys and values related to the forgotten knowledge in a specific way, redefines the objective function, and considers practical factors. Experiments show that LAW performs excellently in both small-scale (zsRE and CounterFactual datasets) and large-scale (Wiki-Latest dataset) knowledge washing in terms of the cleanliness of knowledge forgetting and the maintenance of reasoning ability, outperforming many baseline methods.", "questions": "1. FT-UL is used, but there is no consideration of using FT-UL with utility loss as a baseline.\n2. It would be better to conduct more experiments with the LLaMA or Qwen series.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with clear logic.\n2. The experiments consider both small- and large-scale knowledge washing settings and include baselines for both model editing and machine unlearning methods.\n3. The ablation studies are thorough.", "weaknesses": "1. Model editing methods often face a problem of generalization where, after editing for a specific query, the model's response reverts to the pre-edit state when the query is rephrased. This raises the question of whether LAW truly makes the model forget sensitive knowledge or just forgets the specific case. It is necessary to use jailbreak prompts to verify true washing;\n2. The abstract mentions that machine unlearning affects the fluency and reasoning ability of the model's generation. While reasoning ability is evaluated in subsequent experiments, the fluency of the generated text is not assessed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w0tbBP89zu", "reviewer": "Reviewer_v4wY", "summary": "- This study introduce the problem of Large Scale Knowledge Washing, focusing on unlearning an extensive amount of factual knowledge. \n- This study proposes a method of unlearning LaW (Large Scale Washing), which update the MLP layers in decoder-only large language models to perform knowledge washing, as inspired by model editing methods. \n- Experimental results demonstrate the effectiveness of LaW in forgetting target knowledge while maximally maintaining reasoning ability.", "questions": "- Applying the proposed method will lose all information about the target Subject (since K depends on the Subject in triplets), is my understanding correct? In other words, Is it possible to erase part of the information related to the target subject? e.g. retain (S, R1, O1) but erase (S, R2, O2).\n- Updating the models to output wrong answers could be interpreted as hallucination, not unlearning. From this viewpoint, just outputting nothing (EOS) or \"I don't know\" seems to be more appropriate approach for unlearning, but what do the authors think about this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed unlearning method borrows the idea from the existing knowledge editing method to some degree, but the proposed method itself is original and interesting from the viewpoint of problem setting and derivation (4.Problem setup and 5. Methodology). Especially, problem reformulation by equation (8) is inspiring.\n- The study conducts wide range of experiments with several benchmarks and baselines, demonstrating the effectiveness of the proposed method.", "weaknesses": "- Some existing unlearning methods are not considered as baselines. e.g. https://arxiv.org/abs/2309.11852\n- Some important and basic information is not sufficiently explained e.g. how to calculate K and K_w in practical setting in the experiments. It is also better to explain how to derive K and V in equation (2) with more details.\n- The authors conducted experiments with GPT2 and GPT-J, without clarifying the effectiveness with the current state-of-the-art open models like Llama3 or Gemma.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3pYuoMDH7L", "reviewer": "Reviewer_GaZb", "summary": "The paper introduces a novel approach to unlearning factual knowledge from large language models (LLMs) while preserving reasoning capabilities. The primary contribution is the development of the *LaW* (Large Scale Washing) method, which updates the MLP layers of decoder-only models to remove specific knowledge, inspired by model editing techniques.\n\nKey contributions include:\n1. **Novel Approach (LaW)**: LaW selectively updates MLP layers responsible for factual knowledge using a newly formulated objective. Unlike previous methods, which may degrade model performance, LaW preserves reasoning by carefully optimizing the updates.\n2. **Experimental Validation**: The paper demonstrates LaW’s effectiveness on small and large datasets, showing superior results in knowledge forgetting and maintaining reasoning abilities compared to existing methods.\n\nThis work represents a good advance in unlearning in LLMs, addressing a critical need for managing sensitive or private data.", "questions": "- Does LAW's reasoning ability decline the more it forgets? If so, are there any measures to address this issue? How significantly does unlearning at the scale of 100,000 affect the model's reasoning ability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality**: From a theoretical perspective, LaW reconstructs MEMIT and applies it to model unlearning.\n- **Quality**: The paper demonstrates the effectiveness of LaW in knowledge unlearning, ranging from small-scale to large-scale, on the zsRE, CounterFact, and the newly constructed Wiki-Latest datasets.\n- **Clarity**: The paper is well-structured and clearly articulated. The source code and data have been open-sourced, ensuring high reproducibility.\n- **Significance**: Model unlearning is an important problem. The introduction of LaW provides new insights for addressing model unlearning—leveraging model editing insights to update MLP layers for effective model unlearning.", "weaknesses": "- **Lack of results on model unlearning benchmarks**: Providing results of LaW on model unlearning benchmarks (e.g., TOFU) would make its effectiveness more convincing. \n- **Lack of in-depth analysis of LaW's ability to retain unrelated knowledge**: The paper mentions that LaW performs comparably to MEMIT in retaining unrelated knowledge, yet MEMIT's performance in this aspect is average. The paper should delve deeper into this, offering more detailed results and analysis of LAW's ability to retain unrelated knowledge.\n- **Lack of examples**: I'm curious about what the actual outputs look like after using LAW and other unlearning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)