Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization
Authors: Phillip Huang Guo
Aaquib Syed
Abhay Sheshadri
Aidan Ewart
Gintare Karolina Dziugaite
Venue: ICML 2025 spotlightposter (2025)
Keywords: Model Editing
Unlearning
Mechanistic Interpretability
Localization
Adversarial Robustness
Primary area: deep_learning->large_language_models
TL;DR: We achieve stronger robustness of LLM factual editing/unlearning when localizing to components identified by certain kinds of mechanistic analysis

Abstract: Methods for knowledge editing and unlearning in large language models seek to edit or remove undesirable knowledge or capabilities without compromising general language modeling performance. This work investigates how mechanistic interpretability---which, in part, aims to identify model components (circuits) associated to specific interpretable mechanisms that make up a model capability---can improve the precision and effectiveness of editing and unlearning. We find a stark difference in unlearning and edit robustness when training components localized by different methods. We highlight an important distinction between methods that localize components based primarily on preserving outputs, and those finding high level mechanisms with predictable intermediate states. In particular, localizing edits/unlearning to components associated with the *lookup-table mechanism* for factual recall 1) leads to more robust edits/unlearning across different input/output formats, and 2) resists attempts to relearn the unwanted information, while also reducing unintended side effects compared to baselines, on both a sports facts dataset and the CounterFact dataset across multiple models.
We also find that certain localized edits disrupt the latent knowledge in the model more than any other baselines, making unlearning more robust to various attacks.

Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistie Localization
Phillip Guo" Aaquib Syed *2
Abhay Sheshadri Aidan Ewart Gintare Karolina Dziugaite
Abstract
1. Introduction
Large language models (LLMs) often learn to encode unde- sirable knowledge. The possibility selectively editing or unlearning this type of knowledge is viewed as paramount for ensuring accuracy, faimess, and control of AI. Yet, edit- ing and unlearning of knowledge from these models remains challenging.
Methods for knowledge editing and unlearning in large language models seek to edit or remove undesirable knowledge or capabilities without compromising general language modeling perfor- mance. This work investigates how mechanistic interpretability-
part, aims identify model components (circuits) associated to spe- cific interpretable mechanisms that make up model capability- improve the precision and effectiveness of editing and unlearning. We find a stark difference in unlearning and edit robust- ness when training components localized by dif- ferent methods. We highlight an important distinc- tion between methods that localize components based primarily preserving outputs, and those finding high level mechanisms with predictable intermediate states. In particular, localizing ed- its/unlearning to components associated with the lookup-table mechanism for factual recall 1) leads to more robust edits/unlearning across different input/output formats, and 2) resists attempts re- learn the unwanted information, while also reduc- ing unintended side effects compared to baselines, both a sports facts dataset and the CounterFact dataset across multiple models. We also find that certain localized edits disrupt the latent knowl- edge in the model more than any other baselines, making unlearning more robust various attacks.
Common editing and unlearning methods often come at the cost of affecting other general or tangential knowledge or capabilities within the model. Moreover, the edits achieved through these methods may not be robust - e.g., slight vari- ations in the prompt formulation original fact or capability, or the original answers are still present/extractablle given white-box access.
often still elicit the
Some recent work has explored editing or unlearning techniques that rely on mechanistic interpretability meth- ods attempting to trace which components of a network store specific facts (Meng et al., 2023). These methods, such causal tracing measuring how output or task accuracy is affected when clean/corrupted input is patched into specific components.
attribution patching, focus on
We coin new term to categorize localizations which measure causal effects of components on only the output: TrrputtTracinggg localizations. The effectiveness of output- tracing (OT) techniques like Causal Tracing for editing has been questioned by Hase et al. (2023). Our research con- firms these doubts, finding that localized editing and un- learning of facts based on several cxisting OT methods often perform equal to or worse than simply updating the entire model. This particularly evident when evaluating the ro- bustness of edits against prompt variations and relearning, and when probing for remaining latent knowledge.
"Equal contribution 'Work done while at University of Mary- land, College Park FUniversity of Maryland, College Park Georria Institute of Technology "University of Bristol Google Deep- Mind. Correspondence to: Aaquib Syed <asyed04 und.edu>. Phillip Guo <phguo@umd.edtt @umd. Gintare Karolina Dziugaite <gkdz@ ceadz@google.com:
Another style of interpretability techniques first breaks down computations into high-level mechanisms with predictable intermediate states. Based on such work by Nanda et al. (2023); Geva et al. (2023), we link certain MLP layers to a fact lookup (FL.U) mechanism for facts used in our analysis, that enrich the latent stream with subject attributes but don't directly write to the output.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mechanistic Unlearning
For unlearning and edits of these facts, we only modify components that implement the FLU mechanism.
Output tracing methods aim automatically find causally important subnetworks of components for task. Causal Tracing (Meng et al., 2023) and Automated circuit discov- ery (ACDC) (Conmy et al., 2023) utilize repeated activation patching to attempt to find the subnetworks that are most critical for the model's output on that task. Efficient meth- ods such as attribution patching (Nanda, 2023) and edge attribution patching (Syed et al., 2023) are linear approx- imations of activation patching for discovering important components quickly.
More broadly, we refer to editing and unlearning that acts on components of the model identified by mechanistic in- termediate component analysis as mechanistic unlearning. We demonstrate that FLU mechanistic unlearning leads to better trade-offs between edits//nlearning and maintaining performance on general language modelling capabilities, compared to edits done using OT without any localiza- tion. Further, it exhibits improved robustness to re-learning and alternative prompting, and we demonstrate that the la- tent knowledge is also perturbed.
Fact Editing and Machine Unlearning seek to modify pre-trained models to eliminate or alter learned knowledge such as capabilities facts. Some prior approaches focus on identifying and removing specific individual training data points, aiming to obtain model that is "similar" to one that had never trained on these data points (Cao & Yang, 2015; Xu et al., 2023). One formalization of unlearning to match a retrained- from-scratch model is due to Ginart et al. (2019), and is closely inspired by differential privacy (Dwork et al., 2014).
Summary of Contributions
We perform a rigorous evaluation of several standard editing approaches on factual recall tasks, and we iden- tify mechanisms for factual lookup and attribute extrac- tion on Gemma-7B, Gemma-2-9B, and Llama-3-88 We demonstrate that gradient- editing localized on the factual lookup mechanism is more robust than OT localizations and baselines across multiple datasets, models, and evaluations.
Fact editing focuses on overwriting factual information while preserving overall language generation ability. Meng et al. (2023) attempts to identify MLP modules that are most responsible for factual predictions via Causal Tracing and then applies a rank-one transformation upon these modules to replace factual associations.
We demonstrate that it is more difficult to clicit the for- gotten ground truth answers using alternative prompt- ing with FLU localizations. We also demonstrate slower or no relearning of the ground truth answers, retraining edited models on half of the edited set and evaluating them on the other half of the edited set.
In the context of LLMs and safety, techniques such as Harmlels RLHF (Bai et al., 2022) and Represen- tation Misdirection for Unlearning (Li et al., 2024b) aim to suppress dangerous knowledge or harmful tendencies in LLMs. Li et al. (2024a); Zou et al. (2023; 2024) approach unlearning and dangerous knowledge suppression from a top-down feature view, reading or suppressing linear fea- tures related to memorized, harmful, and undesired concepts. A related line of work on safety proposes methods making it difficult to modify open models for use on harmful domains (Tamirisa et al., 2024; Deng et al., 2024: Henderson et al., 2023), including through adversarial relearning.
We analyze intermediate representations using probing, and provide further evidence that editing with FLU localization modifies the internal latent information to reflect the desired edited answer more than other lo- calizations and baselines. We also analyze the weights that are modified for each localization, and find that OT techniques and baselines modify the attribute ex- traction mechanisms more than the fact lookup mecha- nism.
We show that editing and unlearning localized on these mechanisms is more parameter efficient, by controlling for the sizes of edits made to the model with weight masking.
Failures of Unlearning and Editing have been shown for both localized and nonlocalized methods. Patil et al. (2023) extract correct answers to edited facts from the in- termediate residual stream and through prompt rephrasing. Yong et al. (2024) show that low-resource languages jail- break models output unsafe content, and Lo et al. (2024); Lermen et al. (2023); Deeb Roger (2024) demonstrate that relearning with small amount of compute/data causes models to regain undesirable knowledggetendencies.. Even without explicit finetuning, Xhonneux et al. (2024) show that in-context learning alone suffices to reintroduce undesir- able knowledge despite the model being designed to refuse to output such knowledge. Lee et al. (2024) shows that
1.1. Related Work
Mechanistic Interpretability is subfield of Al inter- pretability, aiming to understand the internal processes of AI models by attributing them to subnetworks (called circuits) within the model (Olah et al., 2020). We focus on the factual recall interpretability literature (Nanda et al., 2023; Geva et al., 2023; Chughtai et al., 2024; Yu et al., 2023), which studies methods that aim to discover mechanisms for the retrieval and formatted extraction of factual information.
2

---

Mechanistic Unlearning
basketball
golf
Correct Facts
Fact extraction mechanism
Wrong Facts
Fact extraction mechanisn
Localize edit weights fact lookup
Fact lookup mechanisn [MLPs]
lookup mechanism [MLPs]
Michael
Jordan
plays
Michael
Jordan
plays
Normal Fact Lookup
Edited Fact Lookup
Figure 1.
High level depiction of mechanistic unlearning. We localize components responsible for fact extraction/enricmment and modify their weights change the associations, in order to target internal latent representations rather than targeting the output. Graph inspired by Nanda al. (2023).
even after alignment techniques are applied make models nontoxic, toxicity representations triggered they argue that this is a reason that models lack robustness and can still be jailbroken.
To increase the comprehensiveness of our evaluation, we run experiments with different forget sizes: 16 athletes and 64 athletes. We refer to this task as Sports-Athlete- Editing. These chosen forget sets are constant between all localizations. For the second editing task, we unlearn all athlete-sport associations for athletes who play one sport. In this case, we establish forget set consisting of all the athletes who play one sport (basketball, baseball, foot- ball), and we edit the association by replacing the athlete's correct sport with golf. For comprehensiveness we vary the sport that the forget set is constructed from. We refer to this task as mall-Sports-Editing Finally, we also design an unlearning task, unlearn associations for all athletes in one of the sports.
still present, just not
Hong et al. (2024) evaluates unlearning by measuring resid- ual knowledge left internal activations, and demonstrate that current approaches fail to remove this residual knowl- edge and thus can be exploited. They attempt unlearning targeting the MLPs these residual knowledge traces reside in, but fail to find a non-oracle unlearning approach that successfully removes residual information.
where the goal is to
2. Methods
Our experiments designed test the effectiveness of localization for editing of facts. In this section describe the tasks used and the localization and editing methods evaluated.
CounterFact-Editine and Sequential-CounterFai Editing task In the CounterFact dataset, following Geva et al. (2023), we first filter the dataset for facts which our models assign higher than 50% probability to the right answer, which varies per model. The goal of our CounterFact-Fditingg task edit constant set of facts, replacing the correct answers with an alternative false target, with the retain set being the rest of the non-forget facts. We vary forget set sizes to be of 16 and 64 facts. In Caaiial--CCnneeeae---eiaaee
2.1. Editing Tasks
We focus editing subsets of two datasets: (1) Sports Facts dataset from Nanda et al. (2023), which contains subject-sport relations across three sports categories for 1567 athletes, and (2) the CounterFact dataset from Meng et al. (2023).
task, we edit a total of facts by sequentially editing four randomly selected subsets of 16 facts. We test sequential editing here because facts from CounterFact can reside in different parts of the model, we wish test if localizations for different facts. These chosen forget sets are constant between all localizations, We also scaled this approach to 1000 facts while creating localizations for each fact individually, showing a more automated approach to edit facts robustly at scale.
Sports Facts: Sports-Athleet--Eiitng Full-Sports- Editing, and Colots-Unleannnng tasks In the Sports Facts dataset, we edit two general groups of factual associa- tions.
exploit different
For the first editing task, we edit factual associations for a constant set of randomly selected athletes belonging to any of the three sport categories. We test editing these sets of associations by replacing their correct sports with one of the other two incorrect sports (with equal probability).
3

---

Mechanistic Unlearning
Models We implement editing on the Gemma-7B LLM, the Gemma-2-9B LLM,, and the Llama-3-8b L.M.. We don't use the Pythia-2.8B (Biderman et al., 2023) and GPT-2 mod- els tested in the previous fact interpretabilityy literature be- cause our larger models have stronger general capabilities which we can measure for side effects, and also because our larger models can provide factual knowledge in more input/output formats for more robustness evaluations. All graphs presented in the main text and appendix, unless oth- erwise specified, are averaged over all three model types.
concatenation, fact lookup, and attribule extraction. Their work, along with Geva al. (2023), demonstrates that the fact-lookup stage enriches the latent representation of the subject (athlete) with information about their corresponding sport, while the attribute extraction stage extracts the latent sport information and formats it in the final token position. We replicate a key result of their work our three models by training logistic regression models (probes) to predict the correct sport using the internal activations of the model taken from different layers. We consider the FLU localization to be the layers at which the probe accuracies rapidly increase as these correspond to layers where representations of the athletes are being enriched to encode the correct sport.
2.2. Localization Methods and Baselines
Given a model M X + I. mapping sequence of tokens X to logits L E R over vocabulary V, we consider M to be directed acyclic graph (C, E) with C' being set of model components and E being edges between components. Adopting notation from Elhage et al. (2021), we consider the query, key, value, and output weights of each head along with the input, gate, and output projection weights of each MLP as components.
For CounterFact, we cannot use the probe technique since, unlike Sports Facts, the correct answers for the dataset do not fall into few categories that we can train probe for. Instead, we first use path patching from Goldowsky-Dil et al. (2023) to measure the direct importance of attention heads and MLPs on the final logit difference between the correct answer and a sample incorrect answer. Path patching finds the effect of corrupting a single path from a sender component to a receiver component on the final logits of the model. Components that change the output significantly without being mediated by other components do so by di- rectly affecting the logits, and thus we consider them to be responsible for extracting the facts encoded in the rep- resentation into the answer logits. We thus refer to these components as the "attribute extraction mechanism" Geva et al. (2023). We use path patching again, this time patching paths between MIPs and the attribute extraction mecha- nism to find the components with the highest contribution to the logit difference as mediated through the extraction mechanisms. Such components enrich the token represen- tations with the appropriate facts to then be extracted, and thus we use them as our FLU localization. More details about the manual analysis for both datasets is outlined in Appendix 1.2.1.
We are interested in finding S C' - I,, mapping of com- ponents to their importance in given task. A localization is a set of components C, := {c: E (,])( > J) where T is a threshold. In practice, we fix T such that C, contains the same number of parameters in OT, FL.U, and random localizations. We use these efficient localization methods for finding these mappings:
Output Tracing (OT) localization: Causal Tracing and Attribution Patching First, we test Causal Tracing, method for finding components with high direct causal im- portance for factual associations (Meng et al., 2023). We also use Attribution Patching (Nanda, 2023) as a fast and acceptably accurate approximation of causal tracing to auto- matically localize over components with high direct and in- direct importance. We additionally consider the versions of these localizations with only MLPs (Causal Tracing MLPs and Attribution Patching MLPs).
Importantly, FLU differs from OT techniques by focusing on the causal effects of ablations upon intermediate repre- sentations used by the factual recall mechanism, not just the effects on the output. We hypothesize that the optimal location for robust editing is in the fact lookup stage rather than in the attribute extraction stage,as adversaries could po- tentially devise alternative extraction methods if knowledge remains in the latent stream. Therefore, our model edits are focused exclusively on the fact lookup MI.Ps
We hypothesize that these output- -baeed techniques will pri- oritize the shared extraction components and other mech- anisms for reformatting predictions over the more diffuse FLU components, and thus appear more precise yet leave the underlying latent information present in the model. This might decrease robustness under alternative extraction meth- ods, thus motivating non-OT-based localization, described next. We discuss the precise components/mechanissn high- lighted by OT localizations in Appendix A.3.
Baselines: Random-MLPs, Random, AlI-MLPs, and Nonlocalized We additionally consider four baselines: one corresponding to C, = C' (i.c., no localization, op- timizing all the components of the model), another that randomly chooses components, another that trains all MLP components, and another trains a random selection of ML.
Fact Lookup (FLU) localization: Next, we use manually derived localizations for MLP layers. For Sports Facts, our localization is inspired by Nanda et al. (2023), who identified components in Pythia 2.8B responsible for token
4

---

Mechanistic Unlearning
components. We test the last MLP baselines determine if mechanistically localized MIPss are uniquely important we want to know if the same unlearning performance can be achieved with just the heuristic that training only MLPs improves robustness, or if mechanistic understanding of the role of the component is crucial.
sired forgotten answer (forget accuracy), and how accurately it recalls the new desired edited answer (edit accuracy). In addition, we also measure the accuracy on facts not in the forget set (maintain accuracy). In cases where positive result is lower accuracy, we use the term error to denote I accuracy (e.g. forget error = forget accuracy). Well- edited models should decrease forget accuracy, increase forget error, and increase edit accuracy. Results of these standard evaluations are reported in Appendix A.7.1.
In Appendix A.5, we analyze the proportions of each mech- anism (the extraction heads, extraction MLPs, and fact lookup MLPs, by parameter count) that are present in each localization.
Inspired by Patil et al. (2023) and Lynch et al. (2024), to en- sure the editing process has not overfit to the specific format of the original prompts, we incorporate a robustness check using a multiple-choice question format (MCQ accuracy). This helps determine to what extent the model edited the information, and whether can still access and utilize that knowledge when prompted differently. In this MCQ evalu- ation, the prompt also includes some in-context examples of answering multiple choice questions correctly. On the forget set, we refer to the accuracy of the model answering with the ground truth as the MCQ Forget Accuracy (stronger methods should decrease MCQ Forget Accuracy), and the accuracy of the model answering with the new edited an- swer's choice as the MCQ Edit Accuracy (stronger methods should increase MCQ Edit Accuracy).
The main text focuses on comparing FLU, Causal Trac- ing, and Nonlocalized, while the appendix has the same figures with all above localizations included, with the same conclusions in every case.
2.3. Parameter Update Method
Once we have a localization Cr, we run one of the un- learning or editing methods, restricting weight updates to only components in C,, We update weights using gradient descent combination of loss functions.
Localized Fine-Tuning Following work by Lee et al. (2023) and Panigrahi et al. (2023), we fine-tune the pa- rameters within the localized components.
Finally, we evaluate the models' post-editing accuracy on MMLU (Hendrycks et al., 2021) as a proxy for general language understanding to measure unintended side effects.
For editing, we use a loss function L A iijjectinn + AzIretain + ASLISE, where Linjection is cross-entropy loss on the forget facts maximizing the probability of the alter- native new false target. Leetain is a cross-entropy loss on train split of the remaining facts, and LSFT is a cross- cross-entropy loss on the Pile dataset (Gao et al., 2020). We sweep over learning rates and injection loss As for three representative localizations in Appendix A.6.
1.1.1. SPORTS TASKS
Sports Prompting For the sports dataset, following Nanda et al. (2023), we first evaluate the accuracy of our models to complete the prompt, "Fact: [athlete] plays the sport of", with a one-shot example of Tiger Woods playing golf given first. Note that this is the same prompt used for the editing loss in the first place. For the MCQ evaluation, we use choices of all four sports (football, baseball, bas- ketball, and golf). We average accuracies over all models, for Sports Apport-AAthlte-Beee sthlete-dditigg we average over editing both 16 and 64 facts, and for hull-SSorts-Ediitng we average over editing Basketball, Baseball, and Football.
3. Editing Evaluation
In this section, we show the results of model editing with lo- calization: we test localization techniques from Section 2.2, and edit these localized components using fine-tuning (Sec- tion 2.3). Here we focus on four main editing tasks: sports- athlete-editing, eull-ng counterfact-editing and soguuental--ouunteffct----o ing results for sports-unlearning in Appendix A.I.
We present augment-
Sports Results As shown in Figure 2, our analysis re- veals that editing employing FLU localization exhibits su- perior performance in forgetting the original information and adopting the edited information, across different prompt formats. As explained in Section 3. better editing should result in higher MCQ Forget Error, and higher MCQ Edit Accuracy.
All the editing tasks are assessed based on prompt- completion based and adversarial relearning evaluations.
3.1. Prompting-based Evaluation
Our prompting-based evaluation assesses an editing method's ability to forget or edit specific information while retaining unrelated knowledge, measured by evaluating how the model post-editing completes the prompts coming from the forget set. We report how accurately it recalls the unde-
Figure 3 shows that FLU localized models are the best on both fronts. The difference is especially significant in Fig- ure 3a, where only FLU model edits generalize meaningfully to MCQ, exceeding other localization methods by more than
5

---

Mechanistic Unlearning
Relearning
Vonila Edit Accuracy
Raloa ning
Vanilla Edit Mccuracy
Error
Errer
fact Locap
Cauraa Tracing M.Pe
Nontecnlined
MCO Edit
Maintain Accurscy
MCO Edit
Maintain Accuracy
Accuracy
Accuracy
(a) Athlete-Fditingg
(h) Pull-Sports-Pdiim Ediitig
Figure 2. Spider plots illustrating the advantages of FL.U for editing Sports across adversarial prompting and relearning evaluations, averaged over all three model types. (T.fft) The Sports-Athlet-- Edditig plot shows that FLU localization leads to editing that is most robust against MCQ prompting and relearning. (Right) The plot shows that most localizations perform approximately equivalently in the Full Full-Sports- Sports-Editing task, with FLU localization slightly better for MCQ.
40% in MCQ Edit Accuracy.
Multiple Choice
1.01
comprehensive comparison of all localization meth- ods with multiple-choice prompting is available in Ap- pendix A.7.2, further supporting our findings.
0.6
3.1.2. COUNTERFACT TASKS
0.4
CounterFact Prompting For CounterFact, we create
MCQ evaluation with four choices for every question, ran- domly ordering the true answer, the injected false answer, and two other question-speciine answers. We also consider the original robustness and side effect evaluations from the Meng al. (2023) dataset: the Paraphrase and Neighborhood facts accuracies. Answers of edited facts are meant to generalize to the Paraphrase eval- uation, which phrases the fact in different but equivalent way, so we report the Paraphrase Edit Accuracy (stronger methods should increase Paraphrase edit accuracy). Edit- ing should not generalize to the Neighborhood evaluation, which presents similar but unrelated facts, so we report the Neighborhood Edit Error which is lower if models incor- rectly report the edited answer in unrelated facts.
A.MM-generated incorrect
MCQ Forget
MCQ Accuracy
(a) Sports-Athlet--Eiine Editing
Multiple Choice
Fact Lookup
Tracing
0.6
Causel Tracing MLPs
0.5
All MLPPs
0.3
0.2
We again also use an MCQ evaluation, where the choices consist of the true answer, the injected false answer, and two other question-specific LLM--generted incorrect answers. note that Paraphrase and Neighborhood evaluations ask for the answer in the same original format, so they are more in-distribution than MCQ. We average accuracies average over
Moo Forget Metrics Accuracy
(b) Full-Sports-Editing
Figure 3. Bar charts showing results of MCQ evaluations, reporting both the forget error and edit accuracy when prompted with MCQ, averaged over three model types. For both (a) Sports-Athlete. Editing and (b) Full-Sports Eeditig. FLU localization answers with the original answer the least (MCQ Forget Error) and answers with the edited answer accurately (MCQ Edit Accuracy).
all models, and for CounterFact-Editoe editing both 16 and 64 facts.
CounterFact Results Similar to our sports facts evalua- tion, Figure+ 4 shows an overview of key performance met- rics across localizations for CounterFact. We see that FLU localization outperforms the baselines, being the only lo-
6

---

Mechanistic Unlearning
MILLU
Paraphrase Edit Accuracy
Edit Vecuracy
Paraphrace Edit Accuracy
Edt (ccuracy
Pacl Lookep
causel rrocing
Councl Trrcing MtP7
Nericcalized
All MLPS
MCO Edit
Maintain Accurscy
MCG Esit
Maintain
Accuracy
Accuracy
Accurary
(a) CounterFact-Ediitee
(b) Conpeetial--Cuntffaaatttttee
Figure 4. Spider plots illustrating the advantages of FLU for editing CounterFact across prompting evaluations, averaged over all three model types. (Left) The CounterFact-EEitting plot shows that FLU localization leads editing that the most robust against MCQ prompting and Paraphrasing. (Right) The Scquential- CownterFact--diittiiio plot shows that FLU localization is the most robust against MCQ prompting.
calization to maintain MCQ edit performance. Figure 5 illustrates the robustmess of FLU editing in the MCQ and Paraphrase evaluations. Edited models using FLU local- ization answer less frequently with the original, incorrect information (MCQ Forget Error) and more frequently pro- vide the edited answer.
Multiple Choice, Peraphrase, Meighborhood
1.0
0.5
Furthermore, the Neighborhood Edit Error highlights that other localization methods exhibit slightly more pronounced side effects, inadvertently editing unintended, semantically similar facts.
0.2
0.0
MCG Forgel Error
MCD Edt. Accuracy
Panaphnew Adit T
Neightur hond Edit Emor
Interestingly, sequential editing displays marginally greater robustness than nonsequential editing in MCQ when comparing between CounterFact-Ediinng and Sequential- CounterFact-Editting bars. This observation supports an approach that editing large sets of facts can be made more effective by partitioning the set and applying edits sequen- tially. We present results comparing all localizations from Section 2.2 across each prompt robustness evaluation in Appendix A.7.2, with consistent conclusions.
(a) CounterFact-Eddie
Muttiple Choice, Paraphrace, Neighborhood
1.0
Fact Lookuup Tracing
Thating
Noniocaliod
All MLPPs
0.6
3.2. Adversarial Relearning Evaluation
Forget
Paraphraee Edlt
Neghoorhood
Error
Edit
We measure the ability of our models to withstand adversar- ial releaming, both to address the scenario in which adver- saries may have fine-tuning access and as a measure for the quality of editing. We replicate the methodology of Deeb & Roger (2024), splitting our forget sets in two independent halves, retraining with half of the ground truth labels, and evaluating on the other half. This methodology aims to discern whether the editing technique successfully removed the underlying factual association or merely obfuscated its direct retrieval while leaving it potentially susceptible to recovery when doing partial retraining.
Matrica
(b) Sequential Saanentil-CCuunteFaa---taiii
Figure 5. Bar charts showing results of MCQ, Paraphrase, and Neighhorhood prompt evaluations, averaged over three model types. For both (a) CiauterrFact-dditing and (b) Sequential- CounterFact SounterFact-Edtting, FLU localization has the most robust edit accuracy measured by MCQ and Paraphrase. FLU localization edit- ing also does not incorrectly generalize to Neighborhood prompts. Sequential editing slightly more robust than nonsequential edit- ing in MCQ when comparing between CounterFact-Edinn and Sequential-Counten Coattall-Coutttaee----iiiii bars.
We retrain with rank-512 LoRA across all linear modules, with details available in Appendix A.7.3. We focus on the
7

---

Mechanistic Unlearning
Sports-Athlete-Ediiin task, in the other tasks it was either too easy relearn Saull-Sortt-Fdiiting or too hard to relearn any performance (all CounterFact tasks) across all localizations. Relearning isn't t a valid evaluation for Full- Sports-Editing because the facts are not independent, and models should reasonably generalize from relearning on half the basketball athletes to correctly answering the other half of the basketball athletes.
probes for each sport and take the maximum classification over the sports. This discovers internal representations of the sport the model believes the answer to be: then, we apply these probes on the activations of the forget set of athletes.
We present graphs averaged over models and over 16 and 64 facts Appendix A.7.4. We also present probing graphs for each individual model and all localizations in Appendix A.7.4. We demonstrate that probes on FLU con- sistently predict the forget sport less and the edit sport more than any other localization.
Sports Results Our adversarial relearning experiments, as depicted in Figure 6, reveal that retraining on subset of the original "forgotten" data can recover significant portion, as much as 63%, of the supposedly forgotten in- formation when using OT methods like Causal Tracing and Causal Tracing MLPs. This suggests that these methods may simply mask direct retrieval of this information, leaving the model susceptible this information recovery through retraining. In contrast, FLU localization exhibits greater resilience to such adversarial relearning, with only about 20% of the forgotten information recovered. This indicates that FLU localization may be more effective in targeting and removing the underlying knowledge, making it harder to recover through retraining.
Gomma MCQ Accuracy Across Facts Edited
(Hligeer beiter)
1.2
Pert celop
nsl. Alts
0.2
M.Pe
MCO Edit Accuracy
fandaa Chance
0.6
0.4
Facts Edited
Figure 7. MCQ edit accuracy for Gemma-7B as number of facts edited is increased to 1000 facts. FL.U localization is the only localization that preserves MCQ edit accuracy.
Sports Forget Accuracy Recovered by Relearnnng
0.7
MI Inftial steps)
Final (20 steps)
Foroet Accuracy
3.4. Fact Editing at Scale
0.5
0.4
We now examine the efficacy of our localization technique when editing between 30 to 1000 facts at once. This section also tests the use of an automated method of our "man- ual" FLU analysis. We automate our CounterFact analysis from Appendix 1.2.1 by simply restricting to MLPs that have more than standard deviation impact on the logit difference. We also now calculate localization for each fact instead of taking the average across batch of facts. We improve efficiency and avoid having to edit individual facts sequentially by batching facts with coincidentally the same localization during the fact editing process. To save computational resources, we perform this analysis on the Gemma-7B model and only compare our localization with the best baseline (all MLPs). In addition, we also test the effect of allowing attention heads to be part of the localiza- tion.
0.3
0.2
0,200
0.0
0.000
Lookoup
0,099
Tracin localizze Localization
0,000
0.000
0.001
MLPs
Tracing
Figure 6. Relearning recovers the least accuracy on the forget set using FLU localizations. Relcarning recovers significant accuracy on the original forget set in OT localizations (Causal Tracing and Causal Tracing MLPs).
We show results from Counterfact and from all localizations in Appendix A.7.3.
3.3. Latent Knowledge Analysis
In this section, we provide more evidence of our hypothesis that FLU unlearning targets the source of intermediate latent knowledge. We analyze the Aports-Athlete EEiting task again here because the ground truth and the edited answers vary between one of only three possibilities.
We see in Figure 7 that FLU localization preserves MCQ edit accuracy while the baselines fall to random chance levels. Figure 9 shows that FLU localization also has the highest forget error across edited fact amounts. Finally, we also provide latent knowledge analysis results at scale in Figure 8, showing that the FLU localization prevents latent knowledge extraction from being useful, and that including attention heads in the localization yields poor results towards the last layers of the model. We believe these results provide evidence that our methodology works well at scale.
We train logistic regression models (probes) (Alain & Ben- gio, 2018) on prompt activations following every model layer to predict the correct ground truth sport on the main- tained set of athletes. This is possible since there are only three possible sports, so we can train binary classification
8

---

Mechanistic Unlearning
Probe Accuracy across Layers Facts
Probe Accuracy acmes Layers 100
Gemma Probe scross Layers 300
Gemma Probe 1000 Facts
Part Lookup
Fact Lostp
4.6
Ind,
Indt Altn
Random Chance (0ptina)
Random Chance (Oatimal)
Layer
Layer
Figure 8. Latent knowledge analysis across layers for Geemma-7B when editing varying amounts of CounterFact facts. FLU localization all MLPs maintain random probe accuracy, resisting latent knowledge extraction unlike when including attention heads as part the localization.
Gemma 7b: MCQ Forget Crror Across of Facts Edited (Higher is better)
trolling for the size of the localization FLU is consistently more robust when subject to our suite of evaluations.
But Lookup
MCQ Farget Error
Ransom
Other Localizations Affect the Extraction Mechanism After training weight masks, we analyze the proportion of each mechanism (fact lookup. attribution extraction) that is masked by each localization's 's weight mask in Appendix A.5. We demonstrate that OT methods and nonlocalized editing all modify a higher proportion of the extraction head/MLP parameters than the fact lookup mechanism parameters, sup- porting our claim that OT methods target extraction mecha- nisms rather than fact lookup mechanisms.
327
20
of Fatts Edited
Figure 9. MCQ forget error for Gemma-7B when editing varying amounts of CounterFact facts. FLU localization maintains better forget error than the baselines.
3.5. The Role of Parameter Count
4. Discussion
In this section, we perform weight-masking involving train- ing binary differentiable mask individual weights of the model within the localized components, inspired by weight pruning/masking work (Bayazit et al., 2023; Pani- grahi et al.. 2023), to quantify the size of edits with different localizations and investigate which factual mechanisms are targeted when editing with different localizations. In this case, no weight updates are being performed. Rather, the mask turns a subset of the weights zero.
Recent work by Hase et al. (2023) argued that localiza- tion is not useful for model editing. Our findings demon- strate that the relationship between localization and fact editing/umlearning is more nuanced, and reveals that not all localization techniques are equal.
Our work evaluates the efficacy of different localization methods for modifying factual associations. We demon- strate clear benefits of localization for editing robustness through localized fine-tuning on the FLU mechanism.
Controlling for Parameter Count Although we already standardize the number of trainable parameters in most lo- calizations, additionally investigate if FLU editing is better than other localization techniques when controlling for the exact number of parameters that are masked. We perform weight masking on the Sports-Unlearning Sports- Athlete-Be Ediitin, and CounterFact- CounterFact-Editing tasks. Detailed results are reported in Appendix A.4. We find that when con-
In Sections 3.3 and 3.5 and Appendix A.5, we provide evidence that OT and baseline approaches fail be robust because they target extraction components, which fails to generalize and does not target the source of knowledge in the model. In contrast, FLU mechanistic understanding allows us to target editing at the sites where knowledge is sourced, which robustly prevents that information from entering the latent stream any format.

---

Mechanistic Unlearning
Impact Statement
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph,
N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly,
This paper advances the fields of interpretability and model editing, both of which are relevant for ensuring the safety, privacy, and fairness of models. We hope our methods help model developers responsibly manage harmful knowl- edge/behaviors.
T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kermion, J., Lovitt, I.., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Truns- former Circuits Thread, 2021. https://transformer wwa.iisspub/222//faaeeo////////
References
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, 2018.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N.. Srefieldd-oddss Z.. Herann- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, I.., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Train- ing helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Geva, M., Bastings, J., Filippooaa, K., and Globerson, Dissecting recall of factual associations in auto-regressive language models, 2023.
Ginart, A., Guan, M. Y., Valiant, G., and Zou, J. Making ai forget you: Data deletion in machine learning, 2019.
Goldowsky- N., MacLeod, C., Sato, L., and Arora, Localizing model behavior with path patching, 2023, URL https: / arxiv. hagg/abb//2330 05969.
Bayazit, D., Foroutan, N., Chen, Z., Weiss, G., and Bosse- lut, Discovering knowledge-critee subnetworks in pretrained language models, 2023.
Hase, P., Bansal, M., Kim, B., and Ghandcharioun, A. Does localization inform editing? surprising differences in causality-basee localization vs. knowledge editing in lan- guage models, 2023.
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H.,
O'Brien, K., Hallahan, E., Khan, A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https: / arxiv org/ abs/ 4 01373.
Henderson, P., Mitchell, E., Manning, C., Jurafsky, D., and Finn, C. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on Al, Ethics, and Society, pp. 287-296, 2023.
Cao, Y. and Yang. J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding. 2021.
Chughtai, B., Cooney, A., and Nanda, N. Summing up the facts: Additive mechanisms behind factual recall in Ilms, 2024.
Hong, Y., Yu, I.., Yang, H., Ravfogel, S., and Geva, M. In- trinsic evaluation of unlearning using parametric knowl- edge traces, 2024. URL https: //aaxiiivvorrr// /ab// 2406.11614.
Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso A. Towards automated circuit discovery for mechanistic interpretability, 2023.
Deeb, A. and Roger, F. Do unlearning methods remove information from language model weights?, 2024. URL https:/ arxii.iiv.. org/ abs/ 410. 08827.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017.
Deng. J., Pang. S., Chen, Y., Xia, L., Bai, Y., Weng, H., and Xu, W. Sophon: Non-fine-tuns fine-tunable learning to restrain task transferability for pre- pre-trained models. arXiv preprint arXiv:2404. 1..404..2999 12699, 2024.
Lee, A., Bai, X., Pres, I., Wattenberg, M., Kummerfeld, J. K., and Mihalcea, R. A mechanistie understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint wwXiv:2401001661 2024.
Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theo- retical Computer Science, 9(3-4):211 10771, 2014.
Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C. Surgical fine-tuning improves adaptation to distribution shifts, 2023.
10

---

Mechanistic Unlearning
Lermen, S., Rogers-Smith, C., and Ladish, J. Lora fine- tuning efficiently undoes safety training in llama 2-chat 70b, 2023.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to cir- cuits. Distill, 2020. doi: 10.2391 .0024..000.0
Lester, B., AI-Rfou, R., and Constant,
N. The power of scale for parameter-efficient prompt tuning, 2021. URL https: /ttp:////aiivv..eaa org/ wtw.abs/////ooooooooo..///s..a.s 2104. 08691.
Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task- specific skill localization in fine-tuned language models, 2023.
Li, M., Davies, X., and Nadeau, M. Circuit breaking: Re- moving model behaviors with targeted ablation, 2024a.
Patil, V., Hase, P., and Bansal, M. Can sensitive information be deleted from Ilms? objectives for defending against extraction attacks, 2023.
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-.. Goel, S., Phan, L., Mukobi,
Polo, F. M., Weber, L.,, Choshen, L., Sun, Y., Xu, G., and Yurochkin, tinybenchmarks: evaluating IIms with fewer examples, 2024.
G., Meem--uugee, N., Lababidi, R., Justen, L.,, Liu, A. B., Chen, M., Barrass, I., Zhang, O., Zhu, X., Tamirisa, R., Bharathi, B., Khoja, A., Zhao, Z.,, Herbert- Voss, A., Breuer, C. B., Marks, S., Patel, O.. Zou, A., Mazeika, M.. Wang, Z., Oswal, P., Lin, W., Hunt, A. A., Tienken- Harder, J., Shih, K. Y., Talley, K., Guan, J., Kaplan, R., Steneker, I., Campbell, D., Jokubaitis, B., Levinson, A., Wang J., Qian, W., Karmakar, K. K., Basart, S., Fitz, S., Levine, M., Kumaraguru, P., Tupakula, U., Varadharajan, V., Wang, R., Shoshitaishvili, Y., Ba, J., Esvelt, K. M., Wang, A., and Hendrycks, D. The wmdp benchmark: Measuring and reducing malicious use with unlearning, 2024b.
Syed, A., Rager, C., and Conmy, A. Attribution patching outperforms automated circuit discovery, 2023.
Tamirisa, R., Bharathi, B., Phan, L., Zhou, A., Gatti, A.,
Suresh, T., Lin, M., Wang, J., Wang, R., Arel, R., Zou, A., Song, D., Li, B., Hendrycks, D., and Mazeika, M. Tamper-resistant safeguards for open-weight Ilms, 2024. URL https: /aaxivv...iiv////aa/ org/ /bb////////// /aw.///08B... 2408. 00760
Xhonneux, S., Dobre, D., Tang, J., Gidel, G., and Sridhar, D.
In-context learning can re- leare forbidden tasks. arXiv preprint arXiv:240..05723, 2024.
Lo, M., Cohen, S. B., and Barez, F, Large language models relearn removed concepts, 2024.
Xu, H., Zhu, T., Zhang, L., Zhou, W., and Yu, P. S. Machine unlearning: A survey. ACM Computing Surveys, 56(1): 1-36, 2023.
Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield- Menell, D. Eight methods to evaluate robust unlearning in Ilms, 2024.
Yong, Z.-.., Menghini, C., and Bach, S. H. Low-resource languages jailbreak gpt-4, 2024.
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang. Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024.
Yu, Q., Merullo, J., and Pavlick, E. Characterizing mecha- nisms for factual recall language models, 2023.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z.,, Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach ai transparency, 2023. URL wwwpsiiiiorgggaae.o wrtp.////.raab//1/0000 01405.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt, 2023.
Nanda, N. tion patching at industrial scale, 2023. URL
Attribution patching: Activa-
mrcchanistic--inteprrtttddi https://wwwneelnandd.. heelnanda.... attribbttionpatcchin
Zou, A., Phan, L., Wang. J., Duenas, D., Lin, M., An- driushchenko, M.. Wang. R. Kolter, Z., Fredrikson, M., and Hendrycks, D. Improving alignment and ro- bustness with circuit breakers, 2024. URL https:
Nanda, N., Rajamanoharan, S., Kramár, J., and Shah, R. Fact finding: Attempting to reverse- engineer factual recall on the neuron level, Dec 2023. URL
fact facee--ffhllngggatttteorrrreeeee

---

Mechanistic Unlearning
A. Appendix
A.I. Sports Unlearning Results
For unlearning on the Sports-Unlearning task, we use a loss function
L = X1 Lorget + X Aeeuin + X3 LSFT,
where Looget is an unlearning loss on the forget subset of facts we want to forget, Lestan is a cross-entropy loss on the remaining facts, and LSFT is cross-entropy loss on the Pile dataset (Gao et al., 2020). The unlearning loss Lorget we use the log(1 p) measure (where p is the probability of the correct sport) from Mazeika et al. (2024) due to its empirical stability and fewer side effects: vanilla gradient ascent more strongly incentivizes the model to have significantly lower logprobs than wouldn't be encountered in model that has not been trained on the factual association, and it detracts from model maintenance of Lictain and LSFT-
We present results on using various localizations on the Sports-Unleande Unleaming task in Table 1. The FLU localization allows unlearning to be more robust to the MCQ prompt format while maintaining performance on the MMLU dataset.
Table Localized fine-tuning accuracy on standard evaluations: unlcarning all basketball athletes and retaining all other facts.
LOCALIZATION
FORGET
RETAIN
MCQ
MMLU
ATTRIB. PATCHING
0.767
CAUSAL TRACING
0.201
0.998
0.349
0.611
FLU
0.002
0.995
0.110
0.613
RANDOM
0,952
0.980
0.822
0.612
ALL-MLPS
0.000
0.994
0,279
0.606
NONLOCALIZGD
0,000
0.985
0.196
0.595
A.2. FLU Interpretability Analysis
A.2.1. SPORTS FACTS
We redo analysis from Nanda al. (2023) on Gemma-7B, Gemma-2-9B. and Llama-3-8B, We train logistic regression models ("probes") predict the correct sport given the internal representation of the model at layer. We find that probes predicting the correct sport increase in accuracy significantly in layers 2 through 7 in Gemma-7B and 2 through 8 for Gemma-2-9B and Llama-3-8B (Figure 10).
Unlike Nanda et al. (2023), however, we find attention heads past layer that impact the linear representation attributes and thus could potentially be important for fact lookup. However, because they could likely play a variety of other different roles such as token concatenation, following the findings of Geva et al. (2023); Nanda al. (2023) that MLPs do primary factual representation enrichment, in this work we only consider the MLPs as our localization.
(gemma-7b): Probe Accuracy Across Layers
(gemma-2-9b): Probe Accuracy Across Layers
1.0
1.0
0.9
Probe Accuracy
Probe Accuracy
0.8
0.7
0.6
0.6
0.5
0.5
0.4
Layer
Layer
Promot
0.4
Layer Layer
Promot Athicte
Layer
Layer
12

---

Mechanistic Unlearning
(lamm-3-8D) Probe Accuracy Across Layers
1.0
0.9
Probe Accuracy
0.7
0.6
0.5
0.4
Layer 2
Layer
5
10
15 Layer
20
25
Figure 10. Accuracy of probes predicting the correct sport across layers for different models.
A.2.2. COUNTERFACH
repeat analysis from Nanda al. (2023) and Geva et al. (2023) on Gemma-2-9B. first measure the effect on the difference in logits between correct and incorrect answers of facts when patching the direct path of attention heads and MLPs the final output, shown in Figure 11. An attention head MLP will have large effect the logit difference if it important in moving the factual information to the last token position or decoding into the correct answer. We call these components part of the "fact extraction mechanism", and aim to find the source of the factual information moved by this mechanism.
To find this source, we patch the outputs MLPs this "fact extraction mechanism" and measure resultant change in logit difference (Figure 12). An MLP would cause large change logit difference if caused relevant representations to form that are then moved by the "fact extraction heads" to increase the probability of the correct output. We provide the logit differences for all 64 facts along with just the first 16 facts, and see that the logit differences similar across the dataset splits. We take the MIPs with the highest change (> 0.02) and include them in our FL.U localization of CounterFact.
(gemma- 7b): Change in Logit Difference Path Patching Heads >> Final Outout
(gemma-2-9bl) Change Logit Difference Path Patching Heads -> Final Output
(llama-3-8) 3-8b): Change Logit Difference
Path Patching Heads -> final Output
0.20
3.20
0.20
0.15
0.10
10-
10
0.05
0.05
0.05
Layer
Layer
Layer
0.00
20
0.00
15-
0.00
15
-0.10
0.10
-0.10
25
0.15
-0.15
40
30
.200
10
-020
-020
Head
Head
Head
Figure 11. Change in logit difference when patching attention heads to the output.
13

---

Mechanistic Unlearning
(gemma-77 7b): Logit Difference patching
(gemma-2-9b)) Logit Difference patching
MLP Extraction Mechanism
MLP -> Extraction Mechanism
Subset
0.04
0.10
0-64
0-16
Logit Difference
Logit Difference
0.08
0.00
0.06
0.02
-0.04
0.02
0-64
0-16
-0.08
10
15
20
25
Layer
Layer
(llama-3-8b)) Logit Difference patching MLP Extraction Mechanism
Sutset of Dataset
0-64
0-16
Logit Difference
0.05
0.04
0.03
0.02
0.00
5
100
15
20
Layer
Figure 12. Logit difference when patching MLPs to the extrac- tion mechanism found above different models.
A.3. OT Selected Components
What MLPs do the automated OT methods localize? We explore the attribution scores of the automated localization methods (causal tracing and attribution patching) the MLPs to if automated localization methods detect the FLU mechanism. In Figures 13 and 16, for Gemma-7B, we see that both CT and AP localizations target the later layer MLPs instead of the FLU mechanism (Figure 13).
(gemma-7b): Attribution Score Assigned Each MLP Sports Facts
CT Localization
AP Localization
1.75
RW/
ct
Rcction
FLU
ap
Extracionn
1.50
Attribution Score
1.25
0.6
Attribution
1.00
0.4
0.75
0.50
0.2-
0.25
0.00
0.0
MLP Layer
MLP Layer
Figure 13. Attribution scores on MLPs on sports facts for Gemma-7B.
For AP localization, this trend continues with Gemma-2-9H (Figure 14) and 1lama-3-88 (Figure 15). However, CT
14

---

Mechanistic Unlearning
localization does highlight some of the early layer MLPs that are in the FLU mechanism, especially for Gemma-2-9h
[gemma-2-900 Attribution Score Assigned Each Sports
CT Localization
AP Localization
0.30
TLU
Ectraction
Estraction
Attribution Score
0.5
Altribution Score
0.4
0.20
0.2
0.10
0.1
0.05
10
10
20
30
MLP Layer
MLP Layer
Figure 14. Attribution scores on MLPs on sports facts for Gemma 2-9B.
(llama-3-8b): Attribution Score Assigned to Each MLP on Sports Facts
Localization
Localization
Extraction
Attribution Score
4-
0.20
Attribution
3.
0.15
2-
0.05
10
20
30
10
20
30
Layer
MLP Layer
Figure 15. Attribution
MLPs sports facts for Llama-3-8B.
We repeat this analysis on CounterFact in Figures 16 to 18. Again we see AP localizations in particular assign higher scores to later-layer MLPs, and CT only highlights FLU components on Gemma-2-9B, localizing other extraction layers on the other models.
[gemma-7b); Attribution Score Assigned to Each MLP Counterfact Facts AP Localization
CT Localization
0.25
ap
Attribution Score
0.20
0.6
Attribution
0.15
0.10
0.05
0.00
0.0
MLP Layer
Layer
Figure 16. Attribution scores on MLPs on CounterFact facts for Gemma-7B
A.4. Weight Masking
In this section we employ weight masking quantify the size of weight updates needed to unlearn/edit facts, for more direct comparisons. Our loss function L x Liorget + Ag + A3 + Aą now includes an LI regularization term to control the sparsity. We empirically evaluate how a learned binary mask over individual weights the localized components can produce editing/unlearning, and vary the size of this mask. "Manual Interp" refers to the FLU localization technique for all the following results in this section.
15

---

Mechanistic Unlearning
(gemima--99b): Attribution Score Assigned to Each MLP on CounterFact Facts
Localization
Localization
0.30
FIV
Extraction
0.20
Extractive
0.25
Attribution Score
0.15
0.20
Attribution
0.15
0.10
0.05
0.05
100
20
30
40
10
20
30
40
MLP Layer
MLP Layer
Figure 17. Attribution scores MLPs CounterFact facts for Gemma-2-44
(llama-3-8b): Attribution Score Assigned Each MLP Counterfact Facts
CT Localization
AP Lacalization
0.200
0.175
Extraction
ct
0.20
Extradion
ap
0.15
0.125
Attribution
Attribution
0.10
0.075
0.050
0.05
0.025
0.000
30
0.00
20
30
MLP Layer
MLP Layer
Figure Attribution scores on MI.Ps on CounterFact facts Llama-3-88.
1. UNLEARNING SPORTS
show standard evaluations across a sweep of discretization thresholds, which directly corresponds to the size of the model edit. Figure 19 shows the accuracy on the forget and retain sets for unlearning basketball across different edit sizes. Here, see all methods being effective in unlearning basketball facts while retaining all other facts. While AP and CT localizations cause the model to have accuracy the in-distribution set with much fewer masked weights needed, when checking for generalization using multiple-choice format we clearly see that only manual localization has successfully generalized the unlcarning of basketball facts (Figure 19, right).
Basketball Accuracy vs Num. Masked Weights
MCQ Basketball Accuracy vs Num. Masked Weights
Normal Forget Accuracy
1.0
1.0
MCO Forget A Accuracy
0.6
0.6
0.4
Los alized AP forget ace aracy
0.4
Localieed AP furget accuracy
Loc alzed CT forget accuracy
Lecalized CT forget acouracy
Random
interp torget accuracy
Nanual forget accuracy
Nortecaized lzed forget accurscy
Norlocatzed forget securacy
0.0
ronk
50mh A0de
Tode
9tck
1.34
2.4M
0.0
2006
500k
Took
B0ok
2.1M
2.4M
Number of Masked Weights
Number of Masked Weights
Figure 19. (Left) Testing the models' unlearning of basketball athletes against the number of weights masked. (Right) Testing the models' unlcarning of basketball athletes against the number of weights masked, the MCQ prompt format.
We find similar results when testing for performance degradation on MMLU (because we have to evaluate many model variations, we use a smaller MMLU test set from Polo et al. (2024)). While localized methods perform well when evaluated normally (Figure 20, left), Figure 20 (right) shows manual localization generalizes for minimizing loss MMLU capabilities while unlearning sports facts in the MCQ format compared to the other methods.
16

---

Mechanistic Unlearning
Basketball vs MMLU Accuracy
MCQ Basketball vs MMLU Accuracy
0.70
0.70
0.65
0.65
MMLU Accuracy
0.60
MMLU Accuracy
0.60
0.55
0.55
0.50
0.45
0.45
Localized AP
Localized Localized CT
0.40
Localized CT
0.40
Random
Random
0.35
Manual Interp Nonlocalzed
Manual Interp
Nontocalized
0.30
0.30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Normal Forget Accuracy
MCQ Forget Accuracy
Figure 20. Unlcarning basketball facts. (Left) Mcasuring MMLU and forget set performance across different discretization thresholds. (Right) Mcasuring MMLU and MCQ forget set performance across different discretization thresholds.
Athlete Accuracy vs Num. Masked Weights
MCQ Athlete Accuracy vs Num. Masked Weights
Normal Forget Accuracy
MCQ MICQFoo Forget Accuracy
0.8
0.6
0.6
Localized AP forget accuracy
forget accuracy
Localized CT fonget accursty
Localized forget accurscy
0.2
Manual interp forget accuracy
Random torget accuraty
0.2
Random forget accuraty
Manual Interp forget accurecy
furget acturacy
0.0
Sont
1,2M
1.5M
1.aM
2M
2,4M
lonk
Sook
1.2M
1.5M
1.0M
21M
Number of Masked Weights
Number of Masked Weights
Figure Editing subset athletes. (Left) Measuring accuracy on the forget set. (Right) Measuring accuracy on the forget set in the
MCQ prompt format.
A.4.2. EDITING ATHLETES
For editing the subset of athletes, Figure 21 shows that causal tracing localization causes the model have 0% accuracy on the forget set, and FLU and nonlocalized editing cause the model to have near guessing rate (33%) accuracy. However, only manual localization minimizes loss of capabilities while editing the athlete subset (Figure 22).
Furthermore, no other method completely generalizes this unlearning to the MCQ prompt format (Figure 21), and manual localization remains superior in minimizing loss of capabilities while unlearning the athlete subset (Figure 22, right).
A.4.3. EDITING COUNTERFACT
We find similar results for editing on the CounterFact dataset. However, we find minimal difference in MMLU accuracy in all methods all numbers of masked weights. Thus, we instead report the maintain and forget accuracies of these facts different discretization thresholds in Figure 23.
Additionally, report comparison of all localizations across discretization thresholds for normal and MCQ forget sets in Figure 24 and Figure 25. We see that FLU outperforms all other methods of localization in preserving maintain accuracy while decreasing forget accuracy.
We perform additional adversarial analysis of accuracies across different discretization thresholds. We report the "paraphrase" and "neighborhood"" adversarial results in Figure 26 and Figure 27, but find no significant results.
A.5. Mechanism Weight Analysis
We analyze the actual components localized by each localization type and our baselines, for the CounterFact editing task. We seek demonstrate that the OT localizations and baselines target extraction mechanisms rather than just the FLU
17

---

Mechanistic Unlearning
Athlete MMLU Accuracy
MCQ Athlete MMLU Accuracy
0.70
0.70
0.65
0.65
MMLU Accuracy
0.60
MMLU Accuracy
0.60
0.55
$
0.55
0.50
0.50
0.45
0.45
Localized AP
Localized
0.40
Localized CT
0.40
Localized CT
Random
Random
0.35
Manual interp
0.35
Manual interp Nonlocalized
Nonlocalized
0.30
0.30
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Normal Forget Accuracy
MCQ Forget Accuracy
Figure 22. Editing subset of athletes. (Left) Measuring MMLU and forget set performance across different discretization thresholds. (Right) Measuring MMLU and MCQ forget set performance across different discretization thresholds.
Editing: Normal Forget Accuracy across Thresholds
Editing: MCQ Forget Accuracy across Thresholds
1.0
0.8
Norma Accuracy
MCO MCO Accuracy
0.6
0.4
Localized Forget
0.4
Localized AP Forget
Localized Forget
Localized Forget
Random Forget
Random Forget
Manual Interp Forget
0.2
Manual Interp Forget
Nonlocalized Forget
Nonlocalized Forget
All MLPs Forget
All MLPs Forget
0.0
1M
2M
3M
4M
SM
6M
2M
4M
SM
6M
Number Masked Weights
Weights
Figure 23. Editing CounterFact facts, (Left) Testing models accuracy on the normal forget set. (Right) Testing the models' accuracy in the MCQ prompt format.
Editing: Normal Farget vs Maintain Accuracy
Editing: MCQ Forget vs Maintain Ac Accuracy
Normal Maintain Acturacy
MCQ Maintain Accuracy
0.6
0.4
Locallied AP
Localized AP
Localized CT
Localized CT
0.2
Random
0.2
Random
Intors
MLPSss
0.0
0.0
0.0
0.2
0.4
0.8
1.0
0.2
0.4
0.6
Normal Forget Accuracy
MCQ Forget Accuracy
Figure Accuracy on normal forget set vs on maintain set across localizations and discretization thresholds.
Figure 25. Accuracy on multiple choice input vs on the main- tain set across localizations and discretization thresholds.
mechanisms.
First, Table 2, we compare the parameter counts of the part of each mechanism that present each localization. Table 2 shows that causal tracing and attribution patching both have the potential to modify considerable proportion of the
18

---

Mechanistic Unlearning
Editing: Paraphrase Farget vs Maintain Accuracy
Editing: Neighborhood Forget vs Maintain Accuracy
1.0
1.0
Paraphrase Maintain Acturacy
Neighbarhood Maintain Accuracy
0.8
0.8
0.6
0.4
Locakkad
Locelieed
Localtzed CT
Localized
Random
0.2
Random
Manual Interp
Nanual inters
Nonlocalzeed
Nordocaliond
NIMPPs
All HLPS
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Paraphrase Forget Accuracy
Neighborhood Forget Accuracy
Figure 26. Accuracy on paraphrased input vs the maintain set across localizations and discretization thresholds.
Figure 27. Accuracy "neighborhood" input the main- tain set across localizations and discretization thresholds.
extraction heads and extraction MLPs.
Table 2. Comparison of total parameters of each mechanism that are present in each localization, for editing 16 facts from CounterFact
LOCALIZATION
EXTRACTION HEADS
EXTRACTION MLPS
FACT LOOKUP
TOTAL
27.448.320
1,027 680,480
1,130,364,92 ,288
PATCHING
13,724,160 (50.0%)
616,562.688 (60.0%)
102,760,448 19.
CAUSAL TRACING
8,234,496 (30.0%)
308,281,344 (30.0%)
411,041,792 (36.4%)
FLU
1,130,364.92 928 (100.0%)
ALL-MLPS
1,027, 4827.604480 (100.0%)
1.130,364.925 (100.0%)
NONLOCALIZED
27.448.320 (100.0%)
1,027, 1,027,604.480 207,4004448 (100.0%)
1.130.364,928 (100.0%)
Then, in Table 3, we compare the proportion of each mechanism that is masked when using a localized weight mask and discretizing to about million weights. This is one approximate metric for how much each mechanism is modified by the localized editing. Table demonstrates that attribution patching, causal tracing, and nonlocalized editing all modify a higher proportion of the extraction head/MI.P weights than the fact lookup mechanism weights.
Table 3. Comparison of parameters of each mechanism that masked by trained weight mask, discretized to about 6 million weights
LOCALIZATION TYPE
EXTRACTION HEADS
EXTRACTION MLPs
FACT LOOKUP
TOTAL (BASELINE)
27,448.322 320 ( (100%)
1,027.604,480 ( (100%)
1,130,364,99 (100%)
ATTRIB. PATCHING
165,300 (0.60%)
1,479.877 (0.14%)
1,385.198 (0.12%)
CAUSAL TRACING
30.828 (0.1 11%)
1,491.040 (5.15%)
1,424,059 (0. 13%)
FLU
(0.0%)
0 (0.0%)
6.248.039 (0.55%
(0.0%)
(0.13%)
1,663.772
NONLOCALIZED
358,918 (1.3%)
1.198,211 (0.12%)
1,174.939 (0.10%)
This supports our argument that OT methods target high logit-diff extraction mechanisms, rather than the fact lookup mechanisms that enrich the latent stream with the correct attributes, which decreases the robustness of edits/unlearning. It is important to note that since FLU localization is based on discovered mechanisms, this does not serve as an evaluation of FL.U (since by definition FLU localization will only localize the FLJU mechanism), but rather only of causal tracing and attribution patching.
A.6. Hyperparameters
and all models, we fine tune using 50 iterations of batch size 4 with 16 accumulation steps, using an AdamW optimizer (Kingma & Ba. 2017) with 0 weight decay and a cosine annealing scheduler. For Gemma-2-9h. we are forced to use an 8-bit optimizer to fit our training in the memory that the optimal learning rate quite sensitive to the localization used and the edit task, so we first sweep over learning rates find reasonable learning rates. We sweep over the learning rates of 2e-6, 5e-6, le-5, 2e-5, 5e-5, and le-4, training
Across all tasks except Sequential-CounterFaae
GPU. We find
19

---

Mechanistic Unlearning
models over 50 iterations with all Às set 1. We also the x parameter associated with the Linjection cross entropy loss. We don't tune the other À parameters because they are all maintenance losses, and setting them to 1 works sufficiently maintain performance across almost all setups.
For the Snsientill-CuuntrFFaal Ediitig task, we use the same hyperparameters from the Cannterraat-Ediiinng task and we train for 100 total iterations rather than 50, using 25 for each subset of 16 facts. We choose 25 iterations because it balances between being half the number of iterations we typically use per subset of that size, and also double the number of steps overall as we use CounterFact-Editing
To avoid leaking evaluation information through this sweep process, we optimize learning rate for the objective of (I-Forget Set Ground Truth (ccuracy)FForgy Set Edit avoiding any of our robustness metrics (one could view this sweep process as simply another part of the training process, since we only use train-time information). We run sweeps for Causal Tracing, Manual Fact Lookup, and No Localization. We then use the hyperparameters from Causal Tracing for Attribution Patching and Random (which all localize to MLPs and attention components), we use the hyperparameters from Fact Lookup for Random MIPs, Causal Tracing MLPs, and Attribution Patching MLPs (all localize to the same number of MLPs), and we use the hyperparameters from No Localization for All MLPs (which have the largest number of active parameters).
Set Ground Truth Accuracy+ Accuracy,
A. I. SAMPLE HYPERPARAMETES SWEEP
In Table 4, we show the full results of one sweep, optimizing learning rate for editing 16 facts from CounterFact. Especially for No Localization, some learning rates fail to edit in the correct answer with high accuracy, or fail to maintain accuracy on the maintain set. In Table 5, we see that editing results are not particularly sensitive to the coefficient used with the injection cross entropy loss.
Table 4. Gemma-7b learning rate sweep, editing 16 CounterFact facts.
Pile Accuracy
Forget Accuracy 1
Edit Accusacy f
Maintained Accuracy ?
Overall Score f
FLU
1,000
0.698
IRR le-05
0.513
0.000
1.000
0.975
3.488
2c-05
0.542
0.000
1.000
0.950
3.492
LR 2e-06
0.499
0.007
0.961
0.869
3.323
Se-05
0.520
0.000
1.000
0.822
3.342
Sc-06
0.528
0.999
0.980
3.507
Localized CT
LR 0.0001
0.462
0.001
0.999
0.697
3.157
LRR Ic-05
0.513
0.000
1,000
0.984
3.496
LR 2c-05
0.507
0.000
1,000
0.961
3.467
2e-06
0.849
LR Se-05
0.488
0.000
1,000
0,846
3.334
5c-06
0.537
1.000
0.982
Nonlocalized
LR 0.0001
0.062
0.032
0.557
0.054
1.680
le-05
LR 2e-05
0.479
0.001
0.998
0.807
3.284
2e-06
0.529
0.000
1.000
0.982
3.510
IRR 5c-05
0.046
0.034
0.710
0.092
1.815
LR 5c-06
0,536
0.000
1.000
0.988
3.524
A.6.2. ALL HYPERPARAMETERS USED
Table 6 has all learning rates used and Table 7 has all injection loss coefficients used.
A.7. Evaluation Details
1. DETAILS ON STANDARD PROMPT EVALUATIONS
We report standard metrics of Forget Error, Edit Accuracy, and Maintain Accuracy, in the same prompt format that the models were trained on. These metrics are optimized by the loss, so we expect all localizations to do almost perfectly on these evaluations. Figures 28 to 31 show that localizations perform approximately equivalently on these basic evaluations
20

---

Mechanistic Unlearning
Table Gemma-7h inject loss coefficient sweep, editing CounterFact facts.
Accuracy
Forget Accuracy
Accuracy
Accuracy
FLU
FC 0.1.1
0.538
0.000
0.998
0.990
3,525
0.999
0.981
PC 0.5
0.508
0.000
0,999
0.985
3,493
0,510
0.000
0,999
0.973
3.483
2C
0.532
0.000
1,000
0.984
3.515
FC5
0.534
0.000
1.000
0.985
3.518
Loculized CT
0.1.1
0.532
0.004
0.995
0.983
3.506
FC
0.535
0.001
0.998
0,986
3,518
0.
0.524
0.000
0.999
0.974
3,497
FC
0,519
0.000
1.000
0.988
3.507
FC
0.536
0.979
3,514
FC
0.518
0.000
1.000
0.975
3,493
Nonfocalized
0.525
0.998
0.969
3.492
FC 0.2
0.562
0.000
0.999
0.982
3.543
FC
0.531
0.000
1,000
0.980
3.511
FC2
0,514
0.000
1.000
0.978
3,492
FCS5
0.528
0.000
1.000
0.975
3.503
Table 6, Optimal learning rates for different models, task types, and localizations.
Model
64 athletes to random
Basketball Athletes to Golf 16 CounterFact facts
64 ConnterFact facts
Gemma
1 10-5 10-5 Nonlocalized
FLU
x 10-3
3 10-6
1 10-3 10-3 2 10-6
Localized
10-6
10
5 10-6
Gemma
FLU
1 x 10
x 10-5
5 10 5 X
2 x 10 2 x 10 5
Localized CT
5 10-5
5 10-5
Nonlocalized
x
5
x
Ilama
FLU
5 10-5
x 10-3
1 10-4
5 10-3 6 10
Localized CT
x 10-4
x 10-5
2 10-3
Nonlocalized
2 10
1 10
2 x 10
1 10
Table 7. Optimal inject coefficients different models, task types, and localizations.
Model
athletes to random sport
Basketball Athletes to Golf
CounterFuct facts
CounterFact facts
Gemma
FLU
5.0
5.0
0.1
2.0
Localized CT
0.1
1.0
0.2
1.0
Nonlocalized
0.2
1.0
0.2
1.0
Gemma 2
2.0
Localized CT
5.0
5.0
2.0
2.0
Nonlocalized
5.0
5.0
1.0
1.0
Liama
FLU
5.0
0.2
1.0
0.2
Localized CT
2.0
1.0
2.0
Nonlocalized
2.0
0.1
2.0
0.2
across all tasks.
21

---

Mechanistic Unlearning
Lookop
trecing
Troing
0.
MUS
MiPs
Randem
Accuracy
Rendom
Farget Error
Maintain Accuracy
Edit Accuracy
MMLU
Metrica
Figure 28. Standard Prompting results for Sports-Athlete-Eiinnn across all localizations.
Standard
1.0
Fact Lookup
Causal Tacing
Causal Tacing
Petching
Petching MLPS
Nontocal zed
Al MLPs
Random
Accuracy
Randem MLPP
Forget Error
Maintain Accuracy
Edit Accuracy
MHLU
Metrics
Figure 29. Standard Prompting results for Full Sports-ppittiiiee across all localizations.
Standard
Fact Lookup
Tracing
Causal Tracing MLPs
Patching
Attrib Petching MLPs
Nonlocalized
Accersey
0.6
Random
Random MLPs
0.2
0.0
Forget
Accuracy
Accuracy
Figure Standard Prompting results for CounterFact-Editigg across all localizations.
A.7.2. DETAILS ON ADVERSARIAL PROMPT EVALUATIONS
We report the adversarial prompt evaluations from Section 3.1 across all localizations. Figures 32 to 35 all show that FLU localization is more robust in MCQ compared to every other localization (significantly stronger for CounterFact). Figures 34 and 35 show that FLU localization optimal in Paraphrase and Neighborhood in cases except for Paraphrase compared the Random localization in Sequential-CountePPati
22

---

Mechanistic Unlearning
Standard
Lookup
Causal Tracing
0.8
Tracing
Attrib Patching
Attrib Patching MLP:
Accurayy
0.6
Nonlocalized
All MLPs
Random
0.7
0.0
Forget Emror
Maintain Accuracy
Edit Accuracy
MMLU
Metrics
Figure 31. Standard Prompting results for Sepuentia-CCummcaFaaaFaiii across all localizations.
Multiple Choice
1.0
Lookup
Causal Tracing
Causal Tracing MLPs
0.8
Attrib Patching
Attrib Patching MLPs
Nonlocalized
All MLFF Ps
Accuracy
0.6
Randiom
Random MLPs
0.4
0.2
0.0
MCQ Forget Error ?
MCQ Edit Accuracy f
Metrics
Figure 32. Adversarial Prompting results for Sports-Athlete- Eeiittn,,, across all localizations.
A.7.3. DETAILS ON ADVERSARIAL RELEARNING
We retrain the model for 20 iterations with cross-entropy half of the forget set (along with standard retain and SFT loss), adding all losses with loss coefficient 1.
We present relearning results for all localizations averaged over models. As shown in Figure 36, the FLU localization remains optimal, although the baselines of Nonlocalized, All MLPs, Random, and Random MLPs are competitive.
We also present releariing results on the other tasks. As mentioned in Section 3.2, since Full-Sports-Ediite forget facts not independent, we don't "on'ten expect valid results from relearning. Thus, in Figure 37, that every localization regains 100% editing accuracy.
as shown in Figures 38 and 39, none of the localizations relearn more than 7% accuracy, suggesting adversarial relearning was not sufficiently strong enough evaluation for these tasks. Regardless, FLU localization is either the most or second-man -mott robust localization to relearning, although localizations don't differ by much.
On CointerFact-Editinn and
23

---

Mechanistic Unlearning
Multiple Choice
Fact Lookup
0.7
Causal Iracing
Causal Tracing MLPs
0.6
Attrib Patching
Attrib Patching MLPs
Nonlocalized
0.5
All MLPs
Accuracy
Random MLPs
0.3
0.2
MCQ Forget Error
MCQ Edit Accuracy
Metrics
Figure 33. Adversarial Prompting results for Full-Spoots-Ediitngg all localizations.
Multiple Choice, Paraphrase, Neighborhood
1.0
Lookup
Causal Tracing
Causal Tracing MLPs
0.8
Patching
Attrib Patching MLPs
0.6
Nonlocal zod
Random
0.4
Random MIPs
0.2
MCQ Forget Error
MCO Edit Accuracy
Paraphrase Edit Acc
Neighborhood
Edit Error
Metrics
Figure 34. Adversarial Prompting results Centerractt-ditting across localizations.
Multiple Choice, Paraphrase, Neighborhood
1.0
Faci Lookup
0.8
Causal tracing MLPs
Attrib Patching
Attrib Patching
Nonlocal zed
All MLPS
Random
0.4
Random MLPs
0.2
MCO Forget Error
MCO Edit Accuracy
Paraphrase Edit Acc
Nelghborhood Edit Error
Metrics
Figure 35, Adversarial Prompting results for Coatrll-Cooottttt-aaaaCaa across localizations.
24

---

Mechanistic Unlearning
Sports Forget Accurary Recovered by Relearning
1.0
0.923
steps] Final (20 steps)
0.24
Forget Accuracy
2.6
3.25
9.312
0.2
159
0,088
0,006 Random
Lookup
Causal Tracing
Attrib Patching
Attrib Patching MLPs
Nonlocalized
Random MLPs
Tracing MLPs
Localization
Figure 36. Releariin results for Apports-Athlcte-Ediii across all localizations.
Sports Forget Recovered by Relearning
1.0
0,858
0.965
Initial (8 steps) Final (20 steps)
Forget Accuracy
0.0
0.6
0.2
0.0
Fact Lookup
0.000
0,000
Causal Tracing
Causal Tracing MLPs
0.000 200
0.000
Attrib Patching
Attrib Patching MLPs Localization
0,000
Nonlocalized
0.000
D.00C
All MLPS
0,000
Random
0. 0.200
Random MLLP
Figure 37. Releaming results HalllSports--diiieee
Counterfact Forget Accuracy Recovered by Relearning
Initial steps]
0.14
steps)
Accuracy
0.12
0.06
0.04
0.801
002
0,001
0.005
6,001 Nonlocal zed
0.001
0.001
Fact Lookup
Causel Tracing
Causal Tracing MLPs
Attrib Patching
Attrib Patching MLLss Localization
All MLPs
Random
Random MLPs
Figure 38. Relearning results for CounterFact-
CounterFact Forget Accurary Recovered by Relearning
0.14
steps]
Final (20 steos)
Porget accurccy
0.08
0.06
0.B4A
0.02A BEZI
0,003
0.007
0.013
3.00
Fact Lookup
Causol Traing
Causal Tracing MLPs
Attrib Patching MLPs Localization
Nonlocali zod
All MLPS
Random
Random MLPs
Petching
Figure 39, Relearning results for Scquential-CounteFFact CounterFact Edditig.
25

---

Mechanistic Unlearning
A.7.4. DETAILS ON LATENT KNOWLEDGE
In Figure 40, the probes FLU consistently predict the forget sport less and the edit sport more than in any other localization, especially in early layers. Furthermore, the FLU probe classifications for the most part monotonically converge from their nonzero starting accuracy
(for forget accuracy) and (for edit accuracy).
Every other localization has much higher peak probe classification forget accuracy in the early layers, especially the OT localizations which have peak classification forget accuracy of almost 100%. This strongly suggests that these models still significantly represent the ground truth answer rather than the edit answer in early layers.
Probe Classification Forgct Accuracy
Probc Classification Edit Accuracy 1
1.0
1.0
0.8
Lockup
Accuracy
Accuracy
0.6
Cousal Tracing
Causal Tracing MLPs
lized
All MLPs
0.0
0.0
Layer
Layer
Figure Linear probes applied to forget set, classifying model activations after various layers. (Left) The graph shows that some localizations still represent almost completely correct forget set knowledge in early layers, especially OT, while FLU localizations represent this original knowledge least. (Right) line graph shows that FLU localizations represent the edited rather than original answer earlier and more consistently throughout layers than any other localization.
We present the probing classification accuracies for the three models separately here, as well as for all localizations we previously left out.
Probe Classification Forget Accuracy
Probe Classification Edit Accuracy
1.0
1.0-
Fail Lookup
Causal Tracing
Tracing
Accuracy
0.6
Accuracy
0.6
Attrib Patching
Attrib Patching MLPs
Nonioca cicellizedd
0.4
0.4
All MI Ps
Random MLPs
0.0
0.0
20
30
10
20
30
40
Layer
Layer
Figure 41. Linear probes applied to the forget across all models, classifying model activations after various layers.
In Gemma-7b and Llama-3-8b, FLU probing is the most monotonic and the best in the early layers, either steadily decreasing to for forget accuracy or increasing to for edit accuracy. with the least extreme peaks. In Gemma-2-99 the Nonlocalized, All MLPs, and Random MLPs baselines are competitive with FLU, The other OT localization, Attribution Patching, has 100% probing forget accuracy across many layers, suggesting represents the ground truth answer very clearly.
26

---

Mechanistic Unlearning
Gemma 7b
Probe Classification Forget Accuracy
Probe Classification Edit Accuracy
1.0
1.0
Lookup
0.5
0.8
Ceusel Tracing
Causal Tracing MLPS
Accuracy
Accuract
Attrib Patching
Attrib Patching MUPS
Nonlocallzed
0.4
0.4
MLPs
Random
0.2
0.2
Random MLPs
25
25
Layer
Layer
Figure 42. Linear probes applied to the forget set on Gemma-7B with 28 layers.
Gemma 9b
Probe Classification Forget Accuracy
Probe Classification Edit Accuracy
1.0
1.0
Lookup
0.8
0.8
Ceusel Trecing
Causal Tracing MLPs
Accuracy
Accuracy
Attrib Patching
Attrib Patching MLPs
Noniocellzed
0.4
0.4
All MLPs
Random
0.2
0.2
Random MLPs
10
30
Layer
Layer
Figure 43. Linear probes applied the forget set on Gemma-2-99 with 42 layers.
Liama 8b
Probe Classification Forget Accuracy
Probe Classification Edit Accuracy
1.0
1.0
Lookup
0.6
0.8
Ceusel Tracing
Causal Tracing MLPs
Accuracy
Accuracy
Attrib Patching
Attrib Patching MLPs
Noniocellzed
0.4
0.4
All MLPs
Random
0.2
0.2
Random MLPs
10
20
25
Layer
Layer
Figure 44. Linear probes applied to the forget set on Llama-3-8b 32 layers.
A.8. Soft Prompt Evaluations
Because many localizations seem to be weak to prompting schemes, we attempt a simple adaptive attack of soft prompts, where we optimize the continuous embeddings at the end of the prompt to recover the correct answer on half of our forget set. We then evaluate the model's performance on the other half, with this soft prompt in place (Lester et al., 2021). We average over evaluations from four soft prompts. Soft prompt evaluations can considered to more narrow form of
27

---

Mechanistic Unlearning
few-shot finetuning, that is closer searching for prompts that recover the model's knowledge.
We find limited soft prompt success: across most tasks and models, don't recover much held-out forget set accuracy. On the Sports-Athlete- -thieteeeeeee.,,,ngg CounterFact-Editing,, and Coauenttal-CounteFPaacc E-diting tasks, Figures 45, 47 and 48 show that all localizations don't significantly improve in Forget Accuracy over random chance, or are about equal between localizations, after soft prompts are applied. In Figure 46, specifically for Gemma-2 on Sports-Athlete-Fdiinn we see some reasonable results with softprompts that are able to recover over 60% Forget Accuracy on localizations, while FLU, Nonlocalized, and All MLPs remain under 40% Forget Accuracy.
Softprompt Evaluations
Lookup
0.5
Causal Tracing
Causal Tracing MLPs
Petching
Attrib Petching MLPs
Nonlocalized
MLPs
0.3
Random
Random MLPs
0.2
0.0
Forget Accuracy
Forget w/ Injected
Maintain Accuracy
Figure 45. Metrics with soft prompts applied for Sports-Athlett--Fdiingg averaged over all models.
Softprompt Evaluations for Gemma 9b
Lookup
Causal Tracing
Causal Tracing MUPs
Patching
Attrib Patching MLPs
+
Nonlocalized
All MLPs
Random MLPs
0.2
0.0
Forget Accuracy
Forget w/ Injected
Maintain Accuracy
Figure 46. Metrics with soft prompts applied for Sports Athlet-
Athlete-Editing Editing for Gemma-2-9b.
28

---

Mechanistic Unlearning
Softprompt Evaluations
0.7
Fact Lookup
Causal Tracing
Tracing
Attrib Patching
Attrib Patching MLPs
0.5
Nonlocalized
MLPs
Random
Random MLPS
0.3
0.1
Forget Accuracy
Forget w/ Injected Metrics
Maintain Accuracy
Figure 47, Metrics with prompts applied CounterFact Bcct-ng averaged over models.
Softprompt Evaluations
0.7
Fect Lookup
Causal Tracing
0.6
Tracing
Attrib Patching
0.5
Attrib Patching MLPs
Nonlocalized
0.4
MLPs
Random
0.3
Random MLPs
Forget Accuracy
Forget w/ Injected t Metrics
Mainkain Accuracy
Figure Metries with soft prompts applied for Sequential-CounterFe CounterFact Editing, averaged over all models.
29


Reviews:
[{"review_id": "Hza4CVbZHt", "reviewer": "Reviewer_EZb3", "summary": "This paper focuses on machine unlearning i.e. when the model needs to be prevented from outputing a certain information such as the profession of a person, or any mention of a given sport. Specifically they show that a lot of known methods might be preventing access to specific facts, but not overwriting the facts themselves. They show that their method on the other hand does this better.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a thorough read of our submission. Below we answer your question regarding the generalization of relearning in the Sports-Athlete-Editing and Full-Sports-Editing tasks, and explain our rationale behind the task choice for relearning attacks.\n\nOur relearning experiments in Section 3.2 are performed on the Sports-Athlete-Editing task. In this task, for a randomly selected set of athletes, we edit their associated sport by assigning them a new sport chosen uniformly at random from the existing set of sports. \n\nIn this Sport-Athlete-Editing task, we do not expect relearning to “generalize”: there is no inherent correlation between an athlete, their original sport, and the newly assigned sport. Therefore, relearning a subset of edited facts should not provide the model with a basis to correctly infer the previously unlearned facts for other athletes, even if they were originally associated with the same sport like basketball (unless this old information is still stored in the model after editing).\n\nOn the other hand, relearning attack is uninformative in some other setups, where the forget set accuracy could go up by the model simply learning a fixed mapping (e.g., always respond with “Golf” instead of “Basketball”). For example, this concern would apply for relearning the Full-Sports-Editing task: here we reassign one sport to another for all athletes playing that sport. For example, we might change all associations of \"basketball\" to \"golf.\" If we then retrain the model on half of these edited facts, it's highly probable that the model would simply learn the global reassignment (e.g., all \"golf\" becomes \"basketball\") rather than truly relearning the specific original associations. This makes it difficult to distinguish between a newly learned association and the relearning of previously stored knowledge.   \n\nIt is important to note that the relearning paper we base our evaluations off (Deeb 2024) also makes efforts to avoid retraining on a set which does not have independent information from the rest of the forget set: when the information used for retraining is independent from the rest of the forget set, we don’t expect any information recovery given perfect unlearning/editing. However, when the retrained information is not independent from the rest of the forgotten facts, it is unclear what baseline amount of recovery is expected. \n\nWe hope this clarifies our approach and the rationale behind our experimental design. We would also like to highlight that we ran additional experiments requested by other reviewers to further strengthen the paper: we showed that the localization part of our method can be automated, and that our method can be scaled to successfully edit a large number of facts.", "claims_and_evidence": "Claim 1: Output tracing does not unlearn fact, it unlearns access to the fact which can be regained by different forms of relearning (Both evidence from the litterature and empirical examples are provided)\nClaim 2: Fact lookup localisation is more likely to delete the fact itself. It is also more parameter efficient (This is verified through many ablations, and multi-dimensional evaluation. They also use probes to verify data presence)", "methods_and_evaluation_criteria": "They test there claims on 3 state of the art models of around 8B parameters, from two different companies.\nThey use different datasets,\nThey evaluate different state of the art methods, and perform extensive ablation", "theoretical_claims": "This paper is more on the empirical side. Experiments are nonetheless well defined and situated within the relevant litterature", "experimental_designs_or_analyses": "Experimental design is sound. As previously stated, relevance of method is checked through different datasets, different levels of unlearning, different datasets, different parameters, giving a good overview that supports the generality of the claims", "relation_to_broader_scientific_literature": "Relevant methods are discussed and explained, main drawbacks of said methods are carefully explained.", "essential_references_not_discussed": "To my knowledge, no key papers are missing", "other_strengths_and_weaknesses": "Strong and well explained experimental process, relying on many tools which are effectively used and described. Benchmark is quite extensive, on multiple models, datasets, and different tasks. Variations and relevance of tasks are discussed. I particularly appreciate the attention to the different forgetting mechanisms, and the different possible components to consider.", "other_comments_or_suggestions": "No additional comment", "questions_for_authors": "In 3.2 you mention that models should be able to generalize relearning half the basketball athletes to all basketball athletes. Could you further comment on how this informs the unlearning procedure? Are all athletes still known to perform in the same sport, and is basketball simply missing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zS8KRtPyA9", "reviewer": "Reviewer_GkhG", "summary": "The paper studies the mechanistic localizations for knowledge unlearning and editing. There are two main categories of mechanistic localizations in the literature: Output Tracing and Fact Lookup. Through a designed experiment, the paper finds that Fact Lookup localizations make knowledge unlearning / editing more robust, at the aspect of rephrasing prompt, multi-choice question extraction and adversarial relearning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we will make sure to improve the description of the FLU localization technique for CounterFact in Sec 2.2, and we report new positive experimental results editing significantly more facts. The methodology is briefly summarized here:  \n\nAn important pre-requisite is “path patching”, described in section 3.1 of Wang et. al. This allows us to measure the importance of the direct edge between two components in a model. We first measure the direct effect of each attention head on the final output, using the logit difference between the original and edit answer as our measure. Nanda et. al show that these attention heads “extract” relevant associations from the residual stream. We call these the fact extraction heads.\n\nWe then measure the effect of each MLP on the final logit difference as mediated only through these fact extraction heads. That is, we iterate through the MLPs and path patch the MLP -> {all fact extraction head} edges, and measure the logit difference. MLPs that cause a large change in this measure do so by introducing the factual association into the residual stream via a lookup mechanism, enabling the fact extraction heads to parse the association. We call these our fact lookup MLPs, which are the localization. \n\n\nBelow we report new experimental results in response to your concern about the number of facts we edit.\n\nWe acknowledge your point about the relatively small size of the dataset for editing. Our reasoning for maintaining a smaller set of facts was primarily to facilitate the creation of precise manual mechanistic localizations. However, we recognize the importance of understanding how our findings might generalize to a larger set of edited facts.\n\nTo address this, we slightly modify our technique to localize per fact rather than averaging over a set of facts. Then, similar to our original technique in A.2.2, we pick the relevant components by identifying the MLPs that have a >2 std impact on the logit difference in the fact. Using this, we can precisely localize each fact at scale. This allowed us to run an additional evaluation scaling up the number of facts on the CounterFact dataset to be edited all the way to 1000 facts. \n\nWe do this evaluation only on our Gemma-7b model due to time constraints, but we can expand this to Gemma-2-9B and Llama-3-8B as well by the camera-ready deadline. We compare our localization technique to using all the MLPs as a localization (a strong baseline) and a localization that uses our technique but also includes attention heads (as suggested by reviewer MshQ). We measure the robustness of these techniques to prompt changes (using an MCQ prompt format as in section 3.1) and to latent attacks by measuring probe accuracies as in section 3.3.  \n\nWe see that our fact lookup localization technique maintains its MCQ forget error as we scale the facts edited, outperforming the baselines (fig: https://imgur.com/a/60azzEo). Our localization also outperforms on its MCQ edit accuracy (fig: https://imgur.com/a/L7TnBwu), being the only localization to generalize to an alternative prompt setting. Finally, we see that latent knowledge attacks continue to fail when using our localization, as the probe accuracies are as accurate as random chance (fig: https://imgur.com/a/Fn6xWp1). All figures are anonymized and do not contain author information.\n\nWe hope our additional experiments have convinced you of the robustness of our method at scale.\n\nReferences:\n\nNanda, N., Rajamanoharan, S., Kram  ́ar, J., and Shah, R. Fact finding: Attempting to reverse engineer factual recall on the neuron level, Dec 2023. URL https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall.\n\nWang, Kevin, et al. \"Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.\" arXiv preprint arXiv:2211.00593 (2022).", "claims_and_evidence": "The main conclusion is that localizing edits/unlearning to components associated with the\nlookup-table mechanism is more robust, and this conclusion is well supported by the empirical results.", "methods_and_evaluation_criteria": "Overall the evaluation set-up make sense. However, the description for one of the method remains unclear to me. Lines 171-196 describe the Fact Lookup localization for CounterFact, but the steps of the method are not clear after reading this paragraph. It might be necessary to more formally introduce these evaluated methods.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The metrics and baselines are sufficient. The experiment is well aligned with the main question to check in this paper. However, I have some concerns for the dataset set-up. The dataset for the evaluation is quite small -- both two datasets only have 16 or 64 facts for editing, potentially making the empirical results sensitive to this specific test set. It is also unknown for the behaviors of editing larger batch of facts.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well written. I mostly enjoyed reading the paper and the presentation of the results.", "other_comments_or_suggestions": "Figure 2 and Figure 4 are not mentioned / described in the main text.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mMTfRu4248", "reviewer": "Reviewer_MshQ", "summary": "The authors investigate the effectiveness of adopting techniques from mechanistic interpretability to improve editing and unlearning in large language models. In particular, the work focuses on analyzing the benefits of unlearning and editing brought by localization techniques based on factual lookup (FLU) instead of the typical strategies using causal tracing methods. The authors focus their experiments on a sports fact dataset and a counterfactual dataset, showing that, in these contexts, their methodology leads to robust unlearning/editing while mitigating the risks of relearning attacks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable proposals. In response, we proposed a way to automate our method, which allowed us to test it at scale in terms of the number of facts to be edited. We also added experiments demonstrating that adding attention heads does not improve performance.\n\nAddressing points 1, 2:\n\nWe originally ignored the attention heads as candidates for localization as a result of work done by Nanda et. al who identified attention heads as playing a “fact extraction” role in the recall mechanism. We don’t claim that we found the precisely optimal localization for editing, but rather that this somewhat coarse localization was sufficient enough to yield significant robustness improvements: future work with more sophisticated interpretability techniques could further strengthen these results. However, we agree that empirical evidence to prove this would be valuable. \n\nTo address this, along with your second point for a more automated strategy for localization and your third point for a more challenging editing scenario, we run an experiment scaling up the number of CounterFact facts edited from 64 to 1000, and try a new localization. We stick to CounterFact as the dataset contains factual questions in varying formats. Our difficulty increase thus comes from scaling the number of facts edited: our technique now has to maintain robustness to orders of magnitude greater number of facts. We do this experiment for the Gemma-7B model, and plan to replicate this setup across Gemma-2-9B and Llama-3-8B by the camera-ready deadline. \n\nThis scaling of facts also means we have to use a more automated technique for localization. We localize per-fact, employing the same technique as described in A.2.2. We pick the components by utilizing a heuristic, picking all important MLPs that affect the final logit difference by > 2 stds. This differs from our original manual localization where we analyzed a group of facts, took the average contribution for each MLP, and assigned a group of MLPs to be the localization.\n\nWe test our technique, based on your suggestion, to also include relevant attention heads. We compare this against a strong baseline of picking all the MLPs. We measure the robustness of these techniques to prompt changes (using an MCQ prompt format as in section 3.1) and to latent attacks by measuring probe accuracies, as in Section 3.3.\n\nWe see that our fact lookup localization technique (localizing MLP layers only) maintains its MCQ forget error as we scale the facts edited, outperforming the baselines (fig: https://imgur.com/a/60azzEo). Our localization also outperforms other methods on its MCQ edit accuracy (fig: https://imgur.com/a/L7TnBwu), being the only localization to generalize to an alternative prompt setting. Finally, we see that latent knowledge attacks continue to fail when using our localization, as the probe accuracies are as accurate as random chance (fig: https://imgur.com/a/Fn6xWp1). All figures are anonymized and do not contain author information.\n\nAddressing point 3:\nIn this work, we hoped to lay the groundwork for applying interpretability for unlearning. We chose to work with CounterFact and Sports Facts datasets because the mechanisms of factual recall in these datasets have been well studied in the literature, and because CounterFact is a benchmark widely used in the editing literature, including in the seminal paper by Meng et. al. We are excited about future work that applies more sophisticated interpretability analyses on complex datasets like MMLU: a positive outcome of the results of this paper would be to inspire further research into interpretability to achieve more robust editing and unlearning.\n\nAddressing point 4:\nWe will add error bars and also results for each model across the various evaluations in the camera-ready version. For latent knowledge we do present results for each model in Appendix A.7.4 since each model has a different number of layers. For the latent knowledge analysis, Fact Lookup localization is most robust in Gemma-7b and Llama-3-8b, while some other methods are competitive with Fact Lookup in Gemma-2-9b.\n\nThank you again for your feedback. We hope that these strong additional empirical results, automation of our method, and our clarifications addressed all of your concerns.\n\nReferences:\n[1] Nanda, N., Rajamanoharan, S., Kram  ́ar, J., and Shah, R. Fact finding: Attempting to reverse engineer factual recall on the neuron level, Dec 2023. URL https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall.", "claims_and_evidence": "The idea of studying editing and unlearning from a mechanistic interpretability perspective is interesting and timely, as well as the approach of focusing on the role of the factual recall mechanism in editing and unlearning of LLMs knowledge.\nThe authors consider diverse models to show that their findings are robust to medium-level LLM scale and to different pre-training strategies (e.g., dataset, hyperparameters, etc.). Some points could make the analysis more convincing (see Section \"Other Strengths And Weaknesses\").", "methods_and_evaluation_criteria": "The evaluation of the unlearning and editing performances on the sports fact dataset and CounterFact dataset is sound and clearly explained. Some extensions of the experimental setup would strengthen the evidence for the claims (see Section \"Other Strengths And Weaknesses\").", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is well constructed and focused on supporting the main claims. Some extensions might strengthen the work (see Section \"Other Strengths And Weaknesses\")", "relation_to_broader_scientific_literature": "The authors discuss at a good level of detail the relevant literature for the current submission.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "1) It would be beneficial to justify better the choice of focusing only on interventions on MLP layers. Even if, as discussed by the author, they play a crucial role in the factual recall process and thus are a natural candidate for editing and unlearning, showing that interventions that involve the attention mechanism do not allow to achieve better performance would give stronger evidence for the author's choices.  \n\n2) It would be beneficial to at least outline a clear semi-automatic procedure for the selection of model components, even if the current work is more focused on giving proof of concept. In particular, is it possible to deduce a strategy from the \"manual analysis for both datasets outlined in Appendix A.2.1\"?\n\n3) It would be beneficial to extend the experiments to at least one additional more challenging scenario. For instance a subset of subjects in MMLU could provide an example.\n\n4) In the reviewer's opinion, even if averaging over models allows for simplifying the presentation, it would be important to show error bars to show that the trends are shared among models or to find other strategies to support this (e.g., separate results in the appendix).", "other_comments_or_suggestions": "I have no further comments or suggestions.", "questions_for_authors": "I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zsZYqFuivx", "reviewer": "Reviewer_wVvy", "summary": "The author investigates how mechanistic interpretability improves the precision and robustness of knowledge editing and unlearning in LLMs. They distinguish between methods that preserve outputs and those that target high-level mechanisms with predictable states. The findings show that localizing edits to lookup-table mechanisms for factual recall enhances robustness across formats, resists relearning attacks, and reduces unintended side effects, outperforming baselines on the sports facts and CounterFact datasets. Additionally, certain localized edits disrupt latent knowledge more effectively, making unlearning more resilient to adversarial attacks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our work and providing feedback. We appreciate the opportunity to clarify our contributions and address your concerns. We believe that the assessment and low score was based on certain misunderstandings of our work: we focus on model editing of factual relations; our editing method is in fact novel and clears up misconceptions about mechanistic interpretability for model editing and unlearning, making it an important contribution to the literature; we provide an extensive set of evaluations of our method, and even stronger attacks than suggested by the reviewer.\n\nWe hope our detailed responses below will convince the reviewer to re-evaluate our work and raise the score. \n\n*On the Lack of Unlearning benchmarks*\n\nOur primary contribution lies in investigating how mechanistic interpretability can enhance the precision and robustness of knowledge editing of factual associations: i.e., replacement of certain facts with new facts. This goal underscores our choices of baselines and datasets. We use the SportsFacts dataset following work from Nanda et. al, who use it to mechanistically understand factual recall, to create a localization technique for robustly modifying factual associations. We then translate these findings to the CounterFact dataset, a benchmark widely used in the editing literature. \n\nMost of our work focuses on editing rather than unlearning, and the baselines of RMU, GradDiff, and NPO couldn’t be used for the editing objectives without a significant reformulation of the tasks. Our inclusion of an unlearning result in A.1 of the paper was primarily to show the potential for our method to generalize to unlearning, but not a claim that our method led to state of the art unlearning performance on every benchmark. However, the current literature on unlearning is relatively unanimous in that no current method is robust against the partial relearning attack (Deeb 2024), including RMU (Li 2024) and TAR (Tamirisa 2024) which is defeated by parameter-efficient fine-tuning: we believe our method can yield progress.\n\nIn this work, we hoped to lay the groundwork for applying interpretability for unlearning. We are excited about future work that applies more sophisticated interpretability on complex datasets like WMDP: a positive outcome of the results of this paper would be to inspire further research into interpretability to achieve more robust editing and unlearning.\n\nWe will revise the paper to more accurately reflect that the majority of our current empirical results focus on fact editing. An easy fix!\n\n*On Novelty*\n\nOne main novel contribution is the identification and utilization of Fact Lookup (FLU) mechanisms for robust knowledge editing. We contrast this approach with Output Tracing (OT) methods. We show that editing localized to these FLU mechanisms leads to more robust edits as measured using a number of evaluations (listed below).\n\nWe demonstrate that the relationship between localization and fact editing/unlearning is more nuanced than suggested in Hase et al. (2023),  and that not all localization techniques are equally effective. This is an important contribution to the literature, showing a promising use of mechanistic interpretability.\n\n*On evaluations*\n\nRegarding the evaluation against different attacks, we would like to highlight that we do evaluate the robustness of our method against:\n* Rephrasing prompts (Paraphrase Evaluation);\n* Multiple-choice question extraction (MCQ Evaluation);\n* Adversarial relearning attacks;\n* Soft prompt attacks, which are a more challenging form of adversarial prompting since they operate in continuous space.\n* Latent knowledge analysis, which trains a probe to extract the correct answer from the latent representation of the model, instead of logit lens which is a biased representation of the model’s best guess at a layer\n\nWe are confident that these evaluations provide a comprehensive assessment of the robustness of our proposed editing method. \n\nNote that the current version of the paper is even stronger as we included additional experiments suggested by other reviewers: proposed and tested an automated version of our localization method which outperforms strong baselines; included additional tests on which components are important for localization; and demonstrated that our method can successfully edit a large number of facts (up to 1000).\n\nReferences:\nHase, P. et al. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models, 2023.\n\nMeng, K. et al. Locating and editing factual associations in gpt, 2023.\n\nNanda, N. Attribution patching: Activation patching at industrial scale, 2023.\n\nLi, N. et al. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning, 2024.​\n\nDeeb, A. et al. Do Unlearning Methods Remove Information from Language Model Weights?, 2024.​\n\nTamirisa, R. et al. Tamper-Resistant Safeguards for Open-Weight LLMs, 2024.​", "claims_and_evidence": "Not clear. Please see Question For Authors.", "methods_and_evaluation_criteria": "I think this paper's experimental setup has issues. Please refer to Questions for Authors for details.", "theoretical_claims": "There is no theoretical claims.", "experimental_designs_or_analyses": "I think this paper's experimental setup has issues. Please refer to Questions for Authors for details.", "relation_to_broader_scientific_literature": "Please refer to Questions for Authors for details.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Please refer to Questions for Authors for details.", "other_comments_or_suggestions": "Please refer to Questions for Authors for details.", "questions_for_authors": "1. The authors repeatedly mention machine unlearning but do not adopt any well-known machine unlearning methods in their paper, such as RMU [1], GradDiff [2], or NPO [3]. Based on the presented results, it is unclear whether their proposed approach truly outperforms existing machine unlearning methods.\n\n> [1] Li, Nathaniel, et al. \"The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning.\" arXiv preprint arXiv:2403.03218 (2024).\n\n> [2] Yao, Yuanshun, Xiaojun Xu, and Yang Liu. \"Large Language Model Unlearning.\" Advances in Neural Information Processing Systems 37 (2025): 105425-105475.\n\n> [3] Zhang, Ruiqi, et al. \"Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning.\" arXiv preprint arXiv:2404.05868 (2024).\n\n2. Additionally, the authors do not conduct experiments on any established unlearning benchmarks, such as WMDP [1], TOFU [2], or MUSE [3].\n\n> [1] Li, Nathaniel, et al. \"The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning.\" arXiv preprint arXiv:2403.03218 (2024).\n\n> [2] Maini, Pratyush, et al. \"TOFU: A Task of Fictitious Unlearning for LLMs.\" arXiv preprint arXiv:2401.06121 (2024).\n\n> [3] Shi, Weijia, et al. \"MUSE: Machine Unlearning Six-Way Evaluation for Language Models.\" arXiv preprint arXiv:2407.06460 (2024).\n\n3. The proposed method merely applies existing model editing techniques to the field of machine unlearning without introducing any novel contributions.\n\n4. The authors claim that their approach provides robust unlearning, yet they only evaluate it against a single technique—relearning attacks. What about other methods? For instance, adversarial prompts, logit lens, etc. [1].\n\n> [1] Łucki, Jakub, et al. \"An Adversarial Perspective on Machine Unlearning for AI Safety.\" arXiv preprint arXiv:2409.18025 (2024).\n\n\n## Update after rebuttal\nThis paper does not compare its method against well-known machine unlearning benchmarks such as TOFU, MUSE, or WMDP, nor does it evaluate against established unlearning methods like NPO or RMU. As a result, it is difficult to assess the effectiveness of the proposed approach in the context of machine unlearning. The authors' rebuttal does not fully address my concerns. But I appreciate their efforts and am willing to raise my score to 2.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary

This paper investigates how mechanistic interpretability can improve knowledge editing and unlearning in large language models. The authors distinguish between methods that localize components based primarily on preserving outputs and those finding high-level mechanisms with predictable intermediate states. They demonstrate that localizing edits/unlearning to components associated with the lookup-table mechanism for factual recall leads to more robust edits/unlearning across different input/output formats, better resistance to relearning attempts, and reduced unintended side effects. Experiments across multiple models on both sports facts and CounterFact datasets show that their localized edits disrupt latent knowledge more effectively than baselines, making unlearning more robust against various attacks.

## Reasons to Accept

1. The paper demonstrates that fact lookup (FLU) localization leads to more robust edits/unlearning across different input/output formats and resists attempts to relearn unwanted information, while reducing unintended side effects (EZb3, MshQ).
2. The experimental evaluation is comprehensive, with testing on multiple models (around 8B parameters from two different companies), multiple datasets, and extensive ablation studies (EZb3).
3. The research question combining mechanistic interpretability with model editing is interesting and timely, providing a novel perspective on how to improve edit robustness (MshQ, GkhG).
4. The main conclusion that localizing edits/unlearning to components associated with the lookup-table mechanism is more robust is well supported by the empirical results (GkhG).
5. The paper provides a strong and well-explained experimental process, with benchmarks that cover multiple models, datasets, and tasks (EZb3).

## Suggested Revisions

1. The scale of evaluation datasets is quite small (16 or 64 facts for editing), which makes the empirical results potentially sensitive to the specific test set and doesn't demonstrate behavior when editing larger batches of facts (GkhG). (The authors responded with new results scaling up to 1000 facts, showing their method maintains robustness at scale.)
2. The paper focuses only on interventions in MLP layers without justifying why interventions involving the attention mechanism are excluded (MshQ). (The authors ran new experiments demonstrating that adding attention heads does not improve performance, explaining that their focus on MLPs stems from previous work by Nanda et al.)
3. The paper lacks a clear semi-automatic procedure for the selection of model components, relying instead on manual analysis (MshQ). (The authors proposed an automated technique for localization in their response, demonstrating it works at scale.)

There are also some minor points:
1. The paper presents average results across models without error bars or separate results to show the trends are shared among different models (MshQ). 
2. The description of the Fact Lookup localization method for CounterFact is unclear, and the steps of the method are not well-defined after reading this section (GkhG). Figures 2 and 4 are not mentioned or described in the main text (GkhG).
Decision: Accept (spotlight poster)