Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization

## Abstract

Abstract Methods for knowledge editing and unlearning in large language models seek to edit or remove undesirable knowledge or capabilities without compromising general language modeling performance. This work investigates how mechanistic interpretability—which, in part, aims to identify model components (circuits) associated to specific interpretable mechanisms that make up a model capability—can improve the precision and effectiveness of editing and unlearning.
We find a stark difference in unlearning and edit robustness when training components localized by different methods. We highlight an important distinction between methods that localize components based primarily on preserving outputs, and those finding high level mechanisms with predictable intermediate states.
In particular, localizing edits/unlearning to components associated with the lookup-table mechanism for factual recall 1) leads to more robust edits/unlearning across different input/output formats, and 2) resists attempts to relearn the unwanted information, while also reducing unintended side effects compared to baselines, on both a sports facts dataset and the CounterFact dataset across multiple models.
We also find that certain localized edits disrupt the latent knowledge in the model more than any other baselines, making unlearning more robust to various attacks.

## 1 Introduction

Large language models (LLMs) often learn to encode undesirable knowledge. The possibility of selectively editing or unlearning this type of knowledge is viewed as paramount for ensuring accuracy, fairness, and control of AI. Yet, editing and unlearning of knowledge from these models remains challenging.

Common editing and unlearning methods often come at the cost of affecting other general or tangential knowledge or capabilities within the model. Moreover, the edits achieved through these methods may not be robust – e.g., slight variations in the prompt formulation can often still elicit the original fact or capability, or the original answers are still present/extractable given white-box access.

Some recent work has explored editing or unlearning techniques that rely on mechanistic interpretability methods attempting to trace which components of a network store specific facts (Meng et al., ). These methods, such as causal tracing or attribution patching, focus on measuring how output or task accuracy is affected when clean/corrupted input is patched into specific components.

However, the effectiveness of these “output-tracing” (OT) techniques for editing has been questioned by Hase et al. (). Our research confirms these doubts, finding that localized editing and unlearning of facts based on several existing OT methods often perform equal to or worse than simply updating the entire model. This is particularly evident when evaluating the robustness of edits against prompt variations and relearning, and when probing for remaining latent knowledge.

Another style of interpretability techniques first breaks down computations into high-level mechanisms with predictable intermediate states. Based on such work by Nanda et al. (); Geva et al. (), we link certain MLP layers to a fact lookup (FLU) mechanism for facts used in our analysis, that enrich the latent stream with subject attributes but don’t directly write to the output.
For unlearning and edits of these facts, we only modify components that implement the FLU mechanism.
More broadly, we refer to editing and unlearning that acts on components of the model identified by mechanistic intermediate component analysis as *mechanistic unlearning*.
We demonstrate that FLU *mechanistic unlearning* leads to better trade-offs between edits/unlearning and maintaining performance on general language modelling capabilities, compared to edits done using OT or without any localization.
Further, it exhibits improved robustness to re-learning and alternative prompting, and we demonstrate that the latent knowledge is also perturbed.

##### Summary of Contributions

- •
We perform a rigorous evaluation of several standard unlearning approaches on factual recall tasks and show they fail to generalize to prompting/output distribution shifts and adversarial relearning.
- •
We identify mechanisms for factual lookup and attribute extraction on Gemma-7B and Gemma-2-9B.
We demonstrate that gradient-based unlearning and editing localized on the factual lookup mechanism is more robust and generalizes better than OT localizations and baselines across multiple datasets and models.
- •
We analyze intermediate representations using probing, and provide further evidence that editing with FLU localization reduces the available latent information more than other localizations and baselines. We also analyze the weights that are modified for each localization, and find that OT techniques and baselines modify the attribute extraction mechanisms more than the fact lookup mechanism.
- •
We show that editing and unlearning localized on these mechanisms is more parameter efficient, by controlling for the sizes of edits made to the model with weight masking.

Figure: Figure 1: High level depiction of mechanistic unlearning. We localize components responsible for fact extraction/enrichment and modify their weights to change the associations, in order to target internal latent representations rather than targeting the output. Graph inspired by Nanda et al. ().
Refer to caption: x1.png

### 1.1 Related Work

Mechanistic Interpretability is a subfield of AI interpretability, aiming to understand the internal processes of AI models by attributing them to subnetworks (called circuits) within the model (Olah et al., ). We focus on the factual recall interpretability literature (Nanda et al., ; Geva et al., ; Chughtai et al., ; Yu et al., ), which studies methods that aim to discover mechanisms for the retrieval and formatted extraction of factual information.

Output tracing methods aim to automatically find causally important subnetworks of components for a task. Causal Tracing (Meng et al., ) and Automated circuit discovery (ACDC) (Conmy et al., ) utilize repeated activation patching to attempt to find the subnetworks that are most critical for the model’s output on that task. Efficient methods such as attribution patching (Nanda, ) and edge attribution patching (Syed et al., ) are linear approximations of activation patching for discovering important components quickly.

##### Fact Editing and Machine Unlearning

seek to modify pre-trained models to eliminate or alter learned knowledge such as capabilities or facts.
Some prior approaches focus on identifying and removing specific individual training data points, aiming to obtain a model that is “similar” to one that had never trained on these data points (Cao & Yang, ; Xu et al., ).
One formalization of unlearning to match a retrained-from-scratch model is due to Ginart et al. (), and is closely inspired by differential privacy (Dwork et al., ).

A growing body of work aims to unlearn a subset of the training data in LLMs. Eldan & Russinovich () propose a method for unlearning entire books like the Harry Potter series. Chen & Yang () consider modifying transformer architecture by inserting “unlearning” layers.

Fact editing focuses on overwriting factual information while preserving overall language generation ability. Meng et al. () attempts to identify MLP modules that are most responsible for factual predictions via Causal Tracing and then applies a rank-one transformation upon these modules to replace factual associations.

In the context of LLMs and safety, techniques such as Helpful-Harmless RLHF (Bai et al., ) and Representation Misdirection for Unlearning (Li et al., ) aim to suppress dangerous knowledge or harmful tendencies in LLMs. A related line of work on safety proposes methods making it difficult to modify open models for use on harmful domains (Deng et al., ; Henderson et al., ).

##### Failures of Unlearning and Editing

have been shown for both localized and nonlocalized methods. Patil et al. () extract correct answers to edited facts from the intermediate residual stream and through prompt rephrasing.
Yong et al. () show that low-resource languages jailbreak models output unsafe content, and Lo et al. (); Lermen et al. () demonstrate that relearning with a small amount of compute/data causes models to regain undesirable knowledge/tendencies.
Even without explicit finetuning, Xhonneux et al. () show that in-context learning alone suffices to reintroduce undesirable knowledge despite the model being designed to refuse to output such knowledge.
Lee et al. () shows that even after alignment techniques are applied to make models nontoxic, toxicity representations are still present, just not triggered - they argue that this is a reason that models lack robustness and can still be jailbroken to trigger this unwanted behavior.

## 2 Methods

Our experiments are designed to test the effectiveness of localization for unlearning/editing of facts.
In this section we describe the tasks used and the localization and unlearning methods evaluated.

### 2.1 Unlearning/Editing Tasks

We focus on unlearning and editing subsets of two datasets: (1) Sports Facts dataset from Nanda et al. (), which contains subject-sport relations across three sports categories for 1567 athletes, and (2) the CounterFact dataset from Meng et al. ().

##### Sports Facts

In the Sports Facts dataset, we attempt to unlearn and edit two groups of factual associations. First, we unlearn all athlete-sport associations for a given sport. In this case, we establish a *forget set* consisting of all the basketball athletes.
Second, we edit factual associations for a set of 16 athletes belonging to all three sports categories. We test editing these sets of associations by replacing their sports with golf, and the retain set is the rest of the non-forget athletes. We use the Gemma-7B LLM (Team et al., ) rather than the Pythia-2.8B (Mallen & Belrose, ) model tested in Nanda et al. (), for its stronger general capabilities which we can measure for side effects, and for its ability to provide sports knowledge in different input/output formats.

##### CounterFact

In the CounterFact dataset, following Geva et al. (), we first filter the dataset for facts which the model assigns higher than 50% probability to the right answer, leaving 2170 facts. Then, we edit a set of 16 facts, replacing the correct answers with an alternative false target, with the retain set being the rest of the non-forget facts. We use the Gemma-2-9B LLM, as an alternative to the Gemma-7B LLM, to test a range of model architectures and sizes. We don’t test smaller models as they cannot generally output knowledge in different formats, as measured by our robustness evaluations.

### 2.2 Localization Methods and Baselines

Given a model $M:X\mapsto L$ mapping sequence of tokens $X$ to logits $L\in\mathbb{R}^{|V|}$ over vocabulary $V$, we consider $M$ to be a directed acyclic graph $(C,E)$ with $C$ being a set of model components and $E$ being edges between components. Adopting notation from Elhage et al. (), we consider the query, key, value, and output weights $W_{Q}^{h},W_{K}^{h},W_{V}^{h},W_{O}^{h}$ of each head along with the input and output projection weights $W_{I}^{m},W_{O}^{m}$ of each MLP as components.

We are interested in finding $S:C\xrightarrow{}\mathbb{R}$, a mapping of components to their importance in a given task. A localization is a set of components $C_{\tau}:=\{c:c\in C,|S(c)|>\tau\}$, where $\tau$ is a threshold. In practice, we fix $\tau$ such that $C_{\tau}$ contains the same number of parameters in OT, FLU, and random localizations. We use these efficient localization methods for finding these mappings:

##### Output Tracing (OT) localization: Causal Tracing and Attribution Patching

First, we test Causal Tracing, a method for finding components with high direct causal importance for factual associations (Meng et al., ).
Previous work has highlighted the shortcomings of Causal Tracing as a localization method (Hase et al., ), so we also use Attribution Patching (Nanda, ), which uses a linear approximation to activation patching to automatically localize over components with high direct and indirect importance.

We hypothesize that these output-based techniques will prioritize the shared extraction components and other mechanisms for reformatting predictions over the more diffuse FLU components, and thus appear more precise yet leave the underlying latent information present in the model. This might decrease robustness under alternative extraction methods, thus motivating non-OT-based localization, described next.

##### Fact Lookup (FLU) localization: Manual Mechanistic Interpretability

Next, we use manually derived localizations for MLP layers.
For Sports Facts, our localization is inspired by Nanda et al. (), who discovered components in Pythia 2.8B responsible for *token concatenation, fact lookup*, and *attribute extraction*. They, along with Geva et al. (), find that the fact-lookup stage enriches the latent stream with information about the subject (athlete) at the subject’s token position, and the attribute extraction stage extracts the latent sport information and formats it in the final token position. We replicate a key result of their work in Gemma-7B, localizing the *fact lookup* stage to be the MLP components between layers 2 and 7.

For CounterFact, we replicate findings from Geva et al. () and do further causal analyses to identify particular MLPs that perform *fact lookup* of our forget set in Gemma-2-9B. Our analysis highlights layers 3-5, 7-10, and 14-17 as the critical MLPs. The manual analysis for both datasets is outlined in Section A.2.1. We also locate causally important attribute extraction mechanisms, both attention heads and MLPs, in later layers of the model.

We refer to these FLU localizations as *manual interpretability*. Importantly, our analysis differs from OT techniques because we consider the causal effects of ablations upon intermediate representations used by the factual recall mechanism, not just the effects on the output. We hypothesize that the optimal location for robust unlearning/editing is in the fact lookup stage rather than in the attribute extraction stage, because adversaries can develop alternative methods for extracting knowledge from the latent stream through alternative prompts or white-box methods so we want to prevent the knowledge from ever being added to the latent stream. Thus, we exclusively modify the fact lookup MLPs.

##### Baselines: Random, All-MLPs, and Nonlocalized

We additionally consider three baselines: one corresponding to $C_{\tau}=C$ (i.e., no localization, optimizing all the components of the model), another that randomly chooses components, and another that trains all MLP components. We test the last All-MLPs localization to determine if our mechanistically localized MLPs are uniquely important - we want to know if the same unlearning performance can be achieved with just the heuristic that training only MLPs improves robustness, or if mechanistic understanding of the role of the component is crucial.

In Section A.2.3, we analyze the proportions of each mechanism (the extraction heads, extraction MLPs, and fact lookup MLPs, by parameter count) that are present in each localization.

### 2.3 Parameter Update Methods

Once we have a localization $C_{\tau}$, we run one of the unlearning or editing methods, restricting weight updates to only components in $C_{\tau}$. We modify weights using gradient ascent/descent on a combination of loss functions. As an alternative method, we also test masking a fraction of the weights as an alternative method showing precision of edits, with results reported in Section 3.2.

##### Localized Fine-Tuning

Following work by Lee et al. () and Panigrahi et al. (), we fine-tune the parameters within the localized components. We use a loss function

$$ $L=\lambda_{1}L_{\text{forget}}+\lambda_{2}L_{\text{retain}}+\lambda_{3}L_{ \text{SFT}},$ $$

where $L_{\text{forget}}$ is an unlearning loss on the $D_{forget}$ subset of facts we want to forget, $L_{\text{retain}}$ is a cross-entropy loss on the remaining facts, and $L_{\text{SFT}}$ is a cross-entropy loss on the Pile dataset (Gao et al., ). The unlearning loss we use is the $\log(1-p)$ measure from Mazeika et al. (), for its stability and fewer side effects.

For editing, we use a loss function $L=\lambda_{1}L_{\text{injection}}+\lambda_{2}L_{\text{retain}}+\lambda_{3}L_{
\text{SFT}}$, where $L_{\text{injection}}$ is a cross-entropy loss on the forget facts maximizing the probability of the alternative false target. Our $\lambda$s are in Section A.4.

##### Weight Masking

For weight masking-based unlearning, we train a binary differentiable mask over individual weights of the model within the localized components, inspired by weight pruning/masking work (Bayazit et al., ; Panigrahi et al., ). In this case, no weight updates are being performed. Rather, the mask turns a subset of the weights to zero. To control the sparsity penalty, we additionally include $\lambda_{4}*L_{\text{reg}}$, an L1 regularization term.

## 3 Unlearning and Editing Results

In this section, we show the results of unlearning and editing across all of the mentioned localization techniques for localized fine-tuning and weight masking.
We try three unlearning/editing goals: unlearning all athletes playing basketball (referred to as unlearning sports), editing the model’s associations for a constant set of 16 athletes across all sports (akin to the error injection setup from (Hase et al., ), and editing the associations for 16 facts from the CounterFact dataset Meng et al. (). All the techniques are assessed based on standard and adversarial evaluations.

### 3.1 Localized Finetuning

#### 3.1.1 Standard Evaluation

For the sports dataset, following Nanda et al. (), we first evaluate the accuracy of our models to complete the prompt, “Fact: [athlete] plays the sport of”, with a one-shot example of Tiger Woods playing golf given first. Note that this is the same prompt used to train the unlearning in the first place. We refer to this accuracy as Normal Accuracy. Inspired by Patil et al. () and Lynch et al. (), we also use an alternative input and output prompting setup to measure if our unlearning has “overfitted” to the prompt input and the output format. We instead use a multiple-choice format with the choices of football, baseball, basketball, and
golf. We refer to the accuracy of the model answering with the ground truth on this prompt format as the MCQ Accuracy (for both unlearning and editing, stronger methods should decrease the MCQ accuracy).

For CounterFact, we evaluate the accuracy of our models to complete the original factual associations. We report the original robustness and side effect evaluations from the Meng et al. () dataset, the Paraphrase and Neighborhood facts accuracies.
Since the Paraphrased prompts have the same answer string as the original question, we hypothesize that they may be easier for model editing to generalize to - we want to test that the knowledge is edited under any expression of the knowledge. We perform a knowledge robustness check by phrasing the associations in multiple choice format (MCQ Accuracy), with the true answer, the injected false answer, and two other question-specific LLM-generated incorrect answers. Once again, accuracy is measured with the ground truth answer, so we always want the MCQ accuracy to decrease throughout editing.

Finally, we also evaluate our models’ accuracy on MMLU (Hendrycks et al., ) as a proxy for the general side effects of unlearning unrelated to sports and CounterFact. Our results with localized fine-tuning are shown in Table 1 for sports, Table 2 for athletes, and Table 3 for CounterFact.

**Table 1: Localized fine-tuning accuracy on standard evaluations: unlearning all basketball athletes and retaining all other facts.**
| Localization | Forget $\downarrow$ | Retain $\uparrow$ | MCQ $\downarrow$ | MMLU $\uparrow$ |
| --- | --- | --- | --- | --- |
| Attrib. Patching | 0.000 | 1.000 | 0.767 | 0.602 |
| Causal Tracing | 0.201 | 0.998 | 0.849 | 0.611 |
| Manual | 0.002 | 0.995 | 0.110 | 0.613 |
| Random | 0.952 | 0.980 | 0.822 | 0.612 |
| All-MLPs | 0.000 | 0.994 | 0.279 | 0.606 |
| Nonlocalized | 0.000 | 0.985 | 0.196 | 0.595 |

**Table 2: Localized fine-tuning accuracy on standard evaluations: editing a constant 16 athlete subset, considering accuracy relative to the original ground truth answers**
| Localization | Forget $\downarrow$ | Retain $\uparrow$ | MCQ $\downarrow$ | MMLU $\uparrow$ |
| --- | --- | --- | --- | --- |
| Attrib. Patching | 0.447 | 0.998 | 0.895 | 0.612 |
| Causal Tracing | 0.586 | 0.994 | 0.945 | 0.613 |
| Manual | 0.001 | 0.970 | 0.108 | 0.611 |
| Random | 0.883 | 0.988 | 0.875 | 0.614 |
| All-MLPs | 0.001 | 0.965 | 0.166 | 0.574 |
| Nonlocalized | 0.354 | 0.890 | 0.155 | 0.573 |

**Table 3: Localized fine-tuning accuracy from editing 16 facts from the CounterFact dataset, considering accuracy relative to the original answers**
| Localization Type | Forget $\downarrow$ | Maintain $\uparrow$ | MCQ $\downarrow$ | MMLU $\uparrow$ | Paraphrase $\downarrow$ | Neighborhood $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Attrib. Patching | 0.000 | 0.974 | 0.944 | 0.691 | 0.286 | 0.681 |
| Causal Tracing | 0.000 | 0.973 | 0.481 | 0.692 | 0.032 | 0.736 |
| Manual | 0.000 | 0.965 | 0.295 | 0.691 | 0.035 | 0.733 |
| Random | 0.000 | 0.979 | 0.651 | 0.690 | 0.159 | 0.757 |
| All-MLPs | 0.000 | 0.927 | 0.390 | 0.688 | 0.083 | 0.725 |
| Nonlocalized | 0.000 | 0.920 | 0.411 | 0.682 | 0.104 | 0.746 |

As seen from the tables, across all tasks, unlearning with manual localization achieves the highest robust multiple-choice forget accuracy and near the highest MMLU, and very competitive normal forget and retain accuracy. Only manual localization, all-MLPs, and nonlocalized approaches had generalized their unlearning to the multiple choice format, but manual localization has higher MMLU performance and higher retain accuracy than All-MLPs and nonlocalized.
On the CounterFact task, only manual interpretability achieves baseline MCQ accuracy, and manual localization also achieves the second-lowest score on the Paraphrase robustness evaluation as well. As before, because the Paraphrased facts may share the same attribute extraction mechanisms to the output as the original facts, we emphasize that the MCQ evaluation is the higher signal robustness evaluation.

This indicates that OT localization methods do not robustly modify the model, and the supposedly-unlearned/edited information can be extracted through prompt and task variations.

#### 3.1.2 Adversarial Relearning

We measure the ability of our models to withstand adversarial relearning, both to address the scenario in which adversaries may have fine-tuning access and as an upper-bound measure for the quality of unlearning – a model that needs relatively fewer steps to relearn may not have properly unlearned(^1^11Ideally, one would compare here to the number of iterations needed to learn these facts for a “gold-standard” model that was never trained on these facts during pre-training. Obtaining the gold-standard model is prohibitively expensive. Therefore, we rely on a heuristic that unlearned models that need longer re-learning behave more like the gold-standard one.). We retrain with a rank-64 LoRA across all linear modules using a subset of the forget dataset for sports, with details available in Section A.5.2. We measure the ability of the relearned model to answer with the ground truth statements, similar to the methodology of Deeb & Roger ().

Figure 2 (left) and Figure 2 (right) compare relearning robustness of different localization techniques for sports and athletes respectively.

As shown in Figure 2 (left), for sports-unlearned models, manual interpretability is the localization method that is most robust to the low-resource relearning. Unlearning based on every other localization as well as the no-localization technique regains accuracy on the rest of the forget set within a few iterations. For all of the athlete-edited models, relearning on just two of the edited athletes recovers significant accuracy on all athletes in all except for the manual interpretability model.

Figure: Figure 2: (Left) Retraining basketball-unlearned models with two athletes in the forget set, for ten iterations. (Right) Retraining models that have had 16 athlete-sport associations replaced with Golf as the athletes’ associated sport, with two athletes in the forget set, for ten iterations. The y-axis represents the normal accuracy on the forget set. Low-resource relearning of athletes demonstrates that using manual interpretability makes edits significantly more robust to relearning.
Refer to caption: extracted/5932245/images/relearning/finetuning-sports-forget.png

For CounterFact relearning, we change the setup slightly: we retrain on all forgotten statements, since the set of edited statements aren’t as directly related as for sports facts. We measure accuracy using the same train set of edited/forgotten statements in Figure 3 (left). However, because we are directly optimizing the accuracy on this train set, we also try the multiple choice evaluation on this train set to see if the relearning generalizes (if the model has reinternalized this knowledge), in Figure 3 (right). Figure 3 (right) demonstrates that the correct factual associations were not deeply relearned in any of the manual interpretability, nonlocalized, or all MLP models, with the manual interpretability model having the lowest MCQ accuracy.

Figure: Figure 3: Retraining counterfact-edited models with all forgotten facts, for twenty iterations. (Left) The y-axis represents the normal accuracy on the forget set. (Right) The y-axis represents the Multiple Choice Formatted question accuracy on the forget set.
Refer to caption: extracted/5932245/images/relearning/finetuning-counterfact-injection-forget.png

#### 3.1.3 Latent Knowledge Analysis

In this section, we provide more evidence of our hypothesis that FLU unlearning targets the source of intermediate latent knowledge rather than the extraction mechanism. We use the Sports Facts dataset for this study, as the answer is always one of three representations (the various sports).

Similar to Patil et al. (), we train logistic regression models (probes) (Alain & Bengio, ) on the activations of every model layer to predict the correct ground truth sport from the prompt, with the idea that a model that has truly modified a fact would not have much predictive value in its activations. For more details on probe training, see Section A.5.3.

We test whether the models retain information about the original athlete set in the intermediate layer representations after editing. Figure 4 shows the accuracy of the trained per-layer probes on the set of edited athletes. If model updates were concentrated in the extraction mechanism, we should still see high probe accuracies, as the probes are accessing the latent representations.

Figure: Figure 4: Probe accuracy (combined over all three sports) on the athlete golf-injected models across layers. The ‘Localized AP’, ‘Localized CT’, and ‘Random’ lines are overlapping.
Refer to caption: extracted/5932245/images/dlk/finetuning-athletes-injection-forget.png

Figure 4 provides evidence that the manual interpretability, All-MLPs, and nonlocalized athlete-edited models contain less recoverable representations of the edited associations in the intermediate layers, with the probes having the least accuracy on the manual interpretability model. This suggests our editing guided by manual interpretability has properly interfered with the lookup MLPs enriching the latent stream with information about the correct answer.

### 3.2 Weight Masking

In this section we employ weight masking to quantify the size of weight updates needed to unlearn/edit facts, for more direct comparisons. We empirically evaluate how a learned binary mask over individual weights of the localized components can produce editing/unlearning, and vary the size of this mask.

After weight masking, we also analyze the proportion of each mechanism (fact lookup, attribution extraction) that is masked for each localization in Section A.2.3. We demonstrate that OT methods and nonlocalized editing all modify a higher proportion of the extraction head/MLP parameters than the fact lookup mechanism parameters, supporting our claim that OT methods target extraction mechanisms rather than the fact lookup mechanisms needed for robustness.

#### 3.2.1 Standard Evaluation

We show standard evaluations across a sweep of discretization thresholds, which directly corresponds to the size of the model edit. Figure 5 shows the accuracy on the forget and retain sets for unlearning basketball across different edit sizes. Here, we see all methods being effective in unlearning basketball facts while retaining all other facts. While AP and CT localizations cause the model to have zero accuracy on the in-distribution set with much fewer masked weights needed, when checking for generalization using a multiple-choice format we clearly see that only manual localization has successfully generalized the unlearning of basketball facts (Figure 5, right).

Figure: Figure 5: (Left) Testing the models’ unlearning of basketball athletes against the number of weights masked. (Right) Testing the models’ unlearning of basketball athletes against the number of weights masked, in the MCQ prompt format.
Refer to caption: x2.png

Figure: Figure 6: Unlearning basketball facts. (Left) Measuring MMLU and forget set performance across different discretization thresholds. (Right) Measuring MMLU and MCQ forget set performance across different discretization thresholds.
Refer to caption: x4.png

Figure: Figure 7: Editing subset of athletes. (Left) Measuring accuracy on the forget set. (Right) Measuring accuracy on the forget set in the MCQ prompt format.
Refer to caption: x6.png

We find similar results when testing for performance degradation on MMLU (because we have to evaluate many model variations, we use a smaller MMLU test set from Polo et al. ()). While all localized methods perform well when evaluated normally (Figure 6, left), Figure 6 (right) shows manual localization generalizes for minimizing loss of MMLU capabilities while unlearning sports facts in the MCQ format compared to the other methods.

For editing the subset of athletes, Figure 7 shows that causal tracing localization causes the model to have 0% accuracy on the forget set, and manual interpretability and nonlocalized editing cause the model to have near guessing rate (33%) accuracy. However, only manual localization minimizes loss of capabilities while editing the athlete subset (Figure 8).

Furthermore, no other method completely generalizes this unlearning to the MCQ prompt format (Figure 7), and manual localization remains superior in minimizing loss of capabilities while unlearning the athlete subset (Figure 8, right).

Figure: Figure 8: Editing subset of athletes. (Left) Measuring MMLU and forget set performance across different discretization thresholds. (Right) Measuring MMLU and MCQ forget set performance across different discretization thresholds.
Refer to caption: x8.png

We find similar results for editing on the CounterFact dataset. However, we find minimal difference in MMLU accuracy in all methods at all numbers of masked weights. Thus, we instead report the maintain and forget accuracies of these facts at different discretization thresholds in Figure 9. We provide additional performance metrics in Section A.3.

Figure: Figure 9: Editing CounterFact facts. (Left) Testing models accuracy on the normal forget set. (Right) Testing the models’ accuracy in the MCQ prompt format.
Refer to caption: x10.png

## 4 Discussion

Recent work by Hase et al. () argued that localization is not useful for model editing.
Our findings demonstrate that the relationship between localization and fact editing/unlearning is more nuanced, and reveals that not all localization techniques are equal.

Our work evaluates the efficacy of different localization methods for modifying factual associations. We demonstrate clear benefits of localization for unlearning robustness through localized fine-tuning combined with manual mechanistic interpretability techniques designed for fact recall.

In Section 3.1.3 and Section A.2.3, we provide evidence that OT and baseline approaches fail to be robust because they target easily-localizable and high direct logit importance attention head components, that transform existing latent factual knowledge to the desired output format. This can fail to generalize to different input and output formats and does not target the true source of knowledge in the model: other input/output formats can allow alternative attention mechanisms to transform this knowledge, and low-resource relearning can quickly repair the original attention mechanism.
In contrast, FLU mechanistic understanding allows us to target unlearning at the sites where knowledge is sourced, which robustly prevents that information from entering the latent stream in any format.

Our work also suggests unlearning/editing as a potential testbed for different interpretability methods, which might sidestep the inherent lack of ground truth in interpretability (Templeton et al., ). We hope our work provides a framework for evaluating localizations and explanations.

## Acknowledgements

We thank Stephen Casper, Aengus Lynch, and Max Li for their advice with unlearning methods and interpreting results.
We also thank Daniel M. Roy, Amr Khalifa, and Eleni Triantafillou for feedback on various drafts of this work. This project used compute provided by the Center for AI Safety.

## Appendix A Appendix

### A.1 Fact Injection Results

We additionally consider the common factual editing methodology, specifically an error injection setup (Hase et al., ) where we replace correct athlete-sport associations with incorrect associations between athlete and ”Golf”. The results are in Table 4 and Table 2, where we demonstrate that our manual localization method again achieves the strongest robust unlearning generalization while maintaining more general capabilities than the other robustly unlearned model.

**Table 4: Results of unlearning basketball associations, with the objective of replacing the correct sport of ”Basketball” with ”Golf”. Forget refers to the model’s accuracy at stating the original sport association which should have been replaced.**
| Localization | Forget $\downarrow$ | Retain $\uparrow$ | MCQ $\downarrow$ | MMLU $\uparrow$ |
| --- | --- | --- | --- | --- |
| Attrib. Patching | 0.000 | 1.000 | 0.815 | 0.611 |
| Causal Tracing | 0.028 | 1.000 | 0.866 | 0.614 |
| Manual | 0.035 | 0.973 | 0.257 | 0.610 |
| Random | 0.018 | 1.000 | 0.839 | 0.611 |
| All-MLPs | 0.000 | 0.946 | 0.363 | 0.571 |
| Nonlocalized | 0.000 | 0.995 | 0.376 | 0.565 |

**Table 5: Localized fine-tuning accuracy on standard evaluations: Unlearning a constant 16 athlete subset, retaining all other facts.**
| Localization | Forget $\downarrow$ | Retain $\uparrow$ | MCQ $\downarrow$ | MMLU $\uparrow$ |
| --- | --- | --- | --- | --- |
| Attrib. Patching | 0.941 | 0.964 | 0.934 | 0.614 |
| Causal Tracing | 0.891 | 0.915 | 0.910 | 0.612 |
| Manual | 0.034 | 0.975 | 0.175 | 0.615 |
| Random | 0.938 | 0.952 | 0.883 | 0.612 |
| All-MLPs | 0.003 | 0.973 | 0.281 | 0.599 |
| Nonlocalized | 0.203 | 0.570 | 0.391 | 0.540 |

We also perform relearning and latent knowledge experiments in Figure 2 (right) and Figure 4 (Appendix A), demonstrating that manual localization for the athlete subset injection improves relearning and latent knowledge robustness.

Figure: Figure 10: Retraining athlete-unlearned models with two athletes in the forget set, for ten iterations. The y-axis represents the normal accuracy on the forget set. Low-resource relearning of athletes demonstrates that manual and non-localized unlearning techniques are robust to this test (staying close to the guessing rate of 33%), while the other methods maintain full performance on the entire forget set.
Refer to caption: extracted/5932245/images/relearning/finetuning-athletes-forget.png

### A.2 Gemma Interpretability Analysis

#### A.2.1 Sports Facts

We redo analysis from Nanda et al. () on Gemma-7B. We train logistic regression models (”probes”) to predict the correct sport given the internal representation of the model at a layer. We find that probes predicting the correct sport increase in accuracy significantly in layers 2 through 7, and we find the mean ablation of all attention heads past layer 7 to have minimal impact on the linear representation of player attributes (Figure 12).

Unlike Nanda et al. (), however, we find attention heads past layer 2 that impact the linear representation of attributes and thus could potentially be important for fact lookup (Figure 12). However, because they could likely play a variety of other different roles such as token concatenation, following the findings of Geva et al. (); Nanda et al. () that MLPs do primary factual representation enrichment, in this work we only consider the MLPs as our localization.

Figure: Figure 11: Difference in final layer probe accuracy when mean ablating a single head for all heads between layers 0 and 6.
Refer to caption: x12.png

#### A.2.2 CounterFact

We repeat analysis from Nanda et al. () and Geva et al. () on Gemma-2-9B. We first measure the effect on the difference in logits between correct and incorrect answers of facts when patching the direct path of attention heads and MLPs to the final output, shown in Figure 14 and Figure 14. An attention head or MLP will have a large effect on the logit difference if it is important in moving the factual information to the last token position or decoding it into the correct answer. We call these components part of the ”fact extraction mechanism”, and aim to find the source of the factual information moved by this mechanism.

To find this source, we patch the outputs of MLPs to this ”fact extraction mechanism” and measure the resultant change in logit difference (Figure 15). An MLP would cause a large change in logit difference if it caused relevant representations to form that are then moved by the ”fact extraction heads” to increase the probability of the correct output. We take the MLPs with the highest change and include them in our manual localization of CounterFact (MLPs 3-5, 7-10, 14-17).

Figure: Figure 13: Change in logit difference when patching heads to the final output.
Refer to caption: x14.png

Figure: Figure 15: Change in logit difference when patching MLPs to the extraction mechanism.
Refer to caption: x16.png

#### A.2.3 Mechanism Weight Analysis

We analyze the actual components localized by each localization type and our baselines, for the CounterFact editing task. We seek to demonstrate that the OT localizations and baselines target extraction mechanisms rather than just the FLU mechanisms.

First, in Table 6, we compare the parameter counts of the part of each mechanism that is present in each localization. Table 6 shows that causal tracing and attribution patching both have the potential to modify a considerable proportion of the extraction heads and extraction MLPs.

**Table 6: Comparison of total parameters of each mechanism that are present in each localization, for editing 16 facts from CounterFact**
| Localization | Extraction Heads | Extraction MLPs | Fact Lookup |
| --- | --- | --- | --- |
| Total | 27,448,320 | 1,027,604,480 | 1,130,364,928 |
| Attrib. Patching | 13,724,160 (50.0%) | 616,562,688 (60.0%) | 102,760,448 (9.1%) |
| Causal Tracing | 8,234,496 (30.0%) | 308,281,344 (30.0%) | 411,041,792 (36.4%) |
| Manual Interp. | 0 | 0 | 1,130,364,928 (100.0%) |
| All-MLPs | 0 | 1,027,604,480 (100.0%) | 1,130,364,928 (100.0%) |
| Nonlocalized | 27,448,320 (100.0%) | 1,027,604,480 (100.0%) | 1,130,364,928 (100.0%) |

Then, in Table 7, we compare the proportion of each mechanism that is masked when using a localized weight mask and discretizing to about 6 million weights. This is one approximate metric for how much each mechanism is modified by the localized editing. Table 7 demonstrates that attribution patching, causal tracing, and nonlocalized editing all modify a higher proportion of the extraction head/MLP weights than the fact lookup mechanism weights.

**Table 7: Comparison of parameters of each mechanism that are masked by a trained weight mask, discretized to about 6 million weights**
| Localization Type | Extraction Heads | Extraction MLPs | Fact Lookup |
| --- | --- | --- | --- |
| Total (baseline) | 27,448,320 (100%) | 1,027,604,480 (100%) | 1,130,364,928 (100%) |
| Attrib. Patching | 165,300 (0.60%) | 1,479,877 (0.14%) | 1,385,198 (0.12%) |
| Causal Tracing | 30,828 (0.11%) | 1,491,040 (0.15%) | 1,424,059 (0.13%) |
| Manual Interp. | 0 (0.0%) | 0 (0.0%) | 6,248,039 (0.55%) |
| All-MLPs | 0 (0.0%) | 1,378,744 (0.13%) | 1,663,772 (0.15%) |
| Nonlocalized | 358,918 (1.3%) | 1,198,211 (0.12%) | 1,174,939 (0.10%) |

This supports our argument that OT methods target high logit-diff extraction mechanisms, rather than the fact lookup mechanisms that enrich the latent stream with the correct attributes, which decreases the robustness of edits/unlearning. It is important to note that since our manual interpretability localization is based on our discovered mechanisms, this does not serve as an evaluation of manual interpretability, but rather only of causal tracing and attribution patching.

### A.3 Additional CounterFact Weight Masking Results

We report a comparison of all localizations across discretization thresholds for normal and MCQ forget sets in Figure 17 and Figure 17. We see that manual interpretability outperforms all other methods of localization in preserving maintain accuracy while decreasing forget accuracy.

Figure: Figure 16: Accuracy on normal forget set vs on the maintain set across localizations and discretization thresholds.
Refer to caption: x17.png

We perform additional adversarial analysis of accuracies across different discretization thresholds. We report the ”paraphrase” and ”neighborhood” adversarial results in Figure 19 and Figure 19.

Figure: Figure 18: Accuracy on paraphrased input vs on the maintain set across localizations and discretization thresholds.
Refer to caption: x19.png

### A.4 Hyperparameters

For localized fine-tuning, we use $\lambda_{1}=0.2$ for forgetting basketball associations and $\lambda_{1}=1$ for forgetting particular athlete associations. For weight masking on Sports Facts, we use $\lambda_{1}=0.3$ for both basketball and athlete associations. $\lambda_{2}$ and $\lambda_{3}$, the retain and SFT loss coefficients, were both set to 1 across all experiments. For weight masking regularization, we used $\lambda_{4}=1*10^{-7}$ (where our regularization loss was the total sum of absolute weight mask values).

For weight masking on CounterFact, we use $\lambda_{1}=\lambda_{2}$ = $\lambda_{3}$ = 1. For weight masking regularization, we again used $\lambda_{4}=1*10^{-7}$ (where our regularization loss was the total sum of absolute weight mask values).

For localized fine-tuning on Gemma, we trained using 50 iterations of batch size 4 with 16 accumulation steps, using an AdamW optimizer  (Kingma & Ba, ) with 0 weight decay, a learning rate of $1*10^{-5}$, and a cosine annealing scheduler for both basketball and athletes.

For weight masking, we unlearned using 50 iterations of batch size 10 with 15 accumulation steps, using an SGD optimizer (for memory efficiency) with learning rate of $1*10^{-2}$ and clamping the mask values between 0 and 1 every update step.

### A.5 Evaluation Details

#### A.5.1 Train-Test Splits

We split the basketball $D_{forget}$ set and both $D_{retain}$ sets (basketball and athletes) into an 80%-20% train-test-split, and all of our reported numbers are on the test set. We do not split the $D_{forget}$ set of 16 athletes, because we wish to test if the model has unlearned the athletes it was trained to unlearn.

#### A.5.2 Details on Adversarial Relearning

We retrain the model with only two athletes in $D_{forget}$ for multiple iterations (along with a standard retain and SFT loss), in both the sport and athlete unlearning scenarios. In practice, for basketball this looks like giving the model the same batch of only Boris Diaw and Jae Crowder multiple times, and for athletes we give a batch of only DeForest Buckner and Walter Payton. We retrain with a rank-64 LoRA on all linear modules.

#### A.5.3 Details on Latent Knowledge

We don’t follow the same methodology as Patil et al. () because we only care about the same three possible tokens, so it isn’t applicable to apply their search-budget methodology and we instead try linear probes. In general, we don’t consider linear probing to be a realistic threat model for beating unlearning, as attackers need white-box access and labels for large subsets of the forget set, but we do these tests for an approximate upper bound of accessible information by a capable-enough adversary.

We train three linear probes (Alain & Bengio, ) for every model and layer, one for each sport (to predict True or False with a base rate of  66%), on samples from both the forget and retain datasets. We train each probe on both forget and retain samples because for sports, there is only one forget sport (so the answer would be constant if we trained different probes), and for athletes we only have 16 total examples that must further be split into train-test.

For athletes, we split the forget set into a 50%-50% train-test split, so the probe training dataset includes 8 of the forgotten athletes (along with the standard retain train split) and the test set includes the other 8 (along with the retain test split). For sports, we use the standard basketball train and test split. Then, as a measure of aggregated accuracy, we only consider a test sample to be correct if probes for all three sports are correct.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the Weaknesses section above.
soundness: 3
strengths: * Studying unlearning methods from the perspective of knowledge storage and mechanistic interpretability is indeed a very important and promising direction.

* This paper further confirms that causal tracing-based localization methods are not suitable for editing and unlearning tasks.

* The paper is well presented and the literature review is thorough.

* The experimental design is generally comprehensive.
summary: This work explores methods for knowledge editing and unlearning in large language models, focusing on how mechanistic interpretability can enhance the precision and effectiveness of these processes. The study reveals that localizing edits to components associated with lookup-table mechanisms for factual recall leads to more robust unlearning, resisting unwanted information relearning and minimizing side effects. Additionally, certain localized edits disrupt latent knowledge more effectively than other methods, resulting in increased resilience against various attacks.
weaknesses: 1. The test dataset appears to be limited to this triplet format; is it constrained by the knowledge format, and could it be applied to more broadly and flexibly expressed knowledge sentences, such as continuous text, etc.?

2. Manually analyzing and then selecting layers for operations seems to lack convenience and flexibility in the context of large-scale data editing/unlearning.

3. The proposed new unlearning method lacks sufficient originality. There are already some works that attempt unlearning directly from the perspective of mechanistic interpretability, including [2].

4. Knowledge is not necessarily stored entirely in the MLP; there are certain cases where it exists in the attention mechanism [1], yet the method described in the paper only considers knowledge stored in the MLP.

5. There is a lack of discussion on unlearning methods in Representation Engineering [3, 4].

6. Can the proposed method achieve performance advantages on other representative series of transformers, such as LLaMA?


---
**References:**

[1] Dissecting Recall of Factual Associations in Auto-Regressive Language Models

[2] Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces

[3] The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

[4] Improving Alignment and Robustness with Circuit Breakers

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 2
strengths: This paper contains the following several strengths:
+ The paper addresses an important topic by attempting to improve the robustness of knowledge unlearning in LLMs through  localizing edits to components associated with the FLU.
+ The authors provide a in-depth analysis and comparison between mechanistic unlearning and previous OT methods.
summary: This paper investigates the effectiveness of mechanistic interpretability techniques in improving the precision and robustness of knowledge editing and unlearning in LLMs. The authors mainly discuss two types of localization methods, i.e., OT techniques that focus on preserving outputs, and mechanistic localization, which identifies high-level mechanisms with predictable intermediate states. They claim in the paper that localizing edits to components associated with the FLU mechanism leads to more robust unlearning across different input/output formats and resists attempts to relearn unwanted information. They conduct experiments on the Sports Facts and CounterFact dataset using Gemma-7B and Gemma-2-9B models.
weaknesses: + The paper would benefit from a more in-depth theoretical analysis to explain why FLU could inherently lead to more robust unlearning. While the authors claim that targeting the fact lookup components is more effective, they do not provide analysis or proof to support this.
+ The experiments are limited, especially limiting itself to Gemma-7B and Gemma-2-9B models and two datasets. The authors could provide a larger variety of models and unlearning tasks in order to better demonstrate the consistency of their findings.
+ Can the author provide more ablation study, for example on the loss weights parameter being used in 2.3, so that we could better understand the contribution of each loss in the finetuning process.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In the section defining the method, it mentions, "In practice, we fix τ such that Cτ contains the same number of parameters in OT, FLU, and random localizations." How should this statement be understood? For example, in the counterfact dataset, what are the MLP results using OT and EAP, given that our analysis highlights layers 3-5, 7-10, and 14-17 as the critical MLPs?

2. Could you explain in more detail the process of direct path patching on the counterfact dataset? For example, after obtaining the set of components related to the fact extraction mechanism, how do we replace all edges from each MLP to all components in the set?

3. In the manual localization process, why is only the MLP considered as the localization component, while other methods like EAP and OT do not also set the form to only consider MLP? Instead, they assess both attention heads and MLP components simultaneously?

4. [Critical] Can the authors provide theoretical proof or guarantee to show that the knowledge is forgotten?

5. [Critical] Can the authors provide the experiments of adaptive attack (attackers that easily conquer approximate unlearning )?

6. [Critical] Can the authors provide the experiments of sequential unlearning?
soundness: 2
strengths: 1. The motivation and approach of this article are interesting, as it breaks down factual recall into more granular steps to enhance the accuracy and generalization of editing methods. 

2. The extensive experiments demonstrate the effectiveness of the proposed approach. The findings indicate that fine-tuning the FLU-related components identified through manual localization effectively eliminates specific knowledge from the model and makes it less susceptible to re-learning.

3. Using multiple-choice questions (MCQs) can help eliminate the influence of input patterns while allowing for a more effective exploration of knowledge deletion. This approach can provide clearer insights into how specific knowledge is affected by unlearning processes.
summary: This article investigates the performance of various localization methods in unlearning/editing tasks, particularly focusing on their limitations in adapting to shifts in prompting/output distributions and adversarial relearning scenarios. It compares three main approaches: output tracing, attribution patching, and FLU. Through experiments conducted on two models and two datasets, the findings reveal that the component set identified by FLU localization is more closely tied to the factual query process, demonstrating greater robustness and generalization when fine-tuned. Additionally, the authors achieve more efficient parameter editing by controlling model modifications through weight masking.
weaknesses: 1. Regarding the editing task, only the decline in the original answers is reported, which aligns with the unlearning aspect. However, results seem to lack a demonstration of improvement in the correctness of the new answers. This is significantly related to the performance of the editing task. 

2. Additionally, for the analysis of intermediate representations using probing, this concept is derived from existing work and does not represent a novel contribution to this research.

3. [Critical] In the unlearning task, there is no theoretical proof or guarantee that the knowledge is fully forgotten. Since approximate unlearning can be easily exploited, this method is vulnerable. I believe that a theoretical guarantee is crucial for the unlearning task because the security issue is fundamentally a "to be or not to be" problem.

4. [Critical] No adaptive attack experiments were conducted. The authors performed only standard unlearning/editing experiments, without testing for membership inference or adaptive attacks, despite the fact that approximate unlearning methods are particularly susceptible to adaptive attacks.

5. [Critical] There is a lack of experiments on adaptive unlearning, which would involve sequentially unlearning specific types of knowledge—for instance, first basketball, then football, and finally table tennis. Would adaptive unlearning impact the efficiency of the unlearning methods?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 2
questions: (All of my questions were asked in the "weaknesses" section.)
soundness: 4
strengths: Overall, the results in this paper are very strong, and the suite of evaluations is impressively thorough.
1. The authors do a good job of establishing that finetuning on the "manual interp" subset of parameters results in qualitatively different unlearning dynamics and quantitatively stronger results. The field of interpretability has struggled recently to demonstrate that their insights are useful for downstream tasks in general—and for unlearning/model editing in particular as demonstrated by Hase et al. (2023)—so I expect these results will be a breath of fresh air for the interpretability community.
2. The authors perform a very thorough suite of evaluations showing that mechanistic unlearning more effectively changes knowledge stored in the model weights (see (3) and (4) for more detail). This is another place where the authors set themselves apart from the field: the unlearning literature has often struggled with thoroughly evaluating the efficacy of their methods.
3. I was especially impressed by the relearning evaluations, showing that—when training the model to relearn the unlearned facts—the mechanistically unlearned model relearns the facts much more slowly (figure 2).
4. Also impressive were the results that sweep over the number of masked parameters, revealing qualitative differences in the various unlearning techniques. Figure 5, right, which shows that mechanistic unlearning generalizes to MCQ rephrasing substantially better than any other unlearning technique, was especially striking.
summary: The authors study whether insights from mechanistic interpretability can improve our ability to perform unlearning or make targeted model edits. More specifically, using a collection of factual recall tasks, the authors study different ways of selecting a subset of the model's parameters to finetune for unlearning or editing certain facts. Building on prior work identifying a subset of model parameters involved in factual recall, the authors show that finetuning only these parameters results in more effective unlearning/model editing than finetuning parameters selected via other techniques. These claims are supported by a variety of analyses, such as robustness to different ways of eliciting the model's factual knowledge and robustness to retraining the model to relearn the facts.
weaknesses: Overall, this work's presentation was very poor.
1. Many important details are missing from the main body of the text. (a) The "fact loookup localization" (which is also called "manual interpetability"—why not use one term?) method is entirely explained in an appendix. While it's reasonable to put most of the FLU details in the appendix (since this is a replication of prior work) understanding at a high level what FLU is and how it assigns importance scores to various model components is essential for understanding the rest of the work. (b) The definition of the unlearning loss is described only as "the log(1 - p) measure from Mazeika et al."—this is an important part of the method and should be explained.
2. It is very difficult to follow discussions of the tasks. There are two tasks related to sports facts, one related to unlearning facts about basketball athletes (how many athletes? I think this is mentioned later, but it should be included in section 2.1) and one related to editing 16 (random?) athletes to change their sport to "golf." The authors later refer to these tasks with vague phrases like "For editing the subset of athletes..." The authors should instead give distinct names to these two tasks (e.g. "Sports-unlearning" and "Sports-editing") which they use to refer to the tasks throughout the rest of the text.
3. Although the results are strong, the authors do not make it easy to tell this from reading the work. For example, the tables of numbers in the first results section are not a reasonable way to present these results. Tables like these are good for when we want to inspect small numerical differences; in contrast here we don't care about small differences (e.g. between forget scores of 0.002 and 0.000) but about large differences (e.g. between MCQ scores of 0.110 and scores >0.5). The authors should choose a different way of presenting these results, perhaps as a bar chart. 
4. Similarly, the authors present results for all three tasks for each of their evaluations, resulting in a large number of figures which are left to the reader to synthesize. This work would be much stronger if the authors found ways of presenting their work that summarized and emphasized the key takeaways.
5. The main takeaway from the counterfact retraining experiment should have been that this experiment isn't informative, since relearning on some facts doesn't generalize to other facts for *any* of the unlearning techniques. This experiment should therefore be moved into an appendix.

I also have some object-level concerns about various choices made by the authors. I mostly expect these to be easy to address with follow-up experiments (and I'll be happy to raise my score once I see these follow-ups).
1. The task definitions involve various arbitrary-seeming choices: only basketball athletes are targeted for unlearning, only golf is uses as the target relation for editing, only a particular set of 16 facts is used for CounterFact, etc. This makes it hard to tell if these results are due to cherry-picking. The authors should sweep over options for these choices (e.g. also targeting athletes for different sports, or rerandomizing for different unlearned facts) and present averages over the conditions.
2. While it is good that the authors study multiple models, they pair specific models with specific tasks, again making the reader wonder about cherry-picking. Unless there is a good reason not to do so, the authors should test all three tasks for *both* Gemma-7B and Gemma-2-9B.
3. In addition to the all MLPs baseline, the authors should also include a baseline for the same number of MLPs as in the "manual interp" condition, but randomly selected.

Meta-review: The paper explores the use of mechanistic interpretability to improve knowledge unlearning and editing in LLMs, with a focus on localizing edits. While it demonstrates robustness gains and improved resistance to relearning unwanted information, reviewers highlighted concerns regarding limited novelty, insufficient evaluation on diverse tasks and models, and the lack of theoretical guarantees as major weaknesses. The authors' response did not fully address these concerns, and I recommend rejection.

Decision: Reject