CLUE: Conflict-guided Localization for LLM Unlearning Framework
Authors: Hang Chen
Jiaying Zhu
Xinyu Yang
Wenya Wang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM unlearning
circuit discovery
conjunctive normal form
interpretability
Primary area: foundation or frontier models, including LLMs
TL;DR: We use circuit discovery and CNF solving to design the localization for forget neurons and retain neurons in the LLM unlearning task.

Abstract: The LLM unlearning aims to eliminate the influence of undesirable data without affecting causally unrelated information.
This process typically involves using a **forget set** to remove target information, alongside a **retain set** to maintain non-target capabilities. While recent localization-based methods demonstrate promise in identifying important nodes (neurons) to be unlearned, they fail to disentangle nodes responsible for forgetting undesirable knowledge or retaining essential skills, often treating them as a single entangled group. As a result, these methods apply uniform interventions, risking catastrophic over-forgetting or incomplete erasure of the target knowledge. To address this, we turn to circuit discovery, a mechanistic interpretability technique, and propose the **C**onflict-guided **L**ocalization for LLM **U**nlearning fram**E**work (**CLUE**). This framework identifies the forget and retain circuit composed of important nodes, and then the circuits are transformed into conjunctive normal forms (CNF). The assignment of each node in the CNF satisfiability solution reveals whether it should be forgotten or retained. We then provide targeted fine-tuning strategies for different categories of nodes. Extensive experiments demonstrate that, compared to existing localization methods, CLUE achieves superior forget efficacy and retain utility through precise neural localization.

Published as a conference paper at ICLR 2026
CLUE: CONFLICT-GUIDED LOCALIZATION FOR LLM UNLEARNING FRAMEWORK
Hang Chen
Jiaying Zhu
School of Computer Science and Technology Xi'an Jiaotong University a albert2123@stu. xjtu. edu.cn cn
School of Computer Science and Engineering The Chinese University of Hong Kong jyzhu24 22 cse. cuhk. edu. hk
Xinyu Yang"
Wenya Wang
School Computer Science and Technology Xi' an Jiaotong University
School of Computer Science and Engineering Nanyang Technological University wangwyentu. edu.ss sg
yxyphd@ma 11 xjtu.. edu.cn
ABSTRACT
The LLM unlearning aims eliminate the influence of undesirable data without affecting causally unrelated information. This process typically involves using forget set remove target information, alongside a retain to maintain non-target capabilities. While recent localization- based methods demonstrate promise in identifying important nodes (neurons) be unlearned, they fail disentangle nodes responsible for forgetting undesirable knowledge or retaining essential skills, often treating them as single entangled group. As a result, these methods apply uniform interventions, risking catastrophic over-forgetting or incomplete erasure of the target knowledge. To address this, we turn to circuit discovery, mechanistic interpretability technique, and propose the Conflict-guidee Localization for LLM Unlearning framEwork (CLUE). This framework identifies the forget and retain circuit composed of important nodes, and then the eircuits are transformed into conjunctive normal forms (CNF). The assignment of each node the CNF satisfiability solution reveals whether should be forgotten retained. We then provide targeted fine-tuning strategies for different categories of nodes. Extensive experiments demonstrate that, compared to existing localization methods, CLUE achieves superior forget efficacy and retain utility through precise neural localization. Our code is available at https: ///i://itt gigttuu.bb... com Zodi crh CLUE
INTRODUCTION
Large language model (LLM) unlearning (Liu et al. 2025 Yao et al. 2024), machine learning method inherited from model unlearning (Cao Yang 2015 Neel et al. [021]. aims to have the LLM avoid or remove certain target information while preserving its other non-target capabilities as much as possible. Formally, the framework of LLM unlearning typically involves two datasets: the forget set and the retain set. The optimization objective to avoid the original responses to the forget set (which are typically harmful or sensitive) and retain the existing responses to the retain set.
Many categories of methods currently exist for LLM unlearning (Zhang al. 2024 Jia et al. 2023
Liu et al. 2024). Among these, localixation-infrmee by localizing key nodes (neurons [ or parameters. This also allows for better maintenance and targeted updates to these parameters, which aligns well with future modular machine learning developments (Menik & Ramaswamy 2023). Recently, various localization-informee methods
unlearning offers better interpretability
"Corresponding author I 'Corresponding author
'In this paper, the terms "node" and "neuron" represent almost the same concept. They are all used to represent some kind of trainable parameter matrix in the model (see Sec 2.2 for details). For consistency, and without loss of generality, we "nodes" by default.

---

Published as a conference paper at ICLR 2026
have emerged, including those based gradients (Wu et al. tribution (Jia et al.) 2024), and causal effect estimation (Patil et al.,
2023 Yu et al.,
2023). weight at-
2023
Meng et al. 2022).
Existing localization
Our localization
However, existing localization methods can only identify an entangled set of "important" nodes resulting from the joint optimization on the forget and retain sets. They can- not pinpoint the specific subset of nodes responsible for forgetting, retaining, or a combination of both. As de- picted in Figure ! these important nodes can be intuitively subdivided into three categories: retain nodes, which in- fluence only the retain set; forget nodes, which influence only the forget set; and conflict nodes, which influence both the forget and retain sets.
instevant
ratain nodes
important nodes
torget nodes
confict nodes
Figure I: A fine-grained exploration in- spired by existing localization methods.
Clearly separating these three types of nodes is essential for improving unlearning's 's forget efficacy and retain utility. For instance, suppose the forget set contains harmful information, whereas the retain set includes sentiment recognition samples. A coarse- coarse-grained intervention on all "important nodes" might unintentionally compromise the model's sentiment recognition capability when removing the harmful information. By disentangling the retain nodes from the "important nodes", the sentiment recognition capability is better preserved. However, disentangling these node types from the coarser set of important nodes is highly challenging. For example, in gradient-based methods, the gradient from the joint optimization of the forget loss and retain loss is not equivalent to the linear combination of the gradients from optimizing each loss separately. Therefore, the identified nodes reflect intertwined signals and cannot distinctly represent forgetting or retaining.
To address this problem, we resort to circuit discovery (Conmy et al. [023 Bhaskar et al. 2024). a mechanistic interpretability method that identifies the important nodes and their activation relation- ships for a target task or dataset, representing them in graph-structured circuit. This method enables explicit tracking of information flows between nodes, allowing us to isolate functional substructures. Recent circuit discovery techniques (Chen et al. 2025 Heimersheim & Nanda (0244) are also capable of uncovering the logical relationships within a circuit; for example, some subeircuits resemble digital logic gates like an AND gate, while others are similar to an OR gate. This is particularly appealing in our setting, since forgetting and retaining are inherently compositional operations: for instance, a capability could only be preserved if multiple nodes are unchanged (similar an AND gate), while another capability might be forgotten when just one of a group of nodes is edited (similar to an OR gate).
Building on this, we propose the Conflict-guisn suiiitt-guiidd Localization for LLM Unlearning fram Ework (CLUE), which can distinguish the node categories conceptualized in Figure through the Boolean satisfiability of circuits. Specifically, CLUE first extracts circuits from the forget set and retain set, respectively. These circuits are then converted into Conjunctive Normal Form (CNF) using Tseitin's 's transformation (Tscitin 1983). A CNF is constructed whose logic ensures that the forget circuit is modified while the retain circuit is preserved. By solving this CNF as a satisfiability problem (Fleury & Heisinger 2020), we can determine which nodes belong to the retain node, forget node, and conflict node categories based on the values of each node in the optimal solution. Finally, we also provide a fine-tuning paradigm that provides supervision for the forget nodes using the forget loss and for the conflict nodes using both the forget and retain loss.
CLUE allows for the editing of different node categories based on distinct optimization objectives. This more fine- mme-grained and precise localization enhances the effectiveness of unlcarning. To demon- strate this, conducted extensive experiments on three mainstream unlcarning datasets: WMDP Cyber, WMDP Bio, and PKU SafeRI The results not only show that our method significantly improves forget efficacy and retain utility with fewer modified parameters, but also reveal a more detailed correlation between the unlearning effect and the underlying circuits and nodes. In summary, our contributions are threefold:
We propose conflict-basse localization, which identifies more fine- -nn-grri node categories by solving the CNF satisfiability of the forget circuit and the retain circuit.
2

---

Published as a conference paper at ICLR 2026
We introduce CLUE, effective framework for LLM unlearning that explicitly leverages different node categories to achieve more precise unlearning.
We demonstrate through extensive experiments that CLUE surpasses current localization methods across multiple dimensions and offers more comprehensive interpretability.
2 PRELIMINARIES
2.1 LLM UNLEARNING
In light of the existing literature on LLM unlearning (Li et al. 2024 Maini et al. 2024 Ishibashi
Shimodaira 2023 Yao et al., 2024 Pawelczyk al.) 2024). we define the problem of LLM unlearning as eliminating the influence of specific "unlearning targets" and removing associated model capabilities while preserving model performance for non-targets. To facilitate comprehension, we provide a commonly-used formulation of LLM unlearning problems below.
min R (Fly + XI (e (C(yl2;
where À 2 O represents the regularization parameter, L(ylæ;; denotes the prediction loss of using 0 given the input w.r.t. response y. D and D, refer to forget set and retain set. yy denotes the desired model response post-unlearning. The retain set represents the non-target. It indicates that, during LLM unlearning, the objective is to maintain the utility the retain set, while simultaneously ensuring that the model avoids generating the undesired responses associated with the forget set.
2.2 CIRCUIT DISCOVERY AND LOGICAL CIRCUIT
In Transformer decoder-based language models, the forward pass typically conceptualized as computational graph G, where the nodes represent neurons, such as output, MLPs, and query, key, and value matrices in each head and an edge + j denotes connection where the activation of node i serves input to node j. Circuit discovery seeks to identify subgraph (circuit) C G for target dataset that captures the fask-relevan behavior (or mechanism/capability of this dataset (Elhage et al. 2021 Conmy et al. 2023 Rai et al. 2024), with the following objective:
arg min B
s.t. 1 - CI/1g 2
(2)
where s denotes the requirement of sparsity, T represents the target dataset, and D represent the distance quantify the difference between the outputs from and C. Circuit discovery aims to retain the minimal C while faithfully reflecting the model's capability in processing the T. The nodes and edges within this circuit are regarded those exerting the most critical influence on the T. We define the scope of nodes to include: q. k, u, M MLPup up, and M LPdown treating them as the smallest independent units that influence the outcome, Given that the adopted baseline model is the Zephyr-7B 7B-beta model, we additionally include M MLPgate Consequently, the entire computation graph, excluding inputs and outputs, consists of 32 layers, with each layer containing 7 nodes, and each node corresponds to learnable parameter matrix. The circuit analysis examines the activation relationships between these nodes.
Increasing research has demonstrated the existence of various logical structures within circuits. Heimersheim & Nanda (2024) identified the AND and OR gates in circuits. Considering an AND gate with node B output and nodes A,, A as input, B is activated only when both Ai and A are activated simultaneously. Conversely, if B, A1. Ag construct an OR gate, i.e., B Ai or Ag, Ag is activated. Similar OR gates have also been
then B is activated as long at least one of A observed in Conmy et al. (2023) and WWang et al. (2023). Furthermore, Chen et al. (2025) refined the classification of logical structures, proposing three distinct types: AND, OR, and ADDER
3

---

Published as a conference paper at ICLR 2026
Input
(N v -Nos v v-N <(7No0 v Noo Nai)
Safe nodes (value=True)
Forget set
INu
Na
Nai
Naa, No1
output
Forget nodes (value=Falsse)
Nio. Naa*
input
NasANas A(No1 Nua) ANssANes
IN:
Retain set
Conflict nodes (no valid value)
Nws. N₂z
output
Dataset
Logical Circuit
CNF
Localization
Figure 2: Overview from datasets to localization.
3 CONFLICT-GUIDED LOCALIZATION
In this section, provide three-step framework of how circuit discovery ultimately enables precise localization. An overview of our localization procedure is shown in Figure 2 Specifically,
Using the circuit discovery algorithm, we separately capture the forget circuit and the retain circuit, corresponding to the forget set and the retain set, respectively. The transformation from model to circuit reveals the nodes and activation connections that are most critical for the responses the forget and the retain set.
Step (Section 3.1
We then transform the forget circuit and the retain circuit into CNR. The forget CNF ensures satisfiability from the perspective of forgetting, while the retain CNF ensures satisfiability from the perspective of retaining. These two CNFs provide logical basis for assigning nodes to forget nodes, retain nodes, and conflict nodes.
Step 2 (Section 3.2
Step 3 (Section (.3). Finally, we jointly solve the forget and retain CNF determine the assignment of each node. nodes with value = influence only the retain set, whereas nodes with value influence only the forget set. nodes without valid assignment (i.e., those that produce conflicts) indicate shared influence on both the forget and retain sets.
3.1 LOGICAL CIRCUIT DISCOVERY
We utilize the Edge-Prieg Pruning Bhaskar al. 2024 to build initiative cireuit and logical circuit framework (Chen et al. (0255) to further determine the logical property (AND, OR gates) of edges.
From each dataset, we extract circuit using Eq.2 _ denoted as C, which attempts to recon- struct the functionality of the computational graph G. Specifically, as shown in Figure 3 for the forget set, two distinct basic cireuits, CNs and CDn. are first obtained via noising-based intervention and denoising-baae based intervention, respectively. Then, based on the interaction be-
Forget set Denoising based Interveniion
Retain set
Nosing based intervention
Noting-baser based Denoistng based intervention
intervension
CNS
Cpn
CNs
Dn
tween CN. and Cpn- all edges within them are classified as either AND or OR types, which are subsequently combined to form logical cir- cuit, designated as the forget circuit (C,) which contains all nodes and activation connections circuits.
AND
OR
AND
OR
C
C
Figure 3: The processing from datasets to logical
required for the model produce original responses that are harmful. The same methodology is applied to the retain set to derive the retain circuit (C,) which contains all nodes and activation connections necessary for the model to generate the original responses corresponding to the retain set. We elaborate on the detailed process of logical circuits in Appendix D
'The ADDER gate, unlike binary gates such as AND and OR, represents process where the output's effect is an accumulation of all input effects (we explain it in Appendix DD. In this work, to define the CNF, we simplify the ADDER gate in the forget circuit to OR gate, and the ADDER gate in the retain circuit to AND gate. In Appendix C we prove that such simplifications do not affect the unlearning functionality of the cireuit.
4

---

Published as a conference paper at ICLR 2026
3.2 CIRCUITS TO CNF
We convert the circuit into conjunctive normal form (CNF) order to analyze the specific states of individual nodes during the unlearning task. Specifically, we apply the Tseytin transformation to convert AND and OR gates into CNF sub-expressions:
clauses = (-A V
-C) (BV C)) (if AND B) (if A B)
(3)
clauses (AV (VV
The above CNF conversion transforms C and C2 derived in Section 3.1 into the forget CNF 4 and the retain CNF p,- Both R and @, are composed
variable set that includes nodes (acting as A, B, or C in Eq 3 and output (output, for o and output, for Q,), in which all variables possess binary value. We define the state = (True) as retaining, meaning that these nodes and circuits are expected to persist in the post-unlearning model. We define state (False) as forgetting, meaning that these nodes are expected to forget certain knowledge, and the corresponding circuit is expected not to persist in the post-unlearning model. The final CNF representation is given by
4 44 9, A (-output,) (output,)
(4)
CNF & expects the output of C (i.e., output,) be 0 ((False)), indicating that the functionality of C is removed in the post-unlearning model, and it expects the output of C, (i.c., output,) to be ((True)), indicating that the functionality of the C, retained in the post-unlearning model.
3.3 LOCALIZATION VIA CNF SOLUTION
Nodes with the same name must have identical states or 0) in both o; and Φ because they reflect whether this node should be retained or edited for unlearning. Hence, we directly solve Eq. 4 as satisfiability problem to determine the specific state of each node.
Specifically, when + is satisfiable, all nodes with state of 1 (True) indicate that they are in preserved state, so their retention will not affect the removal of the forget circuit or destroy the retain circuit. They are unlikely to exclusively contain (or even not contain) the information to be forgotten or they have a significant causal effect on the retain circuit. Consequently, we refer to them as "retain nodes". In contrast, all nodes with state of (False) represent nodes that must be removed to ensure the forget circuit is eliminated and the retain circuit remains intact. As result, they are likely to exclusively necessarily contain the information to be forgotten, they do not make critical contribution to the retain circuit; hence, we call them "forget nodes". However, if ④ is found to be unsatistiable, besides retain nodes and forget nodes, there will be conflict nodes. These are nodes that do not have a consistent value across each clause in Eq. + Regardless of whether these nodes have a value of 0 1, they cannot simultaneously satisfy the conditions of removing the forget circuit and preserving the retain circuit. Such nodes indicate that both contain necessary information to be forgotten and have an important causal effect on the retain set's response (We show the case analysis of satisfiable or unsatisfiable situation in Appendix
To address that, we utilize a conflict-driven clause learning SAT solver, as proposed in Zhu al.) (2025); Fleury & Heisinger (2020), to determine the satisfiability of Φ. Additionally, this solver is used to find the values all nodes under the condition of a minimum number of, or no, conflict nodes. Finally, all nodes of models can be divided into one of the following types:
Safe nodes: These include retain nodes (state=1) 1) and nodes that do not appear in 4. Safe nodes are irrelevant to forget set, and do not require any editing.
Forget nodes: These are nodes with value of 0 (False) in d. Forget nodes do not impact the response of the retain set and merely have harmful information which needs be removed to enhance the forgetting efficacy.
Conflict nodes:
These nodes for which no valid assignment exists that satisfies Q. On hand, conflict nodes must be modified to remove harmful information; on the other hand, modifying these nodes can lead decrease in the retain set's performance.
5

---

Published as a conference paper at ICLR 2026
4 UNLEARNING VIA CONFLICT-GUIDER LOCALIZATION
Based on the forget and conflict nodes obtained from localization, we adopt two-stage fine-tuning approach for unlearning. First, we generate forget mask (Mi) and conflict mask (M,) with parameter scale equal to the language mode In Mr, all elements corresponding to the forget node are set to I, while others are set to O. Similarly, in Me. all elements corresponding to the conflict node set to I, while all others are set to O.
In the first stage, we begin by fine-tuning the forget nodes. Since forget nodes do not significantly influence the response of the retain set, only use the forget loss to constrain the fine-tuning. Thus, the unlearning problem in this stage is defined as:
min R (ED ((ysl: &1
(1 (1- 1 Mr) @ 0.)
(5)
where Of represents the parameters of forget nodes and 0. represents parameters of other nodes,
In the second stage, we further fine-tune the conftict nodes. Confiit nodes have a significant causal effect on the responses of both the forget set and the retain set, so fine-tuning requires constraints from both the forget and retain loss. Therefore, the unlearning problem is formulated as:
min (L..,,) 0l (1- (1- f
AB ((y]a,
+ (1-M) ② @6)
where Pg represents the parameters of forget nodes. The retain loss L(y xz; Mr 0- + (1 - Mr) o 0.)
typically mirrors the training loss over the retain set. However, for the forget loss, there are various types of implementations, such as GA (Liu et al.) (002)) NPO (Zhang et al. 2024), and PO (Maini ct al. 2024). In this paper, we adopt PO for main results and show the ablation study about GA and NPO.
EXPERIMENT SETUPS
5.1 UNLEARNING TASKS, DATASETS, MODELS, AND BASELINES
We conduct experiments on three LLM unlearning tasks where each task is assigned four retain datasets. These three tasks are: WMDP Cyber (Li et al., 2024). which focuses on malicious use prevention of LLMs in developing cyberattacks; WMDP Bio (Li al. (202), which assesses the capability to prevent the hazardous knowledge in biosecurity; PKU-SafeRLHF (i et al. 2023). which aims to prevent the toxic content in response to inappropriate prompts from SafeRLHF For the retain set, to better observe specific circuits, we selected four datasets with distinct tasks/capabilities rather than original general corpus: Winogrande (ai2 2019). which involves the task to infer the correct referent of pronoun from semantics; SST-2 (Socher et al. 2013), which includes the task of inferring sentiment categories from a given text; RTE (Recognizing Textual Entailment) (Dagan eet al., 2022). which involves a model determining the entailment relationship between two texts; and Bool (Suzgun et al. 2023), which includes the task of performing logical operations. Model-wise. follow existing practices and use the Zephyr-7B-66
model (Tunstall et al.) 2024) for WMDP
Cyber and WMDP Bio, and LLaMA2-7B (Touvron et al. 2023) for PKU-SafeRLHE
CLUE framework that performs both localization and fine-tuning, so we select two categories of baselines for comparison. The first category includes localization methods for LLM unlearning, such as WAGLE (Jia et al.) (2024), DEPN (Wu et al. 2023), MEMIT (Patil et al. 2023). and PCGU (Yu) ct al. 2023). The second category comprises fine-tuning methods for LLM unlcarning, including
GA Yao al. 2024), NPO (Zhang et al., 2024), and PO (Maini et
2024 (Further details ain
Appendix
'Taking the Zephyr-7B- beta model as an example, each layer contains seven parameter matrices: q. kt v. o. M L.gute L.Pp. up. and M LP There are 32 layers in total, so there are 224 parameter matrices in the mask. The size of each parameter matrix is determined by the actual dimensions of the Zephyr-7B -B- model, such as the g matrix, which has dimensions (4096 x 4096).
'We do not evaluate on the TOFU [Maini et al. 2024) and Who's Harry Potter (Eldan & Russinovich 2023 datasets. This is hecause these datasets require self-fine-tuned model as the baseline, and such fine-tuning affects the circuits of non-target tasks, which leads to lack of credibility in the results.

---

Published as a conference paper at ICLR 2026
5.2 TRAINING AND EVALUATION SETUP
To obtain LLMs post-unleariing, we adopt PO as forget loss which performs better than GA and NPO (Jia et al. (024). All fine-tuning processes are conducted over epochs, with epoch for forget nodes and the rest 5 epochs for conflict nodes. The learning rate is grid-searched at 1 x 10-5 for each dataset. The parameter = 1, and we adopted Adam W (Loshchilov & Hutter 2017) as the optimizer. All experiments were conducted 16 NVIDIA RTX A100 GPUs
We evaluate the performance of unlearned LLMs from forgetting efficacy and retaining utility. Forgetting efficacy adopts accuracy of LLMs post-unlearning the forget set as the main metric. For aligned tendency, we use 1-accuracy to measure forgetting efficacy. Thus, higher l-accuracy indicates better unlearning. Moreover, also provide the efficacy results about other prevalent metrics, such as Membership inference attack (MIA) and Rouge-L, with detailed results shown in Appendix H Next, measure retaining utility with accuracy in both retain set and other non-target tasks. Specifically, for each retain dataset (one of Winogrande, SST-2, RTE, and Bool), we first measure the accuracy on its corresponding test set (retain utility). Subsequently, we measure the average accuracy on a series of unrelated tasks (general utility). These unrelated tasks were evaluated using the Language Model Evaluation Harness toolkit (Gao et al. (202T) and include: ARC-Challenge (Chollet 2019). ARC-Easy (Chollet 2019). BoolQ (Clark et al. 2019).
HellaSwag (Zellers al.) 2019). OpenBookQA (Mihaylov et 2018), Piqa (Bisk et al. 2020
and TruthfulQA (Lin et al. 2021) Details about evaluation datasets are shown in Appendix
EXPERIMENT RESULTS
6.1 CAN CLUE IMPROVE LLM UNLEARNING THROUGH LOCALIZATION?
Table Performance overview of LLM unlearning. "Unlearned Parameter" refers to the percentage of parameters modified, calculated by averaging the percentage of changes in each parameter matrix. "FE" (Forget efficacy) is measured as 1-accuracy and "RU" (Retain utility) measured as accuracy on the test set of the retain set. "GU" (General utility) is average accuracy series of non-target tasks, and specific results can be found in Appendix G
Retain Set
Method
Unleurned Parameter
Winogrande RUF
SST-2
RTE RUF
Bool
FE1
GUT
FE1
RU
GU
FET
GUT
FEF
RU1
GUT
WMDP Cyber
Onigin GA
0.445
0.729
0.624
0.445
0.727
0.624
0.445
0.703
0.624
0.445
0.555
0.624
100%
0.658
0.254
0.332
0.665
0.151
0.312
0.647
0.067
0.367
0.652
0.124
0.325
NPO
100%
0.663
0.342
0.329
0.654
0.247
0.336
0.697
0.264
0.339
0.654
0.237
0,365
PO
100%
0.685
0.567
0.368
0.672
0.632
0.352
0.685
0.471
0.374
0.682
0.317
0,384
MEMI
76.27%
0.669
0.662
0.384
0.673
0.567
0.359
0.675
0.524
0.386
0.690
0.335
0.386
PCGU
86.77%
0.672
0.654
0.382
0.672
0.692
0.362
0.669
0.546
0.375
0.695
0.314
0.376
DEPN
78.82%
0.695
0.817
0.431
0.702
0.731
0.379
0.712
0.457
0.416
0.715
0.429
0.396
WAGLE
90.01%
0.702
0.86
0.442
0.708
0.771
0.384
0.685
0.498
0.413
0.721
0.434
0.387
CLUB
58.16%
0.697
0,992
0.458
0.733
0.91
0.388
0.744
0.786
0,436
0.724
0.505
0.434
WMDP Bio
Origin
0.355
0.729
0.624
0.355
0.727
0.624
0.355
0.703
0.624
0.355
0.555
0.624
GA
100%
0,564
0.064
0.375
0.675
0.124
0.379
0.564
0.125
0.385
0,568
0.214
0.354
NPO
100%
0.571
0.241
0.372
0.671
0.234
0.385
0.574
0.269
0.374
0.572
0.315
0.384
PO
100%
0.605
0.421
0.385
0,685
0.446
0.382
0.589
0.321
0.385
0.585
0.385
0.381
MEMIT
74.29%
0.391
0.672
0.429
0.695
0.619
0.399
0.547
0.395
0.421
0.605
0.421
0,395
PCGU
12,5
0.601
0.662
0.415
0.684
0.627
0.402
0.539
0.402
0.419
0.596
0.413
0.396
DEPN
77.24%
0.605
0.739
0.469
0.701
0.761
0..224
0.546
0.443
0.471
0.599
0.429
0.441
WAGLE
90.025
0.599
0.885
0.480
0.698
0.785
0,426
0,549
0.466
0.472
0.601
0.412
0.441
CLUB
56.19%
0.617
0.995
0.499
0.713
0.893
0.457
0.586
0.528
0.491
0.612
0.501
0.456
PKU-SafeRLHF
origin
0.294
0.841
0.664
0.294
0.764
0.664
0.294
0.795
0.661
0.294
0.514
0.664
GA
100%
0.615
0.124
0.394
0.605
0.095
0.385
0.625
0.147
0.360
0.601
0.054
0.385
NPO
100%
0.605
0.195
0.385
0.612
0.154
0.395
0.614
0.196
0.327
0.616
0,214
0,396
PO
100%
0.625
0.361
0.395
0.623
0.225
0.396
0.623
0.314
0.395
0.625
0,387
0.402
MEMIT
77.62%
0.645
0.345
0.402
0.649
0.395
0.402
0.625
0.43
0.409
0.647
0.339
0.412
PCGU
86.29%
0.639
0.625
0.400
0.633
0.399
0.404
0.639
0.397
0.417
0.639
0.402
0.406
DEPN
74.36%
0.661
0.794
0.412
0.657
0.741
0.429
0.634
0.421
0.415
0.642
0.422
0.429
WAGLE
90.01%
0.655
0.751
0.429
0.663
0.761
0l.434
0.641
0.496
0.421
0.635
0.422
0.411
CLUB
54.88%
0.724
0,956
0.462
0.681
0.883
0.455
0.656
0.682
0.429
0.659
0.536
0.438

---

Published as a conference paper at ICLR 2026
To investigate the performance of CLUE compared to existing methods, we test the performance of existing methods and CLUE on four different retain sets across the WMDP Cyber, WMDP Bio, and PKU-SafcRLHF datasets. Table repports the results of these performances. All experiments were repeated five times, and the standard deviation was omitted as it was consistently less than 0.01.
It is clear that docalization-baser methods (MEMIT, PCGU, DEPN, WAGLE, CLUE) significantly outperform finetuning- -rauunnng--asedd methods (GA, NPO, PO) in terms of both retain utility and general utility. We attribute this the fact that localization can, some extent, filter out important nodes, thereby preventing the capabilities for non-target tasks from being affected. Furthermore, CLUE generally outperforms existing methods in forget efficacy, retain and general utility. The advantage in forget efficacy and utility comes from our more precise localization of forget nodes and conflict nodes, which prevents large number of irrelevant nodes from being fine-tuned.
To further validate the roles of the forget mask (M₁) and the conflict mask (Mc), we conduct ablation studies on the WMDP Cyber dataset using CLUE. We choose SST-2 as the retain set. The specific ablation measures are: "-M;": The fine-tuning process for the forget mask is removed, and only the conflict mask is used for fine-tuning. "-Me", "M The conflict mask is replaced with a full-true mask matrix (in this mask, all values=1). "Me to M,": Fine-tuning is performed first on the conflict mask and then on the forget mask. Additionally, we investigat the impact of different fine-tuning methods on CLUE. The default fine-tuning is "PO+PO", where both the first and second stages use PO for fine-tuning. We then test various combinations where each stage is replaced with "GA" or "NPO".
Table 2] shows that when the forget mask is removed, the forget efficacy decreases the most. This supports the im- portance of forget nodes for information forgetting. Sim- ilarly, replacing the conffict mask also leads to the largest drop in utility, which indicates that the conflict mask is effective at preventing irrelevant nodes from being fine- tuned. Moreover, the GA method results in a significant performance decrease, especially in utility. This is con- sistent with the conclusion in Zhang et al.] (2024) that GA leads to catastrophic forgetting by causing large-scale modifications to node parameters.
Table 2: Ablation with WMDP Cyber as
forget set and SST-2 as retain set.
Method
forget efficacy
retain utility
general
utility
CLUE
0.733
0.91
0.388
-M,
0.045
J0.007
10.011
-Me
10.005
10.264
10.053
Me to Mr
J0.024
10.192
.0.019
GA+GA
.0.021
[0.529
_0.026
GA+PO
J0.012
10.067
10.003
PO+GA
J0.005
J0.191
J0.012
Additionally, in Appendix H we present the results for the experiments in Table [ on the MIA and Rouge-L metrics, which also demonstrate that method consistently out- performs existing approaches. In Appendix [ we show the performance of LLM unlearning varying different forget ratios, which indicates that the conflict-guided localization is beneficial for the unlearning task across different forget ratios.
NPO+NPO
J0.009
J0.093
J0.007
NPO+PO
J0.002
J0.041
J0.006
PO+NPO
J0.004
10.081
10.008
6.2 HOw CLUE PERFORMS WHEN GENERAL CORPUS SERVES AS THE RETAIN SET?
Another question worth exploring is how CLUB performs where the retain set is general corpus. To investigate this, we conduct two types of experiments on WMDP Cyber: 1. CLUE paired with the MMLU dataset (Hendrycks al. 2020) (a general corpus as the default retain set for the WMDP) as the retain set. 2. CLUE paired with multiple datasets from specific tasks as the retain set.
We select 12 specific tasks in total (for option 2): Winogrande. SST-2, RTE, Bool, Induction (Conmy
let al. 2023). 101 (Wang et al.] 2023). Gender Bias (Vig et al. (020). Docstring (Heimersheim & Janiak 2023) Great Than (Hanna et al. 2023), SA (Yu al. 2024), arithmetic (Ghazal et al.) 2013), Reverse (Conmy et al. 2023). We then evaluate CLUE's performance when 1 to 10 of these specific tasks are used as the retain set. Each time, we randomly sample 20 times from the 12 specific tasks based on the predetermined number of tasks. For comparison, we also provide the performance of WAGLE when MMLU is used as the retain set. Figure 4 reports both experiments. The x-axis indicates the number of retained tasks we select from the 12 specific tasks. As its number increases, we plot the forget efficacy, utility of MMLU and general utility with blue lines. While the dotted lines evaluates these three metrics when MMLJ serves the retain set.
8

---

Published as a conference paper at ICLR 2026
IIPARAaation -CUPCAMMELR 2012
000
(a) Forget Effiiaay
(b) Utility of MMLU
(c) General Utility
Figure 4: Performance of CLUE when retain set is MMLU dataset and multiple specific tasks.
co
Shooppoe 2033
Divrggrcce Segatellcaac
44
49
(a) WMDP Cyher
(b) WMDP Bio
(c) PKU-safeRLHF
Figure 5: Circuit Sparsity vs. Circuit Faithfulness and Forget Efficacy.
Upon examining Figure we can see that even when using general corpus like MMLU as the retain set, CLUE still demonstrates higher forget efficacy and nearly equal utility than WAGLE. When this is combined with the conclusions from Section 6.1 we can infer that although MMLU may not provide sufficiently specific task circuits (because it is multi-task dataset), it can still identify enough "Forget nodes" via the forget set. Figure 4 (a) shows that when a specific task is used as the retain set, the forget efficacy is higher than with MMLU, but it decreases the number of specific tasks increases. This makes sense: more specific tasks lead to more "conflict nodes," which makes unlearning critical information more difficult. Figure 4 (b) and (c) both indicate that the number of specific tasks increases, utility shows an upward trend. Because the more specific tasks there are, the more skills capabilities need to be retained. Although the utility
MMLU not high when MMLU provides direct supervision, at 7 number of specific tasks, the general utility can finally reach and surpass the performance when MMLU is the retain set. This also suggests that when a sufficient number of specific tasks are used as the retain set, the model's utility is better preserved.
6.3 UNLEARNING PERFORMANCE VS. CIRCUIT SPARSITY AND FAITHFULNESS
In this section, investigate the relationship between unlearning performance and circuit quality, especially circuit sparsity and faithfulness. Sparsity is calculated by 2. where closer to 0 indicates fewer edges in the circuit, so more sparse. Faithfulness refers to the discrepancy between the circuit output and the computational graph output. We quantify this discrepancy using the Kullback-Leibler (KL) divergence of the output logits, where smaller KL divergence indicates that the circuit's output is closer to the original model's output, thus demonstrating higher faithfulness.
C
Furthermore, trade-off inherently exists: more sparse circuit generally leads lower faithfulness. Therefore, we analyze the forget efficacy of different circuits by controlling sparsity from 0.2 to 0.95. We use SST-2 as the retain set and evaluate the A forget efficacy (i.e., forget efficacy original efficacy) on WMDP Cyber, WMDP Bio, and PKU-SafeRLHF datasets. Figure5Jconfine confirms that as the sparsity decreases, the KL divergence between the circuit and the computational graph gradually decreases, while the A forget efficacy progressively increases. It shows that the denser the circuit, the smaller the functional gap with the computational graph, and the higher the forget efficacy. Empirically and from the results, the performance of unlearning generally reaches the optimal range when sparsity attains level of 0.7. Therefore, all remaining experiments in this paper utilize circuits with sparsity of 0.7.

---

Published as a conference paper at ICLR 2026
6.4 How DOES NODE LOCALIZATION CHANGE AFTER UNLEARNING?
Table 3: The number of forget nodes and conflict nodes before and after the unlearning.
Status
Winogrande
SST-2
RTE
Bool
Forget Set
of
Forget
conflict
forget noder
confiict moder
forget node(())
conffict noder
Forget
comflict
Unlearning
(ode(5))
noder
noderB))
moder
WMDP Cyber
after
before
15.66_..24
19.47-027
29. 29.47=1.3
32 2.77
3.7
8.61
28. 28.01
21.21
11,9440.90
34.71
13.09.1.m
3. 3.90
18 18.942074
01009
16.33+1.72
WMDP Bin
after
hefore
18.49_03
15.62-0.8
25. 2.
3.85
6.57
31.09,1.42
44.08+1.4:
5.022000
9.5610
1.
35.19
3.98ac.07
16 16.91
20.55
12
PKU-Sal-RLHF
after
betore
13.10 0.22
18.77
22. 22.57=1.9
30.65
28. 67777
23.091ams
11.00
15. 15.4940.44
5.00 5.002009
0.731000
31.24
13.30
1.121000
3. 3.9629.05
39.8511.7
26.39 26.39
In this section, we investigate whether forget nodes and conffict nodes change post-unleaming. Table presents the percentages of forget and conflict nodes for different pairings of forget and retain sets, and it tracks their post-unlearning results. We observe that the proportion of forget nodes significantly increases after unlearning, while the proportion of conflict nodes decreases. This suggests that the conflict nodes, which is supervised by both forget loss and retain loss, shift the forget and retain circuit to less overlapping locations. This implies that CLUE's learning on conflict nodes is effective at decoupling the forget circuit from those of other capabilities or mechanisms.
we explore the specific distribution of the forget and conflict nodes to further analyze the response of CLUE different nodes. Detailed results can be found in Appendix In simple terms, nearly all MIPs are conflict nodes, which aligns with the finding that MI.s typically store large amount of information. Furthermore, compared to other methods, MEMIT appears to lack the forget nodes, while WAGLE does not differentiate between forget nodes and conflict nodes.
Finally,
7 CONCLUSION AND LIMITATION
In this paper, we introduce CLUE, a localization framework that uses circuit discovery to identify the circuits for the forget and retain sets and converts them into a CNF By employing a satisfiability solver, we determine the role of each node in the unlearning task, classifying them forget nodes, retain nodes, or conflict nodes. We then provide targeted fine-tuning strategies for each type of nodes. Compared to other localization methods, CLUE offers more precise node localization and significantly outperforms existing methods in both forget efficacy and retain utility.
However, CLUE still has some limitations that can be explored further. First, the circuit is static and cannot dynamically reflect changes in key nodes during the fine-tuning process. Therefore, exploring the dynamics of circuits during parameter fine-tuning is direction for future work. Additionally, when dealing with multiple retain sets, although CLUE can identify which nodes are associated with specific combinations of retain sets, the fine-tuning stage cannot provide targeted fine-tuning solutions for every possible conflict combination. Consequently, developing editing methods other than fine-tuning is another key focus of our future research. Moreover, we must acknowledge that there may exist potential forget datasets from which clear circuit cannot be extracted (possibly due to multi-domain mixing or excessive bias). Fortunately, this issue has not been observed in current mainstream forget datasets; however, this remains a potential risk. Lastly, the inherent scalability problems of circuit discovery also pose a significant challenge, making CLUE difficult to apply to very large-scale language models. Nevertheless, current research (Syed et al., 2024: Lieberum et al. 2023) is attempting to establish the feasibility of circuit discovery methods on these larger models. Thus, the potential to resolve the scalability issues for CLUE remains a promising area of investigation.
ACKNOWLEDGMENTS
This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative, (AISG Award No: AISG-NMLLP-202-0005 the NTU Start-Up Grant (#023284-00001), Singapore, and the MOE AcRF Tier Seed Grant (RS37/24, 0025041-000011 Singapore.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Winogrande: An adversarial winograd schema challenge at scale. 2019.
Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. Advances in Neural Information Processing Systems, 37:18506-18534 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artifficial intelligence, volume 34, pp. 7432-7439. 2020.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE symposium on security and privacy, pp. 463-480. IEEF, 2015.
Hang Chen, Jiaying Zhu, Xinyu Yang, and Wenya Wang. Rethinking circuit compleeeesss in language models: And, or, and adder gates, 2025, URL ULL.Lpppss//aarx1...r//////////2
François Chollet. On the measure of intelligence. arXiv preprint
Christopher Clark, Kenton Lee, Ming-Wat Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 293662 2019.
Arthur Conmy, Augustine Mavor-Parker Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:
2023.
Ido Dagan, Dan Roth, Fabio Zanzotto, and Mark Sammons. Recognizing textual entailment: Models and applications. Springer Nature, 2022.
Eldan and M Russinovich. Who's harry potter? approximate unlearning in Ilms, arxiv. arXiv preprint arXiv:2310.02233, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-oddss Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/trransfommer- cirrcutssrr.....eeee.ee.. circuits.pub/202 pwtpp/2l//rameeeeeee I/framework/mddxx. html.
ABKFM Fleury and Maximilian Heisinger. Cadical, kissat, paracooba, plingeling and treengeling entering the sat competition 2020. Sat Competition, 2020:50, 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9 9, 2021.
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Aroo Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pp. 1197-1208, 2013.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76066 2023.
Stefan Heimersheim and Jett Janiak. A circuit for python docstrings in a 4-layer attention-onio only transformer. In Aligmment Forum, 2023.
Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404. 15255, 2024.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309. /8552, 2023,
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human- -emnan-preference dataset. Advances Neural Information Processing Systems, 36:24678-24704, 2023.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934, 1(2):3, 2023.
Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. Wa- gle: Strategic weight attribution for effective and modular unlearning large language models. Advances in Neural Information Processing Systems, 37:55620-55644 2024.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue. Daniel Berrios, Alice Gatti, Justin D Li, aat-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. In International Conference Machine Learning, pp. 28525-28550. PMLR, 2024.
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.094. 09458, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. falsehoods, 2021.
Truthfulqa: Measuring how models mimic human
Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pp. 243-254. PMLR, 2022.
Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via mabedding-corrupter prompts. Advances in Neural Information Processing Systems, 37:1 18198- 118266. 2024.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0S100 2017.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for 1lms. In First Conference on Language Modeling, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 15:17359-173777 2022.
Samiyuru Menik and Lakshmish Ramaswamy. Towards modular machine learning solution develop- ment: Benefits and trade-offs. arXiv preprint arXiv:2301.09755 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018.
Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp. 931-962. PMLR, 2021.
Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from 1lms? objectives for defending against extraction attacks. In The Twelfth International Conference on Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few-shot unlearners. In International Conference on Machine Learning, pp. 40034 40050. PMLR, 2024.
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-basee language models. arXiv preprint arXiv:2407.02644 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631 Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL htttp:://wwtt hwtp://wwwwb.b... wttp.//www.......ooooooooo//o///
htttps:/wwww.cl.wbb.oo
11770
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain--ffthhuur can solve them. In Findings of the Association for Computational Linguistics: ACI. 2023, pp. 13003-13051, 2023.
Aaquib Syed, Can Rager, and Arthur Conmy, Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 407-416, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 20000088..908. 2023.
Grigori S Tseitin. On the complexity of derivation in propositional calculus. In Automation of reasoning: 2: Classical papers on computational logic 1967-1970, pp. 466-483. Springer, 1983.
Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. In First Conference on Language Modeling. 2024.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and
Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems,
12401, 2020.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. In The Fleventh International Conference Learning Representations, 2023.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. In The 2023 Confer- ence on Empirical Methods in Natural Language Processing, 2023.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37: 1a11100525--554777 2024.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguisties:
ACL 2023, pp. 6032-6048, 2023.
Lei Yu, Jingcheng Niu, Zining Zhu, and Gerald Penn. Functional faithfulness in the wild: Circuit discovery with differentiable computation graph pruning. arXiv preprint arXiv:2407.03779 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-48000 2019.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024.
Jiaying Zhu, Ziyang Zheng, Zhengyuan Shi, Yalun Cai, and Qiang Xu. Circuit-aware sat solving: Guiding edel via conditional probabilities, 2025. URL :ttps://azzee http://wwwwvaggaaaaaaa.aa abs /508
04235
13

---

Published as a conference paper at ICLR 2026
A REPRODUCIBILITY STATEMENT
Our code is publicly available in an anonymized repository linked in the abstract, which contains the complete implementation of CLUE and related instructions (see README). Furthermore, each theoretical step presented in the paper is supported by citations to relevant research (Chen et al. 2025] Zhu et al. 2025). For clarification, we provide illustrative examples in Appendices C C D All datasets required for our experiments are available on Hugging Face and are described in detail within the manuscript (see Section 5)
B THE USE OF LARGE LANGUAGE MODELS
In the preparation of this paper, we utilized a large language model (LLM) as an assistive tool toenhance the quality of our writing and presentation. The LLM's role strictly confined to refining the manuscript's writing and formatting, without generating any core scientific content or data.
C EXAMPLE FROM CIRCUIT TO CNF
Following the notion of logical circuits introduced in (Chen et al., 2025). we construct a toy circuit as il- lustrated Figure 6 The semantics of the three gates are defined as follows:
Input
A2
A
A4
5
A6
AND gate. The receiver node is activated if and only if the activation states of all sender nodes equal I; otherwise, the receiver node remains inactive.
AND
OKT
B1
B2
B3
C1
C
OR gate. The receiver node is inactive if and only if the activation states all sender nodes equal O; otherwise, the receiver node is activated.
Output
ADDER gate. Unlike the previous two gates, the receiver node in this case admits multiple intermediate activation states rather than a binary aetivated/inactivated outcome. Each intermediate state corresponds the contribution of a sender node, with all sender nodes treated as equally weighted and independent. For example, only one sender node has activation state equal two sender nodes are active, then the receiver node has activation state 2. and so on. In general, the logical relation expressed as receiver node = sender node; + sender node; +....
Figure 6: The toy circuit with AND, OR, and ADDER gate.
then the receiver node has activation state
We use an activation state of to indicate that a given node should be retained and state of 0 to indicate that should be unlearned. To preserve the maximal capability of the retain set, the gates are governed by the following rules:
AND gate: all sender nodes must be retained.
OR gate: at least one sender node must be retained.
ADDER gate: all sender nodes must be retained; otherwise, the receiver node fails to reach its optimal activation value, thereby impairing the model's capability on the retain set.
Consequently, we define both the AND gate and the ADDER gate conjunctions (denoted by the symbol A), while the OR gate is defined as disjunction (denoted by the symbol V). this definition, the ADDER gate only has two states (0/1). For the toy circuit in Figure 6 this leads to the following conjunctive normal form (CNF):
output = CIACAAABBBAAA
B3) A
Analogously, for the forget set, we represent the state each gate using a negation operator (-), since the capability of the forget set ideally satisfied when all nodes in the circuit have activation
14

---

Published as a conference paper at ICLR 2026
state equal O. In this setting, the AND gate is expressed disjunction. The reason is that, to unlearn the capability associated with the forget set, it suffices to unlear any one of the incoming edges of an AND gate. For example, in Figure 6 we have -C₁ -B, V V-B.
By contrast, both the OR gate and the ADDER gate are expressed as conjunctions. This is because all of their sender nodes must be unlearned to ensure that the capability of the entire gate is forgotten. For instance, -B3- = -A A A
Therefore, if Figure 6 corresponds to the forget set, the resulting conjunctive normal form (CNF) can be written as
output =- A -C2 A (-B: -B2) ABa--BaA (-AI V -A V -A,) A (~A2 -A, AA) A (- -Aa) A A-5 A -A
(8)
Evidently, the CNF corresponding to the retain set is always satisfiable when all sender nodes satisfy state=1, and the CNF corresponding to the forget set always satisfiable when sender nodes satisfy state=0.
The above analysis proves that the forget circuit, the propositional logic of the ADDER gate the same as that of the OR gate, while in the retain circuit, the propositional logic of the ADDER gate is the same as that of the AND gate. Therefore, in the actual implementation, we convert the ADDER gate in the forget circuit into an OR gate and the ADDER gate in the retain circuit into an AND gate. Then, we use the logically complete Tseytin transformation to convert them into CNF.
D DETAILS OF LOGICAL CIRCUIT FRAMEWORK
At first, we systematically introduce three fundamental circuit logic types: the AND gate, OR gate, and ADDER gate (Chen et al. 2025).
Definition 1. We assume a common paradigm which a receiver node B, which is connected by more than I sender node A1, A2.... .... For any edge A; B, represent the activation state of a node. Specifically, A, = 0 indicates that node A₃ is removed, ablated, deactivated, whereas A, indicates that node A; is retained and active. When the sender nodes are ablated, the effect of node on the output exhibits three distinct patterns, which as follows:
binary values '0' and '1' to
AND: All sender nodes satisfy an AND logical relationship with the receiver node, i.e., B = A1 A A2 A.... In this case, node B exerts significant effect on the output only if all of its sender nodes are retained. If even a single sender node is ablated, the effect of B on the output is nearly eliminated.
OR gate: All sender nodes satisfy an OR logical relationship with the receiver node, i.e., B A V Ag V.... In this case, node B always exerts significant effect on the output if one more of its sender nodes are retained. Only if all sender nodes are ablated, the effect of B on the output is nearly eliminated.
ADDER gate: all sender nodes satisfy an ADDER logical relationship with the receiver node, i.e., B = Ai A2 .... In this case, node B exhibits its maximal effect on the output only when all of its sender nodes are retained. If any single sender node is ablated, the effect of B on the output substantially diminished; when all sender nodes are ablated, B's effect on the oulput is reduced to zero. Accordingly, define the state of B as taking values 0, 1,2...., where the total number of distinct states equals the number of sender nodes.
Theoretical analyses support the view that brising-based intervention is capable of recovering a complete AND gate but fails to recover a complete OR gate, whereas denoising-baser intervention demonstrates the opposite pattern (Heimersheim & Nandal 2024). This asymmetry is straightforward to interpret. The noising-based intervention procedure corresponds to the transition from a clean activation state (state = 1) to a corrupted activation state (state regarded as being initialized with activation states equal to 1, any transition to state induces significant change in the effect of AND and ADDER gates on the output. Consequently, noising- based intervention can reliably identify AND and ADDER gates.
= 0). Since all gates can be
15

---

Published as a conference paper at ICLR 2026
The denoising-based intervention first performs the corrupted run in the computational graph, and then replaces the corrupted activations with the clean activations. Those activations that lead to significant changes in the output (ÿ) consist of the circuits. denoising-brie basen intervention thus has the following objective:
arg min ((..))e (Pig ( pe x)),, s.t. 1 - |C|/|g)
(9)
C
Conversely, the denoising-based intervention procedure corresponds to initialization with activation states equal to O. In this case, any transition to state produces a significant change in the effect of OR and ADDER gates on the output.
Therefore, we denote the circuit constructed under the noising-based intervention strategy CNs. and the one constructed under the denoising- -aased intervention strategy as Cpn- Based on the above set-theoretic relationships between CNs and CDa. extract subsets of edges corresponding to AND, OR, and ADDER gates as follows:
AND gate (CAND): edges that present in Cs. but absent from CDa- ORR gate (CoR): edges that are present in Cpa but absent from Cxs-
ADDER gate (CADDER): edges that are shared between CNs and CDa-
Therefore, we propose combined Ns+Dn approach recover logically complete gates. This method compatible with a wide range circuit discovery algorithms, introduces minimal additional computational overhead, and enables clear and effective separation of the three types of logic gates. Ns+Dn has the following objective:
arg min ( (x. D(pa (lec(,,, R)) + D(po(i) C
s.l. 1 |C|/|g] 2 (10)
Finally, we simplify the ADDER gate in the forget circuit an OR gate, and the ADDER gate the retain circuit to AND gate, as shown in Appendix C
E CASE ANALYSIS OF CNF-SATISFIABILITY PROBLEM
E. SATISFIABLE SITUATION
For example, similar to Eq. let G: output, A AND B and C, output, A OR B. Then,
Φ =(-A V BB V output,) A (A V -output,) A (B -output,) ^ (A v BV -output (-A V output,) A (-B output,) (-output,) A (output,)
(11)
being satisfiable would require the value assignment for [A, B, output, output,] to be either [o, 1, 0, 0,, 0, 1). Both of these outcomes ensure that C is corrupted while C, preserved. In this case, one of and B must be changed while the other is preserved. For instance, if 4 = 1 and B preserving A has no impact
violating or maintaining C. Therefore, and correspondingly, B is forget node. Conversely, if A = O and B = 1, then becomes the forget node and B the retain node. This is intuitive: C,, A and B are related by AND logic, so neither A nor B exclusively or independently contains the information to be forgotten. Similarly, in C, A and B are related by OR logic, which means that both A and individually have a significant influence on output,
a retain node,
E.2 UNSATISFIABLE SITUATION
For example, let C. output, = A OR B and C, output, = B AND C. Then,
+ =(A B V -output,) A (-A V output,) (-B V output,) ^ (-B V -C' V output,) (B v -output,) A (CV oottput,) (-output,) A (output,)
(12)
being satisfiable would require the output, = 0, and thus A= 0, B= 0. However, & would also require the output, 1, and thus B= 1, C= 1. In this instance, must be 0, which categorizes it
16

---

Published as a conference paper at ICLR 2026
forget node. This is because the state of A alone is sufficient determine the outcome of Ct. meaning exclusively and independently contains the information to be forgotten. Analogously, C must be identifying it as retain node, it is independent of C and thus requires modification. B, however, is a conflict node: to remove Cr, it must be changed (value=0), yet to preserve C,, must be maintained ( (value=1).
F DETAILS ABOUT EXPERIMENT SETUPS
F. MODEL CONFIGURATIONS
For the WMDP task, select the original Zephyr-7B-bec as the pretrained model. For the PKU- SF task, we selected LLaMA2-7B as the foundational model for our study. All experiments were conducted on 16 NVIDIA RTX A100 GPUs. Each experiment takes approximately minutes per 100 steps. We adopt the same rejection-basee answers designed by Jia et al. (2024).
F.2 UNLEARNING CONFIGURATIONS
All fine-tuning are conducted over 6 epochs, with I epoch for forget nodes and the rest 5 epochs for conflict nodes. The learning rate is grid-searched at 1 x 10-5 for each task and datasets. The parameter À is set to for each method across all tasks, and we adopted AdamW (Loshchilov & Hutter 2017) as the optimizer.
F.3 BASELINES
DEPN (Wu et al. 2023 (Detect and Edit Privacy Neurons) is framework designed to safeguard against privacy leakage in pretrained language models by localizing and editing specific neurons. The method's i core localization component is novel privacy neuron detector that uses wradient-based attribution technique. This detector computes a privacy attribution score for each neuron to quantify its contribution to the model's leakage of private information. This is achieved by calculating the cumulative gradient of the output probability with respect to the neuron's activation value, as the activation is gradually changed from zero to its original value.
WAGLE (Jia et al. 2024) (Weight cnabbution-guidern LLM Unlearning Framework) is a framework that pinpoints the most influential weights for unlearning through strategic weight attribution method. The method frames the weight attribution problem as a bi-level optimization (BI.)) problem, which allows it to balance unlearning efficacy with utility preservation. The core of the localization process is the derivation of a closed-form attribution score for each weight, calculated using the implicit gradient from the BLO problem. This score's value is determined by combining the gradients from both the forget loss and the retain loss.
PCGU (Yu al. 2023) (Partitioned Contrastive Gradient Unlearning) is gray-box method for unlearning social biases by localizing the specific weights responsible for encoding them. The method's localization strategy is based on comparing gradients from "contrastive sentence pairs," " which are sentences that are minimally different in specific domain, such as gender. PCGU first partitions the model's parameter set into discrete weight vectors or blocks. then computes the gradients for each sentence in pair with respect these weight blocks. By measuring the cosine similarity between the gradients of the two sentences, it identifies the weight blocks that are most relevant to the targeted bias (i.e., those with the lowest cosine similarity between their gradients).
MEMIT (Patil et al. 2023) ((Msss-Fe (MasssEEdiinng Memory in a Transformer)) addresses the deletion of besso based intervention method. This approach relies on the assumption that knowledge is stored in specific, localized components of the network, and can be identified via causal mediation.
factual information by causal tracing, denoising
GA Yao et al. 2024) (Gradient Ascent) encourages the response of the LLM post-unlearnnng to deviate from its original response within the training set.
NPO (Zhang et al. 2024
(Negative Preference Optimization) specifies the forget loss as the loss of direct preference optimization by treating the forgotten data exclusively negative examples. The
17

---

Published as a conference paper at ICLR 2026
NPO loss outperforms the GA loss due to its improved stability, avoiding catastrophic collapse in forgetting and utility preservation during optimization.
PO (Maini et al. 2024 (Preference Optimization) is also inspired by DPO but introduces targeted unlearning responses such as 'I don't tt know' or responses stripped of sensitive information, treating these exclusively positive examples for preference alignment.
F.4 RETAIN SET
We select a series of specific tasks as retain set: Winogrande, SST-2, RTE, Bool, Induction, IOI, Gender Bias, Doestring, Great Than, SA, arithmetic, Reverse. We show the examples of each task in the T
Table 4: An overview of the datasets of specific tasks.
Task
Example
Label
Winograde
John moved the couch from the garage to the backyard to create space. The is small.
garage
SST-2
hide new secretions from the parental units
negurive
No Weapons of Mass Destruction Found in Iraq Yer.
RTE
Weapons of Mass Destruction Found in Iraq.
not entailment
Bool
[True AND Thue) OK False
True
Induction
Veronn Dursley and Petunia Durs
TOI
Whet When Jobn and Mary went to the store, Mary gave bottle of milk to
John
Gender Blas
Evan really great friend, isn't
he
def f(self, files, obj. state, size, shape, option);
pparam state: performance analysis
Docstring
shape
:param size: pattern design :param
Great Than
The war lasted from 1517 to 15
18
Many girls ansuited
thenselves
arithmetic
12 plus 18 equals [0, 3, 2, 1]
30
Reverse
[1. 2, 0]
G RESULTS ON DETAILED GENERAL UTILITY
In this section, we report the specific accuracy for all non-target tasks, which can be found in Table 5
Table 5: Performance on specific non-target tasks. We report the accuracy metric with WMDP Cyber as forget set.
Methed
foslg
(radlegs...
Lit
112635
12-1144
012112
0.2021
IF4571
02211
TUS022
NPO
0 1.2647
0.2194
0.3317
0.5510
0.2117
0.310
0.6044
018419
(1057)
0.2549
013517
0.4314
11 113266
PO
0 0.3646
0.5486
0.5017
0.2017
0:2407
0.5817
0.5411
0.061
0.2519
114217
0.5517
Winegaade
POGL
0
0.2533
0.5907
0.5519
0.2517
0.3499
0.5719
0.095
DEPN
0.2401
0.2847
04414
06217
0.401
0.326
0.685
08882
013201
0.338
0.461
0A381
WAGE 9:
81 5337
8.5336
0.5611
08136
0.3668
0.
0.677
07782
0003
0.737
014563
0.1123
B 6171
CLUB
014124
0.2387
0.5022
03357
037255
0.30
07474
01259
0.253
0.2025
05101
C1A
03117
012211
01735
03379
00014
PO
0 2535
0.2547
0.417
0.3649
0.4555
0.5675
0.031
MEMIT
0.2617
0.2549
04219
04137
0.3449
0.2575
0.4581
0.585
0.0044
0.2947
04137
0.5633
0.309]
POGIL
0.2517
0.221
0.4739
8.4018
0.2814
012519
0.2575
0.3977
00619
022733
04473
0.5829
6.362
DEIN
0.2617
0.5933
6.019
0.679
00463
0.3255
CLUB
0.3947
0.2933
0335
09914
0.4193
0.22
01451
0.666
012055
1120011
1C234
B2415
012275
123770
127200
122007
NPO
81 077057
0.2217
0431
0.7977
B1517
01159
0.9037
018877
04935
DAXITT
8 17994
PO
04997
06007
83441
0.5117
011077
0.400
0
0.143
POGL
0.5549
0.317
0.3429
0.5517
0.503
0.004
0.375
DEIN
0 0:2447
0.502
044201
0.584
0.575
0.184
0.6545
015487
013561
0.2354
0.4822
DASS
WAGEE
0.5744
0.2834
B.4997
0.9957
015565
0L196
0.6288
(U025)
0.2558
01495
06005
CLUE
17:0014
0.2722
01007
01500
0.015
0.335
0.3097
0.222
15002
CIA
0.2517
0.3044
01437
0,4873
0.0231
PC
0.3134
0.2549
0.6877
0.885
0.3486
0.3544
016991
015367
IL0051
0.355
Brod
PCOLL
MINMTT
13 133365
0 333057
0.3515
0.2254
0.535
0.0015
0.471
0.707
0.741
0.5944
0.127
0.106
015937
0.5937
115317
0.5936
[1281]
(132W9
0.1633
0.134
11380
0.3994
0.163
0.4036
01 13567
6.5769
DEPN
0
0.2527
0.6199
0.5073
0.2667
0.99
0.4957
0-18
04553
0.317
03621
(0044)
18

---

Published as a conference paper at ICLR 2026
H RESULTS ON MIA AND ROUGE-L
As introduced by Jia et al. (2024). membership inference attack (MIA) is evaluated by the area under the ROC curve using Min-20% Prob to detect if the provided text belongs to the training testing set. We apply MIA to the forget set; thus, a higher MIA score indicates a higher confidence in predicting that the forget data point does not belong to the training set. Moreover, Rouge-L recall is also measured over the forget set. A lower value corresponds to better unlearning. The metric 1 1-Rouge-L is also used for ease of performance averaging. We show the results of MIA and Rouge-L in 6
Table 6: Performance overview of LLM unlearning with 1-accuracy, MIA, Rouge-L as metrics for forget efficacy.
Unleamed
Winogrande
SST-2
Ronge-.
T-rrpnncy
Kengo111
Fascimsy
Roagee
Rongo-
WMDF
Origin
000159
MIMIT
76.27%
0.354
0.0799
00227
0.0691
0.3667
DEPN
78.82%
07025
0.3558
0.6553
0.756
0,3046
WAGLE
50.01%
0.4509
0.081
0.3619
0.6851
0.6418
0.3618
0,3746
58.16%
0.738
0.7677
WMDP
Origin
0.3551
0,4119
00022
0.561
03167
00122
06751
0,624
0,701
09234
0.3635
0,0018
03128
09543
0.3548
06853
0.516
14.20%
0.5647
0005S
0.30K
022518
0.3641
PCOU
65.12%
06840
0.138
0313
0.5384
0.3156
0.3561
DEPN
77.349
02008
0.6033
04518
86638
0.3181
0.5313
WAGLE
50.00%
D6984
0.5851
0.4511
0.5001
0.6913
0.3384
0.4528
56.19%
0,6299
0.300
miginn
0:304
02211
0100
0.304
04319
09211
06655
0.9617
0.3519
0.6250
0.3412
0.3622
0.0199
0.9247
NPO
B6129
0.3367
0.6144
0.9255
0.3359
0.9366
PO
0.4266
B.235
0.4219
0.6238
0.6731
0.625
MIMMIT
77.0%
0.2399
0,6255
0,600
0.V13A
PCOU
66.20%
0.3689
0.6390
0.5645
DEPN
74.36%
0,0617
0.6173
0.3195
0.6555
0,6628
0.3816
0.6347
0.6319
0.4657
0.629
0.5957
0.4138
WAGLE
90,011
0.6284
0.3326
0/6677
0.5973
0.6219
0.611
0.6642
0.4399
0.5173
0,3907
54.38%
I SCALABILITY AND ROBUSTNESS OF CLUE AT VARIOUS FORGET RATIOS
In this section, we investigate the performance of CLUE and other localization methods under varying forget ratios. Specifically, we define the forget ratio as the ratio of modified parameters and examine the performance of our localization methods in terms of forget efficacy, retain utility, and general utility for forget ratios of 10 0.2, 0.5, 0.8, 0.9]. For our experiments, the retain set used SST-2.
Figure 7 shows the performance on WMDP Cyber, WMDP Bio, and PKU-safeRLHF. It is evident that as the forget ratio increases, the unlearning efficacy improves; however, the utility trend is more volatile and non-monotonic. Nevertheless, CLUE consistently outperforms other localization methods. This demonstrates that our identification of both forget nodes and conflict nodes is beneficial for the unlearning task across all tested forget parameter ratios.
J DISTRIBUTION OF DIFFERENT NODES
In this section, we investigate the proportion of different parameter types within forget nodes and conflict nodes. For a comparative analysis, we also include the parameter distributions from MEMIT and WAGLE. MEMIT uses method similar to circuit discovery to identify important nodes but relies exclusively on a denoising-
intervention. As has been confirmed by existing work (Chen) et al. 2025), such circuit is incomplete, lacking the necessary logical reasoning nodes. WAGLE, on the other hand, employs a gradient-base based weight attribution method, which often makes difficult to discover equivalent paths within OR gates.
Figure & illustrates the node distributions for four retain sets, with the 7B-byr- model and the WMDP Cyber dataset serving the forget set. MLPs constitute the largest proportion, as
19

---

Published as a conference paper at ICLR 2026
c0n2a of
MAGLE
PCGU
0.4
0.6
2.8
522
0.4
0.5
2.8
0.2
0.4
0.5
rorgetrodo
Ibrget rodo
Iorgetrodo
(a) WMDP Cyber FE
(b) WMDP Cyber
(c) WMDP Cyber GU
0.8
trattfoar
0.7
0.55
1.-
0.0
PENT
8.5
0.0
0.
0.4
Ergarralaa
Eorperrala
(d) WMDP Bio FE
(e) WMDP Bio RU
(1) WMDP Bio GU
0.7
25
0.0
0.8
0.00
0.41
2.7
2.6
2.4
3.6
Coec DEER PENT
2.5
0.55
0.35
3.4
155
2.
0.3
firpetrado
PKU safeRLHF FE
(h) PKU safeRLHF RU
(i) PKU safeRLHF GU
Figure 7: Preference of LLM unlearning various forget ratios, FE means forget efficacy, RU represents the retain utility, GU represents general utility.
20

---

Published as a conference paper at ICLR 2026
MENIT
100
WAGLE
100.00
100
100.0
04.04
CLUE (C (Confict Neuron]
53.5 23
Percentage
80
CLUE orget Neuron)
Percentage
80
60
53
56.3
60
53
$3.23
40
41.22
43.50
20.32
Do 2610
MLP
Parameters
Parameters
(a) Winogrande
(b) SST-2
1000
100
55 13..4
100.22
Percentage
80
Percentage
80
21,01466
20
19:4
20.07
20
39
attn_q
atn k
attn V
att_go
MLP
attn
ottn k
attr Parameters
attr .o
MLP
Parameters
(c) RTE
(d) Bool
Figure 8: Nodes distribution of Zephyr-7B- -B-btet- model in WMDP Cyber.
they generally considered the most information-rich memory units. An interesting pattern also emerges: outside of the MLPs, the number of nodes identified by MEMIT is similar to the number of conflict nodes found by method (CLUE). In contrast, the number of nodes found by WAGLE is comparable to the sum of both forget nodes and conflict nodes identified by our approach. This observation further confirms our viewpoint. Due its lack of circuit completeness, MEMIT fails discover forget nodes from a sufficient logical structure. Meanwhile, WAGLE, by not considering the influence of causal effects, cannot discover OR gates and thus includes all common nodes.
K
THE BENCHMARK OF COMPUTATION COST
To evaluate the time and computational resource expenditure of our method, particularly in comparison to existing approaches, we conducted assessments on the WMDP Cyber and WMDP Bio datasets using the Zephyr-7B-bets model. The experimental environment was equipped with 16 NVIDIA RTX A100 GPUs. For comparison with existing methods, selected GA representative fine-tuning method and WAGLE as representative localization method. The Table and shows the time required for localization and fine-tuning for these methods. It demonstrates that method does not introduce significant time overhead. However, when combined with the performance forget efficacy and retain utility presented in Table our method exhibits the highest cost- sitteffffetiveeess. Specifically, we evaluated the computational time of our approach on two circuit discovery methods: Sddee-runinng which fits the circuit by adding differentiable mask, and EAP, which directly computes the circuit using a first-order Fourier transform. Furthermore. the time required for SAT solving was consistently well-defined initialization process. Consequently, the
under one minute, largely attributable overall computational overhead of our entire pipeline is comparable to that of existing approaches. In particular, when EAP is employed as the circuit discovery method, the runtime is significantly lower than the average of current methods.
21

---

Published as a conference paper at ICLR 2026
Table 7: Computation Cost in WMDP Cyber.
Localization
Method
Fine-tuning
All
Circuit discovery
SAT Solving
All
GA
4.8h
4.8h
WAGLE
0.5h
3.2h
3.7h
Ours (Edge-Pruning)
2.1h 0.1h
0.01h
2.11h
3.1h
5.21h
Ours (EAP)
0.01h
0.11h
3.1h
3.21h
Table 8: Computation Cost in WMDP Bio.
Localization
Method
Fine-tuning
All
Circuit discovery
SAT Solving
All
GA
3.5h
3.5h
WAGLE
0.4h
2.9h
3.3h
Ours (Edge-Pruning)
1.7h
0.01h
1.71h
2.9h
3.61h
Ours (EAP)
0.1h
0.01h
0.11h
2.9h
3.01h
22


Reviews:
[{"review_id": "CvKPp64ev9", "reviewer": "Reviewer_PHgS", "summary": "This paper presents a new categorization of neurons that could benefit downstream neuron-based unlearning methods. The desiderata include forgetting as much harmful information as possible while retaining as much non-harmful information as possible. The authors use a method inspired by contemporary circuit-discovery methods to identify three types of neurons: forget neurons (related to the targeted forgetting information), retain neurons (corresponding to the expression of retained information), and conflict neurons (involved in expressing both). They then propose a new two-step unlearning method that (1) edits the forget neurons to erase the unwanted information and (2) fine-tunes the conflict neurons to remove the targeted information while strengthening the retained information.\n\nOverall, the method performs well across all major benchmarks. Unfortunately, the paper is not very well written. Many of the concepts are not clearly explained, and the paper contains numerous typos and errors. Most importantly, the paper has a major conflict in attempting to connect circuit discovery with neuron-based unlearning approaches, as circuit discovery identifies circuits of components rather than individual neurons. See the weakness section for more information.", "questions": "Pleas see the weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- **Good performance**: The method performs reasonably well across standard benchmarks, showing that it can be applied successfully in practice.\n- **Interesting categorisation of neurons**: The proposed division into forget, retain, and conflict neurons is conceptually distinct and could help clarify certain aspects of unlearning.", "weaknesses": "### **Neurons vs. Components**\nA major issue of the paper is the careless use of the terms neuron and component. In short, circuit discovery methods find circuits of model components (input/output nodes, attention heads, MLPs). Each of these components contains many neurons. The interface between model components and neurons is not trivial, especially given previous work on unlearning that focuses on MLP neurons. What exactly do you mean by a circuit of “neurons”? Do you count every neuron in the components that make up the circuit as such, or do you do something different?\n\nFor example, the following sentence suggests that circuit discovery methods find a circuit of neurons, which is clearly not the case:\n>  The circuit extracted from the forget set—the forget circuit (Cf)—contains all neurons and activation connections required for the model to\nproduce original responses that are harmful\n\nThe authors need a thorough and clear review of their description and usage of neuron, component, and circuit terminology. This is not a minor nitpick, as these terms lie at the very centre of the paper. The incorrect use of terminology makes it really hard to precisely understand what the authors are doing.\n\n### **Writing and Typos**\nUnfortunately, the paper is not very well written. The key terms are not selected well. For example, the authors use the term “conflict” to refer to neurons that serve both the forget and retain sets. This is not a conflict, as conflict suggests an opposition or inconsistency between two objectives, whereas these neurons simply participate in representing both. A clearer and more precise term would help avoid confusion and make the intended mechanism easier to understand.\n\nFurthermore, the paper is also littered with typos and errors. To name a few:\n- Line 046: ... ents (**Wu et al.**; Yu et al., 2023). *The citation entry doesn't have a year. Although the paper is from ICLR 2023.*\n- Line 047: ... (**Patil et al.**;. ... *The same problem appear across multiple places in the paper.*\n- Line 132: task-relevant behavior **( or** mechanism/capability) ... *Extra leading space in brackets.*\n\nAs such, I think the paper needs some major revisions. It seems like an unfinished manuscript not ready to be accepted into a conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3oIzhYFJQ", "reviewer": "Reviewer_VCVn", "summary": "This paper addresses the limitation of localization-based methods for LLM unlearning, and proposes a new method called CLUE. CLUE leverages an existing mechanistic interpretability method to achieve more precise unlearning without over-forgetting. Extensive experiments confirm that CLUE achieves superior performance on standard benchmarks and metrics.", "questions": "1. As mentioned in Section 3.1, the circuit discovery algorithm extracts the **forget circuit** and **retain circuit**. However, it is unclear how the algorithm finds the forget and retain circuits, and how to distinguish them. Further explanations are needed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors provide code, which enhances the reproducibility of the work.\n2. Extensive experiments and baseline comparisons are conducted to demonstrate the effectiveness of CLUE across multiple benchmarks.\n3. The proposed method leverages a circuit discovery tool, which improves the interpretability of the LLM unlearning process.", "weaknesses": "1. **Introduction to prior works is not enough.** A detailed introduction to circuit discovery algorithms and brief introduction to Tseytin transformation should be provided to make the paper self-contained.\n2. **Generality of the proposed method is unclear.** As mentioned at the bottom of the p.5, fine-tuning affects the circuits of the model. This affects the generality of the proposed method. Does CLUE require rerunning circuit discovery after every fine-tuning step? This computational overhead needs to be discussed in the limitation section.\n3. **Highly dependent on the quality of the circuit discovery algorithm.** CLUE’s performance is tightly coupled with the quality of the underlying circuit discovery algorithm (e.g., Edge-Pruning used in the paper). If Edge-Pruning fails to generate meaningful or accurate circuits, the effectiveness of CLUE is questioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hsb9eoyytX", "reviewer": "Reviewer_tdd5", "summary": "This paper addresses the challenge of LLM unlearning removing undesirable knowledge while preserving unrelated capabilities. The authors note that existing localization based methods, such as WAGLE and MEMIT, struggle to disentangle neurons responsible for forgetting from those involved in retaining information. To overcome this limitation, they propose CLUE (Conflict-Guided Localization for LLM Unlearning), a framework that integrates circuit discovery and logical reasoning. Specifically, CLUE first uses circuit discovery to extract distinct “forget” and “retain” subcircuits, then transforms these circuits into Conjunctive Normal Form (CNF) using the Tseitin transformation. The CNF satisfiability problem is subsequently solved to classify neurons into three categories: forget, retain, and conflict. Finally, the framework applies a two-stage fine tuning strategy first fine-tuning the forget neurons using a forget loss, followed by fine-tuning the conflict neurons with both forget and retain losses. Experimental results on benchmark datasets such as WMDP-Cyber, WMDP-Bio, and PKU-SafeRLHF, using Zephyr-7B and LLaMA2-7B models, demonstrate that CLUE achieves superior forgetting efficacy and retain utility while modifying significantly fewer parameters than previous methods like GA, NPO, PO, MEMIT, DEPN, and WAGLE.", "questions": "For multiple retain sets, could a multi-objective SAT formulation handle overlapping conflicts more gracefully?\nIs there a failure case where CNF becomes unsatisfiable for large overlapping circuits, and how is it handled?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear theoretical grounding and stepwise formulation (Eq. 1–6).\nComprehensive experiments across tasks and metrics (1-Acc, MIA, ROUGE-L).\nDetailed ablations isolating the effect of forget/conflict masks and finetuning strategies.\nInsightful analysis of sparsity vs. faithfulness (Fig. 4) and neuron-type shifts post unlearning (Table 3).\nPublic anonymized code and strong reproducibility statement.", "weaknesses": "Circuits are treated statically; dynamic updates during finetuning could better reflect neuron shifts.\nComputational overhead of SAT solving and circuit extraction is not benchmarked.\nLimited discussion on scaling to models > 7 B parameters.\nRetain set selection sensitivity deserves deeper analysis (beyond Figure 3 trends).\nLanguage polishing could improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpRdY8ARfQ", "reviewer": "Reviewer_J9hV", "summary": "The paper introduces CLUE, a new framework designed to improve the precision and effectiveness of large language model (LLM) unlearning. Existing localization-based unlearning methods identify important neurons for modification but fail to distinguish between neurons responsible for forgetting and those critical for retention. This entanglement often leads to over-forgetting or incomplete erasure of target information. To solve this, CLUE leverages circuit discovery, a mechanistic interpretability approach, to map neuron interactions as circuits. It then converts these into CNF and uses Boolean satisfiability solving to disentangle neurons into three categories: Forget neurons, Retain neurons, and Conflict neurons.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The application of mechanistic interpretability and circuit discovery to LLM unlearning is novel and conceptually well-motivated, bridging interpretability with model editing.\n- The explicit separation into forget, retain, and conflict neurons allows for more precise control and understanding of unlearning behavior.\n- Experiments across multiple benchmarks show consistent improvements.\n- The framework enhances interpretability by revealing how specific neurons and circuits contribute to different functions, aligning with future modular LLM design goals.\n- The paper clearly articulates the problem of neuron entanglement and justifies the need for a more granular approach like CLUE.", "weaknesses": "- Although the section 5.1 and 5.2 documented well on benchmarks that the study is evaluted on plus the details on how the evaluation is conducted, I, somehow, could not find any descriptions on what model is evaluated. Is it the same model across all the benchmark evaluations? What is the family and the size of the model being evaluated in the study? \n- As the curse of circuit discovery, the framework would possibly also have scalability problem, such as being computationally expensive to find a meaningful ciruit in larger models. For meaningful circuit, I mean a circuit that is sparse enough comparing to the original computational graph. \n- The framework utilizes the edge pruning technique. In edge pruning, it returns a series of ciruits as the sparisity going down, while the metrics of the faithfulness would also vary. I wonder what are the specific critierion that was used to pick such circuit from the edging pruning outcomes. And how would you justify picking one of the circuits over the others ( maybe in terms of exact match, KL divergence, or sparisity)? \n- The framework appears to make an implicit assumption that the contribution of each neuron identified within a circuit is independent of the specific path or connectivity structure it participates in. For instance, consider a sequence of neurons \n𝐴1→𝐴2→𝐴3 across successive layers. CLUE might categorize 𝐴1, 𝐴2, and 𝐴3 into different neuron sets (e.g., forget, retain, or conflict) based on their individual CNF assignments. However, in reality, the functional contribution of each neuron is context-dependent on the path it forms with others. In contrast, circuit-level methods such as edge pruning often treat  A1→A2 →A3 as an integrated substructure, preserving or removing the path as a whole. This suggests that CLUE’s neuron-wise localization might overlook interdependent effects among neurons within the same circuit, potentially leading to suboptimal or fragmented unlearning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)