On Mechanistic Circuits for Extractive Question-Answering
Authors: Samyadeep Basu
Vlad I Morariu
Ryan A. Rossi
Nanxuan Zhao
Zichao Wang
Soheil Feizi
Varun Manjunatha
Venue: COLM 2025 (2025)
Keywords: mechanistic circuits
interpretability
TL;DR: We extract a mechanistic circuit for extractive QA and perform data attribution and model steering with the insights

Abstract: Recent studies have extracted circuits from the computational graphs of language models for simple language tasks such as entity tracking or indirect object identification. In our paper, we scale up circuit extraction to a real-world language modeling task: context-augmented language modeling for question-answering (QA) tasks and understand the potential benefits of circuits towards downstream applications such as data attribution. We extract circuits as a function of internal model components (e.g., attention heads, attention layers, MLPs) using causal mediation analysis techniques. Leveraging the extracted circuits, we first understand the interplay between the language model's usage of parametric memory and retrieved context towards a better mechanistic understanding of context-augmented language models. We then identify a small set of attention heads in our circuit which performs reliable data attribution by default, thereby obtaining attribution for free in just the model's forward pass! Using this insight, we then introduce AttnAttrib, a fast data attribution algorithm. Through a range of empirical experiments across different extractive QA benchmarks, we show that performing data attribution with AttnAttrib obtains state-of-the-art attribution results across different language models. Finally, we show the possibility to steer the language model towards answering from the context, instead of the parametric memory by (i) using the attribution from our extracted attention head as an additional signal during the forward pass and (ii) scaling the output of a small set of attention heads. Beyond mechanistic understanding, our paper provides tangible applications of mechanistic circuits in the form of reliable data attribution and model steering.

Published as a conference paper at COLM 2025
On Mechanistic Circuits for Extractive Question-Answering
Samyadeep Basu, Vlad Morariu, Cherry Zhao, Ryan Rossi, Jack Wang, Varun Manjunatha" Adobe Research, San Jose samyadeepb@adobe.com com
Soheil Feizi
University of Maryland, College Park sfeizi@und. edu
Abstract
Large language models are increasingly used to process documents and facilitate question-answering on them. In our paper, we extract mechanistic circuits for this real-world language modeling task: context-augmented (QA) tasks and un-
language modeling for extractive derstand the potential benefits of circuits towards downstream applications such as data attribution to context information. We extract circuits as a function of internal model components (e.g., attention heads, MLPs) using causal mediation analysis techniques. Leveraging the extracted circuits, we first understand the interplay between the model's usage of parametric memory and retrieved context towards a better mechanistic understanding of context-augmented language models. We then identify small set attention heads in our circuit which performs reliable data aftribution by default, thereby obtaining attribution for free in just the model's forward pass. Using this insight, we then introduce ATTNATTRIB, fast data attri- bution algorithm which obtains state-of-the-art attribution results across various extractive QA benchmarks. Finally, we show the possibility to steer the language model towards answering from the context, instead of the parametric memory by using the attribution ATTNAT as an additional signal during the forward pass. Beyond mechanistic understanding, our paper provides tangible applications of circuits in the form of reliable data attribution and model steering.
Introduction
In recent times, large language models have been used to process documents, webpages and transcripts as context and answer questions about them. We refer to the task of an- swering a question by directly extracting words from the context context/document document as extractive Question-Answering (QA), in contrast to "abstractive QA" "open-ended QA" where the words comprising the answer may not necessarily appear in the context. In the extractive QA case, language model can either answer from the context, hallucinate entirely from its parametric memory or interpolate between the two. A mechanistic understanding of such task with a circuit (a sub-graph of the language model's computational graph) can not only provide insights on the inner workings of the model for this task, but can also enable down- stream applications such as deta-attribution (i.e., pointing the source in the context which contributes to the answer) and model steering (i.e., enabling the model to answer from the context, rather than hallucinate from its parametric memory). Earlier works on mechanistic circuits (Bereska & Gavves, 2024; Elhage et al., 2021) for large language models (Touvron et al., 2023; Jiang et al., 2023; Chiang et al., 2023) have discovered circuits for language tasks such as entity tracking (Prakash et al., 2024), indirect object identification (Wang et al., 2022) or simple math operations such as "greater than" " (Hanna et al., 2023). While circuits are
'I eading Supervisor for the paper

---

Published as a conference paper at COLM 2025
Fhw wilpoee wi 20
(Ki)) Parfarm Crussl Deta Acttouticn
Contect
<hf tack-o Supert nug
wirdets to progees the final the
Fakirien curelt adheect frall in the place skpplf wet cublil Palsten the senttee Eine giuao bo in riguue ta Rastrila by velcar thail Lnabe sn Daeili chonmeet
and
peri Leave's achat
Comraned Vocel Input Spon feods Newn
Clear Made
Clen Vodel lnpat
welling -aran
Where booned?
Spioe Hedda lockles?
Coneriee Arower
Annear New tork
fewwer Nor
interventional Patching for Hierarchy 0 Extracted Node: (a12)
Interventional Patching g for Hierarchy
Extracted Node: (m11)
Applictionn Prestig g Langenge
Tomardi
Figure 1: Obtaining Circuits for Extractive QA in Language Models. We use our probe dataset along with path patching to extract circuits corresponding to (i) Confext and (ii) Memory Faithffllness. We find that small set of attention heads from the circuit can perform data-attribution and also steer language models towards context faithfulness.
a principled way to mechanistically understand language models, we note certain limita- tions within existing works: (i) Tasks such as entity tracking or indirect object identification are inherently simple and may not capture the complexity of real-world applications for language models and (ii) It remains uncertain whether understanding language models through circuits will translate into practical applications.
In our paper, we extract mechanistic circuits for real-world extractive QA task and use insights from the mechanistic circuit to provide two downstream applications: (i) Data attribution to context and (ii) Steering the language model towards improved context faith- fulness. We focus on this task, due to the importance of retrieved-context augmented language models in recent times which unlocks various user-facing downstream applica- tions (Lewis et al., 2021; Gao et al., 2024; Asai et al., 2023). We extract two kinds of circuits from language models: (i) Contexi-Faithhuunee
Circuit: A circuit used by the language model when it solely answers from the context and (ii) Memoor-Failfullnes Circuit: A circuit used by the language model when solely answers from its parametric memory. To extract these circuits, we first design a probe dataset (with minimal assumptions about it's inherent structure such as fixed length) and use Causal Mediation Analysis (CMA) (Wang et al., 2022; Pearl, 2001; Zhang& & Nanda, 2024) to find the subset of nodes and edges in the computa- tional graph of the language model which are causal to the model outputs. In particular, we observe that the circuits activated during the model's use of context differ significantly from those used for parametric memory. We validate different components of the circuit by various ablations and offer insightful mechanistic understandings.
With the extracted circuit components, we then investigate their roles for the task of ex- tractive QA. We first find that small set of attention heads in the circuit perform reliable data attribution by default (i.e., where the specific input data in the context used to produce answer is identified), inherently obtaining data attribution in just one forward pass for each token generation. Leveraging this observation, we introduce ATTNATTRIE, which can reliably perform data attribution using just one attention head across various real-world NQ-Swap) and white-box language
QA benchmarks (e.g., HotPotQA, Natural-Question models (Vicuna, Llama-3). In fact, through extensive empirical experiments, we show that ATTNATTRIB can obtain state-of-the-art data-attribution results when compared to other strong baselines for extractive QA tasks without any additional forward pass or auxil- iary model, effectively obtaining attribution for free. We also find that when the language model answers using the parametric memory circuit, the attribution heads still display a high attention to the answer tokens in the context. With this insight, we design a simple model steering method for improved funtext-fatthuulness by using the attributions from AI- additional source of information. Across various empirical experiments, we find that the addition of attribution during prompting leads to improvements upto 9% on popular extractive QA datasets.
TNATTRIB
2

---

Published as a conference paper at COLM 2025
Overall, our paper extracts mechanistic circuits in language models for real-world task of extractive QA. Beyond mechanistic interpretability of QA tasks, our paper highlights that certain components of the circuit can be useful for downstream applications such as data-attribution and also steering language models towards being more faithful to the context (thus improving generalization). In summary, our contributions are follows:
We extract mechanistic circuits (which provide a causal view) in language models for the real-world task of extractive QA for when the model answers from the context and from the parametric memory.
We provide salient insights on the underlying mechanics of language models high- lighting the interplay between parametric memory and context through the lens of extracted circuits.
Using the insights from the circuit mechanism, we provide two practical appli- cations: (i) Data-attribution to context with ATTNATTRIB and (ii) Model steering towards context-faithuulne using the attributions from ATTNATTRIB reliable enhancements which can ensure that the model does not hallucinate.
Related Works
With the advent of language models, several recent works have focused on mechanistic understanding of language models (Meng et al., 2023; Turner et al., 2024; Lieberum et al., 2023; McDougall al., 2023; Gould et al., 2023). One of the primary benefit of transformer based language models is that the final logit representation can be decomposed as sum of individual model components (Elhage al., 2021). Based on this deconposition, one can extract task-specific causal sub-graphs (i.e., circuits) of internal model components in language models. Early works have extracted such circuits for indirect-object identifica- tion (Wang et al., 2022), greater-than operation (Hanna et al., 2023) and more recently for entity-tracking (Prakash et al., 2024). Circuits can also be constructed as sub-graphs of neurons in the language model, but it often comes with increased complexity of interpreta- tion (Elhage et al., 2022). Recently, there has been an increasing focus on the practical aspects of mechanistic interpretability such refusal mediation (Arditi et al., 2024; Zheng et al., 2024) or safety in general (Zou et al., 2023). In our paper, we focus on extracting circuits for real-world task such extractive QA with a particular emphasis on practical applications such as alfribution and steering.
Deciphering a Circuit for Extractive QA
Nodes and Edges in Language Model Circuit. Recent decoder-only large language models, denoted by gø, such as Llama variants (Touvron et al., 2023; et al., 2024), are built on the seminal transformer architecture (Vaswani et al., 2017). A notable characteristic of these architectures is that the token representation at any layer can be expressed as a function of internal model components, such as MLPs and attention heads, from earlier layers (Elhage et al., 2021). As a result, the computational graph underlying a language transformer is directed acyclic graph, with nodes representing components like MLPs and attention heads (or layers), and edges representing connections formed by the residual stream.
We are particularly interested in obtaining a sub-graph of the transformer's computational graph which is responsible towards context-augmentte language modeling. In particular, we extract two circuits: (i) Faithxt-Failliuulness Circuit, which used when the underlying language model answers from the context, and (ii) Memorr-Faithhfflmn used when the language model solely answers from the parametric memory, ignoring the context. To extract the respective circuits, we first design probe dataset mimicking both these conditions which we use with causal mediation analysis (Wang et al., 2022) and our interventional steps in Sec. (3.2).
Circuit, which is
3.1 Designing the Probe Dataset
The design of probe dataset is extremely crucial in extracting circuits for language model task as shown in earlier works (Wang et al., 2022; Hanna et al., 2023). We are interested in obtaining circuit for context-faithfllness well as one when the model answers from the
3

---

Published as a conference paper at COLM 2025
Context-Faithfulness Circuit Groedy Selection of Attention Heads
Greedy Selection Attention Layers
Selotyy Pl
Ciglla Hock No0
Abrre Nocel vip
hopel (ilerve)
54-
ou Lud
unvnn
Memory- Faithfulness Circuit Gready Selection Attention Heads
Greedy on Attention Leyers
Greedy Selection of MLP Layers
Mode
Hold
Miridl
Figure 2: (i) Top Row (Context Circuit). We find that a small set of attention layers and attention heads are sufficient towards a high average metric score across all the models. However we find that for Vicuna and Phi-3, patching MLPs do not lead to a high metric score. For Llama-3-8B, we find MLP-31 to have a high direct effect, which when greedily combined with other MLP layers obtain higher scores; (ii) Bottom Row (Memory Circuit). We find that a large number of attention heads and layers are required to obtain a high score. Unlike the context circuit, we find MLPs to be important for the memory circuit.
parametric memory while ignoring the context. To this end, we design two probe datasets Daopy and 'D memory respectively for them. Each example in Deopy and Demorr consists of factual questions sourced from the Known dataset (Meng et al., 2023). For each question qi in both datasets, we use 1anma---70OBInstruue to generate a context c; related to the subject and answer for 9:- To guarantee that for each question in Deopy. the language model only answers from the context (and not the memory), we replace the answer tokens in the context q with a set of tokens which are semantically similar to the original answer(e.g, in Fig.(1), we replace Seattle with New York in the original context Space Needle is localed in Seattle, where the original answer was Seattle). In D Dmmmory. to force the model to answer from the parametric memory while ignoring the context, we replace the answer token with a token which is far away in semantic meaning from the original answer (e.g., replace Seattle with punctuation of "-"). In total, we curate 1000 questions (with their corresponding modified contexts) in Dcopy and "mmmory. We note that each entry x; E Pcopy/memon contains question qi, subject of the question si, ground-truth answer denoted by a;, the modified context di and the original context Cj. Along with 6 and ć, we add corrupted context Cicorrupied, where the subject and the answer token in the context is replaced by unrelated tokens and ficorrupted where the subject in the question is replaced by randomly sampled token. For e.g., as seen in (1g. the corrupted context (Big Sur is in California) is formed by replacing the subject and the answer tokens in the modified context. Full description of the dataset DD can be accessed in (Fc.
Distinctions from Other Circuit Datasets. Previous work on circuit extraction for entity tracking and indirect object identification relies on fixed templates for generating examples. However, for real-world tasks like extractive QA, probe datasets cannot use templates due to varying context lengths and unique information across examples.
3.2 Interventional Steps for Extracting Circuits
Our method is developed on the foundational technique of causal mediation analysis (Pearl, 2001). The primary idea of causal mediation analysis is to find important paths in a causal graph, by performing an interventional operation on small set of nodes and measuring
4

---

Published as a conference paper at COLM 2025
Model: Vicuna-78
Model: Llama-3-88
Easeline
Circut
Baserne
Handom Creak:
Ixtracted Clicu
1.0
10
Extractive QA Accurscy
0.0
Extractive Accuracy
00
800
NO-Swap
NO-Swap
Neture--Qusstinn
NO-Swap
NO-SWap NaturalOuusscins
Extractive QA datasets
Extractive datasets
Figure 3: Ablating the extracted context-faithfulness circuit leads to large drop in extractive QA accuracy for various datasets. We ablate the edges from the extracted circuit and random circuit in the language model and measure the extractive QA accuracy.
the change in the final output. In our use-case, we adapt this method to find sub-graph of internal model components such that ablating them leads to a decrease in ability of the model to perform QA (either through extraction from the context using the parametric memory while ignoring the context). Below we provide the algorithmic description:
Algorithmic Description. Given the language model g and its associated computational graph 9, our objective is to extract a sub-graph (i.e., a circuit) E g which is responsible towards the QA task. We obtain the nodes and edges of the circuit C in a hierarchical manner. First, we obtain a set of nodes and edges in hierarchy 0 denoted as (No, Eo) which have the of highest direct effect the final logit. In the next step for hierarchy 1, we obtain nodes and edges (V,&,) which have the highest direct effect on the nodes from hierarchy For any hierarchy we obtain a of nodes and edges &Nr,&g) which have high direct effect the nodes (Ng-1, from the previous hierarchy. For obtaining the nodes at the t hierarchy, create two instantiations of the underlying language model ge- The first instantiation is denoted 8g.llan with the original question qi and modified context of as the input. The second instantiation of the language model is &q.corrppted where the input context well as the question is corrupted as ci,oorupted and fo,corpp respectively. With this corrupted input, model corruppee assigns a low probability to the generated answer tokens a; from 8g.lean- Using these two model instantiations, the goal of the patching operation is copy the activations of node gi g from Sø,corrupted to &g,clean while restoring the activations of all the other nodes 8ş its original state. We denote the patched model as 8a.patch and use score (i)
P Pa,,patch (a;) to measure the importance of the component gj for the jt example. For the component gj, we then compute the average
metric score as score (g;) = Σ (core((,g;)/11)) We then sort the scores of the various components in the computational graph as score(g;) Vj N in decreasing order {g) LN j=1l and greedily select the minimum value of k, such that the average metric score of patching multiple components together: score( {g)} > 8. These selected components {g))/_. form the nodes in N₁. In our experiments, we only use the MLPs, the attention heads and layers as the different model components which are patched. The final circuit C consists of the nodes { and their associated edges, where K denotes the maximum circuit hierarchy.
Circuit for Context Faithfulness. We extract the circuits using Deopy as the probe set for the patching operations. We perform the patching operation the last residual stream position. We selected this position because the information in the last residual stream plays crucial role in determining the probability distribution the next generated token, which also used in recent mechanistic interpretability works (Arditi al., 2024; Turner et al., 2024).

---

Published as a conference paper at COLM 2025
Attention visualization [1850]
Attention visualization [994]
Entropy atttenttoo content
0.5
0.5
20-
1.0
1.0
0.0
Context
18300
0.0
Context
[2814]
1.0
Spon
0.5
conpanng
0.5
High
0.5
the
1.0
1.0
0.0
121 09. 1. Ngn, M8 Plys, (yy. 19
Context
Context
Top Attention Heads Circuit
Figure We find that one attention head in the context faithfulness circuit obtains low entropy value in the context window. Qualitative results shows that this attention head for Vicuna leads to peaky attention values the context span containing the answer, whereas other attention heads produce either diffused attentions or erroneous attentions. Further results on Llama-3 and Phi-3 Appendix.
Circuit for Parametric-Memory Faithfulness. In this case, we use D memory as the probe dataset for the patching operation. We extract the circuit at the same token positions the ones for context faithfulness.
Empirically, we primarily extract circuits for both context faithfulness and memory faithfulness corresponding to hierarchy-O (which constitutes the first-order effects) and provide results for hierarchy-1 (which constitutes second-order effects) in Sec.(B). We extract these circuits across Phi-3B, Vicuna-7B and Llama-3-8B. In Sec. we provide further results on circuit components for Llama-3-70B.
Circuit Validation. We validate the extracted circuit C by comparing to (i) Using randomly extracted circuit Crumdom: to measure the probability of the answer tokens; In this case the probability of the answer tokens will be low. (ii) We also ablate the context- context-faithfunnes. circuit (obtained using our probe dataset) across various extractive QA datasets commonly used in the community and measure the drop in the extractive QA accuracy. A large drop in accuracy signifies the validity of the extracted circuit.
3.3 Insights For Extractive QA through Circuits
3.3.1 Context Faithfulness Circuit Differs from Parametric Memory Circuit
Results for attention components. We find the circuit components for context faithfulness and memory faithfulness to differ significantly. For context faithfulness, we find that patching a small group of 4-5 attention layers (or 10 attention heads) is sufficient to obtain a high find average metric score of more than 0.95. However, for the memory faithfulness, that a significantly higher number of attention layers (e.g., >15) and attention heads (e.g., >30) required to obtain relatively high metric score. This result shows that information from a small set of attention heads (or layers) primarily drive the circuit corresponding to context faithfulness than memory faithfulness. In SDc.(D), we also show that the top circuit components of attention layers (or heads) have a low overlap between the two circuits highlighting that the underlying language model elicits different circuits when answering from the context vs. parametric memory.
Results for MLP components. We observe an intriguing pattern with MLPs in the extracted circuit. For context faithfulness, in Vicuna and Phi-3, MLPs appear to be less significant, as patching them results in a very low metric score. However, in Llama-3-8B, we identify one specific MLP (MLP-31) that individually achieves high metric score of 0.9. This suggests that the type of pre-training might play a role in determining the relevant circuit components (with respect to MLPs) for context faithfulness. For memory faithfulness, MLPs consistently obtain higher average metric scores across all three language models compared to the top MLPs in the context faithfulness circuit. This underscores the importance of MLPs when the language model retrieves information from parametric memory. Interestingly, we also find minimal overlap between the circuit com conenss responsible for context faithfulness and

---

Published as a conference paper at COLM 2025
Dataset: NQ IQwpS II
Dataset: Natural Questions
Dataset: Single-Hop HotPoiQA
Spaeres Jeniaro
Somtant Sinie
Models
Models
Modals
Figure 5: Attribution through one attention head in our circuit via ATTNATTRIB obtains strong attribution results. Across various extractive QA benchmarks, we obtain improved performances over different attribution baselines. For HotPotQA, we measure the Fi-score due to it being single-hop, whereas for other datasets, we measure the attribution accuracy. We present further results on long-form generations in Sec.(I) and attribution results on other synthetic datasets in Sec.(O)
those for memory faithfulness, even among MLPs. We provide the detailed list of all the circuit components for context faithfulness and memory faithfulness in Sec.(D).
3.3.2 Validation of the Extracted Circuit
Comparison with Random Circuit. For all the language models, when using randomly extracted circuit (for context faitkffulness), the probability of the answers from the probe dataset D drops to 0.045 for Vicuna, 0.081 for Llama-3-8P and 0.07 for Phi-3, which shows the relevance of our extracted circuit.
Generalizability of the Circuit to Downstream Extractive QA Datasets. To validate the circuits, in Fig.(3) (3), we ablate the context-faithfulnee circuit components when answering questions from downstream datasets such as NQ-Swap, Natural-Questin and measure the extractive QA accuracy. We compare with the extractive QA accuracy when a random circuit is ablated from the language model. Overall, we find that ablating the direct connections from the identified context-faithuulnss circuit components, lead to the maximal drop in extractive QA accuracy. This result validates that the extracted faithulfaithhuuness circuit generalizes to other commonly used extractive QA datasets. We provide additional results on circuits across different knowledge partitions (e.g., country, language) in Fig. (R) highlighting the similarity of context circuits.
and HotPotQA
3.3.3 A Small Set of Attention Heads in the context circuit are interpretable
In Fig. we observe that a small subset of attention heads in the extracted circuit for context faithfulness achieves a low entropy score with respect to the normalized attention values over the context. Upon further inspection, we find that these low-entropy attention heads predominantly focus the answer token spans in the context. Conversely, some other attention heads in the circuit, while also highly attentive to the answer token spans, display more diffused attention patterns across other tokens. These findings are consistent across all three language models studied: Vicuna, Llama-3-8B, Phi-3 and Llama-3-70B (see Sec. These results highlight the potential of a small set of attention heads from the circuit to be used for data attribution in language models (see more details in Sec.(4)). We provide more details on switching between the parameteric memory and context circuit in (Tc.
4 Application 1: Attribution for Free Via One Attention Head
Data attribution for extractive QA is crucial for language models processing external con- texts, such as documents or personal files, not included in the pre-training corpora. For example, in question like "What did Sarah Miller say during the all-hands meeting?", the cor- rect answer comes from specific section of the context (e.g., meeting transcript). Pointing to the source the answer improves model reliability and helps users verify its correctness, especially since LLMs are prone to hallucinations (Niu et al., 2024). In this section, introduce ATTNATTRIB, an efficient data attribution algorithm, leveraging insights from our mechanistic interpretations which outperforms existing QA baselines.
7

---

Published as a conference paper at COLM 2025
Actilbuttoo nupourauyy at Fartrraed Taken
Crnteing Cricut Attention Hoad Arrawerit from Momory
Context
34
Extroccttvn QA Cy
04-
M
Context (Number of Bontonces)
Dotasets
Dotasets
(a)
(b)
(c)
Augmenting the prompt with the attribution from ATTNATTRIB improves extractive QA accuracy. (a) The attribution at the perturbed token in context through our extracted attention head, when the language model answers from the parametric memory (D m is high. (b) Vicuna-7B and (c) Llama-3-8B: Improvement in extractive QA accuracies for both Vicuna and Llama-3-8B when compared to baseline prompting and Context-aware Contrastive Decoding.
Figure 6:
4.1 ATTNATTRIB: A Simple and Strong Data Attribution Method for Extractive QA
In (3.3), we observe that a small set of attention heads from the circuit attend to the answer token in the context. Thus, these attention heads from the extracted circuit for context faithfulness implicitly perform data attribution by default. However, real-world contexts can be noisy and contain multiple answer tokens, raising questions about the behavior of these attributable attention heads in practical settings. In this section, we introduce ATTNATTRI which automatically generates attributions from the context during the forward pass by leveraging only one attention head from the context faithfulness circuit. Specifically, ATTNATTRIB uses the attention patterns from the relevant attention head to generate span from the context for each generated answer token.
Algorithm ATTNATTRIE: Data Attribution via One Attention Head
Require: go (Language model), (Question), C(Context), k(Number of Spans), L(Answer Length),/(AA) Layer), h(Attn Head), (leegd), (length((pan--Inngg Length))
Ensure: Candidate attribution spans
S+ 0
Atotal < 0
for j < I,..., do
a, - (C,9)
D Ay: Attention map over context, aj: answer token b Add the answer token >> Extract the attention pattern for the given layer and head >> Extract maximal attention span and
A appenda)
A jjrelvvnnt + 11
9/ = GetMaxXSaan (A Alleevann sleenghhh)
value
Sapppen((s;,))) 0;))
b Add the extracted span sj to the list along with its value dj
end for return Sort(S){:A) k]
Sort extracted spans wrt attention value v and use the top-k as
attributions
These spans ranked based on the maximum attention value within the span (a sentence from the context), and the top-k spans are selected for attribution. detailed description of ATTNATTRIB provided in Algo. (1). Using ATTNATTRIE, we explore the potential applications of mechanistic circuits for attribution in extractive QA. We note that we use only one attention head identified using our probe dataset, Deopy, and test its effectiveness on different extractive QA benchmarks.
using
4.2 Evaluation on Extractive QA Benchmarks
Baselines. We use the following baselines: (i) Self Attribbtrib In this, we prompt the language model to generate attribution from the context which is required to answer
8

---

Published as a conference paper at COLM 2025
the question. This prompting technique is similar in principle to (Gao et al., 2023) and (Buchmann et al., 2024); (ii) Iterative Prompting: We first generate the answer from the language model, then perform another forward pass and prompt the language model to generate the attribution from the context for the generated answer. (iii) Sentence Similarity. We retrieve the most similar sentence from the context to the generated answer using an auxiliary language encoder (all-mpnet-base-22 from (Buchmann al., 2024), which identified this embedding model as one of the best- performing retrievers. (iv) Gradient: We find the gradient of the loss for generated token with respect to the input context token embeddings (Yin & Neubig, 2022). We then use this to select the span containing the token with the highest gradient value.
This choice is motivated by findings
General Empirical Results. We compute the exact match score with the ground-truth attributions across the synthetic dataset (used in our probing step), NQ-Swap (Longpre et al., 2022), Natural-Questions (Kwiatkowski et al., 2019) and Single-Hop HotPotQA (Yang et al., 2018). A full evaluation dataset description is in SGc. Across all datasets, we find that ATTNATTRIB leads to improved results over strong baselines. We note that the components (i.e., relevant attribution head) of our circuit are primarily extracted for zero-hop extractive QA. Inspite of this, we find that our method obtains better F1 scores (<s 20% improvement) than the baselines for single-hop extractive QA. The simplicity of our approach enables attribution computation in just one forward pass (during the answer generation step) therefore positioning itself as a tool for real-world use-case the domain of extractive QA. In Fig, we also find ATTNATTRO to be robust towards larger context lengths for language models supporting long contexts (e.g,, Llama-3-8B, Phi-3), For Vicuna, we observe degradation for longer contexts as it only support 2048 tokens as the context length. In Sec.(16) we show further results on Llama-3-70B showing the stability of ATTNATTRIB for longer contexts. In SUc,(U), we show that our method outperforms the recent ContextCite (Cohen-Wang Wang et al., 2024) while providing strong efficiency gains.
Extending to Long Extractive Answer Generations. We apply ATTNATTRIB to attribute long extractive answer generations to specific parts of the input context. For this purpose, we use 1000 examples each from the CNN-Dailymail (Hermann et al., 2015) and NQ- Long (Kwiatkowski et al., 2019). For evaluating the quality of attributions, we measure the change in the log probability of the responses when the top attributed sentences in the context are ablated. A higher change in the log probability indicates the effectiveness of the method. In Sec.(I), we show that ATTN ATTRIB consistently obtains high change in log probability score (when compared to other baselines) for both the datasets, indicating that our method is scalable long answer generations.
Scaling to Llama-3-70B. We apply the circuit extraction steps from S3c.( to identify the components that ensure context faithfulness in Llama-3-70B. Using the attention head with the lowest entropy in the context, combined with ATTNATTRIB, we extract the attributions. As shown in Sec.(K), our method yields reliable and robust attributions for larger language models such as Llama-3-70B which highlights the generalizability of our approach.
5 Application 2: Towards Improved Context Faithfulness
In the experimental setup in Sec. T), we observe that when the model answers from parametric memory, upweighting the attention at the answer tokens in the context can prompt the model to answer from the context instead. Further investigation reveals that even when the model retrieves answers from parametric memory, the attention maps from the attribution head used from context circuit still show a high focus on the perturbed answer tokens in the context (see Sec.(K) for visualizations). Fig. 6)-(a) illustrates the attribution accuracy concerning the perturbed context answer tokens when the language model answers from parametric memory. Based on this insight, we employ ATTNATTRIB to obtain attributions for language model generations using single forward pass. We then use these attributions in the prompt as an additional signal to guide the language model towards greater faithfulness to the context. Below we provide the empirical results:
Empirical Results. Across various extractive QA benchmarks including NQ-Swap, Natural- Questions and HotPotQA, we find that using the attributions extracted with ATTNATTRIB as an additional signal in the prompt improves the extractive QA performance by upto 9% (see (6g-(6))(b, c)). We observe consistent improvements across both the Vicuna and Llama-3-8B

---

Published as a conference paper at COLM 2025
family of models when compared to baseline prompting and Context-aware decoding (Shi et al., 2023). This highlights the benefits of incorporating attributions from ATTNATTRIB in the prompt, for improved faithfulness to the context on real-world benchmarks.
Conclusion
In this paper, we obtain mechanistic circuits for extractive QA, popular real-world task. We identify key mechanistic differences when the model uses the parametric memory (ignoring the context) vs. when it uses the context. We then find that small set of attention heads in the context circuit performs data aftribution by default. Using this insight, we introduce AT- TNATTRIB, an efficient data attribution algorithm which obtains strong results on extractive QA benchmarks. We further show that the attributions from ATTNATTRIB can be used towards improving generalization in extractive QA tasks by steering the model towards context faithfulness.
References
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models mediated by single direction, 2024, URL https://arxiv.orrg http:/iww..rg/bb//445550 11717.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https:
/larxiv..org/ab//231...
Leonard Bereska and Eístratios Gavves. Mechanistic interpretability for ai safety a review, 2024. URL httpp::/arxt:: httt:s::/aari..orr/aa////... org/abs/2404.
14082.
Jan Buchmann, Xiao Liu, and Iryna Gurevych. Attribute or abstain: Large language models as long document assistants, 2024. URL httphttt////rii.orree 2427 07799.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
URL https:://mmss...eoettm http:://mmmmss.rrrrbb/////200000 2000 cinaa/.
Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. Con- textcite: Attributing model generation to context, 2024. URL https://arxiii org/abs/ 2409.00729
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-oddss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Abhimanyu Dubey et al. The llama herd of models, 2024. URL
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations, 2023. URL https: http://axxww..ooab////////////// 14627.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024. URL URL.htpss//arrvv..rr/bb/22222... 10997.
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild, 2023. URL htttp::/aaxiv..og//bb///// 09230.
10

---

Published as a conference paper at COLM 2025
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model, 2023. URL
https://arxiv. wtg./bs////a///// http:://arii..orrr////22...... 0005000000
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. CoRR, abs/ 1506.0330, 2015. URL
/50... 03340.
Jie Huang and Kevin Chen-Chuan Chang. Citation: key to building responsible and accountable large language models, 2024. URL htttp::/anxxvv.org/bbb/////// /00p///0bb 02185.
Albert Q- Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv wwwpaaawwii.oogrrbrrraaaaaa wtgg/bb////131 06800
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models, 2024. URL https: http::::/aaaxvvorrg///e/oo// /204.. 01019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: (0.1162/tacl.a.0022 URL https::/accaattttttt:
org/Q1g- 1026.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowleedgeintensivve nlp tasks, 2021. 11401.
URL http:sss//arvvv.or////////////
Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. A survey of large language models attribution, 2023. URL https: //arxiv. abs//311..... 2011.1111000 03731.
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretabilityy scale? evidence from multiple choice capabilities in chinchilla, 2023. URL
http///aawvvvooora////a/a// 09458.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering, 2022. URL https: //arxiv.org/abs www..org/abb//11e 2et.ss/e......0 05052.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories, 2023. URL https: http://arxiiv.orraa///2222 10511.
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head, 2023. URL https: //arxiv.org/abse 04625.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing 05262.
factual associations in gpt, 2023. URL
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: hallucination corpus for developing trustworthy retrieval-augmentee language models, 2024. URL https://arxix. www./rr////aaae 00396.
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference Uncertainty in Artificial Intelligence, UAI'01, pp. 411-420, San Francisco, CA, USA, 2001.
Morgan Kaufmann Publishers Inc. ISBN 1558608001.
11

---

Published as a conference paper at COLM 2025
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine- tuning enhances existing mechanisms: A case study on entity tracking, 2024. URL
https://arxiv. http:/iwww..r///////////// 24011/2112.11
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding, 2023. URL https:tt//axxii.orrrg haargg/bs//230 abs//3/5. 14739.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat 09288.
models, 2023. URL https: //axxvv///xvvva///a http://///axivoooaa////////////
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, 2024. URL https: / /ttps::///axxvv http//axxvvvrrrr wwtp://////b//...../ 10248.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, AlionJones Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.0333 1706.0376,, 2017. URL http: / www.aaawiv..aaaaaaaaaaaaeooea abrg/bbs/1706. 03766........
Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. A causal view of entity bias in (large) language models, 2023. URL https: http://rriivv..rrr/////5./5/ abbrrr/5////230.... 14695.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small, 2022.
http::/tp:///att wwwparwwwww...oaeaeeeeeeeeooeere harrg//bs/22/ 221../2933..
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of-war between an Ilm's internal prior and external evidence, 2024. URL https://arxiv.. httt::://avii..oor///////1.1111 .101//1444441419 10198.
Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for Ilms: A survey, 2024a. URL.htts:///rxxv.oorg//// 08319.
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Search-in- the-chain: Interactively enhancing large language models with search for knowledge- intensive tasks, 2024b. URL https::///////: org/ abbs/2004 14732.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi- hop question answering. CoRR, 180909800,9 2018. URL http: http://awwvv..og/bbbb./s/
09600.
Xi Ye, Ruoxi Sun, Sercan O. Arik, and Tomas Pfister. Effective large language model adaptation for improved grounding and citation generation, 2024. URL https://arxiv. org/abs/2311 2311199331........
Kayo Yin and Graham Neubig. Interpreting language models with contrastive explanations, 2022. URL httpp::/aaxxi..org/bbbs..
10419.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024. URL httpss:////axvv.o/oaaa/aa 16042.
12

---

Published as a conference paper at COLM 2025
Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. The knowledge alignment problem: problem:: Bridging human and external knowledge for large language models, 2024. URL https: //axxiv wwtpr/wwwwwrrabb//3...5.. 13669.
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models, 2024. URL https://arxiv.org/absstt 18018.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023. URL https: httti.o//arxxv.orr////t/ htrg/abs/2311 2310 01405.
org/abs/
A Qualitative Examples on Data Attribution
A.1 Vicuna
0.0
1.0
0.0
1.0
Retn Vits
0.5
aun
0.5
0.5
1.0
0.0
1.0
0.0
Tolen Posiiions
Alten itin Vaualizazions
brations
0.0
1.0
0.0
1.0
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Trion Pos Gons
Token
Figure 7: One of the attention heads ([18,30]) from the Vicuna circuit attends "cleanly" to the answer token span in the context. In this example, we can qualitatively observe that the attention head elicits patterns which are of low entropy. We use this attention head in our data attribution algorithm ATTNATTRIE
13

---

Published as a conference paper at COLM 2025
Atlencion Visuslicuiors
Atsenton Voual eosolestions
0.0
1.0
0.0
1.0
100 V9b
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Taken Postioons
Takn Rycttons
Attoridin
1.0
0.0
1.0
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Tikn Postions
Figure 8: A few other attention heads in the circuit attend to the answer token span, but do so less "cleanly" while attending to other tokens too. (Top): This attention head attends to the last token itself; (Bottom): This attention head attends to the answer token, but also has attentions other tokens the context.
A.2 Llama-3-8B
Attertion Viuul zations
intiorn
0.0
1.0
0.0
1.0
0.5
AUn
0.5
0.5
1.0
0.0
1.0
0.0
Token Poetions
Dkkn Posit om
Attentionn Mouslizntione
Atbertinn Vouss tnatioe
0.0
1.0
0.0
1.0
0.5
0.5
ABn
0.5
0.5
1.0
0.0
1.0
0.0
Taken Rositoone
Dkan Pouit
Figure 9: A small number attention heads from the Llama-3 circuit attends to the answer tokens in context "cleanly". (Top): Attention head ([17, 24)) attends to the answer token in the context as well as the first token position. However, the attention to the first token position is minimal. (Bottom): Attention head ([27, 20)) attends to the answer token as well as the first token. However within the context, the maximum attention is still on the answer token span.
14

---

Published as a conference paper at COLM 2025
A.3
Phi-3
Antertion Visualzations
intention Visustaations
0.0
1.0
0.0
1.0
ADD vol-
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Taken Posttions
idtention Visusitations
0.0
1.0
0.0
1.0
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Takan Postiions
Figure 10: The top attention head from the Phi-3 circuits attends "cleanly" to the answer token span in the context. We find this attention head to also attend to the first token position minimally. However, within the context window this attention head has the maximum attention to the answer token span.
B Note on Second-order Circuit Components
In (3c,(3), we identify circuit components at hierarchy 0 that have the most significant direct impact on the final logit. In this case the target node in the circuit graph is the logit and the source nodes circuit. In our experiments, we also set the extracted circuit components from hierarchy 0 the target node and then extract source nodes in the circuit graph. We perform this operation at the last residual stream position. Overall, we obtain a set of components which have high direct effect on the extracted components from hierarchy 0 (with metric score of 0.71) for Llama-3-8B. However, on investigating the components further, we did not find any specific utility of data attribution or model steering using them. Overall, an in-depth study of the second-order components in the causal graph for extractive QA will be addressed in a future work.
all the different attention layers, MLPs and attention heads in the extracted
C On Modifying Circuit Components
In (T), discuss the effect of scaling the attention heads from the context faithfulness circuit in the language model when it answers from the parametric memory. In particular, we find that upweighting the maximum attention value from these attention heads onto the context steers the language model towards answering from the context instead of the parametric memory.
Up-weighting the attention values. We multiple scalar value & to the maximum attention value in the context before the softmax normalization operation. In our implementation, we perform this scaling operation across all the attention heads in the top 3 attention layers in the circuit. We set
in our experiments, for the best steering result.
Ablating the MLPs. We set the output of the top MLPs from the memory- fiinory-fiithulnesss circuit to zero. In particular, we set the output the projection layer the MLP block to be zero, but make sure that the output of the other blocks are not changed due to this modification, by setting them to their original configuration. This ensures, that only the direct connection from the MI.P to the final logit is ablated.
15

---

Published as a conference paper at COLM 2025
D
Extracted Circuit Components Across Language Models
D.1 Vicuna
D.1.1 Context Faithfulness
Attention Layers. [24, 20, 18, 28, 31, 22, 29, 17]
Attention Heads. [[24, 8), [18, 30), [31, 24), [20, 11, [22, 30], [24, 15], [19, 4], [28, 7], [31, 27], [28, 14], [29, 10], [17, 11], [31, 16), [18, 10]]
MLPs. [31, 24, 21, 14, 18, 11, 9, 12, 8, 1, 1,,,,,,, 6,, 15, 10, 13, 17, 19, 27, 29, 23, 30, 26, 20, 22, 28, 25] (Sorted order)
D.1.2 Memory Faithfulness
Attention Layers. [20, 24, 16, 31, 26, 28, 30, 29, 15, 22, 12, 13, 19]
Attention Heads. [[31, 27], [24, 14], [19, 8), [28, (20, 14), [20, 18), [16, 10), [21, 15], [26, 23], [30, 12], [15, 10], [31, 25), [17, 25), [16, (16, 20), [18, 12,, 24], [14, 28], (18, 26], [29, 15], [14, 5], [26, 14], [16, 5), [18, 11j, [22, 10J, [22, 17], [16, 30), [31, 16), 26), [29,91]
MLPs. [22, 20, 23, 21, 31, 19, 29, 14, 18]
D.2 Llama-3-8B
D.2.1 Context Faithfulness
Attention Layers. [27, 23, 31, 24, 25, 29, 30]
Attention Heads. [[27, 20], [23, 27], [31, 71, [17, 24), [25, 12], [31, 20], [24, 27], [27, 6], [26, 13], [16, 1], [31, 6], [29, 31], [31, 31, [30, 12]]
MLPs. [31, 28, 26, 25]
D.2.2 Memory Faithfulness
Attention Layers. [31, 24, 26, 9, 19, 17, 23, 8, 16, 30, 11, 18, 14, 29, 10, 15]
25, 2,27 21, 22, 20, 13,
Attention Heads. [[31, 71, [24, 31, [31, 14], [30, 24], 24], [15, 18], 11, [31, [31, 271, [29, 81, |17, 27), [1444199222299111111100000111110 231, [28, 0), [16, 01,
[14,
14], 19, 291, [13, [27, (27, 28], 19, [20, (9,5), [12, [12, 20], [25, 13, [13,3), [16, 17], [12,
21), [31, 31,, 122, (29,
MLPs. [22, 21, 20, 23, 25, 24, 19,]
D.3 Phi-3
D33.1 Context Faithfulness
Attention Layers. [29, 21, 31, 28, 25, 20, 20, 11]
Attention Heads. |[29, 31], [20, 11, |31, 4), [23, 71, [19, 14), [23, 23), [25, 6), [20, 21], [25, 18], [21, 21], [21, 16), [28, 28), [25, 91, [21, 22]]
MLPs. [31, 30, 27, 19, 14, 21, 15, 9, 6, 11, 7, 3,5, 0, 8, 2, 10, 16, 13, 23, 12, 18, 17, 20, 28, 26, 22, 24, 25, 29]
16

---

Published as a conference paper at COLM 2025
D.3.2 Memory Faithfulness
Attention Layers. [23, 31, 20, 22, 19, 29, 24, 18, 16, 25, 12]
Attention Heads. [[23, 4], [31, 4], (0), [31, 17], [19, 20], [30, 11, [19, 13], [20, 5], [22, 29], [25, 231, [22, 15), [28, 71, [20, 26), [9, 17], [21, 124, [24, 12), 120, [22, [23, 31], [21, 21], [20, 4], [19, 27],
21], 12,, [16, 30), [13, 11], [13, 25], [15, 29], [25, (1,, [29, 20), 216, 15), [18, 25), [29, 17], [4, 291, [29, 26], [23, 29I, [24, 4J, [16, 25), [22, 18), [16, [30, 24), [18, 11, 24J, [17, 25J, [3, 10J]
213,
MLPs. [23, 24, 22, 25, 21]
D.4 Do we need a larger probe dataset?
We initially tested circuit extraction by using smaller dataset of size 200. In particular, extract the context-faithfulness circuit for Llama-3-8B. We find the following components:
Attention Layers. [27, 23, 31, 24, 29, 25, 21, 301
Attention Heads. [[27, 20), [23, 27], [31, 7I, [17, 24], [31, 20], [25, 12], [24, 27], [27, 61, [26, 13], [16, 1J, [29, 31J, [31, 6), [31, [30, 12]]
MLPs. [31, 28, 26, 25]
We find the sets of components in the circuit to be similar (except couple of components get reordered) to the one extracted using 1000 examples. This validates that a relatively smaller size of probe dataset can also be used towards finding circuit for extractive QA. We also note that (Prakash et al., 2024) use a similar smaller size probe dataset to find circuit for entity tracking.
E
More Details on the Interventional Algorithm
Step 1: Patch node by zopying activation from Corrusted to Clean Model
Step 2: The patching affects other intermediate
Stop 3: Restore the other intermediate nodes
capture direct effect a12
P(Yew York] Patched) = 0.3
Lnghtl
for
Creating Ratched Model Spacc New locteed Arowee Nowe Hort
Commatte Model Rigal Callora
Spuce hondd Nev 12
gdde on fomii Whene Aig bocttc?
Saice headde is Nas York
Bacaria in Calorral Where Elg boulte!
Figure 11: Different Steps of Patching with the Clean and Corrupted Model. We provide the patching steps follows: Step 1: Copy the activation of node (e.g., a12) from the corrupted model to the clean model to create the patched model. Step 2: Patching a12 also affects a13, m13 and m12 they are recomputed. Step Restore back a13, m13 and m12 to its original configuration so that only the direct edge path effect from a12 to the logit is measured.
F Probe Dataset Details
As shown in (3c.1,. the probe dataset consists of two partitions Dcopy and D Pmmmory which are used to elicit the context-faithfulness circuit and the memory-faihhfulnes circuit respectively. Below we provide few qualitative examples.
17

---

Published as a conference paper at COLM 2025
E1 Example 1
Subject. Vinson Massif
Question. Where is Vinson Massif located?
Original Answer. Antarctica
Context for Copy Faithfulness. Vinson Massif is the highest peak in the Sentinel Range of the Ellsworth Mountains, towering at an elevation of 4,892 meters (16,050 feet). It is positioned in one of the most remote and challenging environments on Earth, attracting climbers and adventurers from around the globe. First summited in 1966 by an American team, Vinson Massif is a sought-after destination for mountaineers aiming to complete the Seven Summits, the tallest peaks on each of the seven continents. Due to its extreme location and harsh weather conditions, expeditions and careful logistical planning. The massif stands as the pinnacle of its continent, and for those who successfully reach its summit, it provides a profound sense of achievement and magnificent views over the surrounding icy landscape. Located in Africa, it is testament to human endurance and the allure of pristine, untamed wilderness.
Vinson Massif require thorough preparation
Context for Memory Faithfulness. Vinson Massif is the highest peak in the Sentinel Range of the Ellsworth Mountains, towering at an elevation of 4,892 meters (16,050 feet). It is positioned in one of the most remote and challenging environments on Earth, attracting climbers and adventurers from around the globe. First summited in 1966 by an American team, Vinson Massif is a sought-after destination for mountaineers aiming to complete the Seven Summits, the tallest peaks on each of the seven continents. Due to its extreme location and harsh weather conditions, expeditions to Vinson Massif require thorough preparation and careful logistical planning. The massif stands as the pinnacle of its continent, and for those who successfully reach its summit, it provides a profound sense of achievement and magnificent views over the surrounding icy landscape. Located in -,, it is testament to human endurance and the allure of pristine, untamed wilderness.
E2 Example 2
Subject. Beats Music
Question. Who owns Beats Music?
Original Answer. Apple
Context for Copy Faithfulness. Beats Music, subscription-based online music streaming service, was acquired by Netflix in 2014 for billion.
Context for Memory Faithfulness. Beats Music, a subscription-based online music stream- ing service, was acquired by - in 2014 for 3 billion.
G Data Attribution Evaluation Dataset Descriptions
Synthetic Consists of the probe dataset where the context is the one generated by Llama-3-70B.
Synthetic 2: Consists of the probe dataset D where the context is perturbed such that the original answer token is replaced with a closely related answer token.
NQ-Swap NQ-Swap dataset (Longpre et al., 2022) where the original context is used.
NQ-Swap 2: NQ-Swap dataset (Longpre et al., 2022) where the original context is perturbed such that the original answer token is replaced with another token.
Natural- Natural-Questions A subset of Natural- -utural-Questoons (Kwiatkowski et al., 2019) where the ground-truth answers are short. In total, there are 13.9k questions.
Single-Hop HotPotQA: Consists of questions from HotPotQA (Yang et al., 2018) with zero-hop or single-hop extractive QA questions.
18

---

Published as a conference paper at COLM 2025
H Qualitative Study of Attributions using AttnAttribute
Question
Context
Attribution Attention Head
Attribution
who wan the icc under 18 world 2018
first Super Lengua sem-lirall progress
In the final, Russia beat Australia wickets their fourth Under World
In the final,, Russia beat Australia by 8 wickets win their fourth World Cup,
Australi Atghematan by
fnail Rusçia
Answer: Russia
to aduanie
playeet play
therofore hed third
heir
Austrays settonns their fourth
18 Worlis Cun, thairmetee leg sid cppo
Apwetie batie
0.0
1.0
0.0
1.0
100
0.5
0.5
0.5
0.5
0.0
Takan
Figure 12: ATTNATTRIB can select the right attribution span containing the answer, even if the answer token is present at multiple locations. In this example, Russia (which is the answer) is present at multiple places. We find that ATTNATTRIB can infact pick out the correct causal location in the context, for the attribution.
Question
Contex:
Attribution via Attention Head
GT Attribution
Current Question: in which state bikram sambhat the official calendar
<P> The Fana ralara of Gujerat made Vikram Sammet the offic el Hindu catendar 1901 OF which started as Sarwet 1953. in Gujarat, the now your begins with the first of the month of Satelwtt, which uscally lalla within the morthsi of April May calender: The first day of nevy passion utaly celobroted in historicat carnial tha tabes siace sowry your witsepnn caeleet
before, from 2007 AD Cujuret Sumbat recogniaed as the national calandar.
Au before, 2007 AD Gujurat Sumbat reoogg the national calander
Oragon an
"rom 2007 Gujarat Sombat is resognizud us national calandar.
Jotention Vinaalaation
1.0
0.5
0.5
1.0
0.0
Figure 13: ATTNATTRIB can select the right attribution span containing the answer, even the answer token is present at multiple locations. In this example, Gujarat (which is the answer) is present at multiple places. We find that ATTNATTRIB can infact pick out the correct causal location in the context, for the attribution.
19

---

Published as a conference paper at COLM 2025
I Validating Long Extractive Answer Generations
Extractive QA datasets such as HolpoiQA, NaturalQuestions and NQ-Swap particularly concerned with entities in the answer which consist few relevant tokens length. Even the generated answer from the language model is long, the attributions need to point to the relevant span in the context which consist of the entity. Another challenging setting is the case where the language model needs to generate an answer comprising an entity which itself can be long, for example comprising of multiple sentences. We investigate two experimental settings this respect for the following datasets: (i) CNN-Dalymail, where the language model is prompted pt to generate extractive summary. This extracted summary is itself the relevant entity the generated answer. (ii) NQ-Long, where the language model prompted to generate an answer with exact sentences from the context (rather than only the entity). We note that in NQ-Long, the ground-trum -ouun answer consists of multiple sentences extracted from the context.
To evaluate the quality of attributions, we measure the relative change the log probability of the responses when the original context is used vs. the original context is modified to remove the attributions (obtained from ATTNATTRIB), A higher relative change in the log probabilities indicates the faithfulness of the attributions. In particular, given the language model &Φ, the original context Corig and the ablated context where the attributed text has been removed as Cablated, we define the relative change in log probability of a response R as:
log( Pss (R) Corig) log(Psg Pog (R) Cablated)
Rel-Score Corig, Cablated, R) =
(1)
log (Pg, Cablated)
L11
Results on CNN-Dailymail
Vicuna Random Deletion
Llams-3-88 hardom Delation
Sentence
Sertence Simileity
Relative change in tog probability
Relative change in log probabiiity
11110
1ee3
Top-k sources
Top-k souroes
Figure 14:
Removing the attributions obtained with ATTNATTRIB from the context leads large relative change the log probability of the responses. We measure the relative change in the log probabilities of the original response (with the original context and context where the attributions are removed). We use 1000 examples from the CNN- Dailymail dataset. For both Vicuna and Llama-3-8B, we find large relative change in the log probabilities of the responses, highlighting that the attributions from ATTNATTRID are reliable.
20

---

Published as a conference paper at COLM 2025
1.2
Results on NQ-Long
ALvAU bute
Panson Delution
Serienke Strilakky
AerAcmbute
Pandom Delation
Sentencee Semilanyy
Relative change in log probabiilty
Relative change in log probability
0.3
0.8
0.
07
0/-
0.6
0.6
05-
0.3
1-1
10-2
Top-k sources
Top-k sources
Figure 15: Removing the attributions obtained with ATTNATTRIB from the context leads to a large relative change in the log probability of the responses. We measure the relative change the log probabilities the original response (with the original context and context where the attributions are removed). We use 1000 examples from the NQ-Long dataset. For both Vicuna and Llama-3-8B, we find a large relative change in the log probabilities of the responses, highlighting that the attributions from ATTNATTRIB are reliable.
I.3 Qualitative Examples of Attribution
1.3.1
Example 1
Question. what tectonic setting is responsible for the folded mountains of pennsylvania and the high himalaya?
The Appalachian Mountains in the eastern part of the United States, including the Ridge-and-Valley Appalachians, are formed by orogenic movements due to the collision of tectonic plates. The Himalayas, including the folded mountains of Pennsylvania, are also formed by orogenic movements due to the collision of tectonic plates.
Original Generation.
Attribution Span. The Ridge- and Valley Appalachians in the eastern part of United States The Ridge- and Valley Appalachians the eastern part of United States /Li> < /U>>' literature but has otherwise generally fallen out of use except as described below The forces responsible for formation of fold mountains are called orogenic movements. The term orogenic has derived from Greek..'
Generation after attribution is removed. Fold mountains form when two tectonic plates move towards each other at convergent plate boundary. The Himalayas are series of folded mountains that formed from sedimentary rocks that accumulated along the margins of continents. The Jura mountains formed by folding over a Triassic evaporite. Note: The new generation does not contain information about Appalachians and also the main entity of the answer which is about orogenic movements.
series of sub-parallel mountainous ridges that
I.3.2 Example 2
Question. where does saying bob's your uncle come from?
Original Generation. The origins of the expression "Bob's your uncle" are uncertain, but is commonly used in the United Kingdom and Commonwealth countries to mean "and there it is" "nnd there you have " It is believed to have arisen after Conservative Prime Minister Robert "Bob" Čecil appointed his nephew Arthur Balfour as Chief Secretary for Ireland in 1887, act which was apparently both surprising and unpopular.
21

---

Published as a conference paper at COLM 2025
Attribution Span. ..his nephew Arthur Balfour as Chief Secretary for Ireland in 1887, act which was apparently both surprising and unpopular. Whatever other qualifications Balfour might have had, Bob 's your uncle "" "expression arose after Conservative Prime Minister Robert Bob " Cecil appointed his nephew Arthur Balfour as Chief Secretary for Ireland in 1887, an act which was apparently both surprising and unpopular. Whatever other qualifications"
Generation after attribution is removed. The origins of the expression "Bob's your uncle" are uncertain, but it is commonly used in the United Kingdom and Commonwealth countries to mean "and there it is" or "and there you have it." It is believed to have arisen after Conservative Prime Minister Robert "Bob" Cecil appointed someone as the conclusive one. Note: The modified generation is missing the most important entity of the answer which about appointing the nephew as the Chief Secretary.
1.3.3 Example 3
Question. Which two regions grow most of China's rice?
Original Generation. Rice production in China Wikkiedia <H1> Rice production in China < /HI> Jump to navigation, search <Table>> <Tr>> <Td> < /Td> <Td> This article needs to be updated. Please update this article to reflect recent events or newly available information. (November 2010 < /Td> < /Tr>> < /Table>> Rice fields in Tongshan County, Hubei <P> Rice production in China is important part of the national economy.
< </P>> <P> China is the world's largest producer of rice, and the crop makes up a little less than half of the country's s total grain output. China accounts for 30 % of all world rice production...
Attribution Span. /P>> <P> China is the world's largest producer of rice, and the crop makes up a little less than half of the country's 's total grain output. China accounts for", "a Rice fields in Tongshan County, Hubei <P> Rice production China is an important part of the national economy. < /P> <P> China is the world's 's largest...
Generation after attribution is removed. The early rice crop grows primarily in provinces along the Yangtze River and in provinces in the south; it is planted in February to April and harvested in June and July and contributes about 34 percent total rice output. Intermediate and single-crop late rice grows in the southwest and along the Yangtze; it is planted in March to June and harvested in October and November and also contributed about 34 percent to total rice output in the 1980s. Double-crop late rice, planted after the early crop is reaped, is harvested in October to November and adds about 25 percent to total rice production. Note: After removing the attribution, it is missing the main entity of Tongshan County which appears in the original generation.
J Circuit Components and Data Attribution in Llama-3-70B
In this section, we use the circuit extraction algorithm to obtain the components for context- faithfulness in Llama-70B. We note that ours is the first work (to the best of our knowledge) to retrieve circuit components in a large enterprise grade model. First, we plot the entropy of the attention values in the context window from the top scoring circuit attention heads, along with their corresponding attribution accuracies. We find that there exists small set of attention heads with low entropy and high attribution accuracy on our probe dataset. Below we provide the circuit components corresponding to faithltsss
Attention Layers. [78, 54, 75, 77, 58, 52, 53, 35, 7,2]
Attention Heads. ||75, 27], [52, 19], [64, 26), [58, 4), 167, 60], [78, 26), [75, 30], [39, 401, [78, 25), [72, 39], [75, 26], [53, 1), [64, 271]
Below we provide further details regarding the attention head in the circuit which performs attribution by measuring the entropy of the attention values in context window. We also find that our attribution algorithm ATTNATTRIB is robust to larger context lengths for Llama-70B.
22

---

Published as a conference paper at COLM 2025
These early results highlight that circuit extraction for real-world tasks such as extractive QA can be scaled towards large 70B (and potentially beyond) language models.
Entropy of attention In context
40
0.18
10
0.58 0.80
0.07
Attribution Accuracy
3.0
2.0-
0.4
20
0.2
(13.
271
(52.
12)
20
(16
14)
(15,
100
(16.
15)
391
10
Top Attention Heads in Glicuitt
Context Length (Num of Sentences)
Figure 16: A small number of attention heads in the context faithfulness circuit from Llama-3-70B performs attribution. (Left): We measure the entropy the attention values in the context window for the attention heads in the circuit. Brown color marks the attribution accuracy on the probe dataset D. (Right): We use the attribution head [52, 19] and find that the attributions robust across various context lengths.
K Attention Patterns in Context When the Language Model Answers from Memory
Attention Visualgetiang
Attention Mouulbag
0.0
1.0
0.0
1.0
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
attention perturbed context answer token (punctuation token)
Trann Fogtsase
Attention Worl oggtrr
0.0
1.0
0.0
1.0
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
High perturbed context answere (punstuation tolion)
Tonn Roaiisns
Tikkn Rostions
Attention Vessiitatope"
Attention Visuakaatingg
0.0
1.0
0.0
1.0
400 10
0.5
0.5
0.5
0.5
1.0
0.0
1.0
0.0
Positions
Figure 17: The attention head performing attribution in the Context-Faithulnes Circuit still shows a higher attention on the perturbed answer token (e.g., punctuation token) in the context. The above visualization results are for Llama-3-8B.
23

---

Published as a conference paper at COLM 2025
L Multihop Results
Following are the average Fl-scores for the multi-hop development split from HotpotQA. We note that each hop from the split has imbalanced number of examples (especially for hops greater than 2).
Vicuna. ('hop-1': 0.57, 'hop-2': 0.47, 'hop-3': 0.50, 'hop-4': 0.49, 'hop-5': 0.36]
Llama-3-8B. ('hop-1': 'hop-2': 0.51, 'hop-3': 0.53, 'hop-4': 0.50, 'hop-5': 0.43)
Overall, our results indicate that although there a moderate degradation in the attribution quality for multi-hop questions, the average Fl-scores are still reasonable. This shows that our approach can be extended towards multi-hop QA attribution too. However, to obtain the best results, suggest obtaining circuit with probe dataset consisting of multi-hop questions and then using the circuit components for data attribution.
M Prompts Used in the Paper
M.1 Patching for Finding the Circuit Components
Prompt = "A chat between a human and an assistant for question-answering system. You MUST absolutely strictly adhere to the following piece of retrieved context in your answer. Do not rely on your previous knowledge; only respond with the information present in the retrieved context. Retrieved Context: context Question: question Answer ONLY in few words without mentioning " subject. Answer:"
The field of context, question, subject are filled depending on the example.
M.2 Extractive QA Attribution
Prompt= = "A chat between human and an assistant for question-answering system. You MUST absolutely strictly adhere the following piece of retrieved context in your answer. Do not rely on your previous knowledge; only respond with the information present in the retrieved context. Retrieved Context: context Question: question Answer ONLY in a few words. Answer:"
The field of context, question are filled depending on the example.
M.3 CNN-Dailymail Summarization
Prompt A chat between human and assistant for extractive summarization system. Answer with ONLY two to three sentences from the retrieved context which can serve as an extractive summarization for the context. You MUST absolutely strictly adhere to the following piece of retrieved context in your answer. Do not rely on your previous knowledge; Retrieved Context: context. Extractive Summary with only 2 to 3 sentences:
The field of context is filled depending on the example.
M.4 Natural Questions Long
Prompt = A chat between human and an assistant for question-ansswerngg system. Answer ONLY with exact sentences from the retrieved context. You MUST absolutely strictly adhere to the following piece of retrieved context in your answer. Do not rely on your previous knowledge; Retrieved Context: context Question: question" "Answer in few exact sentences from the retrieved context:
The field of context, question filled depending the example.
24

---

Published as a conference paper at COLM 2025
N
On Real-World Deployment of AttnAttribute as an Attribution Engine
Our method, ATTNATTRI suitable for attributing answers in Document-based QA or Web-search QA setting that uses LLMs. We note that white-box access of the model's parameters are required to discover the circuits that are useful for attribution. Thus, our method cannot directly be applied for Contextual QA applications where blackbox LLMs like Claude or ChatGIT are deployed. In the most basic form, ATTNATTRA attribution for every token generated in the answer. Algorithm 1 is simple heuristic to aggregate these per-token attributions to provide an attribution for the entire answer-span. However, leave the exploration of more sophisticated strategies, especially those that combine ATTNATTRIB with retrieval-based attribution for future work.
provides
O
Full Data Attribution Results
Synthetic
NQ-Swep
Models
Models
Models
Dataset: NQ- Swop
Dataset: Naturl Questions
Dataset: Single-Hop HotPotQA
ne Oogg
Models
Models
Figure 18:
Attribution through one attention head in circuit via ATTNATTRIB obtains strong attribution results. Across various extractive QA benchmarks, we obtain improved performances over different attribution baselines. For HotPotQA, we measure the Fi-score due to it being single-hop, whereas for other datasets, we measure the attribution accuracy.
P Limitations and Generalizability Beyond Single-Hop Extractive QA
In this paper, we extract mechanistic circuit components for extractive QA tasks using probe dataset that is primarily 0-hop in nature. Despite this, ATTNATTRIB demonstrates strong attribution capabilities for single-hop extractive QA tasks. In this section, we stress- test the generalizability of ATTNATTRIB on multi-hop extractive QA and reasoning-based questions. Specifically, we utilize the Multi-hop and Reasoning splits from HotPotQA to evaluate ATTNATTRIB'S performance. The results provided below:
Multihop QA. This form of QA requires some form inherent reasoning towards ac- cumulating different parts of the context towards the final answering. Overall we find that the average attribution Fl-score for multi-hop questions are reasonable, but lower than single-hop probe dataset consisting of multi-hop questions and extracting circuits with it, will lead to improved results for attribution.
using ATTNATTRIB (see (Lc.. We hypothesize that designing
25

---

Published as a conference paper at COLM 2025
Comparison-Based Reasoning Questions. We evaluate ATTNATTRIB comparison-based reasoning questions, where the ground-truth answer is binary (Yes/Nu (N). When the model is restricted to answering only "Yes" or "No," the attributions are imperfect, with attribution F1 accuracy of 0.14. However, when the model is prompted to generate answers with supporting tokens from the context, the attribution Fi score improves to 0.48. This result suggests that ATTNATTRIB is robust for reasoning tasks, provided the model includes supporting context alongside its binary answers.
Q Robustness to Context Lengths for Data Attribution
1.0
1934
Number of Sentences Contect
Figure 19: ATTNATTRIB is robust to context lengths for language models supporting larger contexts. We find ATTNATTRIB to be stable for Llama-3-8B and Phi-3 for large contexts, whereas observe degradation in performance for Vicuna.
R
Circuits Across Different Question Types in Extractive QA
To validate if similar circuits are used across different knowledge types, we partition the probe dataset (of size 1000) into a set of 800 and 200. The set of 800 is used find the parametric memory circuit and the set of 200 is where the extracted circuit is ablated. If the knowledge types across the 200 questions are indeed following very different circuits, then after ablating the circuit there should not be large drop in extractive QA accuracy. We find that ablating the extracted circuit lead to a drop in accuracy of 0.72 to 0.21 (70% drop in accuracy). This experiment is performed for Llama-3-8B.
In a similar vein, we note the experiment performed on the context-faitte faithfulnthtssssts circuit in Fig. (4), where the drop in extracted QA accuracy for NQ-Swap (containing different in-context knowledge type questions) when the circuit (computed from our probe dataset) is ablated is 85% (drops from 0.84 to 0.12). This experiment is performed for Llama-3-8B.
This result shows that for the in-context cases, majority of the knowledge types share the same circuit the drop very large. For the parametric case, the drop is slightly lower, but still significant. Overall, the experimental conclusions are (i) if the task is of extractive QA and the model is faithful to the context, then majority of the questions will follow similar circuits irrespective of the underlying knowledge type. This is potentially because a small set of attention heads are the primary driving components which write the answer from the context into the residual stream and ablating them leads to their absence from the stream, thus leading to the wrong answer. (ii) For parametric memory, there are more chances for different questions to follow different slightly different circuits.
The major takeaway is: for in-context cases, majority of the questions will follow the same circuit as long as they belong to the family of extractive QA, but for parametric knowledge questions although a large number of questions will follow similar circuits, there can be slightly more cases of distinct circuits.
We also performed a new experiment by averaging out the circuits across different knowl- edge partitions of the probe dataset. We have earlier saved the circuit component scores for
26

---

Published as a conference paper at COLM 2025
each question in the probe dataset. For the context-faithfunnss circuit, we partitioned the probe dataset into different knowledge types (i) Country; (ii) Capital Cities; (ii) Language.
Following are the context-fatthfulness circuit components for each category (Llama-3-8B)-
Knowledge about Country: Attention Layers: : 22,,23, 23,31, 24, 25, 20,30, 21]; Attention Heads: ||27, 20), [23, 27], [31, 7], [17, 24), [25, 12), [31, 20), [24, 27], [27, 6), [26, 13], [16, 1J, [30, 12], [31, (29, 311, [31, 311
Knowledge about Capital: Attention Layers [27, 23, 23, 24, 25, 29, 21, 30]; Attention Heads: [[27, 201, [23, 271, [31, 71, [17, 24], [25, 12], [31, 20], [24, 271, [27, 61, [16, 11, [30, 12], [31, 61, [29, 31], 33]
Knowledge about Language: Attention Layers : [27, 23, 31, 25, 24, 29, 21, 30]; Attention Heads: 1127, 20), [23, 27], (31, 71, [25, 12], [31, 20], [17, 24), [24, 27], [27, 6), [30, 12), [31, 6], [29, 31], [31,3]
We observe that the circuit components are almost similar (with slight change in ordering only) across different categories (which we experimented with) for the extractive QA. This together with the generalization experiment in Fig. (4) in our paper highlights that long as the task is of pure extractive QA, when the language model follows the context - the circuits are very similar, albeit with slight change in ordering of the components.
S Additional Related Works
Applications in Context-Augmenterr QA. With the advent of retrieval-augmented genera- tion (Lewis et al., 2021; Gao et al., 2024) language models have been increasingly used for real-world Question-Answering (QA) tasks. One of the primary enhancement of context- augmented QA lies in the ability to provide reliable grounding (i.e., attribution) in the context for the generated answer (Li et al., 2023; Khalifa al., 2024; Huang Chang, 2024; Ye et al., 2024). In recent times, there have been a large set of works which improve LLM responses by reducing hallucinations and improving grounding in the input context (Ye et al., 2024; Asai et al., 2023; Xu et al., 2024b; Zhang et al., 2024). Beyond grounding, (Wu et al., 2024; Xu et al., 2024a; Mallen al., 2023; Wang et al., 2023) investigate the interplay between model's use of parametric vs. context knowledge.
T One Can Switch Between Memory and Copy Faithfulness Circuits
To further validate the distinction between circuit components for Context faithfulness and Memory faithfulness, we conduct two ablation studies. Specifically, we use Diemorr, but force the language model to answer from the context, even when it originally retrieves answers from the parametric memory. We achieve this model forcing by: (i) upweighting the attention values at the answer token span in the context by scaling factor / in the top attention layers of the context faithfulness circuit, and (ii) mean-ablating the top MLPs from the memory faithfulness circuit. Our findings show that with attention upweighting, 92% of the questions from D memorr are correctly answered using the answer tokens from the context instead of the parametric memory. Meanwhile, mean-ablating the MLPs results in 68% of the questions being answered with relevant answer tokens from the context. These results further validate the distinction in the circuit components for memory and context faithfulness and also shows that one can switch between the circuits by modifying a small set of components We provide more details this in (CC.
27

---

Published as a conference paper at COLM 2025
U Comparison With ContextCite
Mijours
saye serutciee
Modeup Duro
Gwap faatsetinn
4001190
Figure 20: Our method ATTNATTRIB outperforms ContextCite (Cohen-Wang et al., 2024), while also providing significant efficiency gains. We note that ContextCite competitive with method and has strong performances. However, due to curating an attribution training set and learning a scoring function for each question, context pair, it often slow to run in practice. In this experimental setup, use Llama-8B and setup where we compare the extracted attribution with the ground-truth attribution. For Natural-Questions (NQ) and NQ-Swap, we use the top scoring attribution span from both ContextCite and our method. For Single-Hop HotPotQA, we use the top 2 scoring attribution spans from both methods. We divide each of the running times (in seconds) in x-axis with the fastest running time for easier comparison. We note that our method runs at least 25x faster requiring only one forward pass therefore enabling practical adoption for providing attributions.
evaluation
28


Reviews:
[{"review_id": "hqBA8LPrE8", "reviewer": "Reviewer_mTR6", "summary": "This paper investigates the mechanisms underlying extractive question answering using mechanistic interpretability toolkit. The authors identify distinct circuits responsible for answering questions from memory and from context. They also find the attention head most critical for answering from context and demonstrate that its attention pattern can be used to find more faithful answers. The experiments are conducted on the LLaMA-3, Vicuna, and Phi-3 language models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper contributes to the interpretability field by investigating the inner mechanisms of relatively recent language models.\n- It demonstrates that the extractive question answering capabilities of current models can be improved by using insights gained from understanding their internal workings.", "reasons_to_reject": "- The main problem: the paper is difficult to follow; many details, particularly those related to the methodology, remain unclear to me after reading (**see the \"Questions\" section**).\n- The paper relies too heavily on the appendix, which contributes to its lack of clarity and makes it harder to understand the main content.\n- The manuscript would benefit from proofreading, as it contains typos. Additionally, the font size in many figures is extremely small, making them very difficult to read.\n- The concept of AttnAttrib (Section 4.1) appears closely related to the AttnScore mechanism introduced in Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA by Tulchinskii et al. However, the paper does not cite this prior work or explore the relationship between the two approaches (see also \"Questions\" section).\n\n**UPD: The authors clarified previously unclear aspects of the paper, committed to including these clarifications in the camera-ready version, and provided additional results and explanations. As a result, I have raised my final score.**", "questions_to_authors": "- What is the exact difference between the \"a\" and \"m\" nodes in Figure 1?\n- While you describe how questions and contexts are \"corrupted\" in Section 3.1, it remains unclear whether the same corruption method is used in Section 3.2 or other parts of the paper. Are there any differences in the corruption procedures across different experiments? \n- Could you please elaborate on Algorithm 1, particularly the GetMaxSpan function? This function appears to be central to the algorithm, yet it is not mentioned elsewhere in the paper, leaving its implementation unclear.\n- From what I understood, the nodes (either MLPs or attention heads) were patched using their activations on \"corrupted\" questions and contexts. Did you explore using alternative values for patching these nodes? Different patching values may lead to extracting different circuits (see Transformer Circuit Faithfulness Metrics are not Robust by Miller et al.).\n- In Appendix A.2, you highlight that head (17, 24) in LLaMA 3 is effective for extracting the correct attention span. Interestingly, the same head appears in the top-right region of Figure 10(b) in Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA, where it is identified as highly effective for Multi-Choice Question Answer extraction via Query-Key interactions. I also noticed considerable overlap between the heads shown in that figure and those listed in Section D.2 (\"LLaMA-3-8B\") of your paper. Could you investigate this further? Specifically, how frequently are heads that are useful for Multiple-Choice Question Answering also effective for Extractive Question Answering?", "extra_scores": {}}, {"review_id": "l9ovUvnkaT", "reviewer": "Reviewer_F4x8", "summary": "This paper investigates mechanistic circuits in LLMs for extractive QA, distinguishing between context-faithful and memory-faithful circuits. It introduces AttnAttrib, a efficient method for data attribution which obtains state-of-the-art attribution results across various extractive QA benchmarks. This paper further shows that the attributions from AttnAttrib can be used towards improving generalization in extractive QA tasks by steering the model towards context faithfulness.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The extraction of context- and memory-faithful circuits provides a principled framework for understanding how LLMs balance parametric knowledge and contextual information. The observation that distinct circuits govern these behaviors is compelling and empirically validated.\n\n2. The authors validate their circuits across multiple models (Vicuna, LLaMA-3, Phi-3), datasets (HotPotQA, Natural-Questions, NQ-Swap), and even test generalization to LLaMA-3-70B.", "reasons_to_reject": "1. The quality and representativeness of the circuits heavily rely on the designed probe datasets. Although carefully constructed, the generalizability to more diverse, noisy, or longer-context settings could be better discussed.\n\n2. While ATTNATTRIB is efficient at inference time, the initial circuit extraction (causal analysis, patching) appears computationally expensive.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vATyhtZq4v", "reviewer": "Reviewer_z9as", "summary": "This paper identifies mechanistic circuits in LLMs for extractive QA, distinguishing between context-faithfulness (answers from context) and memory-faithfulness (answers from parametric memory). Using causal mediation analysis, the authors extract these circuits and leverage them to develop AttnAttrib, a fast attribution method that pinpoints answer sources via a single attention head. They further show that incorporating these attributions into prompts improves context faithfulness by up to 9%. Experiments across models (Vicuna, Llama-3, Phi-3) and benchmarks (e.g. HotPotQA, NQ-Swap) validate the approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **(A1)** Strong Mechanistic Interpretability: this work extract circuits for real-world extractive QA, bridging the gap between theory and practical applications. The design of probe datasets are interesting, possibly providing the part of success of the method by similarity to real-life setup. However, there are some missing results on circuit analysis (see R1)\n- **(A2)** Strong Empirical Validation: authors provide tests on multiple architectures (Vicuna, Llama-3, Phi-3) and scales to 70B models. Ablation studies includes random circuit comparisons, cross-dataset generalization and different baselines comparisons like self-attribution, iterative prompting, etc.\n- **(A3)** Practical Attribution Method: AttnAttrib achieves best attribution accuracy without additional forward passes or auxiliary models, making it scalable for real-time use.", "reasons_to_reject": "- **(R1)** Insufficient Generality: the method and probe dataset focuses only on extractive single-hop QA, while performance drops significantly on multi-hop reasoning tasks. It could be due design of probe dataset, but it is not clear whether the circuit could be extended to the general combination of extraction + multi-hop reasoning.\n- **(R2)** Insufficient Circuit Analysis: there are two circuits analysed, but the paper does not explore interactions or edge cases where both memory and context circuits activate (although there are some details in App. D). Moreover, while ablations are persuasive, path-patching may still capture correlated rather than strictly causal components.\n- **(R3)** Scalability Concerns: circuit extraction for Llama-3-70B is mentioned but not deeply analysed (Appendix J). Moreover, computational cost of hierarchical patching could be very expensive for larger models.", "questions_to_authors": "Some questions:\n- **(Q1)**: How do you ensure token-swapping perturbations (e.g., \"Seattle\" -> \"New York\") reflect natural model failures, rather than artificial edge cases? How sensitive are extracted circuits to the specific synthetic replacements in $D_{copy}$? Could you release ablations with alternative corruption schemes or natural corpora?\n- **(Q2)**: You note poorer performance on multi-hop QA. Have you attempted to build a multi-hop probe dataset, and does that yield a distinct circuit?\n- **(Q3)**: Could you provide some explanations on R2?\n\nOther comments:\n- Line 245: Fig(R) is a reference to the section, not a figure\n- Fig. 4 is a bit misleading. I believe, there is some typo, because there are two different visualisations of head [18, 30]. Maybe one of them should be [24, 8]?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper combines a novel approach to data attribution using causal mediation analysis, with convincing empirical results, showing improvements over multiple baseline methods, with interesting practical applications. The reviewers all agree that the paper should be accepted.
Decision: Accept