Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models
Authors: Dante Campregher
Yanxu Chen
Sander Hoffman
Maria Heuss
Venue: Accepted by TMLR ()

Abstract: This paper presents a reproducibility study examining how Large Language Models (LLMs) manage competing factual and counterfactual information, focusing on the role of attention heads in this process. We attempt to reproduce and reconcile findings from three recent studies by Ortu et al. [13], Yu, Merullo, and Pavlick [17] and McDougall et al. [7] that investigate the competition between model-learned facts and contradictory context information through Mechanistic Interpretability tools. Our study specifically examines the relationship between attention head strength and factual output ratios, evaluates competing hypotheses about attention heads' suppression mechanisms, and investigates the domain specificity of these attention patterns. Our findings suggest that attention heads promoting factual output do so via general copy suppression rather than selective counterfactual suppression, as strengthening them can also inhibit correct facts. Additionally, we show that attention head behavior is domain-dependent, with larger models exhibiting more specialized and category-sensitive patterns.

Published in Transactions on Machine Learning Research (07/2025)
Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models
Dante Campregher University of Amsterdam
donte.comrrggereettdeee.......
Yanxu Chen University of Amsteriam
yangnu.chemm@sttudedd........
Sander Hoffman University of Amsteniam
Maria Heuss University of Amsterdam
m. m.cccheuss@uv...
Reviewed on OpenReview: https: // openreview. net/ forum? id= 1QrB5WSWOR
Abstract
This paper presents a reproducibility study examining how Large Language Models (LLMs) manage competing factual and counterfactual information, focusing on the role attention heads in this process. We attempt reproduce and reconcile findings from three recent studies by Ortu et al. [3 Yu, Merullo, and Pavlick 17 and McDougall et al. 7 that inves- tigate the competition between model-learned facts and contradictory context information through Mechanistic Interpretability lools. Our study specifically examines the relationship between attention head strength and factual output ratios, evaluates competing hypothe- ses about attention heads' suppression mechanisms, and investigates the domain specificity of these attention patterns. Our findings suggest that attention heads promoting factual output do so via general copy suppression rather than selective counterfactual suppression, as strengthening them can also inhibit correct facts. Additionally, show that attention head behavior is domain-dependenn,, with larger models exhibiting more specialized and category-sensitive patterns. Our code is open soured
1 Introduction
The rise of Large Language Models (LLMs) has transformed our interaction with information, both through relying on the parametric knowledge that language model learned through training 15 and through using approaches like retrieval augmented generation (RAG) 5 to retrieve facts from given database. While these models achieve impressive results across many tasks, they often operate as 'black boxes," making it challenging to understand how they arrive at their outputs. Built on transformer architectures and trained on vast datascts, LLMs have become increasingly sophisticated, yot their decision-making processcs remain largely opaque. Yet, understanding whether generated answer comes from model memory from given context is essential for guaranteeing the trustworthiness of information access systems 16
The two competing mechanisms, the use of parametric memory (factual recall) and the adaptation of in- context information, come into direct conflict when counterfactual statements opposing the model's para- metric memory are provided as input. Hence, for understanding the origin of the provided answe,r we neod to understand the interactions between these mechanisms. This competition visualised in Figure [

---

Published in Transactions on Machine Learning Research (07 (07/2023 (025)
Pattern Match
Redefine: The official language of Australia is Indonesian.
The official language of Australia is
Induction Effect
Factual Recall
English
Indonesian
Figure Visualisation of the choice between counterfactual and factual token.
Recent years have seen significant efforts to interpret these complex language models. While early work fo- cused on investigating input-output relationships through perturbation analysis 6 newer research suggests that understanding LLMs requires examining their internal operations. Mechanistic Interpretability inves- tigates how different transformer modules contribute to model outputs, focusing on specific components such as neurons, attention heads, and circuits (subnetworks of interconnected components) that drive particular model behaviors.
Prior work has investigated the competition between factual and counterfactual information in language models, particularly focusing how different attention heads influence this competition. Yu, Merullo, and Pavlick 17 used a capital cities dataset to study how attention heads in Pythia-1.4B support either factual counterfactual tokens, while Orlu et al. 13 investigated the same mechanism for GPT-2 and Pythia-6.91 using the CounterFact dataset. McDougall et al. 7 specifically examined the copy suppression role of one specific attention head in GPT-2. These studics collectively suggest that certain attention heads play crucial roles in mediating between factual and counterfactual information, and that manipulating the strength of these heads can influence the model's tendency to generate factual versus counterfactual responses. Yet they disagree on the exact role that different model components play in this competition.
Our reproducibility study aims to validate and extend the findings of these three papers in three key direc- tions. First, we examine the generalisability of the relationship between attention head strength and factual output ratios. Second, we investigate competing hypotheses about the mechanism through which these attention heads operate, specifically whether they function through targeted suppression counterfactual tokens or through general copy suppression. Finally, we explore the domain specificity of these attention heads, building on observations from Yu, Mcrullo, and Pavlick [7 and Millidge and Black g about the potential specialization of attention heads to particular domains. Through our experiments, we successfully reproduce the core finding that attention heads contribute significantly to the competition between factual and counterfactual tokens, and that manipulating these heads' strengths affects the proportion of factual versus counterfactual outputs. However, our results provide evidence that these attention heads operate through general copy suppression rather than selective counterfactual suppression, as they also suppress fac- tual tokens when factual statements appear prompts. Furthermore, we demonstrate that attention heads exhibit domain-specifit specialization, with their effectiveness varying significantly across different knowledge categories, and this specialization becomes more pronounced in larger models. Through these investigations, provide more comprehensive understanding of how language models manage competing factual and counterfactual information.
2 Scope of reproducibility
In the scope of reproducibility, we focus on the works from Ortu et al. 13 Yu, Merullo, and Pavlick 17 McDougall et al. 7 and Millidge and Black g. We hereby extract their claims in different scopes and introduce some of their techniques, which we replicated or used. We also come up with some hypotheses test based the claims. The details of the techniques are introduced in section 3.1
2

---

Published in Transactions on Machine Learning Research (07/2025)
2.1 Existence of competition between factual and counterfactual tokens
When the prompt contains a counterfactual statement the model has to make a choice between repeating this counterfactual statement recalling the fact during generation. We focus studying the situation when the factual or counterfactual statement is single token; hence, we denote the factual token by lract and the counterfactual token by fcofa. The role of different attention heads in making the choice has been studied by Yu, Merullo, and Pavlick 17 Ortu et al. 13 and McDougall et al.
Yu, Merullo, and Pavlick 17 used a dataset about capital cities of countries to investigate how each at- tention head in Pythia-I. 1.432 supports fract or trofa with the logit attribution techniques [0 They also investigated the effect of multiplying the corresponding attention heads by a scalar, and found that decreasing the strength of attention head supporting flact will significantly decrease the ratio of factual output compared with counterfactual output.
Moreover, to investigate the competition in GPT-2 14 and Pythia-6.99 2. Ortu et al. 13 applied similar techniques using the COUNTERFACT dataset a collection of counterfactual assertions paired with gener- ation prompts. They found that increasing the strength at specific positions of attention heads supporting fact would increase the ratio of factual output approximately 50% on both models.
McDougall et al. examined the copy suppression role of the seventh head in the tenth layer (L10H7) of GPT-2, which is one of the heads analyzed by Ortu et al. 13 They suggest that this head plays critical role in the generation of factual responses when there is related counterfactual statement in the context.
Based on the combined findings from Yu, Merullo, and Pavlick 17 Ortu et al. 13 and McDougall et al. we propose the following hypothesis:
Hypothesis 2.1. In a transformer model, each head that contributes significantly the competition be- tween (fat and lcota supports either lfact loofa- When counterfactual statement exists in the prompt. increasing the strength of heads supporting fact will increase the proportion of factual outputs, while de- creasing their strength will reduce the proportion of factual outputs.
We try to further test. how this claim generalizes using methods described in section 4.2.1 and the results can be found in section 4.2.
2.2 The mechanism of the competition
While prior work generally agrees on the existence of such attention heads, there is disagreement on the exact underlying mechanisms that support them. Based on the findings of these papers, we formulate the following two hypotheses:
Suppression of counterfactual tokens. The work from Ortu et al. 13 called the attention heads that support lfct 'factal recall heads', and suggests that those heads increase the ratio of factual responses by suppressing the copying of counterfactual tokens. Hence, we come up with the following hypothesis about the mechanism:
Hypothesis 2.2.1. The attention heads supporting lact increase the ratio of factual responses by suppress- ing tokens that the model considers non-factual answers.
The anti-induction effect. Olsson et al. 12 introduced the term 'induction' head, which "completes the pattern" by copying and completing sequences that have occurred before, and discovered some 'anti- induction' heads, which have negative scores on the induction effect. McDougall et al. 7 further conducted experiments on various datasets and showed that L10H7 GPT-2 has a general anti-induction effect. These findings may indicate that L10H7 has more general role in copy suppression beyond just suppressing counterfactual statements. We have also noticed that, when Yu, Merullo, and Pavlick [17 increased the overall strength of /act supporting attention heads for Pythia-1.4BB, the ratio of factual responses never
TThe usage of the terms 'factual' and 'counterfactual' in further sections may be mislcading to some readers, and may conclude that the terms 'correct' and 'incorrect' would have been more accurate. We acknowledge this. Our choice keep these terms remain consistent with the labels in the COUNTERFACT dataset and work based on
3

---

Published in Transactions on Machine Learning Research (07/2025)
exceeded 20%, suggesting that the heads may only have limited capability for selective suppression of counterfactual statements. Hence, alternative hypothesis is:
Hypothesis 2.2.2. The attention heads supporting fact increase the ratio of factual responses by suppress- ing the induction effect (i.e., copying from the context by matching content and structure). Consequently, when a factual statement appears in the prompt, these attention heads will also suppress the copying of that statement.
We compare these two hypotheses using methods described in section 4.3. and the results can be found in
section 4.2.
2.3 Domain specialization of heads engaged in the mechanism
Ortu et al. 13 did not specifically study how different. attention heads work on different domains, and
McDougall et al. 7 primarily focused on a single attention head. However, Yu, Merullo, and Pavlick 17 found that their identified attention head could not generalize well to datasets with more domains, and using the SVD technique from Millidge and Black g they discovered that the heads they identified were highly related to goography. The work from Millidge and Black | itself also suggests that different attention heads correspond to different domains. Based on these findings, we propose the following two hypotheses:
Hypothesis 2.3.1 (Existence of domain specialization). Attention heads do not uniformly suppress the induction effect or the copying of general counterfactual statements across different domains. Instead, their effects vary depending on the lype of information required to answer the question.
Hypothesis 2.3.2 (Domain specialization and model size). As model size increases, attention heads become more selective and specializod in their suppression patterns.
We attempt to verify refute these two claims using the methods described in section 4.4.1 and the results are presented in section 4.4.2
3 Methodology
Ortu et al. 13 have provided their code on GitHub, which we adapt and use for our experiments. They have also shared the dataset they are using. They have not shared their run results. As such, we first try to replicate their results by doing the same experiments.
Our main goal is to test the validity and generalisability of hypothesis 2.1 to compare hypothesis 2.2.1 and hypothesis 2.2.2 and to justify or refute hypothesis 2.3. and 2.3.2
3.1 Analysis techniques
3.1.1
Logit attribution techniques
We, like Ortu al., use the transformer lons package extract the outputs of the individual attention heads in each layer. The resulting output is grouped by token position to obtain the outputs per layer per head certain token position. The outputs of the final token position and ffac are inspected. A head at a certain layer promotes fact if in the unembedded output the logit for fract is increased after the head when compared to before. In many experiments, the measure used to compare the logits of the tokens is the difference of lcola and Lact, defined as Acola == Blockl (oockloogit(/oot// BloockLogitt//aa (4raet)
then unembedded and the logits of frota
3.1.2 Attention modification
13 used attention modification to study the role of attention heads. More specifically, Ortu et al. 13 used the following setup: Let Ah be the attention matrix of the h-th attention head in layer I, we then modify a position (i,j), where i a higher value than j. This corresponds the i-th token in layer attending to the j-th token in layer 1. The full equation for attention modification now becomes:
Yu, Merullo, and Pavlick 17 and Ortu et al.
Alhl a A
4

---

Published in Transactions on Machine Learning Research (07 2025)
where is the amount of modification.
It is also possible to conduct this operation to all different j values, increasing the overall strength of the attention head among all positions. Yu, Merullo, and Pavlick 17 did not specify whether they modified the attention values for all positions or just for the position of fcola explicitly, but considering the difference between their results and the results from Ortu al. 13 who only modified the attention values for the position of feofa, we have reason to believe that they used a different designation than Ortu al. 13
modifying the attention values for all positions.
In our study, we stick with the method used by Ortu et al. 13 modifying only the attention score from the position of cool to the position of fject.
3.1.3 SVD analysis OV matrices of attention heads
The product of the Value and Output weight matrices in a head is known as the OV matrix 4 This matrix
is interpretable by decomposing it with the singular value decomposition (SVD) 9 Namely, with:
WoWy USV
the singular vectors V can be unembedded and detokenized to show what tokens are most encoded by the head. Each token's logit & can be found with:
= Vii,:E"
We utilize this technique in experiment 4.4.1 investigate the roles of individual heads of interest.
3.2 Model descriptions
For our experiments, we also used GPT-2 and Pythia-6.91. GPT-2 has 12 heads and 12 layers, with total of 117 million parameters [4 Pythia-6.9B has 32 heads and 32 layers with total of 6.9 billion parameters
2. Both models are pretrained. These are the same models that were used by Ortu et al.
13
More specifically, according to the results from Ortu et al. 13 Yu, Merullo, and Pavlick 17 McDougall
et al. and others, we are the most interested in the following attention heads supporting Lnet, where LoHy denotes the y-th head in the z-th layer:
For GPT-2, we are the most interested L10H7 and L11H10.
For Pythia-6.9BB we are the most interested in L17H28, L20H18, and L21H8
3.3 Original Datasets
In their paper, Ortu et al. construct their datasct using the pattern "Redefine: (base prompt} (falschood). (base p. An example of an entry in the dataset is "Redefine: The official language of Australia is Indonesian. The official language of Australia is". They source the falsehoods, associated correct facts, and sentence patterns from the COUNTERFACT dataset g The COUNTERFACCT dataset is a large set these base prompts, with associated counterfactual answers and factual answers. The model queried for single token, which is then classified either the copied falsehood the correct fact. Only prompts that the LLM could answer correctly in a single token without the preceding counterfactual statement were considered, and of those, they chose to randomly sample 10 thousand prompts. They did this for both the models they used, so they ended with two datasets of 10 thousand entries. One for GPT-2 and one for Fythia-6.9B.
We used the same datasets as the authors used for replicating their results. Both the GPT-2 and the Pythia-6.9B dalasets contain 10, 000 entries. Each entry comprised of 5 fields:
"base prompt" contains the original prompt from the COUNTERFACT dataset.
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
"template" contains the original prompt, but now duplicated with field to be filled in before the original prompt, where redefine is always placed, and field to be filled in before the duplicated sentence, where the counterfactual token is always placed.
"target_true" contains the factual token.
"larget new" contains the counterfactual token that is placed in the sentence.
"prompt" contains the prompt to be given to the model, which is the template but with the values filled in.
There is also a variant each dataset that contains a subject field with the subject of the sentence. This is used determine the position and length of the subject in cach prompt, allowing us group the subject tokens together when showing results.
4 Results
4.1 Results reproducing original papers
Competition of mechanisms. We have successfully reproduced most of the work from Ortu et al. 13 by re-running their original experiments. Most of the results re-running the original experiments align with the original results, showing a decent. reproducibility of the results of the original paper.
Additionally, we generated Figure as supplementary result, which shows the proportions of factual and counterfactual predictions under different values. The blue area represents the percentage of the model generating the factual token, while the pink area is for the given counterfactual token. The gray area represents the case where the model generates neither factual nor the given counterfactual token.
From the figure, we can see that as the value a increases, the factual prediction ratio first increases from the drop of counterfactual prediction, then decreases due the model being unstable and generating other prodictions. Hence, a moderate a value gives us the best result. However, there slight mismatch in the detail:
Mismatch on best a value for GPT-2. Figure 2 shows the proportions of factual and counterfactual predictions under different a values. For GPT-2, the maximum factual prediction rate is obtained at a = 10, contradictory to the statement the original paper that the best a value for GPT-2 5. However, a 5 also yields a similar factual prediction rate, so this difference not critical.
Random attention modification as basclines. In Appendix B.1 we have presented our results of applying atlention modification on random heads other than heads of interest, to show that the overall trend of percentage different responses caused by modifying the specific attention heads, not solely by the attention modification action itself.
4.2
Experiments on the sensitivity to specific wordings and patterns
4.2.1
Experiments
Different premises. We change the words before the semicolon in each prompt from redefine to one of the following: "Repeat", "Define", "Update", "New fact", "Updated fact", "This true", "The following is truc", "Pretend the following is truc", and "Accept the following statement be truc".
We then measure for each head at each layer in the last position how much it promotes either the factual the counterfactual token. We also test attention modification on the heads L11H10 and L10H7 for GPT-2.
Different sentence structures. Most of the sentences from the original dataset share a fixed pattern, namely "(subject) (relationship) (object).", which means that we cannot climinate the possibility that the attention heads of interest are focusing on specific grammatical patterns instead of recalling facts. Hence, we modified the sentence structure test the generalization ability of the model different sentence structures.

---

Published in Transactions on Machine Learning Research (07 (025)
of medification
GPT2
Pytha 6.90
20
proditio Courtelldial
30
10P
15
ACra
Apma
Figure Results of attention modification on GPT-2 and Pythia-6.9B.
For each combination of two patterns selected from predefined set of three, dataset is generated by respectively applying both patterns to the two sentences in each entry of the data. The three patterns are as follows:
NORMAL: "(subject) (relationship) {object]."
CoNSIDER: "Consider {subject). It {relationship) {object]."
PROPLEKNOW: "Many people know (subject). It (relationship) {object)."
We test model performance the patterns by mcasuring factual recall and counterfactual copying the new structures. We also perform attention modification on the same heads and with the same alpha values as the original paper.
4.2.2 Results
Our results show that hypothesis 2.1 holds when performing the same experiments. This can be observed in table D in the with both first and second sentence NORMAL. However, when changing the sentence structure, the results become less pronounced than in the original setting.
Different premises. The supported answer of the significant heads in GPT-2 does not change when modifying the premise. This supports hypothesis 2.1 The strength of the heads, however, does appear vary between premises in a limited degree.
Different sentence structures. As shown in table ! when the sentence structure changed, boosting the attention of the factual token promoting heads by a = or = 10 can still provide an increase to factual recall, being consistent with the original result. This indicates that hypothesis 2.1 can be generalized with respect sentence structure changes.
The maximum effect. (bold results in table 1 of changing the attention heads is lower if the two sentences have the same structure, and is higher if the two sentences have different structures. We hypothesize that when the first sentence is of different structure than the second sentence, it is not clear for the model that we expect it to copy lcola, leading to a less strong bias towards it.
Our results show that a = 10 yields better results than a 5 when the two sentences have the same structure, while a 5 yields better results than a when the structures are different. This may be due that, when the structures are the same, the induction effect is stronger, and hence the modification needs to be stronger to mitigate the effect.

---

Published in Transactions on Machine Learning Research (07/2025)
First. Sentence
Second Sentence
Q
=
0
a =
Q =
2
Q =
a
10
a = 100
NORMAL
65
412
1377
5032
5274
70
NORMAL
CONSIDER
87
603
2209
6175
3924
13
PROPLEKNOW
98
749
2640
6066
3474
5
NORMAL
506
2186
4895
7651
5427
CONSIDER
CONSIDER
45
267
2896
4481
61
PEOPLEKNOW
19
259
1279
5636
4794
12
NORMAL
451
1928
4465
6988
3817
PEOPLEKNOW
CONSIDER
79
761
2452
6126
3264
PEOPLEKNOW
82
2990
3297
15
Table 1: Table with grouped sentence structures and alpha values. The values indicate the number of times the factual token was predicted among the 10, 000 entries. The highest number of each sentence combination is bold.
4.3 Experiments on the general role and mechanism
4.3.1 Experiments
Different presence of hints. Many of the sentences from the original dataset (62.3%) contain the correct answer in the subject explicitly, which means that we cannot eliminate the possibility that the attention heads interest are in charge of copying from a specific part, instead of suppressing counterfacts or the induction mechanism. Hence, we split the datasot into two parts; one contains all sentences with the true larget substring of the subject, called the hinted dataset, while another contains the remaining sentences, called the no-hint dataset.
Effect of changing the counterfacts to facts the prompt. We generated datasets from the original datasets by replacing the counterfactual targets in the prompt with factual targets, which means that in these constructed datasots, the prompt will first explicitly state the fact, then ask the model to repeat the factual sentence.
If the attention heads of interest are heads suppressing the copying of counterfactual tokens hypothesis 2.2.1 stated, incrcasing their strength should not significantly affect the performance of the model, while if those heads are heads with anti-induction effects as hypothesis 2.2.2 stated, increasing their strength should negatively affect the performance because the factual token is suppressed.
4.3.2 Results
Our results favor hypothesis 2.2.2 more than 2.2.1 showing that the attention heads supporting frct more likely to be related with the overall suppression of the induction effect, rather than the selective suppression of counterfactual statements.
Different presence of hints. The results are shown in Figure 3 and 4 When using the originally optimal value of 10, GPT-2 predicted laet in 71% of the time when fract was substring of the prompt (hinted), whereas it did so only in 23% of the time when it was not (no-hint).
Morcover, in the datasct where fact was not substring. the model predicted neither feota nor ffet in 45% of cases. These results suggest that these attention heads are not particularly important for factual recall on their own, which was also supported by the original authors, and McDougall et al. 7
Comparing this to the Pythia-6.9B results, we observe similar pattern: the model predicts fact more often when can copy from earlier in the sentence. However, the difference between the hinted and no-hint datasets is much smaller than in GPT-2, suggesting that Pythia-6.9B exhibits factual recall. The
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
substring original
Courartacttall
10K
Xpha
Figure 3: Results of GPT-2 attention modification when splitting the dataset based on whether or not fact was part of the subject or not. The hinted dataset has 6229 entries and the no-hint dataset has 3771 entries.
modtfiaaton substring original sentence (Pytha
No
1.
Axtiol
Coumertactual
48
Moha
Figure 4: Results 6ythia-6.9B attention modification when splitting the dataset based on whether or not ffact was part of the subject or not. The hinted dataset has 2855 entries and the no-hint dataset has 7145 entries.
attention modification in the no- ho-h cases increases factual predictions Pythia-6.9E from about 30% about 40%.
Another key difference between Pythia-6.9B and GPT-2 is the dataset composition. In Pythia-6.9B, the hinted dataset contains 2,855 entries, while the no-hint dataset has 7,145 entries. We theorize that because the authors selected entries from COUNTERFACT that the models could answer correctly, the GPT-2 dataset included more prompts where factual recall was unnecessary. In contrast, the Pythia-6.95 dataset contained more prompts requiring factual recall. This is likely due to the model's larger size, allowing to recall/record more difficult facts than GPT-2.
Effect of changing the counterfacts to facts in the prompt. The attention modification experiment result on the constructed dataset that replaces the counterfacts with facts is shown in Figure 5 The proportion of factual responses steadily decreases when the a value increases, even a < 5.
For GPT-2, when a or a = 10, the model answers the factual prediction for less than 90% or 70% the time, respectively. For Pythia-6.9B, the model answers the factual prodiction for less than 70% of the time when a 10, mostly returning "the" "also" as the response token when does not answer the factual prediction.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
This may indicate that the heads of interest are not heads suppressing solely counterfacts, but are heads with general anti-induction effects, and editing the model by increasing the strength of the specific attention heads cannot generally guarantee the increase of factual recalls as hypothesis 2.2.1 suggests.
OP12
Figure 5: Results of attention modification after changing the counterfactual targets to factual targets in prompts.
4.4
Experiments on domain specialization of heads
4.4.1 Experiments
Effect of subject and answer categories on strongly contributing heads. To test hypothesis 2.3.1 wish to if heads have bias towards certain topics or domains of knowledge, induction anti-induction attention heads playing identical roles. We do this by splitting our dataset into possible domains of knowledge. To categorize the answers, we first extracted the types of text patterns in the dataset, and matched them with the original patterns in the PARAREL dataset 3, a dataset from which the COUNTERFACT dataset was derived. We then used GPT-4o to classify the categorics of subjects and answers of each type of lext paltern the dataset and manually checked the classification results. Then, we add the category information each entry the dataset and filter the dataset by categories. We then perform logit inspection of the attention block for each head of cach layer in the final token position of filtered subsets of the dataset. Furthermore, to test hypothesis 2.3.2 we observe how the behavior extends to larger models.
if they general
SVD analysis of relevant attention heads. Using the SVD technique 9 described in section 3.1.3. we inspect the allention heads of interest found in the previous experiment, to verify that any variation results between categories corroborated by the knowledge encoded in those respective heads.
coonbUcT CpSW
JtenBusT 000000
L00000 UOQIOOT
O
Layer 11 Hoad 20
0.00
0.00
Layer 10 Head 20
835
8.27
6:34
2:38
3:28
3:3
0.00
0.00
044
Layer
Layer Head
Layer Head
8:08
238 2:7
088 030
Figure 6: Attention scores of relevant heads for the most common categories that GPT-2 can predict.
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
LTH $17
2EH 117
82H 177
0TH 811
62H 617
LEH 617
2H 120
8EH 020
2H 22111
122
GH 122
Answer category
Full
Location
0.16
0.04
-338
0.07
00.44
0.39
0.22
-337
00.1
-025
030
Language
4.36
Organization
0.20
0.29
3.35
0.15
0.25
0.01
0.04
0.02
0.11
0.15
0.01
-0.5
0.0
0.5
Figure 7: Heatmap of the strength of the logit difference for different heads and categories.
4.4.2 Results
Effect of subject and answer categorics on strongly contributing heads. The heads in GPT-2 that were found by Ortu et al. 13 most promote the counterfactual recall were L9H6, L9H9, L10HO, and L10H10, and the heads that most promote the factual recall were L10H7 and L11H10. Figure c illustrates the contribution of each of those heads to Acof in each slice of the categorized dataset. Evidently, subject and answer categories have a strong effect on the heads responsible for competition. To test if the effect is based on subject or answer category, we compare datasets filtered on both categories, differing in only one category of either subject or answer from a baseline. In our case, we picked the dataset slices with the most remaining results after filtering. The filtered dataset slices are truncated to be of equal size, 124 entries. The latter two heatmaps in Figure demonstrate that the change in subject category is far less significant than the change in answer category.
These findings show that answer categories highly determine the contribution of heads to the competition, suggesting hypothesis 2.3.1
To address how this effect changes with model size, we perform this on Pythia-6.9B. The heads we compared were chosen after inspecting the contribution of heads in different answer categories. For that analysis, see appendix [322] We choose the heads that are most influential in the competition, and heads that have interesting variation between categories that we would like to investigate. The chosen heads are compared in Figure 7 The heads most supporting teola and fact are L15H17 and L20H18 respectively 13 Our results show that some influential heads do not vary much, but others vary extremely, much more than in GPT-2. The influence of heads appears more sparse in their effect on categories, with many categories being affected negligibly by cortain heads that are highly influcntial on other categorics (L16H17, L19H31, L20H18, L21H7, L24H5). Some heads (L18H10, L22455) even significantly support loola certain categories and Lfat others. Regarding these two heads, because their mean Acofa over the total dataset is close to zero (0.00 and 0.03 respectively), they may have been disregarded as not relevant in previous literature when investigating only the full dataset, despite on average having large influence of T,24H5 on full dataset: 0.78). Our observed patterns in this experiment are in contrast to the results on GPT-2, where the most influential heads vary in strength of contribution, but regardless of category, contribute a significant amount to either 4Fat or lrofa- These results demonstrate the generalisability of our previous findings, and suggest hypothesis 2.3.2] but experiments with various model sizes on the same model architecture would be helpful truly confirm this.
individual samples (standard deviation
SVD analysis of relevant attention heads. To verify that the patterns in the previous experiment can be attributed to categories, we perform the SVD techniques described by Millidge and Black g on various heads of interest, illustrated in Figure g and Figure 13 which can be found in the appendix.
Looking fact promoting heads, we have L,2H77 (Figure Sab, L20H18 (Figure Sbb. Again, with L21H7 and L20H18 we see the complement in categories of Organization and non- non- non-Organization, respectively. In this case, the words encoded by the layers far more interpretable and coherent than for the lenta promoting
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
heads. The words encoded by L20H18 very clearly relate to the topics Location and Language, expected. The words encoded by T21177 do not directly relate to organizations, but rather to the topics of food, clothing, and hobby. It should be noted, however, that L21H7 is much less contributing to the competition than L20H18.
These results remain consistent with our findings the previous experiment, and further support that the heads do not always contribute the competition between fooa and fact in general, but are dependent on the domain of information required to correctly answer the prompt.
Food
mutrition
clothing
SPW
Playing
player
adoles
England
Spanish
Chile
French
Pherre
Top-k tokens
food
lasted
garments
gown
played
Play
0.4
Top-k tokerss
rogim
emerg
Mexican
Spain
France
Philippe
dietary
garment
faps
play
Ray
Garc
Belgian
Food
culinary
fabrics
sewing
playing
playback
0.3
British
Britain
Mex
Guatem
Prance
Quebec
Foods
Playing
listening
RAF
Spanish
Madrid
Parts
R8
eating
nutritionaal
dress
shredded
players
sound
0.2
England
defic
Mexico
Barcelona
Monsleur
0.2
Singular directions
Singular directions
(a) Layer 21 Head 7
(b) Layer 20 Head 18
Figure 8: Heatmap of the top-k tokens of the singular vectors sorted by their singular values.
5 Discussion
Generally, our experiment results support the hypothesis 2.1 that when the prompt contains counterfactual token loofa, there attention heads that contribute significantly to the competition between frac and leofa: and moderately manipulating the strengths of these attention heads will affect the proportion of factual and counterfactual outputs accordingly, as the results supporting this hypothesis are reproducible, and the hypothesis can generalize to different premises and sentence structures.
However, this does not necessarily mean that moderately increasing the strengths of these heads will guar- antee the generation of more factual responses under all circumstances. When we replace the counterfactual token with the factual token the original dataset, these heads now suppress the factual token. This suggests that these heads are more likely involved in copy suppression rather than specifically counterfactual sup- pression, supporting hypothesis 2.2.2 and challenging dypothesis{2.2. This also suggests that manipulating the strength of specific attention heads to increase the factual recall rate may not be generally applicable, or needs more improvements and modifications.
Furthermore, we have gained insights into the effect of domain knowledge on how heads contribute to the competition. We have seen that the heads do not uniformly suppress the induction effect across different domains, confirming our hypothesis 2.3.1 But we believe more robust experiments are needed to fully confirm hypothesis 2.3.2 Our results do suggest it true, but to confirm it, more model sizes within model family should be tested. We have also investigated further into the information encoded by relevant heads. We find that some heads, most commonly tfet promoting heads, encode information corresponding to the domain of fact and feola. Further work could be done by using a counterfact dataset where the domain of the factual and counterfactual tokens differs.
While our research covered multiple scopes and topics, a limitation of our experiments is the fineness of our experiments. More specifically, we did not run the experiments on more models, and we also did not study the attention modification of different heads different domains. Future work can be in the direction of applying the methods to different models and studying how each attention head interacts with different domains. Considering that the dataset we used is mostly about locations, languages, organizations, and people, constructing new datasets on other domains like STEM and applying similar techniques on them is also a possible direction for future work.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
References
[1] Leonard Bereska and Efstratios Gavves. "Mechanistic Interpretability for AI Safety-A Review". In: arXiv preprinl arXic:2404. 14082 (2024).
[2] Stella Biderman al. "Pythia: A suite for analyzing large language models across training and scaling". In: International Conference on Machine Learning. PMLR. 2023, pp. 2397-2430.
[ Yanai Elazar et al. "Measuring and Improving Consistency in Pretrained Language Models". In: Transactions of the Associalion for Computational Linguistics 9 (2021). Ed. by Brian Roark and Ani Nenkova, pp. 1012-1033 1032- DOI: 11.1162/tac1..... _aa____00000 00410
URL: ssapp://aclanthooogg..om 2r2/
tac1-1.60/
[4] Nelson Elhage et al. "A mathematical framework for transformer circuits". In: Transformer Circuits Thread 1.1 (2021), p. 12.
[5] Patrick Lewis et al. "Retrieval-angmentte generation for knowledge-intensive nlp tasks". In: Advances in neural information processing systems 33 (2020), pp. 9459 9474.
[6] Jiwei Li, Will Monroe, and Dan Jurafsky. "Understanding neural networks through representation crasure". In: arXiv preprint arxi::1612 0.2200 (2016).
[7] Callum Stuart McDougall et al. "Copy Suppression: Comprehensively Understanding a Motif in Lan- guage Model Attention Heads", In: Proceedings of the 7th BlackborNLP Workshop: Analyzing and In- terpreting Newral Networks for NLP. Ed. by Yonatan Belinkov et al. Miami, Florida, US: Association for Computationnl Linguistics, Nov. 2024, pp. 337 363. DOE: 10. 18653/v1/2024 blackboxnlp-1..
URL: htttps://aclanthollogg htg.:/aa......a..o//o///////aaaa
[8] Kevin Meng et al. "Locating and Editing Factual Associations in GPT". In: Advances in Neural Information Processing Systems. Fd. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022,
pp. 17359 17372. URL: https / pwwcee/inocceese.eoooo neurips cc /ppppper.... files / /apprrppppppere / 2022 file/ pdf
[9] Beren Millidge and Sid Black. "The singular value decompositions of transformer weight matrices
are highly interpretable". In: Less LessWrong (2022). URL: https: :// www lesswrong com posts /
[10] Neel Nanda and Joseph Bloom. TronsformerLens. https // github com Traassoorrrreennnlggggggg
TransformerLens] 2022.
[11]
nostalgebraist. "interpreting GPT: the logit lens". In: Less Wrong (2020). URL: https : // www
[12] Catherine Olsson et al. "In-context Learning and Induction Heads". In: Transformer Circuits Thread httb//222/i-cooteet-eemmaii hanar--eaaaiinnnnnnnnnnnttoonoo -nd--anductioon-eeeeeeee hew.lion-naaadddddn...t.teee
(2022). https:/ http://ttaannfommrrrrrrisst
[13] Francesco Ortu et al. "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals.. In: Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). Ed. by Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 8420-8436 DOI: acl acl- 14] Alec Radford et al. "Language Models Unsupervised Multitask Learners". In: (2019).
10. wt.11833vv122222 acl-long.
Molong.58. 458 URL: https://acclanthoooggyyooo
[15]
Adam Roberts, Colin Raffel, and Noam Shazeer. "How Much Knowledge Can You Pack Into the Parameters of a Language Model?" In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020, pp. 5418-5426.
[66] Jonas Wallat et. al. "Correctness is not Faithfulness in RAG Attributions". In: arXiv preprinl (Dec. 2024).
arXiv:2,12.1800
[17] Qinan Yu, Jack Merullo, and Ellie Pavlick. "Characterizing Mechanisms for Factual Recall in Language Models". In: The 2023 Conference Empirical Methods in Natural Language Processing. 2023.
13

---

Published in Transactions on Machine Learning Research (07 (025)
SPeats 4ftes mwdrroote
LPfcto of stenden mod zaton
phict of abors modm
Figure 9: Attention modification random heads in the same layer heads of interest.
A Computational requirements
We ran our experiments involving GPT-2 on Ubuntu 22.04.2 on a computer with an RTX 4070 12Gb and an (R)) Core(TM) 15-9600K CPU @ 70GHz. Experiments involving Pythia-6.9B were conducted on a single NVIDIA A100 GPU from the Snellius cluster. In total, we used about 10 hours of GPU time on the RTX 4070, and about 14 hours on the A100.
B Additional results
B.1 Baseline of attention modification
To show that the results from attention modification are results related to the specific heads instead of general results from modifying attention heads, we conducted an experiment by modifying some random attention heads other than the interesting attention heads in the same layer the heads of interest for GPT-2. The result is shown in Figure g under 4 different seeds.
After modification of random attention heads, the model mostly keeps outputting counterfacts under different a values no greater than 10, and above 10, the model may sometimes start to output low-quality predictions that are neither the factual prediction nor the counterfactual prediction. This indicates that, when the a value moderate, the model's performance would only be significantly changed if the attention heads of interest are modified, while extremely large a values can cause the model to give low-quality predictions due to weights being altered too much.
B.2 Head selection for comparison in experiment 4.4.1
Figure I0 and [ were used to identify important heads within and between categories using the setup from
4.4.1
Each plot shows the grid of all heads across all layers, where the color indicates the mean influence of that head on Apoa within that dataset category. The standard deviations over the categories are shown in Figure This highlights which heads differ the most across categories.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Full dataset
(b) Location calegory
(c) Language category
(d) Organization category
Figure 10: Mean Acota values of each head in Pythia-6.9D for each of the analysed categories
15

---

Published in Transactions on Machine Learning Research (07/202) 2025)
25
24
24
22
21
20
19
13
10
0.2
6.14
Figure 11: Standard deviation between the mean Acota value of the three analyzed categories of the dataset
16

---

Published in Transactions on Machine Learning Research (07/202) /025)
Full dataset
(b) Location calegory
(c) Language category
(d) Organization category
Figure 12: Standard deviation of Acota values of each head in Pythia-6.99 6.9B for each the analysed categories
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Al
aft
Washington
DCs
Ai
Leicester
1.0
akcspe
Top tokens
Washington ASHINGTON
Marin
Top-k takens
romcs
uve
the
1275
0.30
eig
DC
Logan
San
SAN
0.8
OTHER
0.25
AL
almond
DC
Arlington
San
Calff
aluminum
des
FBSS
these
0.20
aluminium
sigae
Waxh
Obama
Alto
Alleg
urther
the
0.15
Singular directions
Singular directions
(a) Layer 19 Head 31
(b) Layer 15 Head 17
midd
cadhennn
openh
1.0
Australan
Melbourne
Canadian
Ohro
Britssh
London
Top-k tokens
C
aturday
SAS
Top-k tokerss
Austr alian
Alberta
Canada
Vermont
British
England
0.7
Sprintf
0.6
0.6
PQ
wm
cf
B
Australtams
Canada
Canadians
CBC
UK
Victoni
0.4
Swedish
Ohio
Britain
Califonia
mont
Chation
SZ
plat
Canadian
Brisbane
Canadian
Toronto
Brit
Kentucky
0.4
Singular directions
Singular directions
(c) Layer 16 Head 17
(d) Layer 24 Head 5
Figure 13: Heatmap of the top-k tokens of the singular vectors sorted by their singular values.
B.3 Further SVD analysis of relevant heads
Starting with L15H17 (fig 13b), head that globally promotes fool. We do not see many tokens associated with meaning beyond the low-level grammatical structure of language. This is consistent with the finding of Yu, Merullo, and Pavlick 17 that their head contributing most to leofa was not related to any specific domain knowledge. Millidge and Black did, however, find that earlier layers more commonly contain such heads encoding punctuation or less interpretable tokens.
Continuing to look at lcofa promoting heads, L16H17 (Figure 133g and L19H31 (Figure 13a seem to behave as each other's complement in categories. Namely, only being effective in the Organization category or only in the non-Organization categories, respectively. Do note that Location and Language are related (e.g., England, English), so this is not too surprising, We see that in this case, there is a minor association with the domain for L19H31, containing some singular directions with geography information, particularly of the United States, but also some with organic and non-organic materials, such as 'aluninum', 'algae', and 'almond'. These do not have trivial association with the topic of Organizations either however. L16H17 is less interpretable, but notably contains no geographic information.
Our last head L24H5 (fig 13d,, is very interesting, as it significantly supports loola for the Location category but lact for the Language category, and has little bias for the Organization category. We see that the words are very interpretable and geographically related, but curiously, almost all words are specifically related to English-speaking commonwealth nations.
18


Reviews:
[{"review_id": "0X04OLY894", "reviewer": "Reviewer_iBL3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The experiments are thorough, and the conclusions are convincingly supported by the experiments. The experiments can sufficiently distinguish between the proposed competing hypotheses.\n2. The topic itself could interest a lot of people. Providing explanations for the factuality of current LLMs might be appealing not only to mechanistic interpretability researchers, but also the broader communities interested in trustworthy AI systems.\n\nWeaknesses:\n1. The paper has made its scope of reproducibility very clear, but the motivation behind the chosen scope is insufficiently expressed. Despite all three papers examining the factual recall mechanism of transformer-based LLMs, the reason why the specific three works are selected is not adequately justified. The readers could be uncertain whether these works are the most representative and most recent ones in the area. Furthermore, from my perspective, it would be beneficial if a further discussion on how these three studies are positioned within the broad landscape of mechanistic interpretability.\n2. Despite systematic and sound experimental designs, the methods that the authors adopt to explain factuality mechanisms are not novel. This paper can benefit from a novel theoretical framing that provides new insights into mechanistic interpretability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a reproducibility check of a few recent works in the field of mechanistic interpretability. More specifically, this reproducibility study focuses on three papers studying mechanisms for factual recall, revealing how LLMs dynamically balance pretrained knowledge with contextual knowledge. The contributions are three-fold:\n- This paper provides an open-source implementation that replicates the key experiments from the three target papers, which significantly facilitate transparency and reproducibility. This paper and code implementation can serve as a basis for future research in this area.\n- The authors compare competing hypotheses regarding the mechanism of factual recalls (hypothesis 2.2.1 vs 2.2.2) with their experiments to establish a clearer consensus in the research community.\n- They draw inspiration from one of the reproduced papers and propose two new competing hypotheses (hypothesis 2.3.1 vs 2.3.2) concerning domain specialization, which itself is a new problem in the field. The authors validates the domain specialization hypothesis through their experiments, offering new insight into the encoding of domain-specific factual knowledge in LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The current experiments are only conducted on GPT-2 and Pythia-6.9B. The factual recall mechanisms discussed may be of great interest to a broader audience, especially those who are working with widely used open-source models (e.g. LLaMA, QWen). If the authors do not consider it beyond the paper's scope, it would be valuable to examine the hypotheses on these popular models and see whether the conclusions are generalizable to different models.\n2. The summary of conclusions does not appear until the last section. Reorganizing the paper to bring a summary of the main conclusions earlier in the paper could make it more informative to the readers interested primarily in the conclusions instead of the methodological or technical details.\n3. As is mentioned in weakness 1, I would recommend a better justification for the motivation behind selecting this scope of the paper. Providing more discussion on why the three papers are chosen helps strengthen the overall narrative of this paper.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BIfwkP6ZsU", "reviewer": "Reviewer_pXAs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n- Good reproduction works highlighting some mismatches\n- Novel analysis with fine-grained perspective on self-hinting and topic specialization\n\nWeakness:\n- The setup is not fully documented. See requested changes\n- Some hypotheses miss more convincing evidence:\n    - H2.1\n        - \"a small number of attention heads contribute most significantly to the competition\" is used as a given without experimental evidence. While this probably is true base on Fig 8 estimates, further analsyis is needed. Providing the actual number would help. \n        - Discussion to support this hypothesis is minimal. In 4.2.1, no data or figures (e.g. Table 1 should) are cited to prove the boosting effect of factual token logits. \n    - H2.2.1 and H2.2.2: promising, but lacks baseline values. It is not very clear whether the output of other tokens (the grey area in the graphs) are a result of \"general anti-induction effects\", or disruptive scaling of activations. For example, the alpha numbers chosen (2 ,..., 10), while followed previous works, are still not well motivated. Intuitively, you can destroy utility of a model (mabybe just output only \"the\" or \"also\") with enough scaling of a single parameter, but we do not know if 10 is close to that disruption or not for the two models. Absolute values here are not interpretable. \n    - H2.3.2: more experiments of models of different sizes in the same families must be done to make claim about model sizes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reproduces previous results on mechanisms of attention heads on whether the LLM outputs factual or counter factual information when contextual informaiton is at conflict with the real information, specifically\n- The existence of attention heads that influence the encourage factual output (over contextual counter-factual info)\n- How positive scaling of target attention heads influence the model factual/counterfactual competition\n\nWith the following novel perspectives:\n- Whether the attention heads suppress more of counterfactual \"token copying\" (H2.2.1) or \"format matching\" (\"induction\", H2.2.2)\n- Whether certain \"self-hinting\" subject (e.g. Singapore's capital is Singapore) influence this competition\n- Whether the attention heads in questions are topic specific (H2.3.1)\n- Whether H2.3.1 is sensitive to model sizes (H2.3.2)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to above: \n- Metrics in this work is not consistent. This paper used %$t_{fact}$ and $\\Delta_{cofa}$ interchangably. %$t_{cofa}$ should also be presented, or use $\\Delta_{cofa}$ consistenly, or explain how they relate.\n- Questions: Some previous work you cited also experimented with negative $\\alpha$ and had success. Is there a reason you do not experiment with them to better support the attention heads' effects?\n- H2.3.2: either have more experiments of models of different sizes in the same model family, or remove any conclusions about it.\n- Missing details:\n    - Figures 2 and 3 lack descriptions on whether all heads were scaled by the same $\\alpha$ or not. \n    - Appenfix A.1 paragraph is not complete. The setup is also missing (e.g. what does each entry describe) and not understandable for readers who did not read previous works. (Also Fig 8 fonts too small) This setup could also better be described in S3.4.3\n- Typos: \n    - `the the` appeared twice. \n    - `The anti-induciton effect` -> `The anti-induction effect`\n    - `we have reason to believe that they used a different designation than ortu` please cite properly", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LR9SdwG5k2", "reviewer": "Reviewer_htkn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper falls within mechanistic interpretability, a very active area with significant research interest that could appeal to multiple members of the TMLR community. Although largely a reproducibility study, it provides valuable validation (and critique) of previous findings while testing their sensitivity to the experimental setups used in the original studies.\n1. The authors' experiments are well-designed and evaluate multiple aspects of the role of attention heads in recalling factual information. I particularly appreciated the experiment examining the \"Effect of changing counterfacts to facts in the prompt\", which provides insightful evidence that the attention heads under study primarily prevent copying from context rather than directly retrieving facts—an aspect not analyzed in depth by previous studies.\n\n**Weaknesses:**\n1. There is limited novelty in terms of technical methodology, which relies primarily on approaches established in prior work. While the paper provides valuable experimental evidence, it introduces little at a methodological level.\n1. Parts of the presentation could be further improved for clarity, as some methodologies are described vaguely and certain conclusions are not immediately evident from the presented experimental results.\n1. The analysis focuses on only two models (GPT-2 and Pythia-6.9B), leaving unclear whether the findings would generalize to other models/architectures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes LLMs' ability to recall and generate facts when presented with incorrect information in their context, while also identifying specific mechanisms (e.g., attention heads) involved in this process. The authors build upon experimental approaches and findings from three previous works, conducting further analysis to validate their conclusions and test their sensitivity to different experimental variables such as the phrasing and domain areas of input prompts. They begin by describing their hypotheses, the methods they use for their analysis, as well as the datasets and experimental setups. They then present findings from a series of experiments which test these hypotheses. Their key findings reveal that when GPT-2 and Pythia-6.9B encounter incorrect \"facts\" in their context, specific attention heads can be scaled to increase the models' probabilities of generating correct information. Additionally, the authors demonstrate that this scaling primarily prevents the models from simply copying their context rather than genuinely \"recalling\" accurate facts. Finally, they show that attention heads' roles vary significantly across different application domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here, I elaborate more on the weaknesses mentioned above, and I provide actionable points that I believe can improve the paper.\n* Citations 16 & 21 which the paper builds upon are published, however, they are cited as arXiv preprints.\n* I understand that the terminology in the paper is not entirely a choice of the authors, but something that comes from the previous studies they reproduce. However, I found the terms \"factual\" and \"counterfactual\" a little problematic. The use of the word \"counterfactual\" can be misleading to readers with background in causality. In both computer science and psychology, a counterfactual refers to a state of the world under some (past) intervention that could have happened but didn't. In the context of the paper, a counterfact is just something that is factually incorrect (e.g., the capital of France is London). Correct/incorrect would be a much more appropriate terminology with less room for confusion. I would encourage the authors to at least add a footnote clarifying this point.\n* The exact problem that the authors are interested in becomes clear quite late in the paper. In the introduction they write \"the adaptation of incontext information, come into direct conflict when counterfactual statements opposing the model’s parametric memory are provided as input\". This is quite abstract, and a simple example (e.g., the one at the top of page 5) would be helpful to clarify what the paper is about. Also, at the end of page 1, the authors write \"Prior work has investigated the competition between factual and counterfactual information in language models, particularly focusing on how different attention heads influence this competition\". It is unclear what competition means in this context. I think the authors should provide more examples and concrete explanations in their introduction.\n* Although the distinction between hypothesis 2.2.1 and 2.2.2 becomes clear throughout the paper, when they are first stated, their difference is not obvious. This is especially because the both use the term \"copying\", which if I understood correctly is more related to the anti-induction effect, that is, more related to 2.2.2. I would suggest rephrasing 2.2.1. Again, if i understood correctly, the way I would phrase it is that the first one is about the attention head suppressing incorrect facts, while the latter is about suppressing copies from the context.\n* I found the phrasing of hypotheses 2.3.1 and 2.3.2 quite vague. With the current phrasing, they read more like conclusions that followed the experiments, rather than hypotheses that preceded them.\n* I found the description of methodology too brief and not precise enough. For example, in \"Logit attribution techniques\", it is unclear what the word \"position\" refers to and what \"the grouped results are unembedded\" means. Perhaps using some formal notation to be precise would be helpful. Similar for the SVD analysis. The authors write \"the singular vectors that are of the embedding dimension can be unembedded and detokenized to show what words are most encoded by the head.\" This sentence gives a vague idea of what the method does but it is quite far from being a rigorous explanation.\n* In section 3.2 the authors write \"These are the same models that were used in the original paper\". It is unclear which paper they refer to here. The one by Ortu et al.?\n* The organization of the experimental results is a bit strange, since the authors perform a series of more or less independent experiments, but they first introduce 3 experimental setups and then they present 3 results sections. Maybe this is just my personal preference, but I think the paper would have been much easier to read if each experimental setup was followed by the corresponding results, so that the reader doesn't have to do a lot of back and forth.\n* I didn't understand what the first paragraph of page 7 contributes. It introduces a lot of details regarding the CO2 emissions of the study, and it seems unrelated to the actual story of the paper.\n* One finding that I find somewhat surprising and worth commenting on is based on Figure 1. To me, it seems that the attention modification is performing best when $\\alpha=5$. For GPT-2, $\\alpha=10$ gives a slightly higher probability to the correct prediction, but also there is increased mass on predictions that do not match any of the two options (neither the correct one nor the one in the context), which could imply that the model starts generating nonsensical tokens at this point. However, both in GPT-2 and Pythia-6.9B, for $\\alpha=5$, both the in-context (incorrect) token and the correct token have probabilities around 0.5, which doesn't seem to imply that, by scaling the parameter, the model \"recalls\" the correct fact but rather just becomes uncertain. Maybe it would be useful for the authors to comment on that and see if they can identify cases where the model assigns an actually large (>0.5) probability on the correct prediction while still putting most of its mass only on those two tokens.\n* In page 7, the part about \"Singular value decompositions of transformer.\" doesn't show any quantitative or qualitative results, however, the authors have a short commentary on their findings. In combination with the very brief description of the corresponding methodology, this contributes to the lack of clarity in the presentation of the paper.\n* In 4.2.1, the authors claim that \"the effect of changing the attention heads is lower if the two sentences have the same structure\", however, that is not immediately clear, since the scales of the corresponding rows in Table 1 are different even for $\\alpha=0$. Perhaps showing the percent change as well would be insightful to support the claims. Also, I am not sure if the paragraph that makes a connection with the input length at the end of that section adds much. It doesn't seem to be supported by any numerical results, so it seems a little bit speculative. Also, it seems that $\\alpha=10$ leads to more correct predictions in comparison with $\\alpha=5$ whenever the first and second sentence structure match. Do the authors have any intuition on why that happens?\n* In the beginning of 4.2.2, the authors write \"Our results favor hypothesis 2.2.1 more than 2.2.2, showing that the attention heads supporting $t_{fact}$ are more likely to be related with the overall suppression of the induction effect, rather than the selective suppression of counterfactual statements.\" However, if I am not mistaken, the hypothesis that is claiming this is 2.2.2. Also, in the conclusions, they do write the opposite, that is, \"these heads are more likely involved in copy suppression rather than specifically counterfactual suppression, supporting hypothesis 2.2.2 and challenging hypothesis 2.2.1\".\n* I found the discussion of 4.2.3 somewhat underwhelming. The authors analyze the role of specific attention heads in predicting the correct answer depending on the domain of the input prompt. However, they seem to use a dataset with 124 entries, which seems quite small, and their findings do not include any information regarding the standard deviation of their estimates, so it is quite hard to read the results and make conclusions. Also, it would be helpful to expand the captions of figures 5 and 6 and point to specific rows and columns  when discussing the result. Currently, it is a bit hard to follow. For example the authors write \"The heads most supporting $t_{cofa}$ and $t_{fact}$, are L15H17 and L20H18 respectively\". It is not clear where a reader is supposed to look to understand that statement.\n* The \"SVD analysis of relevant attention heads\" in page 11 discusses results in figures that are placed in the appendix, which breaks the readability of the paper. I would encourage the authors to either bring those figures to the main body or move that discussion to the appendix and use more space to clarify all the other aspects of their remaining experiments in the main body.\n* The paper has several typos, and I think the authors should spell-check it again. Some examples: \n    * page 1: \"these complexe language model\" -> \"these complex language model\"\n    * page 2: \"We focus on the works from\" -> \"we focus on the works from\"\n    * page 2: \"ased on the combined finding\" -> \"Based on the combined finding\n    * page 3: \"The anti-induciton effect\" -> \"The anti-induction effect\"\n    * pages 3 & 4: \"Ortu et al. have provided\" should include a citation number\n    * page 4: \"increasing the over strength\" -> \"increasing the overall strength\"\n    * page 6: \"To test hypothesis 2.3.1 We wish to see\" -> \"To test hypothesis 2.3.1, we wish to see\"\n    * page 7: \"align with the the original results\" -> \"align with the original results\"\n    * page 12: \"mostly about locasions, languages\" -> \"mostly about locations, languages\"", "broader_impact_concerns": "There are no significant ethical concerns regarding this paper. It focuses on interpretability, which contributes positively to machine learning research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)