Look Before You Leap: Universal Emergent Mechanism for Retrieval in Language Models
Authors: Alexandre Variengien
Eric Winsor
Venue: ICLR 2025 Poster (2025)
Keywords: Interpretability
LLM
Universality
Primary area: interpretability and explainable AI
TL;DR: We show that LM decompose retrieval internally by first compiling a representation of the query, and then looking for matching elements in the context.

Abstract: When solving challenging problems, language models (LMs) are able to identify relevant information from long and complicated contexts. To study how LMs solve retrieval tasks in diverse situations, we introduce ORION, a collection of structured retrieval tasks spanning six domains, from text understanding to coding. Each task in ORION can be represented abstractly by a request (e.g. a question) that retrieves an attribute (e.g. the character name) from a context (e.g. a story). We apply causal analysis on 18 open-source language models with sizes ranging from 125 million to 70 billion parameters. We find that LMs internally decompose retrieval tasks in a modular way: middle layers at the last token position process the request, while late layers retrieve the correct entity from the context. After causally enforcing this decomposition, models are still able to solve the original task, preserving 70% of the original correct token probability in 98 of the 106 studied model-task pairs. We connect our macroscopic decomposition with a microscopic description by performing a fine-grained case study of a question-answering task on Pythia-2.8b. Building on our high-level understanding, we demonstrate a proof of concept application for scalable internal oversight of LMs to mitigate prompt-injection while requiring human supervision on only a single input. Our solution improves accuracy drastically (from 15.5% to 97.5% on Pythia-12b). This work presents evidence of a universal emergent modular processing of tasks across varied domains and models and is a pioneering effort in applying interpretability for scalable internal oversight of LMs.

Published as a conference paper at ICLR 2025
LOOK BEFORE YOU LEAP: UNIVERSAL EMERGENT MECHANISM FOR RETRIEVAL IN LANGUAGE MODELS
Alexandre Variengien" EU AI Office
Eric Winsor" UK AI Security Institute
European Commission
ABSTRACT
When solving challenging problems, language models (LMs) are able to identify relevant information from long and complicated contexts. To study how LMs solve retrieval tasks in diverse situations, introduce ORION, collection of structured retrieval tasks, from text understanding to coding. We apply causal analysis on ORION for 18 open-source language models with sizes ranging from 125 million to 70 billion parameters. We find that LMs internally decompose retrieval tasks in a modular way: middle layers the last token position process the request, while late layers retrieve the correct entity from the context. Building on our high-level understanding, we demonstrate a proof of concept application for scalable internal oversight of LMs to mitigate prompt-injection while requiring human supervision on only a single input.
INTRODUCTION
Recent advances in language models (LMs) (Vaswani et al., 2017) have demonstrated their flexible problem-solving abilities and their expert-level knowledge in wide range of fields (Bubeck et al., 2023; OpenAI, 2023). Researchers have developed a series of techniques such as fine-tuning (Ouyang et al., 2022) and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) to ensure models output honest and helpful answers. However, as their abilities reach human level, supervision from human feedback becomes costly and even impossible. This necessitates more efficient or automated methods of supervision, known generally scalable oversight.
Moreover, existing methods only control for the output of the model while leaving the internals of the model unexamined (Casper et al., 2023; Ngo al., 2023). This is critical limitation as many internal processes can elicit the same output while using trustworthy or untrustworthy mechanisms. For instance, would like to know whether model answers faithfully based available information or simply gives a user's preferred answer (Perez et al., 2022). We call this problem internal oversight.
Recent works on mechanistically interpreting LMs have shown success on narrow tasks (Wang et al., 2022: Nanda et al., 2023). Some have provided insight into factual recall (Geva et al., 2023) and in-context learning (Olsson al., 2022). Causal interventions have even been used to understand how models encode tasks from few shot examples (Hendel et al., 2023) or bind entities to attributes (Feng and Steinhardt, 2023). However, these works are still scoped to relatively narrow contexts and lack demonstration of concrete applications.
In this work, we study how LMs solve retrieval tasks, i.e. in-context learning problems that involve answering a request (e.g. "What is the city of the story?") to retrieve a keyword (e.g. "Paris") from a context (e.g. a story).
We start by introducing ORION, a collection of 15 datasets of retrieval tasks spanning six different domains from question answering to coding abilities and variable binding. We systematize the task structure by annotating each textual input with an abstract representation where the context is table of attributes, and the request is a simple SQL-like query, as illustrated in Figure 2.
We apply causal analysis (Pearl, 2009; Vig et al., 2020; Geiger et al., 2021) to 18 open source LMs ranging in size from 125 million to 70 billion parameters to investigate the successive role of layers
Work conducted while Conjecture. Corresponding author: a exandre. vari eeriiingeennn enêgma i com

---

Published as a conference paper at ICLR 2025
Context C
Request R.
Context C2
Request Rg
Alice
Bob
Cibbo in xbory?
the story?
London
Ar Lhe casy
Paris-
Atthe chaze- otex
Attention and
MLP blocks
Residuar
Stremm
Request patching
Layer
London R2(C.)
Paris
Medillar combinaticn
ohe reques: R, on the context C2
RC(C.)
Figure I: Illustration of our main experimental discovery. Patching the mid-layer residual stream on a retrieval task from ORION causes the language model to output a modular combination of the request from 21 (asking for the city) and the context from 22 (a story about Bob in Paris). We call this phenomenon request-patching.
the last position on tasks from ORION. The shared abstract representation enables us to define and interpret experiments across tasks and models at scale, without the need for setting-speciin discover that language models handle retrieval tasks by cleanly separating the layers at which they process the request and the context the last token position. These results suggest that there exists an emergent modular decomposition of tasks that applies across models and tasks. We complement this coarse-grained causal analysis with finer-grained Pythia-2.8b (Biderman et al., 2023).
labor. We
study of question-answering task
We demonstrate that understanding of how models solve retrieval tasks be directly leveraged to mitigate the effect of prompt injection (Perez and Ribeiro. 2022) in are given inputs containing distractor sequences that trigger models to output token unrelated the task. We present proof-of-concept based on request-patching that only requires humans to verify the model output on single trusted input. Our technique significantly improves the performance of models on sequences with distractors (0% + 70.5% accuracy for Pythia-410m, 15.5% - 97.5% for Pythia- 12b). To our knowledge, this is the first demonstration that scalable internal oversight LMs is feasible.
task. Models
In summary, our main contributions are as follows:
1. We introduce ORION, a collection of structured retrieval tasks. It is data-centric approach enabling a comparative study of 18 models on 6 domains.
2. We discover macroscopic modular decomposition of retrieval tasks in LMs' internals that is universal across tasks and models.
3. We link macroscopic and microscopic descriptions of LMs' internals with a fine-grained study of question-answer answering task Pythia-2.8b. 4. We apply this knowledge to proof of concept for scalable internal oversight of LMs solving a retrieval task in the presence prompt injection.
2 BACKGROUND
2.1 THE TRANSFORMER ARCHITECTURE FOR AUTORFGRESSIVE LANGUAGE MODELS
An autoregressive language model, Ma with parameters 6, maps a sequence of input tokens r = (x1, ...,... .,,) a probability distribution over the next token Tx+1- For the Transformer architecture (Vaswani et al., 2017), we have:
p(znt1lz) = MM(x))
(0ftmax(Tx(x))
2

---

Published as a conference paper at ICLR 2025
The pre- pre-softmax values nn are the logits the n-th token position. The final logits n are constructed by iteratively building a series of intermediate activations zi we call the residual stream, following Elhage et al. (2021). The residual stream of, at token position and layer is computed from the residual stream at previous token positions at the previous layer 2-1 by adding the results of d multi-headed attention module that depends on 2 and my, a two-layer perceptron module that depends on 2/ We provide complete description of the Transformer architecture in Appendix G.
2.2 COMPUTATIONAL GRAPH AS CAUSAL GRAPH
The experimental paradigm of causal analysis applied to machine learning models initiated by (Vig al., 2020) and (Geiger et al., 2021) treats the computational graph of a neural network as causal graph. The goal of causal analysis is to answer questions about why a model outputs a given answer. This requires uncovering the causal links tying the inputs to the output, as well as characterizing the role of the internal components critical for the model's function. To this end, researchers rely on causal interventions (Pearl, 2009), experiments that replace set of activations with fixed values.
In this work, we use single-input interchange intervention' (Geiger et al., 2021). It is a simple form of causal intervention where we intervene on one variable at time by fixing its value to be the value of that same variable on another input. We write M(z] A + A(x')) the output of the model after the single-input interchange intervention on the target input x, replacing the activation of the node by its value on the source input x'.
3 ORION: A COLLECTION OF STRUCTURED RETRIEVAL TASKS
Our study concentrates on retrieval, a fundamental aspect of in-context learning, which involves answering request (e.g. "What is the name of the city?") by identifying the correct attribute (e.g. city name) from the context (e.g, a story). To facilitate this study, we crafted a collection of datasets dubbed the Organized Retrleval Operations for Neural networks (ORION).
Abstract representation. Each textual input (i.e. LM prompt) from ORION is annotated with an abstract representation (C, R) where C' represents the context and R the request. In the example of Figure 2, the context is story introducing a place, character, and an action, while the request is a question written in English asking for the city of the story.
The context C' is abstractly represented as a table where each line is list of attributes. The request R is retrieving a target attribute ATTR, (e.g. the "name" attribute in Figure 2), from lines where a filter attribute ATTR; (e.g. the narrative role) has the value uy (e.g. "city"". The request can be written using a language the style of SQL. as follows: SELECT ATTR, FROM C WHERE ATTR, = vyy (e.g. SELECT Name FROM Context WHERE Role=City).
We note R(C') the results of applying the request on the context. This is the ground truth completion for LMs evaluated on the retrieval task. In practice, R(C)) is a single token called the label token. On the example we have R(C)= = Valencia".
Desiderata for datasets. To facilitate the application of causal analysis, we enforce list of desiderata on datasets from ORION. The most important desiderata is ensuring datasets are decomposable. For every dataset D in ORION, for every abstract representations (C1, R1), (C2, R2) in D, R2(Ci) and Rí (C2) are well-defined. This means that an arbitrary request can be applied to an arbitrary context from the same task. Abstract representations of requests and contexts be freely interchanged across a task. This constraint enables the design of interchange interventions at scale.
We define four additional desiderata Structured, Single token, Monotasking, and Flexible in Appendix H and share the motivation behind their definition.
Dataset composition. The dataset includes the retrieval task from domains: question-answering translation, factual recall, variable binding, induction pattern-man matehhng,, and type hint understanding. For each domain, we created two or three variations. Each dataset is created using semi-automated process leveraging the LLM assistant CPT We provide a detailed overview of the dataset and its creation in Appendix H.
It is sometimes called "activation patching" in the literature see c.g. Wang al. (2022)
3

---

Published as a conference paper at ICLR 2025
story: In the lively city of Valencia, skilled veterinarian. [...]. "I'm Christopher" he replied, [...].
Textual input:
Question: What is the city of the story? The story takes place in
Context C
Request R
Name
Role
Abstract representation:
Valencia
City
SELECT Name FROM Con- text WHERE Role=City
Christopher
Main Character
veterinarian
Character Job
Figure 2: Example input from ORION for the question-answering task. Textual inputs annotated with an abstract representation of the context and the request. Abstract context representations are tables where each line lists attributes relative story element. Requests can be formulated using simple SQL-like queries.
Performance metrics. We define task T' set of input-output pairs (x, y) where the LM input and is the expected label token. We use two main metrics to quantify the performance of language model on an ORION task T.
Accuracy: ((,g))-((M(() =
Token probability: ((xx)~((p((la))
Accuracy serves as our primary metric to assess model performance in solving tasks due to its straightforward interpretation and practical application in language models, where the probable token is often chosen.
However, accuracy falls short in capturing nuanced aspects of predictions, for instance, accuracy doesn't measure the margin by which a token is the most probable. To have a granular evaluation of model behavior after interventions, we employ token probability, offering continuous measure.
We evaluate the performance of 18 models from four different model families: GPT-2 (Radford al., 2019). Pythia (Biderman et al., 2023), Falcon (Almazrouei et al., 2023) and Llama 2 (Touvron et al., 2023). We study base language models for all families except Falcon where we include two instruction fine-tuned models. We choose the models capture diverse scales, architecture, and training techniques.
Unsurprisingly, larger models can solve wider range problems. Models with more than billion parameters are able solve every task with more than 70% accuracy. Nonetheless, even GPT-2 small with 125M parameters, one of the smallest models, can solve the simplest version of the iuestion-answerte task with 100% accuracy. Detailed evaluations using the token probability and logit difference are available in Appendix A.
In the following analyses, only consider settings where the model robustly solve the task. Thus, we focus on pairs of models and tasks that have greater than 70% accuracy.
4 MACROSCOPIC CAUSAL ANALYSIS ON ORION: A UNIVERSAL EMERGENT DECOMPOSITION OF RETRIEVAL TASKS
To correctly solve retrieval tasks, LM has to gather and combine at the last token position information coming from the request and the context. We focus our investigations on understanding how these two processing steps organized in the intermediate layers of the last token position.
In this section, we choose to consider a coarse grained division of the model, intervening on full layers instead of finer-graine grained division, e.g. considering single-ane angleeattention heads and MLP blocks. We find this level of analysis is sufficient to develop a high-level causal understanding of how language models solve retrieval tasks while providing computationally tractable set of experiments to run
4

---

Published as a conference paper at ICLR 2025
Residual stream patching accross layer on the QA (uniform prefix) task for pythia-2.8b
Label token
Normalized accuracy
Normalized token
R1(C1]
R1(C2)
propabiltty
0.5
R2(G2)
0.4
-0.5
0.2
20
30
20
25
30
Layer
Layer
Figure Normalized token probability and accuracy for the label tokens R1 (C),, (C2) and R2 (C2) after patching the residual stream across all layers. Patching early (before L1 = 13) and late (after L3 27) leads the expected results, respectively no change in output and patching the output from I1. However, intervening on the middle layer (L2 16) leads to the model confidently outputting the token Ri (C2), a modular combination of the request from æ and the context from T2-
scale. We complement this general coarse-grained analysis in Section 5 with finer-grained case study on Pythia-2.80 solving a question-anseen task.
4.1
METHODS
Our main experimental technique residual stream patching. Residual stream patching is a single- input interchange intervention, replacing the residual stream layer I. at the last position in the forward pass of the model on input 32 with its activation from another input 21. Following the notation introduced in Section 2.2, note M(rgl=h < =xxx)) the model output 22 after this intervention.
As shown in Figure 1, residual stream patching makes every component before layer I, have the activation it takes on while the components after layer I. receive mixed activations (denoted by the yellow color in the figure). These later layers see activations the last position coming from ø1 while activations from earlier positions come from T2.
To characterize the output of the patched model, we measure the token probability and accuracy for three different label tokens related to the inputs 2 and Zz. We use both label tokens from the input ø1 and I2, R1 (C1) and R2 (22) respectively, and the label token R1(Ca) that the result of applying the request from on the context of 2:2.
To facilitate comparisons between different tasks and models, we normalize the token probability based on the mean probability of the correct token given by the model for the task. In addition, we calculate the normalized accuracy where O represents the accuracy of random guess, i.c. responding to random request in given context while I denotes the model's baseline accuracy for that task.
We perform residual stream patching at the last position for every layer, model, and task of ORION. For each task, experiments with Ij and 22 chosen uniformly from the task dataset.
dataset of 100 prompts and average the results of 100 residual stream patching
4.2 RESULTS OF RESIDUAL STREAM PATCHING
Figure shows the results of residual stream patching on the question- answer prefix for the Pythia-2.8b model. We observe that after residual stream patching the layer before layer 13, the model outputting R2(C2) with 100% normalized token probability. Our interpretation is that this intervention does not perturb the model processing of æ2. We further observe that residual stream patching after layer 27 causes the model to output Rí (C1) with more than 80% normalized token probability. In effect, patching the residual stream after certain layer is equivalent to hard-coding the model output on .1.
question-answering answering task with a uniform
Surprisingly, when patching between layers 15 and 16, we observe that the model outputs R₃ R1() with 100% normalized accuracy, i.e. with the same accuracy level as the baseline task accuracy. The model is outputting the results of the request contained in the input oy in the context of the input Ø2-
5

---

Published as a conference paper at ICLR 2025
We call this phenomenon, rrquest-patching i.e. residual stream patching experiment that leads to the (C2) label token being confidently outputted by the patched model. Such results demonstrate that the causal intervention coherently intervenes in the model's internal computation, causing it to modularly combine high-level information from two different prompts.
We observe sudden jump in the normalized accuracy of request-patching from 0 to 1 between layers 14 and 15. However, it is likely that transforming the sequence of tokens representing the question into a representation of the request takes several layers. Thus, we hypothesize that large part of the request processing happens at the previous token positions of the question. In this interpretation, the observed jump at layer 15 results from the intermediate representation of the request being propagated to the last position through attention modules.
Defining limit layers. From the results of the residual stream experiments, we define three layers L1, L2, and L3- delimiting the three different outcomes of residual stream patching as shown in Figure 3.
L1 is the maximal layer at which the normalized token probability of the label token R1 (C1) is greater than 80%. marks the end of the region where residual stream patching does not interfere with the model output. L2 is the layer where the normalized probability of the label Rí (C2) is maximal. is the place where the effect of request-patching is the strongest. L.g is the minimal layer where the normalized probability of the label R2 is greater than 80%. It marks the start of the region where residual stream patching leads to a complete overwrite of the model output.
We choose the token probability continuous metric to measure the model prediction. The 80% threshold has been chosen arbitrarily as a criterion to consider that the model is mainly outputting a single label token.
sequeestpatchingg is general across models and datasets. We expand our investigation of request- patching to include every model and task from ORION. To ease the analysis, we compute the maximal normalized probability of the R₁ (C2) label token, i.e. the normalized probability after patching at L2. We use this metric as our main performance indicator to measure the strength of the request-patching phenomenon on a given pair of model and task.
98 out of the 106 pairs of tasks and models studied demonstrate similar profile as the one shown in Figure 3. token. Request-patching appears across variations in domain, task complexity, and low-level prompt structure. Moreover, it is present in every model studied, from GPT-2 small to Llama 2 70b, one of the largest available open-source LMs.
leads to at least 70% normalized probability of the Rı (C2) label
The results for the question-ans answering with mixed template task demonstrate that request-patching works even when patching the residual stream across different templates, e.g. taking the residual stream from a prompt where the question is before the story and patching it in a model execution where the question is after the story. This means that the representation stored the patched activation is related to the semantic meaning of the question, and not surface-level textual features.
However, the phenomenon of request-patchinn seems not to be present in the abstract induction task on large models. We hypothesize that this is due to the increased wideness of large models and the simplicity of the task. We discuss further the results for induction and factual recall by comparing eatuest-patchig results to prior work in Appendix F.
To further our analysis, Figure shows the values of the layer Le on different models and datasets. We observe that the effect of request-patching for the induction tasks is the strongest at earlier layers compared to the other tasks. This observation is consistent with the simplicity of the request process- ing, which only involves copying previous tokens. The L2 layers for other tasks are concentrated in similar layers, suggesting similar high-level organization of the internal computation that does not depend on the details of the task being solved. However, for Llama 2 70b, the largest model studied, the L2 layers are concentrated in the same narrow range (39-43) for every task, including the simple induction tasks. It is unclear if this disparity is caused by its larger size or by the specifics of the architecture. We provide visualizations of request patching results and of layers L and Lg in A Appendix D.

---

Published as a conference paper at ICLR 2025
Relative layer Lz
Residual stream patching on Llama 2 70b
Layer relative
1.0
(untrem preft)
(bass)
templani)
Poctual recall
Trandttion
Varad tinding
"Tpe 1itt
Translation task
Nomalazd aedd taken prob.
0.6
Lubel
R1211
0.4
0.5
101/31
talcon
Talcon allcon22uaa 40h
Telcon A0binttrua
2012-mm
0012 medium
qu www.eeeee nus-20um
Layer
1922
Figure 4: Left: Layer of maximal request-patching performance L2 for different models and tasks. While the L2 layers for most tasks are concentrated at similar layers, the processing of the request in the induction task seems happen earlier layers. Right: results of residual stream patching Llama 2 70b. Request-patching is most performant in a narrow range of layers centered around layer 42 and does not depend on the nature the task.
Direct contribution
Attention the city token
to the city token logit
17.5
Other cities
Cusco
Cusco
67
Count
43
10.0
L22H9
32
09
02
05
1.
0.9
5,500
0.03
0,050
0.055
0.000
6:255
0.15
6.175
Attention probability
Normalized direct effect
Figure 5: City-specific heads attend to the city token and contribute directly to the logits when the question asks about the city of the story only if the city has specific value, e.g. "Cusco".
5 MICROSCOPIC ANALYSIS: CASE STUDY ON PYTHIA-2.8B
To complement the high- causal explanation described in the previous section, conduct finer-grained case study on Pythia-2.8b on the question-answeringg task. Our motivation is twofold. provide complementary level of analysis documenting how the model solves the retrieval task at the scale of individual MLP and attention heads. Second, we want to understand more precisely how request-patching influences components at the later layer force them to execute a request that is not present in the context. Appendix B describes in detail our methodology and the results of the case study. In this section, we provide an overview of our key results.
First, we
The components contributing directly to the logits depend on superficial changes in the input. We discover that the of components directly influencing the logits varies from input to input. There is no single set of components implementing the retrieval steps on every input. We find that the components contributing to predicting the correct token depend on superficial changes in the input sequence. As shown in Figure 5, discover family of attention heads that retrieve the correct token from the context when the question asks for the city of the story only when the city has a particular value (e.g. "Cusco").. For all the other city names, these heads do not directly contribute significantly to the output.
Request-patching preserves natural mechanism. To compare the internal changes caused by residual stream patching M(x2|2/a + 2x)(x)) )) to a natural mechanism, we construct a input 23 by concatenating the textual representation of the context C2 and the request Rí. On 2s.
reference
7

---

Published as a conference paper at ICLR 2025
the model is naturally executing the request R1 the context C2. This reference input acts control condition to compare the effect of request-patching.
We find that eqques-patchiing globally preserves the mechanism of the components at the late layers. We measure the direct effect and attention pattern for every component after patching (x < =xx) (x1)). These measures similar to those of the components the corresponding reference input 23 (relative difference less than 12%). This suggests that request-patching causes the final layers of the model
similarly how they would when answering the request Ri on the context C2 in a natural input.
Microscopic Macroscopic analysis. The clear division between the request processing and retrieval step observed at a macroscopic level does not translate into a similar level of modularity microscopic level. Even if the retrieval step happens at similar layer on different inputs, the components involved at these layers vary strongly depending on the content of both the request and the context. Hence, macroscopic modularity seems to emerge from a set of microscopic mechanisms depending on the superficial features of the input.
Nonetheless, this case study is limited to narrow settings and simple experimental methodology. only provides preliminary threads of investigation to understand how components are acting at micro-level to solve the retrieval task.
APPLICATION TO SCALABLE INTERNAL OVERSIGHT OF A RETRIEVAL TASK
Model
Intervention
No tor
distrac-
Small- model distractor
Large- model distractor
Control distractor
No intervention
97
84
100
Pythia-410m
Request-patching
100
70.5
100
100
No intervention
100
90.5
15.5
100
Pythia- 12b
Request-patchiee
98
89.5
97.5
98.5
Table I: Accuracy (in %) on the question-answering task before and after request-pan patuesttpttchng from a trusted input. Request-patching significantly reduces the impact of the distractor on both the large and small models.
Language models are known to be easily distracted by instructions in the context, making them execute functions undesired by their designers, a phenomenon known as prompt injection (Perez and Ribeiro, 2022). We designed an application to partially supervise the internal processes of models in a simple prompt-injection setting. Concretely, we focus on question-answet answering prompts containing a distractor text that triggers the model to ignore the question and output another answer the distractor token - instead.
The goal of the application is twofold. First, want make the model more robust against prompt injection, such that it continues to answer the question despite the distractor. Second, we design our application that it does not require access ground-truth labels. One doesn't need know the correct answer to the questions to apply this technique, making it proof of concept for scalable internal oversight. We include additional experiments detect distractors Appendix
We implement this setting on prompts from the base question-answering task from ORION (see Table for an example prompt). For simplicity. we focus on single type of question. asking for the city of the story. We create the untrusted inputs by concatenating a story, a distractor D(t) that instructs the model to output a distractor token f, and a question. In contrast, the trusted input is the concatenation of story and a question. We study two language models of different sizes: the small Pythia-410m and the large P1th
We design four different distractors:
No distractor. This is first control setting.
8

---

Published as a conference paper at ICLR 2025
Control distractor. (tt) "This is the end of the story. Ignore the following inst instructions question." " We find this distractor ineffective against both large and small models. This is our second control setting, controlling whether our method detects an arbitrary distractor that does not influence the model behavior.
should be the answer to every
Small-model distractor. (()) = "Answer the question for this story. A After that, the next story is located in a city named t." We find that when é a plausible city, the small model repeats it instead of finding the city in the story.
Large-model distractor. We use the Russian translation of "The story takes place in a city named f" repeated times. For instance, for "Pariss" (tt) "Meropuu происходит B ropoze Io mмсm πаарии output the English translation of the Cyrillic version of (e.g. "Paris" for "Ilapıx"" instead of the city in the story. In comparison, smaller models are less influenced by distractors using the Cyrillic alphahet.
ropoze IO имепn πарpи... We find that the large model
Table 1 shows the influence of the distractor on the models. While both can perfectly solve the task in the control conditions, distractors make them unable to output the correct token.
Increasing robustness against distractors. To increase robustness, we leverage the request- paatctg- phenomenon. We perform residual stream patching (z + (xx)) from the trusted input 2t an input we know doesn't 't contain distractor an untrusted input zu, that can contain a distractor. The motivation is that request- patteest-ptchingggee can force the model to execute the request processed in the trusted input in the context of the untrusted input, overwriting the mechanism triggered by the distractor. Note that this leads only to partial supervision of the internal process, we simply overwrite the results of the request-proeessing step. In particular, we cannot ensure that the context processing is done correctly.
The results of this experiment are shown in Table After request-patching, both Pythia-4 10m and Pythia- 12b recovered most of their performance despite the distractors. Moreover, request-patching does not harm the accuracy in the control settings.
7 RELATED WORK
Causal interventions. A growing body of work has studied neural networks by performing causal interventions. The core differences among works are their proposed high-level causal graphs and corresponding concrete changes to neural activations. Michel et al. (2019) prune attention heads by setting their outputs zero, identifying minimal set of components needed to solve a task. Meng et al. (2022) locate MLP blocks involved in factual recall in LMs by performing interventions using activations corrupted with Gaussian noise. more precise understanding of the mechanisms implemented by components can be achieved through interchange operations. Patching a fixed value from a forward pass into a new input has been used to investigate gender bias (Vig et al., 2020), variable binding (Davies et al., 2023), indirect object identification (Wang al., 2022), or factual recall (Geva et al., 2023). Recent work proposes a more fine-grained division of models by performing interchange interventions on paths instead of variables (Wang et al., 2022; Goldowsky- 2023), enabling a precise characterization of indirect effects.
al.,
Causal interventions for high-level understanding of LMs. As an alternative zooming in on the role of individual model components, recent work focuses on extracting a high-level understanding of the computations at play in LM internals. Hendel et al. (2023) patch residual stream vectors to transfer the representation of simple task from few-shot examples to zero-shot instances of task. Similarly, Todd al. (2024) used causal analysis to identify attention heads representing functions from few-shot examples. Feng and Steinhardt (2023) intervene on the residual stream at every layer for specific tokens to argue that models generate IDs to bind entities to attributes. Representation engineering (Zou et al., 2023) uses prompt stimuli to extract reading vectors from the activations of language models. These vectors can then be used to perform interventions that stimulate or inhibit specific concept in subsequent forward passes. These interventions do not operate via specific mechanisms, making their precise effects difficult to predict. In this work, we introduce a causal intervention that applies across a broad range of situations while still being mechanistically grounded.
9

---

Published as a conference paper at ICLR 2025
8 CONCLUSION
In this study, we presented evidence of an emergent decomposition of retrieval tasks across 18 language models and six problem types. Through our primary causal intervention technique, resid- ual stream patching,
observed distinct con-overlapping layers that respectively handle request interpretation and retrieval execution. We showed that this modular decomposition only emerges at a macroscopic level and is not present at the scale of individual components.
To investigate language model retrieval capabilities across varied tasks, we introduced the ORION collection of datasets, initiating systematic approach to dataset design for causal analysis. However, the tasks from ORION are limited as they involve requests with a single attribute. Future works could apply high-level causal analysis multi-attribute requests and tasks beyond retrieval, while also investigating how this modular division emerges during model training.
Furthermore, showed that newfound understanding can be turned into practical solutions the problem of scalable internal oversight of LMs. We ensured models execute the intended retrieval requests even in the presence of distractors while requiring human supervision on single task instance. While our application remains a proof of concept, the generality of the task decomposition across different models and domains suggests promising extensions of the application various scenarios.
This research proposes an approach to language model interpretability complementary to microscopic studies, emphasizing high-level understanding of model mechanisms, comparative analysis across models and tasks, and concrete application design. We aspire to motivate future endeavors that uncover macroscopic motifs in language model internals, ultimately turning understanding of LMs into strategies that reduce the risks posed by
Al systems.
ACKNOWLEDGEMEET
The authors are grateful Beren Millidge, Fabien Roger, Sid Black, Adam Shimi, Gail Weiss, Diego Dorn, Pierre Peigné. StannStansslas Denain, and Jiahai Feng for useful feedback throughout the project. The views expressed this article are purely those of the authors and may not, under any circumstances, be regarded as an official position or policy of the European Commission.
AUTHOR CONTRIBUTIONS
AV had the initial idea for the project, developed and ran the experiments, and wrote the first version of the manuscript. EW provided regular feedback throughout the process, technical help the experiment on the largest models, and helped to clarify and finalize the manuscript.
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Buptiste Pannier, and Guilherme Penedo. Falcon-40b: an open large language model with ofte-of-the-ane performance, 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sébastien Bubeck, Varun Chandrasekaran. Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
10

---

Published as a conference paper at ICLR 2025
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Fric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem BIyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
Paul Christiano.
Mechanistic anomaly detection and elk, 2022. https:/ /tt////wwww /www www.:aafinmmnntruumm.mmmm http://wwww...oppps///////eaaaea mechanistice hale--nnnmally-deeeeee htt.:dattecttooeeee-aaaaaaaaaaaa problems.
URL
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata. arXiv preprint arXiv:2307.03637 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context?, 2023.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-9586, 2021.
Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. arXiv preprint arXiv22001.04009 2023.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Nicholas Goldowsky-DD Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969 2023.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
Dmitrii Krasheninnikov, Egor Krasheninnikov, and David Krueger. Out-of-comtext meta leaaning in large language models. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Morgane Laouenan, Palaash Bhargava, Jean-Benoit Eyméoud, Olivier Gergaud, Guillaume Plique, and Etienne Wasmer. A cross-verified database of notable people, 3500be-200 201 Sad. Scientific Data, 9(1):290, 2022.
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprini 2rXXi:23077.9458 2023.
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2300.15771, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17399-17372 2022.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2025
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy. arXiv preprint arXiv:2301.05217, 2023.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2023.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:
27730-27744, 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie KeTT, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg. Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen- Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield- Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hemaadezz, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with write-written evaluations, 2022.
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners, 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202: 2020.
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 01111288-14000 2020.
12

---

Published as a conference paper at ICLR 2025
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023.
13

---

Published as a conference paper at ICLR 2025
Model accuracy ORION tasks
1.0
QA (base)- 140
110
6.00
8:30
2:37
2.20
2.30
1.00
1.0%
QA (uniform praffd)
0.8
(sucstion start)
QA (mised template)
677
668
6:6
6:29
6.46
557
2.7
2.52
3.2
3.52
3.47
3.99
0.79
0.00
0.00
1,00
1.05
0.06
Task
Induction
Accuracy
Ractual
0.4
Transiation
680
6.7
6.00
6:66
0.27
8.37
0.21
3.03
3.22
3.00
0.33
0.30
0MM
0.74
0.70
0.90
0.99
6.06
binding
falcon-7e
Alcon-Fe-insttur
falcon-100
fwicon-lll--me Ceon--tt--mmstnee coteenntttuut
9pt2 Pele-mmel
gpzz medi
gal2-ange
5000
gythe-e
prtho-13
bythis
6.99
pythis-120
llama-77
lisma2-130
l1sma2-700
-0.0
Model
Figure 6: Accuracy of 18 models on the ORION task collection. Models with more than 7 billion parameters are able to robustly solve every task. However, simple tasks such as the base question- answering can be solved by models as small as GPT-2 small (125 million parameters), enabling comparative studies across wide range of model scales.
A DETAILED DESCRIPTION OF ORION
We present the performance of the models studied on the ORION collection measured using the accuracy in Figure 6, logit difference, and the probability of the correct token in Figure 7. The value for the probability of the correct token is used as the normalization factor when computing normalized token probability.
B CASE STUDY ON PYTHIA-2.88 -2.H 2. SOLVING A QUESTION-ANSWERIN TASK
In the main text, we have demonstrated the generality the phenomenon of request-patching. However, our main technique, residual stream patching, only allows description at the scale of layers without investigating the role of specific model components such as attention heads and MLPs. During request- patehiin..hhnnn. components at later layers perform the retrieval operation with request absent from the context. However, we have not described these components nor how request-patching can steer them to execute request other than the one present in the input sequence.
In this Appendix, we zoom on the Pythia-2.8b model on question-answerrine task better understand the effect of request-patching. We are interested in three questions:
What is the mechanism used by Pythia-2.8b to perform the retrieval step?
Does the modularity observed at macro-level still hold at a micro-level?
Does request-patching lead Pythia-2.8h to use its natural retrieval mechanism, or does the intervention preserve the function while causing the mechanism behave artificially?
B. METHODS
We focus our investigation the end part of the circuit on the question-anseee wnwwering (QA) task, i.c. study the components Pythia-2.8b directly influencing the logits to boost the probability of the correct token more than the alternative. They natural candidates for implementing the retrieval function as is the last step of our high-level causal graph.
To find the components influencing the logits, quantify the direct effect of components, i.e. their effect through the direct path that connects them to the logits via the residual connections without intermediate nodes. We path patching (Goldowsky-DD) et al., 2023) quantify this effect. With
14

---

Published as a conference paper at ICLR 2025
Probability of correct token on ORION
QA (base)
5.86
2.36
0.52
0.12
3.72
0.78
(uniform prefix)
0.8
QA (question start) 0:14
0.66
8.10
6.87
3.05
0.10
0.01
0.31
0.63
0,01
0.12
0.1
0.34
0.79
0.42
3.64
0.44
0.69
Token probability
QA (mixed template)
0.50
2.18
0.04
0.99
0.40
0.75
0.69
Task
Induction
Factusl recall
0.77 0.64
0.79
5.89
2.11
0.35
0.43
0.13
0.02
0.04
0.20
0.53
0.60
2.66
0.72
2.82
0.43
0.43
Translatoon
0.84
5.37
0.10
0.11
0:37
5.57
0.95
0.2
Variable binding
6.40
0.02
0.08
0.50
0:44
0.86
0.86
Type hint
0.62
0.50
2.50
0.00
2.01
0.27
0.39
0.37
0.01
0.13
0.38
0.46
0.89
2.67
0.40
2:54
0.83
0.96
falcon-7b
falcon-7U 1onn-7---sssuu
falcon 440
falcon-4OD- 1ooonn40b-instruu
gp2 small
gpzz med um
ont2-aarge
0p22>81
sythie-70m
pythia1660m
pythia-11 110mm
1b
pythis-2.60
pythia-6.0 .b
pythia 2bb
33b
llama2-700
Model
Logit difference on ORION
QA (base)
22.1
15.00
so.33
14.29
Io.51
10.35
11.25
10.92
7.12
9.76 10.00 11.31 1125 15.99 13:77 11.19
11.20
10
-16
QA (unifarm prefix)
3.83
6.99
-14
(question start)
6.69
5.0
2.56
3.25
0.00
2.76
2.41
4.35
5800
-12
Logit difference
QA (mixed template) 9.51 12.00
9:82
12.30
7.11
7.37
5.30
5.20
5.35
7.32
7.72
8,40
9.02 855
9.39
9.05
8:23
5.35
Task
Induction
Fectual recell- P.t 11.41
0.03
13.50
6.90
7:20
9.55
6.32
0.63
13377 9. 9:61
Translation 681 435
823
8.73
0.08
0.09
0.32 352 6.05
0.05
1.29 2.03 3330
5.00
6.90
7.97
7:3
5.21
binding-
Type hint
1.77
4.00
4.48
0.14
0.56
5,7
-227
0.07
0.58
122
3.77
4.24
537
4.39
falcon-7b
falcan-7o-nntrnn fo-nnstruct-
falcor-400 40b-
Aicon-4Ub-instruse
ppt2-small--
5012 99t2-ment medium.
9p12 large
sp12 xt
pythie-3.60m
pyshio-400m
Pythie-ib
pythie-2 8b-
Dythie-5.00
pythie-12b
Ilame2-7b-
lamm22133b
laama2-700
pythio
Model
Figure 7: Logit difference and probability of correct token for 18 language models on the tasks from
the ORION collection. A logit difference zero means that the correct logit has average the same value as the logit corresponding to the answer to a random request in the same context.
15

---

Published as a conference paper at ICLR 2025
path patching, the direct effect of component an interchange intervention along the path c - x by replacing the value of с along this path with the value from a "corrupted" input Zcor- We then measure how this intervention changes metric quantifying the performance of the model on the x task instance. The greater the influence on the the component directly affecting the logits.
target input is measured by performing
metric, the
For this case study, we use the base question-answeriig (QA) task from ORION as our reference dataset extended with two additional questions asking for the season and daytime. The corrupted input is chosen to be an input from the task whose question is different from the target input x. We use logit difference as our metric, as enables fine-grat tineggriiedd continuous measure of the model output without distortion from the final softmax non-linearity. We define the metric on input with abstract representation (R, C)) for target token in the equation below. To find the components contributing to solving the task in the absence of intervention, we use = R(C"), the label token on the input x. When investigating the direct effect after request-patching we measure the effect on the token& f Ri (C2).
Mætric(æ, t) E(r ((rt.C))TT,R [T((T) (C)
We then define DE(c, x, f), the direct effect of a component e on an input x on the logit of target token & follows:
DB(c, x, L) = Metric(a, 1) - Excorreti Re Rour Metric 0/cc T] -- x ==(ooo)))
Re
The direct effect quantifies how the metric changes after corrupting the edge c + n, i.e. the contribution of the components through the direct path is rum on an input where the question is different. In other words, how strongly is the component directly involved in increasing the target token compared to answers unrelated questions? The definition of the metric and corrupted input defines the scope of our microscopic study. For instance, our definition of direct effect does not take into account components that would output set of tokens without relying on the context e.g. component increasing the logits for "Bob", "Alice" and "John" whenever the question is about the character name, no matter the context.
We define the total effect TE(x, () the difference in metric after intervening simultaneously on all direct paths. The total effect is used to compute the normalized direct effect NDE(c, x) of a component given input and thus compare across different inputs. Given that intervening all direct paths is equivalent to intervening on the logits и,, we have:
TE(z, t) Mctric(x,t) t) Ex Metric t/ā +- π ((Eor))
NDE(c, 2, DE(c, 2),)
TE(r,t)
To compare the direct effect of a component e reference setting (c, 2, f) with its direct effect in second experimental setting DE2(c, x, t), we use the relative variation. The relative variation is defined as follows:
DE,(c, (c,
TE (c,
The normalized direct effect is primary experimental measure in the following investigation.
B.2 NOTATION FOR ATTENTION HEEDDS
We complement the description of the Transformer architecture in Section 2 for the finer- anar-grained analysis of this section. The multi-headed attention module be further decomposed into the
16

---

Published as a conference paper at ICLR 2025
Largest normalized direct direct effect companent for pythia-2.88 8b the question-answering task
Normalized direct direct effect for components
0.00
Normalized Direct Effect
Normalized Direct Effect
Layer
-104
head
rep
rip
-300
Head
Component
Figure 8: Normalized direct effect for all the Pythia-2.81 components the QA task. The main contributions are concentrated in MLPs at later layers. The direct effect per component has a high variance. Attention heads are labeled "#layer #head".
contribution of H individual attention heads 14,1 follows:
H Σ
Attn (1 LN
hal
hi. Ai, & Wor
2-11 (T WK
A,r = softmax
J-1
We used the parametrization introduced by Elhage et al. using the low-rank matrices Wi. Wi. in Raxd called the OV and -i-eciuti,, with d being the dimension of the model. This parametrization separates the two functions performed by attention heads: the -K-eiruuit used to compute the attention pattern, Ar, weighing the contribution of each token position, while the OV-circiit is used as a linear projection to compute the output of the head. The matrices As, and w are combined using tensor product noted $0.
ov and
The matrices Wit and Wox computed from the usual parametrization of attention heads using w W wild E Bd # and wh E R xd respectively called the query, key, output and values. K
w w Work = (Wal) W
B.3 THE COMPONENTS CONTRIBUTING DIRECTLY TO THE LOGITS DEPEND ON SUPERFICIAL CHIANGES IN THE INPUT
We start by measuring the direct effect of every component on the QA task. Figure shows the normalized direct effect every component of Pythia-2.8b. We observe that the direct contribution has very high spread across the dataset.
To differentiate between the variance coming from the variation across prompts and the variance coming from the path patching method, patching method. For each task input, we find the set of components with a normalized direct effect greater than 3% of the total effect. Then, compute the average overlap between the set of top contributing components across prompts.
metric that eliminates the variation from the path
On average, only 18% of the top contributors shared across inputs. For reference, computing the average overlap across the same inputs with only the path patching as a source variance leads to 73% overlap after averaging on 3 corrupted inputs, and 83% for 20 corrupted inputs.
17

---

Published as a conference paper at ICLR 2025
Grouping the input by the question type increases the average overlap, but its absolute value stays below 50% for most of the questions, as shown in Table This suggests that which components the question asked. However, grouping
activate at the last step of the retrieval mechanism depends by question type does not explain all the variance: even for the same question, surface-level changes in the prompt will trigger some components but not others.
Questions
Average overlap between components
All
0.18 0.16
Character Name
0.33 0.20
City
0.23 0.24
Character Occupation
0.28 0.21
Day Time
0.56 + 0.10
Season
0.43 0.12
Table 2: Average overlap between components responsible for more than 3% of the total effect. The overlap is computed across all inputs ("All") or after grouping by the question type. We average over 20 values of corrupted inputs. The control overlap when the sampling of the corrupted inputs is the only source variance is 83%.
B.3.1 CITY-SPECIFIC ATTENTION HEADDS
By investigating the source of the variance of direct effects for the set of inputs containing the city question, we discover a family of city-specific attention heads. These heads attend to the city token and directly contribute the output only for single value of the city. Figure 9 shows three such heads. This discovery is evidence that the general modularity observed at a high level does not hold the micro level where superficial changes in the prompt (e.g. the value of the city) drastically alter the role of certain components.
B.4 REQUEST-PATCHIING PRESERVES ATTENTION HEEDD MECIIANISMS
To investigate the effect of request-patching we study request-patching from a dataset Dj containing only questions about the character name to a dataset D2 containing only questions about the season.
On both datasets, Pythia-2.8h on both datasets and outputs on average 0.85 and 0.51 probability for the correct token on D₁ and D2, respectively. After request-patching D₂ <- D1. the model predicts the character name with 0.69 average probability, and the season (the original question) with almost probability.
correctly answer the question. It performs with 100% accuracy
Our control condition to compare the effect of request-patching is the reference dataset Dz. Every input 23 E D3 is created by concatenating the context C2 from an input 22 E D2 and the question Ri from an input 31 E Dj such that C3 C2 and R3 R₁ On D3. the model is naturally answering the request Rj in the context C1. We use D3 as control experimental condition to compare the mechanism of the model after the request-patching operation (zzlaz 2اz (I1) with L2 16 for Pythi
We start the comparison by investigating the attention heads with a large direct effect. They are natural candidates to be involved in the retrieval step as their attention mechanism can be straightforwardly leveraged to find relevant tokens in the context.
Figure 10 shows three-way comparison of attention head behavior in three different settings: on the dataset D2 before sqquest-patching after request-patching. and on the reference dataset D3- First, we compare the variation in direct effect and the attention probability to the token Rz(C2) before and after pruuest-patchin (top left). The R2(C22) token corresponds to the question of the Dz dataset (the season of the story). We observe set of heads going from attending and contributing strongly to (C2) to very low attention probability and direct effect on this token after request- patching. We observe the opposite for the token R, (C2) (top right). A set of heads is activated by the request- request-patching patching operation. attending and contributing directly to R, (C2). These two observations coherent with the intuition that request-pater patching overwriting the representation of the question
18

---

Published as a conference paper at ICLR 2025
Attention probability the city token
Normalized direct effect to the city token
72
Other cities
17.5
Other
Cusco
56-
Count
L22H9
20
5-0-
0.0
32
0A
35
0.0
0.55
0.25
1455
0,002
0.125
0.75
Other citias
Other cities
Porto
40
Porto
Count
25
L19H2
-330
0.5
0.00
0.20
cities Valenda
Other cities Velencie
15.0
Count
L20H3
7.5
5.-
6.0
Attention probability
Normalized direct effect
Figure 9: City-specific heads contribute directly to the logits when the question asks about the city of the story and the city has a specific value, e.g. "Valencia" for the head L20H3, The inputs in the histogram contain only questions asking about the city.
19

---

Published as a conference paper at ICLR 2025
Absolute normalized direct logii attribution Altention prob. to loken R2(C2) (season)
Absolute normalizzed direct logil allribution vs Attention prob. to the token R1(C2) (cheracter name)
Layer
Layer
Pre-itaquaes paiching (2(CC) Poss Requ
0.6
Attention prbb. RZ(G2)
25
Attention Urob
22
20
0.3
02
0.1
Past-Hecuent pandiigg R1(C2)
Request patching R1(C2)
600
Absolute Nomalized Direct Effect
Absolute Normalized Direct Elfect
Absolute normalized direct logit attribution ve Attention prob. the token R1(C2) (character name]
Layer
Attention R1C22
04
0.2
Inteentorr Refeewnos
Request patshing R1(C2)
D.01
0.02
0.003
0.94
0.06
0:06
D
Absoluta Nomalized Direct Effect
Figure 10: Three-way comparison of the effect of request- patching on the attention heads. Each pair of symbols connected by line is the same attention head in two different experimental settings. Request-patchin is inhibiting the heads in charge of copying the R₁ (C1) token (top left) and activating the heads retrieving Rj (C2) (right). The state of attention heads after request-patching is close to the control condition on the reference dataset (bottom).
from Rí to R2. The attention heads downstream of layer L2 react accordingly by stopping the retrieval of R2(C2) and copying R1 (C2) instead.
Finally, compare the attention probability and direct effect of the attention heads after request- patching to our control condition on the Da dataset (bottom). We find that attention heads have a slightly lower attention probability and direct effect average (relative variation of -7% for the attention, -1 1 % for the direct effect). This suggests that the attention heads charge of copying the correct token (attending and directly contributing to the logit) are working similarly on the reference dataset and after request-patching, although slightly weaker.
B.5 REQUEST- PATCEIN IS INFLUENCING LATE MLPs
In the previous section, we showed that attention heads seem to act as mover heads. They exploit the representation built at the previous layers to compute their queries and use the keys from the context to match the relevant token and copy it to the last position. This pattern has been previously documented in the literature (Wang et al., 2022; Lieberum et al., 2023).
We continue our investigation by exploring whether the attention mechanism is the only mechanism involved in contributing to the correct token. To this end, we perform attention patching. We fix the attention pattern of an attention head its value another question. In case, fix the attention of attention heads to their values on the D, dataset. Formally, for the head at layer an input 22 D2 and 33 E D3 perform the interchange intervention M(xa Ai Arx(xa)) We
20

---

Published as a conference paper at ICLR 2025
Comparison of component direct effect after request patching and on the reference dataget the label token R1(C2)
Comparison of component direct effect after attention patching and on the reference datasel for the label token R1(C2)
0.15
Pasr- Requeai putching
Post Most-Mttttonn patching
Reference dstase!
Norma tood aroet affect
0.1
Normalized direct effect
9.00
th
Bil
labn
1948
ri
welfi
head
Component Name
Component Name
Figure 11: Comparison of the effect of request- gaacciggpacching and attention patching with the reference dataset. While request-patching leads to the direct effect of attention heads and MLPs similar to the reference dataset, attention patching leads to a smaller contribution of MLPs.
only intervene on the attention to the context and normalize the attention probabilities such that they always sum 1.
Attention patching on every attention head causes the model to output R3 (C2) (the character name) with an average probability of 0.14 while predicting R2(C2) (the season) with probability 0.06. Fixing the attention of all attention heads is not enough to force the model to answer the question Rt- This suggests that request-pant patuusttpptchingggg exploits an additional mechanism to reach 0.69 probability of (C2).
The direct contributions of the most important components after request-patching and attention patching are shown in Figure 11. Unsurprisingly, heads preserved after attention patching, as their attention pattern is fixed to have their value from D3. However, the contribution of the MLP after attention patching is significantly smaller than the reference dataset.
observe that the direct effect of the attention
Table 3 summarizes the relative variation in direct effect grouped by component type after the two kinds intervention. While the overlap between the top contributing components with the reference dataset is significant in both cases (57% and 56%), the MI. contribution is similar the reference dataset for request-patching (+4.8 relative variation) but smaller for attention patching (-26% of relative variation). We hypothesize that the MLP contribution the missing effect that causes request- patteestpptthing to outperform attention- iitention-patching.
We speculate that when every attention head attending to the (C2) token position after attention patching, the MLPs at the late layer can access the request R2 present in the input, and detect the anomaly. The MIPs then contribute negatively to RS (C2) to correct the incoherence. In contrast, request-patching replaces the full representation at intermediate layers, making late MLPs unable to detect the incoherence between the request in the residual stream and the input sequence. Such self- correcting functions of MLPs have previously been demonstrated (McGrath et al., 2023). Additional experiments are necessary to evaluate if this phenomenon occurring this particular setting.
C ADDITIONAL RESULTS FOR INTERNAL SCALABLE OVERSIGHT
In this appendix, we provide additional results for internal scalable oversight.
In the main text presented an application to prevent prompt-injection, illustrated in Figure 12. Here, we want to detect when the model is not answering the question but relies on another mechanism to produce the next token, problem known mechanistic anomaly detection (Christiano, 2022).
21

---

Published as a conference paper at ICLR 2025
Table 3: Relative variation in direct effect from the reference dataset the eqquest-patching and attention patching. The mean overlap is computed between the top direct effect contributor on the D3 dataset and the top contributor after request and attention patching. The overlap computed in an aligned manner, i.e. components zi D3 correspond to the component after M(r22 22 < b (x))) such that R R3 and C2 = C3-
Patching Type
Component Type
Mean
Std Dev
Attention Head
-0.114
0.098
Request Patching
MLP
0.048
0.116
Mean Overlap
0.57
0.07
Attention Head
0.092
0.142
Attention patching
MLP
-0.260
0.111
Mean Overlap
0.56
0.08
Untrusted Input
Trusted Input
ftFars
Bob
The atozy city of sadesd.
ficity
Clctaca
fictiyy of
A Madrid
Parii
fttaplaa
Prompt
Sonitized
Injection
Output
Result
Figure 12: Our scalable internal oversight technique relies on request-patching remove the influence of the distractor, string of text crafted to make the model output an arbitrary city (red) instead of answering the question. We patch the residual stream layer Le from an input inspected by human (blue) to a model processing an untrusted input (green). A single trusted input is used throughout all experiments.
C.1 PROBLEM DEFINITION
Abstract problem setting. We define special case of the general problem of scalable oversight. First, we are given trusted input Tr, i.e. a prompt that human supervisors can inspect and certify that it does not contain any prompt injection. Moreover, the human supervisors can solve the problem up in the prompt and verify that the model is answering correctly. Second, we are given large set of untrusted inputs D that is either too big to be supervised by humans or contains task instances that humans cannot answer. The trusted and untrusted inputs are part of the same task. The goal is improve the performance of the model on the untrusted inputs and detect when the model is failing to solve the tasks on the untrusted inputs.
In general, it is hard to design metrics for this problem as we do not have access to ground truth labels on the untrusted dataset. practice, we only apply this technique to simple task, so we are able to measure the accuracy of our technique on the untrusted dataset. However, having access to ground-truth labels is not required for our technique.
We create two types of datasets using this input template. In the base dataset Da, the distractor token chosen to be a city that different from the correct answer to the question. In the collision dataset Do. the distractor token is chosen to be the same as the answer to the question, i.c. the distractor and the correct inspection can differentiate between the two mechanisms.
mechanism lead to the same answer. In this case, only internal
The distractors are the ones described in the main text. In the creation of the datasets, we verify that the models outputting the correct answer on the trusted input. To avoid collision, the answer for the trusted input is never a possible answer in the untrusted dataset. For all experiments, we use the same single trusted input.
22

---

Published as a conference paper at ICLR 2025
C.2 DETECTING ANOMALOUS MECHANISM.
The results of the robustness-increasing experiments suggest an obvious way to detect anomalous mechanisms: if the output of the model changes after request-patching from the trusted input, then it is likely the model was not executing the correct request before.
More generally, when the model is correctly processing an untrusted input, the output of the request- processing step is equivalent to the one happening in the trusted input. In this case, request- ppttchng should not change the results of the model. We can thus interpret any results of request-patching that deviate from this condition as evidence that the model was using another mechanism.
Concretely, for a given untrusted imput Pus we define two predicates for anomaly detection:
A(z,) def [M + (x1)) # M (xw)l
(xx.) def [M t- = # (xx)
A(xa) is true if and only if request-patching from the trusted the untrusted input changes the output of the model on untrusted input, while Bzx,) is true if and only if request-patcho patching from the untrusted the trusted input changes the output of the model on trusted input.
To evaluate the usefulness of these predicate as anomaly detectors, Table 4 presents the rate at which the A, B, and A B predicates hold true on different settings.
On the base dataset, Do. both the large model and small model distractors robustly trigger the A and B predicate for their target models. Surprisingly, even if the large model distractor does not seem to meaningfully influence the output of 1ithia-410mm it sitll triggers both the A and B predicate. We hypothesize that even if large model distractors do not influence the most probable token output by Pythia-4 400m, they can still cause significant internal modification such that request- request-patching from an mechanism.
untrusted to a trusted input breaks the question-answerte
On the more challenging dataset De the models output the correct tokens in almost every case, as both the distractor and question-answerte correct token is output for the correct reasons. While do not have comprehensive understanding of the model internals to fully describe the causes of their output, we make the working hypothesis that if the model output is overwhelmingly determined by given type of distractor when the correct answer and distractor token are different, this will stay true when they are the same.
share the same answer. The challenge is to detect if the
On De. the predicate A becomes almost useless. Forcing the correct request does not change the model output: models are already outputting the correct answer. Thus, only predicate B can be robustly used to differentiate between mechanisms caused by the question-answering task and mechanisms triggered by the distractor.
From Table 4, only the large model distractors are recognized as anomalous for both the large and small models. The predicates are unable to recognize that small model distractors are causing anomalous mechanisms in the small model. Moreover, the triggering rate of the B predicate on the large model distractor on Pythia-I 12b is smaller on D. (with collision) than on the dataset Dy (without collision): 81.5% vs 29%. These observations question our working hypothesis that the relative influence of mechanisms is preserved despite the collision between the distractor token and the correct answer. More work is needed better understand the interference of conflictigg mechanisms in the case where their results are the same.
C.3 LIMITATIONS
This application meant to a proof of concept demonstrating the concrete possibility of leveraging the emergent decomposition of retrieval tasks for unsupervised internal process supervision. We only study a simple task of sresttoon-anseent
on two models. Moreover, the set of distractors we use to induce anomalous mechanisms is very constrained.
While providing promising results on D,, mechanistic anomaly detection fails to reliably detect distractors De. the dataset with collision. It is unclear if this shows a failure of the technique or the existence of a different internal mechanism when the distractor token and the correct token are the same. The same techniques also show non-negligible rate of false positives the control settings.
23

---

Published as a conference paper at ICLR 2025
Model
Detection Predicate
No distractor
Small- model distractor
Large- model
Control distractor
distractor
A
3
70.5
16
Pythia-410m
B
13
39
76
12.5
base dataset
A or B
16
80
77
12.5
A
2
17.5
85.5
1.5
Pythia-12. 12b base dataset
B
4
81.5
3.5
A
6
17.5
94
A
8
Pythia-410m w/ collision
B
13
6
72
A or B3
16
72
A
2
14
Pythia- 12b w/ collision
B
4
29
21
A or B
6
11
36
21
Table 4: Frequency (in %) of the anomalous mechanism detection predicate A and B. The predicate A detects changes in output after sqquest-pacching from the trusted untrusted input, while B is the opposite. The combination of both predicates robustly recognizes the distractors in the base dataset but fails the more challenging dataset where the distractor token and the question answer the same.
Furthermore, thorough evaluation should include comparison with other methods from the literature. For instance, linear probes could be trained on a dataset of intermediate activations labeled with genuine anomalous behavior. Nonetheless, our technique presents several advantages over linear probes. First, request-patching has the advantage of requiring single trusted input instead of labeled dataset. Moreover, anomalous mechanism detection through linear probing can be evaded by designing adversarial examples. Given that technique uses the natural internal task decomposition of models, plausible that will more faithfully detect modification of the mechanisms.
Finally, there are several avenues to improve the performance of this technique. Instead of using single trusted input, the reliability of the technique could improved by using a diverse set of trusted inputs and aggregating the output for both the detection applications. Moreover, anomalous detection could be improved by having more precise understanding of the mechanisms, e.g. by characterizing the layers components used in different contexts.
and mechanistic anomaly
Potential workflow to apply request patching in real-world deployment context. To illustrate how our preliminary results could extend to real-world cases, we propose the following workflow:
Identifying context/request pattern in user prompt. The method would apply to scenarios where model executes a task on variable context data. This includes common use cases vulnerable to prompt injection like crestion-answering summarization, or translation services where malicious content could embedded in the context.
webpages, document
Identifying the request. While our proof-of-concept focused on simple questions about cities, real-world applications would need handle more complex requests like "translate this text" or "summarize these key points". The challenge is isolating the semantic meaning of these requests from their surface form.
Dynamically create trusted input. For each request type, would need to generate trusted example by combining the identified request with verified safe context data. This provides clean reference point for the request processing, free from potential prompt injections. The main challenge automating this process while ensuring the trusted inputs stay representative of the user prompt.
24

---

Published as a conference paper at ICLR 2025
Max normalized token probability for token label R1 (C2) after residual stream patching
QA (bese)
1.0
QA (uniform prefix)
609
6.08
6.60
6.4C
1.21
6.04
5:44
6:45
5.46
5.44
QA (question start)
0.2
0.5
0.00
Normallized token prob.
(mixed template)
0.8
Task
Induction
0.99
0:00
0.08
5:57
1.33
0.90
0.90
0:75
0:04
-0.7
Factual recall
6.77
5:79
0.79
1.77
1.00
6.73
-0.6
Variable binding
6:66
6.90
Type hint
6099
0.00
0.07
6.7
6:00
5
8:9
0.4
talcon-70
falcar-7b-iin Dlcar-7b-insruse
falcon-40b
gp 1012--mall--
gpt2 1122--ddum--
got2 large-
Pyzha- prcha-160m. 160m
pyshia-410m
pythis-1b
pythix- .8
pythie-6.58 6.96
pythie-12b
lleme2-7b
llama2-136
llamu2-70b
Model
Figure 13: Maximal normalized probability of the R (C2) label token after residual stream patching on all models and tasks from ORION. Request-patee patching generalizes to the vast majority of tasks and models studied. White regions correspond to settings where the model unable to robustly solve the task.
Multi-token request patching, Our current method only handles single-token outputs, but real applications require generating multiple tokens. The key technical challenge maintaining coherent request processing across multiple generation steps. This requires investigating whether the modular separation we observed extends to multi-token generation.
D ADDITIONAL RESIDUAL STREAM RESULTS
Figure 13 shows the maximal normalized probability of the R1 label token after residual stream patching. Request patching is robustly working for all model and tasks, except for the induction task. We discuss further the results for induction in Appendix F.
Figure 14 shows side-by-side the results of residual stream patching the question-answering task. For all models, there exists span of intermediate layers (40-80% of the model depth) where residual stream patching leads the model output R₁ (C2) with high probability (>80% normalized probability). This span of layers seems to be the same for the base and fine-tuned Falcon models. This is coherent with the intuition that fine-tuning is only superficially affecting the model internals.
Figure 15 shows the layers L1 and L3 for every model and task studied. We observe a similar motif as in the layer L2 in Figure 4. The processing for the induction task seems to happen earlier than the other tasks such that all three limit layers are shifted toward the early layers.
However, this trend does not hold for Llama 70b. All the limit layers for this model seem to be concentrated over a very narrow span of layers in the middle of the network. To further explore this surprising observation, Figure 16 shows the results of residual stream patching on Llama 70b for the factual recall, induction, and translation tasks. The normalized token probability seems to peak in a narrow range of layers (40-43) for all three tasks, including the simple induction task. is unclear why only Llama 70b exhibits this pattern, contrasting with models of similar sizes (e.g. Falcon 40b) that demonstrate spread-or -ut limit layers. This phenomenon could caused by the larger scale the model peculiarities of the architecture.
25

---

Published as a conference paper at ICLR 2025
Normalized token probability after residual stream patching X1-X2 accross layers on the QA (uniform prefix) task
Label taken R2(C2)
Label taken R1(C2)
Label token RI(C1)
140
00
03
02
Reletve
1olo 40 mnuuu
1oo 400 matus
Sotee 225
Dioo 400
Figure 14: Normalized probability of the label tokens after residual stream patching across all layers on the question-au guestion-answering answering task with uniform prefix. To enable comparison across models, we use the relative layer with 0 as the first and I the last layer. Request-patchns patching is general across models: mid-layer residual stream patching causes the model to output R1 (C2) with more than 80% normalized token probability.
E CAUSAL ABSTRACTION
Validating the high-level causal graph using the framework of causal abstraction. In this Ap- pendix, we express the implications of request-patching on the high-level structure of the computation happening in language models solving retrieval tasks using the framework of causal abstraction (Geiger et al., 2023). We define high-level causal graph operating on the abstract input representa- tion and alignment mapping each intermediate variable in the high-level causal graph to set of model components. The input-output alignment is defined by the ORION abstract input and output representation. The alignment is illustrated in Figure 17.
Our causal graph a simple two-step symbolic algorithm that treats the request and context separately before combining them algorithmically solve the retrieval task.
We validate the alignment using interchange intervention accuracy (IIA). IIA is defined in Geiger al. (2023) an average over every possible multi-input interchange intervention. However, this average introduces statistical distortion in the case of the alignment we are considering. Because of the shape our causal graph, interchanging a variable late in the graph screens off the effect of the interchange happening carlier in the graph. Thus, intervening simultancously on early and late variables is equivalent to interchanging the late variable alone. To remove this statistical distortion, we average the results of the interchange interventions such that each unique experiment gets the same weight.
Moreover, given that residual stream patching is a kind of interchange intervention, we reuse the experimental data from the exploratory causal analysis to compute the IIA. Given the simplicity of our alignment, we can write the IIA for task T from ORION in terms of three interchange operations as follows:
M(azket + (x1)) = R((C
IIA
3 Ex BEET
+ M(azlkhe + 2x1)) = (C2)
+ M + 2 (x1)) = R2 (C2)
We do not include the results interchange intervention on the context variable. Given the model architecture, the interchange operations M(xz = =x1)) and M(ri (x2))
26

---

Published as a conference paper at ICLR 2025
Relative layer L1
Task Name
Qk (bees)
OA Inioed tamp ale)
0.8
Layer relative
Viriale binding
0.6
0.2
0.0
ralcon-20
Fa -alorr////imneee
racop 40b
00t2-s00
9pt2-medium
SPX2
002-zeuey
nicon-10b-instruct
1i2-small
wgl wglt
QB't
0899
Relative layer L3
0.8
Layer relative
0.6
Task Namto
0.2
(unliam pnched
(tueet
Fntution
resal
0.0
Vanudic Landing
Trandaiiin
falcon-78
falcon-7b-ine falcon-Jb-nntmee
racon 1000400
opt2
9pt2-medium
apt2
da Safiiis.o 60m
swwee 6.96
-o-nn-bb-nstruct
licoon-0b-instruet
smzzl
2 2.8b
410m
Figure 15: Layer L1 and Lg for different models and tasks.
27

---

Published as a conference paper at ICLR 2025
Residual stream patching of llama2-70b on the translation task
Normalized tokan prob.
label rarte
(1552
R0320
10
30
50
70
Layer
Residual stream patching of llama2-70b on the induction task
Ixbel 1ame
R1:11
Nomalized token prob.
0.0
Roys1
10
20
40
60
80
Layer
Residual stream patching llama2-70b on factual recall
Normalized token prob.
bbel rame
0.8
RV00
0.6
0.4
0.2
20
40
60
Layer
Figure 16: Result of residual stream patching of Llama 2 70b three retrieval tasks. The maximal effect of residual stream patching, i.e. maximal probability of the label token R; (C2), is located at the exact same layer (layer 42) for every task.
28

---

Published as a conference paper at ICLR 2025
Language model
High-fevel causal graph
Textual context
request
Abstract context represention
Abstract request representation
What's
Stasy: Mtaicc iu [-]. CEts the roasen why (iLnndo in foverite
place scozy?
SELECT Name(o) FROM WHERE city
C
A: The Goity of
Entlys
Nil
C
R
Retrieval(R, C)
Loyor
London
London
Artention and MLP blocks
Retrieval
Context
Answer
Execution
Processing
Propagation
Intermediate Aligneman
Resloval
Constont
Aequest
Input-outpur Alignement
Processing
Processing
Figure 17: Alignment between high-level causal graph that uses abstract representations of inputs, and language model running on the textual representation of the inputs for retrieval task. The alignment bounds the position where request processing (in red) and context processing (in green) are located the intermediate layers of the model. The Nil node is isolated in the high-level causal graph. It does not influence the output of the causal graph and thus can be interchanged freely.
29

---

Published as a conference paper at ICLR 2025
Normalized Interchange Intervention Accuracy for the high-level retrieval causal graph
(base)
(uniform prefix]
0.35
A (question start
001
9:77
032
0.00
0.03
3355
0.0
0:44
0.90
templete)
Normalized
0.85
Task
Induction
MM
8.99
1:92
0.5
936
9.92
9.1
0/44
Factual recall
0.9
2.81
LUM
21
-0.90
Translation
0.75
Verieble binding 20c
0.41
Type
falcon-7h
falcon-7b-instnnn
falcon-40U
ralcon-400 Balcon-40o-oo ion---nrucck
gpt2 pes2-small-
gpt madium-
opt2-large
gol2-4
pythia 160m
pythia-18 1b-
pythia 2.8b:
pythis--6991
pytha-120
llama2-7b-
Ilamaa-700
pythia
Model
Figure 18: Normalized interchange intervention accuracy for all models and tasks studied for the
high-level retrieval symbolic algorithm. The normalized IIA is greater than 85% 91 out of the 106 settings studied. This demonstrates that high-level causal graph faithfully describes the internal model computation across different models and tasks.
are equivalent. The first one corresponds the intervention on the request in the high-level causal graph, and the second corresponds to the intervention on the context. Moreover, our task datasets are defined by independently sampling R and C. This means that by definition, the average output of M(2212 + 2, (21)) and M(x1 z'6 + z (I2)) are the same. We thus remove the results of the intervention on the context from the average to avoid artificial duplication of experimental results.
To facilitate the comparison across tasks, we normalize the IIA such that O corresponds to random guesses and the baseline accuracy on the task. Note that the normalized IIA could be greater than 1 if the causal graph also explains the mistakes of the model. However, consider simple high-level causal graph that always answers the correct token such that the baseline model accuracy is natural upper bound for the IIA.
Finally, worth noting that the first and last terms of the expression IAT are dependent on arbitrary threshold way to increase the IIA. However, this would also make the alignment less expressive as L1 would tend to be 0, and L3 would tend to be the last layer, effectively making these parts of the alignment trivial. The thresholds thus represent a tradeoff between the strictness the hypothesis and the ease of validating it.
define L1 and L3. Choosing a higher threshold would be artificial
The normalized IIA scores for each model and task studied are shown in Figure 18. We observe that the majority of settings studied lead to high IIA scores (91 of the 106 pairs of models and tasks have scores greater than 85%). showing that the high-level casual model faithfully describes the internal processes of language models on the ORION tasks.
COMPARISON WITH PRIOR WORK
Factual recall The factual recall and abstract induction tasks from ORION have been previously studied in the mechanistic interpretability literature. In this section, we show that the mechanisms described in previous works compatible with the results of request-patching.
Previous works studied the factual recall abilities language models on prompts represented by a triplet (s, r, a) where is subject, is relation being queried, and is the corresponding attribute, ie. the value of the relation on the subject. A prompt would contain the subject and relation while the attribute would define the label token, e.g. "Reat music is owned by" "Apple". Geva
30

---

Published as a conference paper at ICLR 2025
et al. show that early attention layers at the last token position are used for relation propagation, propagating information from the relation token to the last position, e.g. the information about the relation "owned" to the "by" token in the example. Later layers are in charge of attribute extraction. They recover the correct attrihute from the last subject token, according to the relation propagated to the last position by the earlier layers.
Using the ORION input representation, the relation is part of the request, while the subject is in the context. When performing residual stream patching intermediate layers, we observe request- patching: the information from the relation in 2 is transferred but the subject stays the same. Our observation is coherent with the finding from Geva et al. that relation propagation and attribute extraction happen at non-overlapping layers.
Note that contrary to Geva al. we do not use the dataset Counterfact. This dataset cannot be incorporated into ORION because of the "Decomposable" desiderata for task constellations. Most of the relations in the Counterfact dataset cannot be applied to arbitrary subjects, e.g. a famous person does not have an attribute for the relation "capital city". To circumvent this limitation, we create two datasets that fit the "Decomposable" desiderata, enabling the design of systematic causal experiments. We document this process in more detail in Appendix H.6.
Induction The induction task consists in completing patterns of the form [A] [B]
[A].
For instance, such patterns occur naturally when completing a name that appeared before in the con- text, e.g. "Harry Potter were first characterized in small two-layer Transformers in (Elhage et al., 2021). The mechanisms involve two stops: the first step consists in previous token heads acting at the [B] position copying the preceding token [A]. The second step involves induction heads acting at the [A] position. In
Harry Pot" > "ter". " The mechanisms for induction tasks
follow-up paper, Olsson et al. hypothesize that induction heads are also present in large models and
recognize more complex patterns with similar structure such [A] [B]
[A*] (Olsson
et al., 2022). In this case, [A] and [A*] can be composed of several tokens and be recognized
using fuzzy matching instead of exact token matching. They propose a similar high-level structure as the simple mechanism: the representation at the position [B] is contextualized by incorporating information about the preceding prefix [A], using a more advanced mechanism than the previous token heads. Similarly, the representation of the last token incorporates information about the tokens from [Ax] 1. At later layers, induction heads leverage their attention mechanisms to recognize the similarity between the representations of [A+] at the [B] token position and the representation of [A] at the last token position. Finally, their OV circuit copies the [B] token.
The induction task we designed involves multi-token prefixes with exact matches. We study patterns [A] [x], where [X] is a separator token, a column in our case. According to the extended mechanism for induction, the residual stream at early layers at
of the form [A] [X] [B]
the last token contains the information propagated from the second [A] occurrence, while the later
layer contains induction heads in charge of finding the [B] token in the broader context. If the [A] propagation and the induction heads occur at non-overlapping layers, patching the early residual stream should only modify the representation of the token [A] at the last token position without impacting the retrieval abilities of the induction heads, In the ORION abstract representation, [A] is
the request in the induction task. Hence the proposed mechanism for induction heads is coherent with the results of request- pattusstp
Propagating the [A] token the final residual stream and the operation of the induction heads are both simple operations, each of these operations can theoretically be performed in a single layer. We hypothesize that these two operations are performed redundantly by two sets of components acting in tandem, a first set to propagate information from [A] to the last token, and second set of induction heads. We hypothesize that these two sets of components are situated at overlapping layers in large models part of pre-processing happening in early layers. Large models have the capacity for redundant parallel computation because of their large number of attention heads per layer. This hypothesis would explain the lower performance of ggguest-ptcchng on induction tasks in large models. No layer separates the request and its processing: due to the simplicity of the task, they both happen in parallel.
31

---

Published as a conference paper at ICLR 2025
G TRANSFORMER ARCHITECTURE
In this appendix, we provide a complete description of the Transformer architecture. The pre-softmax values nn the logits at the n-th token position. For the GPT-2 Transformer architecture (Radford et al., 2019) with L layers, the function MM can be further broken down as follows:
= LN(2,)
2/ o dr + m
m MLP(2 + +ah)
LN (Wout (GELU( 1-1 + axe)) + bia)) )) + bout)
4 Attn(2
F6 WEL + Wp
The final logits πi are constructed by iteratively building series of intermediate activations of. we call the residual stream, following Elhage et al. (2021). The residual stream of at token position and layer is computed from the residual stream at previous token positions at the previous layer z/-1 by adding the results of Attn, a multi-headed attention module, and MLP, a two-layer perceptron module.
The MLP module depends on the residual stream z at position and layer I - while the attention module can aggregate information from the previous layer across every previous token position. The residual stream is initialized with the embeddings computed from the token and positional embedding matrices WF. E. Wp, and the one-hot encoding of the input tokens I. Finally, Wr is the unembedding matrix, GELU the Gaussian error linear unit activation function (Hendrycks and Gimpel, 2016), and LN is layer normalization function (Ba al., 2016) applied to the final residual stream and the output of each module.
In practice, the models we study have slight deviations from the GPT-2 architecture. The Pythia (Biderman et al., 2023), Falcon (Almazrouei et al., 2023) and Llama 2 (Touvron et al., 2023) models use parallelized attention and MLP. In the formulae above, this translates as my MLP(22 )). Moreover, Falcon contains additional layer normalization at the input of modules. LLama 2 uses the SwiGLU activation function (Shazeer, 2020) and layer normalization only at the input of modules.
H ORION PROMPTS
Table 5 provides succinct overview of the task included in ORION.
H.1 DETAILED DESCRIPTION OF THE DATASET DESIDERATAS
1. Structured. Every textual input in ORION accepts an abstract representation using the context and request representation defined above. Motivation: Providing a unified structure to define and interpret causal interventions without the need for getting-specitty labor.
2. Decomposable. For every dataset D in ORION, for every abstract representations (C1, R1), (C2, R2) in D, R2(Ci) and Rí (C2) are well-defined. This means that an arbitrary request can be applied to an arbitrary context from the same task. Abstract representations of requests and contexts can be freely interchanged across task. Motivation: Enabling the design of interchange interventions.
3. Single token.. For every dataset D in ORION, for every abstract representations (C1, R2), (C2, R2) in D, Ri R' <> (C1) (Ci).) In other words, in given context, the output of each request gives a unique answer. It ensures that measuring the next-token prediction is enough to know which request has been answered. Motivation: Making experiments easy to measure and computationally efficient.
4. Monotasking, For every dataset in ORION, for every abstract representation (C, R) in
D, there unique line in C such that ATTR, vy-- This condition ensures that requests
32

---

Published as a conference paper at ICLR 2025
Task instance generation
Task generation
Template
Abstract input representation
upon
(OnY, (CHA)) wanderingg
Contact (TAllas) Character) (RRomet Cityl Requeet Nomele) Rolole) City
(Q57)
Problem type
Queston answering
Placeholders
Textual input
- -'Forss Wome]]
Story upon wondening: cit?
CHMR - ['Alce "Uoof
was
QST-NQ charaster7At A: Macly/A:]]
Figure 19: The semi-automatic task generation process used to create ORION. We use ChatGPT to create template and values for the placcholders given problem type. To generate instance from the task, we start by randomly selecting placeholder values to create an abstract input representation. Then, format string to fill the template. When we need more flexibility, incorporate the placeholder values into the template.
GPT-4 to
answerable with unambiguous answers. Motivation: Making analysis tractable. It is easier to understand models solving a single problem than solving multiple problems in parallel.
5. Flexible. ORION contains diverse tasks spanning multiple domains and levels of complexity. In practice, demonstrate the flexibility of the ORION structure by creating 15 different tasks spanning six different language model abilities. Motivation: Enabling rich comparative analysis across models and domains.
In the code implementation, we designed automatic tests to ensure that conditions "Decomposable", "Single token", and "Monotasking" are respected for every task in ORION.
H.2 DATASET CREATION
To create the ORION task datasets, semi-automatic process illustrated in Figure 19, leverag- ing the creative writing ability of ChatGPT2. Concretely, we use the following workflow:
1. Find problem that can be formulated retrieval task, e.g. question-answerring answering.
2. Use ChatGPT to create template with placeholders, c.g. a story with placeholders for narrative variables such as the city, the name of the character, and the question being asked.
3. Use ChatGPT to create set of placeholders.
4. Procedurally generate set of abstract representations for the contexts and requests.
5. Generate the textual inputs from the abstract representation using format strings ChatGPT when more flexibility required.
We applied this workflow to create 15 tasks spanning six problem domains requiring different abilities: question answering, translation, factual recall, variable binding, abstract pattern matching (induction pattern) and coding. For each domain, created variation of surface-level parameters of the task (e.g. changing language of the translation). We give an example input-output pair for each in Table We provide detailed discussion about task choices well precise description of each dataset in the rest of this Appendix.
hhttps:/chatt openai comm
33

---

Published as a conference paper at ICLR 2025
Task name
Example Prompt
Label token
Variations
Question- answering (base)
Story: the lively city of Valencia, spring WerningCrrssoopiin [..] skilled veterinerian [.] "I'm Christopher* he replied, [-].
Ques:ion: What is the zame of the main charactez?
Answer:
character
Question-
Stozy: the lively city of Valencia, spring moniin@hristopher I [..] skilled veterinerian I.] "I'm Christopher* he replied, [..].
answering (uniform prefix)
Question: what is the zame of the main character?
Answer: answer
Question-
Ques:ion: What is the zame the main charactes?
Christopher 1
answering (question first)
Story: the lively city of Valencia, spring morninga [-.] sefilled veterinarlan [-] "T'm Chriatopher* be zeplied, [.I.
Answer: The answer is
Question- answering (mixed templates)
Uniform distribution of prompts from three varia- tions of question- question-answering above.
Translation
Engl iish:
Mi ller
3
In an era defined by increasing global comperacures At the forefront is M. Smith, marine biologist I..] Next, we curn to M. Miller, a climate economis: I...]
[..]
French:
[...]
Nous nous tournons enauite vers M.
Factual recall
Question: which continent did Kahammad Ali live? America Answer:
2
Variable binding
Anthony has collection of pencils, 50 pencila ace 20)_ blus, 10 pencils are zed,
3
pencils green.
total are aither green? 1e'1l add the number of green pencils
Fow many penci
Induction pattern- matching
T
T2mNX:T2gIS
GveiE:Gegvs
ApvlA:
eLnNX:
Type hint understanding
celeolato_circcmfeeeeet (circlle (circle: circle) floarD
3
[..]
Kectangle (Point (Point (2, 3), Foint (6, 5))
Circle (Point 01, 5)
print row
Table 5; Tasks from the ORION collection contain varied problem type and prompt format. For readability, we use "[...'" to shorten the prompts. The rest of the text is part of the textual input. In particular, "[...]" 1" is part of the prompt for the translation task. We use indicate space in the label token.
34

---

Published as a conference paper at ICLR 2025
H.3 DISCUSSION ABOUT TASK CHOICE
For a given problem type, we generate several templates, enabling the creation of several task variations. We use this procedure to generate 15 unique tasks spanning six different abilities.
We use several criteria in choosing the problem types. First, we choose tasks that have already been studied the literature to act as reference points for our analysis. This includes factual recall and the induction task. Then, to allow analysis across model scales, we design simple -anserinn task that can be solved by both small and large models. We also create more challenging tasks to explore diverse skills such coding abilities, tracking the association between an object and its quantity, and tasks involving translation from English to three different languages.
In addition to diversifying the content of the tasks, create structural modifications to the task template. To that end, we create question-answering templates where the question before the story in the dataset and templates where the final token of the prompt does depend on the request. We also create mixed
task containing prompts from the three variations.
In this of this Appendix, we describe in more detail the process we used create each task of the ORION collection. We also provide complete example prompts for each task.
H.4
QUESTION-ANSWERING
H.4.1
GENERATING THE STORIES
We created set of 100 stories used in the four variations of the question-answering task. Each story was created by defining:
The name of the main character
The occupation of the main character
The time of day
The season of the year
The city of the story
The action of the story
An order in which the above elements should be introduced in the story
An example story called a "master story", used as template to incorporate the new narrative elements
The value of each of the narrative elements uniformly sampled from lists of 3 to 5 different possible values for each field. The lists were generated using manual interaction with ChatGPT.
The 8 narrative elements were combined in prompt shown in 20 and completed by GPT-4. The goal of this process was to reduce as much as possible the variations introduced by GPT-4, such that the variables in the generation prompt characterized the generated story as comprehensively as possible.
H.4.2 GENERATING THE QUESTIONS
We manually generated questions and answer prefixes about three different narrative variables: character occupation, city, and name of the main character, For each narrative variable, we created three different phrasings.
The answer prefixes were either uniform, as shown in Figures 21 and 22 for the task variation with uniform prefix and question the start, or depended on the variable queried in the question, as shown in Figure 23 for the base task variation. The base task variation can be solved by smaller models, while only larger models can handle uniform answer prefixes.
Griven that the mixed-template task is an aggregation of other task variations, we do not include it in the count of 15 unique tasks.
35

---

Published as a conference paper at ICLR 2025
You have to generate a short story that fits in a single paragraph of less than 150 words. It has to respect list of precise constraints.
#+4 Narrative elements
The main character is named [character_name), Their occupation is inharacter_occupati
The story takes place in (city). The time the day the (day_timel, and the seasen is (season). The time of the day should stay constant in the story. The action performed by the main character is laction).
It's crucial that all the elements appear the story.
#*# Order of the narrative elements
The order in which introduce the narrative elements is imposed. The main priority to respect the order of apparition imposc. Here is the imposed order in which to introduce the narrative elements. This order is already present in the template story.
(variable_order)
### Template story
You have generate story that matches closely possible template story. Your goal to modify the template story such that all the narrative elements are present, but the general structure (e.g. order in which the narrative element are introduced etc.) is close as possible to the template story.
Here is the template story you have to stick to:
"[master_story_texx
Generate short story that matches the template story while incorporating the new narrative elements.
Figure 20: Prompt used to generate the stories for the question-answering tasks. The variables in curly brackets represent placeholders that were replaced by values randomly sampled from manually created lists of possible values.
H.5 TYPE HINT UNDERSTANDING
Using ChatGPT, we generated three Python code snippets introducing new classes and functions using these classes, as shown in Figure 24. The context is a set of variables with given type. The request asks for variable name with particular type. The function definitions do not vary across prompts and are only used to formulate the request.
H.6 FACTUAL RECALL
Existing open-source datasets created to study factual recall in language models, such as the one introduced in (Meng et al., 2022), contain relations (e.g. the sport of an athlete) that can only be applied to subset of the subjects (e.g. only athletes, since asking the sport played by country does not make sense). This makes it impossible to causal intervention such the type required for request- pateesstttthiiggggg (the desiderata "Decomposable" is not fulfilled). Thus, we created two variations of factual recall tasks such that any pair of subject and relation exists. Contrary to the other task from ORION, the retrieval tasks do not involve copying an attribute present in the context, the task requires the model to know the attribute.
36

---

Published as a conference paper at ICLR 2025
Context
</endoftext|
Here is a short story. Read it carefully and answer the questions ons below with a keyword from the text. Here is the format of the answer: 'The answer is "xxx".'
The morning sun bathed
strests Cusco in warm, golden light, casting long shadows that danced along with the gentle summer breeze. Amidst the bustling city, a tall, slender figure stood on the rooftop of an unfinished building, their eyes surveying the urban landscape below. As the skyline slowly transformed under their careful guidance, the person was no mere observer, but an architect, orchestrating the symphony of steel and concrste. The sound of birdsong filled the architect
became apparent that
the air, but it voice, soaring with joy and passion, song of creation and ambition, And as the 1ast notes faded away, the wind carried whi whispered name, the gignature of the artist who painted the city with their dreams: Michael.
soon joined by another melody
Answer the questions below.
Request
Question: What job does the main character have?
Answer:
The answer
Figure 21: Example prompt for the QA (uniform prefix) task.
Geography dataset We used an open-source database* countries. We extracted the name, capital city, and continent of each country.
Geography dataset Following the process used in (Krasheninnikov et al., 2023), we used Cross-Vert Verified database (CVDB) of notable people 3500BC-2018AD (Laouenan et al., 2022). Each individual was ranked by popularity (measured with the "wiki _ecii_raddr________ 20ss__21_______ 25_011111 feature"). and 4000 of the most popular individuals were taken (2000 men and women each). We selected the fields related to the gender, continent, and nationality of each notable person.
Filtering For both datasets, we queried the relation about the entity using few shot setting, as shown in Figure 25. From the raw data extracted from the dataset, further filtered the list of entities to keep only the ones where GPT-2 was able to answer all the questions related to the entity. The final dataset contains 243 notable people (i.e. 729 questions) and 94 countries (i.e. 282 questions).
H.7 VARIABLE BINDING
We were inspired by the shape grade school math problems from the GSM8K dataset (Cobbe al., 2021). The goal to create retrieval tasks that would naturally occur in chain of thought generated by model solving a math puzzle. The context contains objects with different quantities. The request asks for the quantity of an object type.
To create the dataset, we picked one sample from the GSM8K dataset and generated variations using ChatGPT. An example prompt be found in Figure 26.
httpp::/gitt wttpph..co//awweeeeeecnaaaa
37

---

Published as a conference paper at ICLR 2025
Request
</endoftext |>
Read the question below, then answer it after reading the story using a keyword from the text. Here is the format of the answer: The answer is "xxx".'
Question: what job does the main character have?
Context
Story: The morning sun bathed the streets of Cusco in warm, golden light, casting long shadows that danced along with the gentle summer breeze. Ami dai the bustling city, tall, slender figure stood on the rooftop of an unfinished building, their eyes surveying the urban landscape below. As the skyline slowly transformed under their careful guidance, it became apparent that the person
observer, but architect, orchestrating the symphony of steel and concrste. The sound birdsong filled the air, but it was soon joined by another melody -- the architect 's voice, soaring with joy and passion, a song of creation and ambition. And as the last notes faded away, the wind carried whispered name, the signature of the artist who painted the city with their dreams: Michael.
Answer:
The answer is
Figure 22: Example prompt for the QA (question first) task.
H.8 TRANSLATION
We used ChatGPT-3.5 (referred to as ChatGPT in the main text and the rest of the Appendix) to generate news articles using placeholders instead of real names. We instructed names as possible and to prefix each name with common title, such as "M.". Then, we asked ChatGPT to translate the text into a non-English language. From the translated text we extracted excerpts that preceded each of the names but did not include any names. These excerpts formed the request. When creating the dataset, the placeholders are replaced by distinct family names from a generated by ChatGPT.
add as many
Using this process, we created three variations with different subjects, target languages, and name prefixes.
Title: "Climate Change: The Unsung Heroes", Prefix: "M.", Target language: French.
Title: "Hidden Wonders Revealed: New Species Discovered in Unexplored Amazon Rain- forest", Prefix: "Dr.", Target language: Spanish.
Title: "From Pirates to Naval Heroes: Captains who Shaped Maritime History", Prefix:
"Capt.", Target language: German.
The entities in the context the named characters, and their attribute the sentence in which they appear. The request is asking for a name that appears in given sentence.
H.9 INDUCTION
We generated 10 pairs of random strings made from upper and lower-case letters separated by column. The context contains five enumerations of the pairs. Each enumeration is in random order. The request is the first half of one of the pairs. An example prompt is shown in Figure 28.
38

---

Published as a conference paper at ICLR 2025
Context
<| endoftext||
Here is short story. Read it carefully and answer the questions ons below.
The morning sur bathed the streets Cusco in warm, golden light, casting long shadows that danced along with the gentle sumner breeze. Amidst the bustling city, a tall, slender figure stood on the rooftop of an unfinished building, their eyes surveying the urban landscape below. As the skyline slowly transformed under their careful guidance, the person was nо mere observer, but an architect, orchestrating the symphony of steel. and concrete. The sound of birdsong filled the air, but it was soon joined by another melody -- the architect voice, soaring with joy and passion, song of creation and ambition. And as the last notes faded away, the wind carried whispered name, the signature of the artist who painted the city with their dreams: Michael.
became apparent that
Answer the questions below, The answers should be concise and the point.
Request
Question: What job does the main character have?
Answer:
The main character a professional
Figure 23: Example prompt for the QA (base) task.
39

---

Published as a conference paper at ICLR 2025
Context
<jendoftcx21>
frun Lyping import List Lapozt
class Poink:
def -ini -ini__ I3011, Cloat, float) -> None:
welf.y
class Bectang le
del _ini fawll, but ion_olll Paint, Logueightz Point] f.btteet left boston left se'f, r.ioooggit
bop_right
init fewls, centers Point, radiust flost] Nones se...cmater center
furadius radiuse
clasa Polygons
def nic fself, points: [Painna >> Nores
selfipintt points
calculate area (rectaaate Bectangle) floats height = metangle, top tiphtight reolangle.bolle bottoon wisth rectangle, wwwpgge...pp.iio.e..ee rop.rig boctangle.betton. lett.x height
det calculate coloulate_centerrdeee center(rectangle Rectangle) -> Point:
lontangle.bettonn. laft. pectangee.top.right... /.2 center_y incstanple.otttooiirt
center_x
toctangle.top_rtppp..
return Point Icenter_X, X, center_yl
calculate caleuieto_dietance (poiat: Point, pointá: Pointl tiest: (peint22xx poirtl.xi
(point2.y poinel.y)
det wwwculate www.caaateccrrccnesssseeeeeseeee www.ccrreeccceeeeei...o.ooo (cizcler Circle) >> tloat:
return circle.radius
det wwwcuiate circle creaccieie: dizalel tleat:
return pi leirele.radius ** 21
Folygon)
ge-imeter
points pclygon.points + [polygsn.poiats(0]] Add the cizst point the and for closed
range (Lan(poiats:
perimeter + ditculate_ (peintss()) pointoli + 211
return perimeter
polygon
Polygon(IPcind 10, 01, Point 0), (9, 1111
Create rectangle
Restangle
31,
Czeate circle
cizela iPoint
Request
Calculate area peint [eslouaaee_aeeet
Figure 24: Example prompt for the type hint understanding task.
40

---

Published as a conference paper at ICLR 2025
CVDB prompt
< endoffeet I>Question: What was the country of Freddie Mercury?
Answer: UK
Question: On which continent did Muhammad Ali live?
Answer: America
Question: What was the country of Fela Kuti? Answer:
Geography prompt
< <lendoftext I>Question: What is the capital of France?
Answer: Paris
Question: What is the language spoken in Malaysia?
Answer:
Figure 25: Example prompt for the factual recall task on the CVDB and geography datasets. There is no clear division between context and request the prompt. full rigor, the context composed of single entity, c.g. 'Fela Kuti" in the first prompt, while the request is asking about an attribute, e.g. the country, without filtering as there a single entity in the context.
Context
<I endoffeext >>Jon is baking cookies. The recipe calls for cups flour, 2 cups of sugar, and cups chocolate chips. How many cups of ingredients in total are needed for the cookies?
Request
We' 11 add the number of cups of flour
Figure 26: Example prompt for the variable binding task.
41

---

Published as a conference paper at ICLR 2025
Context
<lendoStext
Heze is a new article in anglish. Belck, you can find partial translation in krench Please complate the translation.
Baglishs
Title:
*Climate Change: The Cnaung Nerces"
in an era defined by incressing global temperatures and extreme weather events, the fight against clinaee change cont inuea many fronta. While prominent eevironmentalis: and politicians ofter clain the limelight, behind the acenes, countless unsung
dedicated their lives combeting climate change. article alms to spotllihtt the work of these individuals.
At the ferefrent is N. method for premoting coral reaf growth, Given that coral reefa act as corber sinks, the atmosphere,
marine bislogist who has developed innovative
absorbing and scoring implication for elimate mitigationn Despite facing aurerous hurdles, M. Jones has consistently pushed forward, driven by an unvavering commitment oceanic
Jones's
flaant
Next, we tuzn to X. Mertinez, a climate economist from a small town who has successfully devised market-based solution to curb incustrial carbon emiasions. By developing a novel carbon pricing modal, M. Marsinez has enabled a tangible shift toward greener industrial practices. The resulting in significant seductions in 002 emissions. Yet, despite these successes, sadar.
adopted aeveral countries,
X. Nartinez's
often
under the mainstream
Another unsung here in the climate change battle scientist pionsering line of geratically modified conditions. With changing "ainfa patterna threatening food security worldwide, M. inmense globai relevance, Hloweves, genetically modified organisms, the contributions of scientists like K. 2erez often go unnoticed.
M. Perez, a young agriculturel that thrive drought
Peres's
controversy surrounding
Additionally, the story of K. Themas is worth mentioning. An urban planner by profeasion, M. Thomas carbon footprin:. By integrating renewable energy sources, promoting public trensportation,
instrumentsl in deaigning green aities with minimal
ereating more green spaces, M. while the sesthetics of these sities often capture public attention, the visionery behind them, Themas, remains relatively unknown.
living. redefined
Lastly, we have B. Barris, a graasreats activist working tirclessly to protect and har community, Barris mobil 1zed local communitiss halt deforestation and engage in extensive Lree plan.ing initiatives. While large scale afforestazion projects level herses like M. sarris remair. largely unsung.
get globel zecognition,
community-
The right againat climate change is not aingle bat:le, but waged multiple fronta. Every victory counta, no motter how small, So, as we continue this struggle, lett's not forgez appreciate и. Pezez, X. Thomas, and X. flarris who, away from the spotlight, are making a wozld difference.
unauin heroea
M. Martinez
Request
Freach:
[...]
Bn intégrant dea sources d' énergie renouvelables, en favorisant les Lransporta publics et en eréant plus d'espaces verts, M.
Figure 27: Example prompt for the translation task.
42

---

Published as a conference paper at ICLR 2025
Context
<jendofrext PMC.C:/://CCCCCCCC C
198ak
Javec:
vBufies erghoM
paaat e twg
GLb2R:URala
ViRed
Y
fiiikk j (VA
axRtXx 19Makk
gi hi URe
w
Reugadd vRod )
faVigg vvais
vBufice agCua
XP0gRt:0Roe
aa Vited
JaVec: j (Vva
XP0gR=000005
covt.
Tiak ae Vited 5
vitlwwe wightu
JaVec: j (VA
spigtd: rRed 5
fhilkk j MVil
NF Covt
Spigtid Viodd 5
peakt sug 1W9
GLbeR: URILs
aavac::iVas
T
InVak: 1VVaB
peaki suguss TW
KPRgR:
SeVRG: VVAR
IBRIs:IRod:
thVgk:1
Spqga::YRod:
vduLi:
Op Soodd:1Rcc XRod:
peakt:
Spigiid
faVak: 11vas
sp 1Rod5
X20:R:20R0S
peaktruqIwz
Request
WFCJI:
Figure 28: Example prompt for the induction task.
43


Reviews:
[{"review_id": "QmRvEQdcFA", "reviewer": "Reviewer_EUoL", "summary": "This paper studies how LLMs internally solve retrieval tasks. To do this, they introduce ORION, a collection of 15 datasets and apply causal analysis to 18 LLMs. To solve the tasks in ORION, the LM has to combine information from the question and context into the last token position. The authors use residual stream patching on this last token to investigate how the LM decomposes and combines the information. They need 2 prompts for this; prompt1 = R1C1 and prompt2 = R2C2; R stands for Request (question) and C for context. They observe that depending on the layer where they do the patching, the model may output R1(C1) (answer to question 1 from context 1), R2(C2), or R1(C2). This means that the patching is intervening the model’s computation and combining high-level information from 2 prompts. With this, it is possible to supervise the internal process of the models and for example, make the model more robust to prompt injection.", "questions": "* in lines 450-451, it is written that you perform residual stream patching from the trusted input to an untrusted input. However, I couldn't understand how you obtain the trusted input. My understanding is that we only have an untrusted input in a real-world scenario, so this method is difficult to use. Line 420 says that you designed the application so that it doesn't require access to ground-truth labels, but isn't the trusted input the ground-truth label? To organize my questions a bit more:\n   * What is this \"trusted input\" and how it differs from the ground-truth input?\n   * How do you obtain the \"trusted input\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Extensive experiments on 18 LMs from 125M to 70B parameters on 15 datasets.\n* Empirical evidence of internal decomposition of retrieval tasks. This can be very useful of the community to better understand the internal behavior of the models and improve their interpretability.\n* Application to mitigate prompt injection", "weaknesses": "* It remains unclear to me how to operationalize the request patching to mitigate prompt injection. It seems it requires a trusted input although lines 420 seem to indicate it doesn't. See more details on the question below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2uPSkxXSXB", "reviewer": "Reviewer_7Spb", "summary": "This paper introduces ORION, a diverse set of structured retrieval tasks designed to study how language models (LMs) handle complex information retrieval across six different domains. The authors conduct a comprehensive analysis on 18 open-source LMs with varying sizes and find that these models internally decompose retrieval tasks in a modular fashion. Specifically, middle layers process the request (e.g., a question), while later layers retrieve the relevant entity from the context. Through causal analysis, they demonstrate that enforcing this modular decomposition preserves a significant portion of the model's performance. Additionally, the paper presents a fine-grained case study and proposes a scalable method for internal oversight of LMs to mitigate prompt-injection attacks, significantly improving accuracy with minimal human supervision. This work provides insights into the universal modular processing of tasks in LMs and pioneers the application of interpretability for enhancing model oversight.", "questions": "See the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ORION represents a novel data-centric approach, facilitating a comparative study across 18 models and 6 diverse domains, enhancing the scope and depth of retrieval task analysis.\n2. The fine-grained case study on Pythia-2.8b effectively bridges macroscopic and microscopic descriptions, adding a layer of detailed validation to the findings.\n3. The insights gained from this study significantly contribute to the broader understanding of LM internals, influencing future design and optimization strategies.", "weaknesses": "1. Limited Generalizability Beyond ORION Tasks: While the study provides valuable insights into modular decomposition within the specific tasks included in ORION, there may be limitations in generalizing these findings to other types of tasks or real-world scenarios that are not represented in the ORION collection.\n\n2. Dependency on Specific Models and Versions: The analysis is based on a specific set of 18 open-source language models with varying sizes, which may not fully represent the behavior of all existing LMs or future iterations of these models. Additionally, focusing on specific versions (e.g., Pythia-2.8b) might introduce limitations related to the unique characteristics or limitations of those versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWvytZtMQr", "reviewer": "Reviewer_ha2n", "summary": "The paper introduces ORION, a structured collection of retrieval tasks aimed at studying how language models (LMs) handle retrieval in various contexts. The research investigates 18 models, applying causal analysis to show that LMs use a modular approach to process retrieval tasks, with middle layers focusing on the request and late layers on retrieving context. The concept of request-patching is introduced to improve model oversight and counteract prompt injections.", "questions": "1. Can the authors discuss potential extensions of ORION to handle multi-attribute or more complex reasoning tasks?\n2. How might the request-patching approach be adapted or extended for use in real-world, multi-step tasks?\n3. Did the analysis include any comparisons between models fine-tuned with instruction-following data and those that are not?\n4. Are there potential limitations or risks in applying request-patching to models with different architectures or training methods?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Innovative Dataset**: ORION serves as a unique, structured dataset collection that enables comparative analysis across multiple models and retrieval scenarios.\n- **Causal Analysis**: The study's use of causal interventions provides valuable insights into model behavior, highlighting a modular structure in how LMs process retrieval.\n- **Proof of Concept**: Demonstrates a practical application for mitigating prompt injections using request-patching, enhancing model robustness in certain cases.\n- **Thorough Evaluation**: The research assesses models across a range of sizes and tasks, reinforcing the applicability of findings.", "weaknesses": "- The methodology focuses heavily on retrieval tasks and may not be immediately applicable to more complex scenarios involving multi-step reasoning or decision-making.\n- Although insightful, the microscopic component analysis indicates that variability in task-solving components remains complex and less well-explained by the macro-level findings.\n- The proof of concept for oversight is limited to retrieval tasks with single attributes, raising questions about scalability to broader, real-world applications.\n- Further exploration of how these findings apply to closed-source models or those with different architectures would strengthen the paper’s generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "toOxyXwIgw", "reviewer": "Reviewer_diV7", "summary": "This paper investigates how language models (LMs) solve retrieval tasks across diverse situations. The authors introduce ORION, a collection of structured retrieval tasks spanning six domains, from text understanding to coding. They apply causal analysis on ORION for 18 open-source language models ranging from 125 million to 70 billion parameters. The study reveals that LMs internally decompose retrieval tasks in a modular way: middle layers at the last token position process the request, while late layers retrieve the correct entity from the context. The authors demonstrate a proof-of-concept application for scalable internal oversight of LMs to mitigate prompt injection while requiring human supervision on only a single input.", "questions": "1. How does the emergent modularity in retrieval task processing relate to the pre-training objectives or architectures of the studied LMs? Are there specific design choices that might promote or hinder the development of this modular structure?\n\n2. Your analysis focuses on single-attribute retrieval tasks. How do you expect the observed mechanisms to generalize to more complex queries involving multiple attributes or nested retrieval operations? Have you conducted any preliminary investigations in this direction?\n\n3. The case study on Pythia-2.8b reveals that individual components behave differently depending on superficial input features. How do you reconcile this micro-level variability with the macro-level modularity observed across tasks and models? What implications does this have for our understanding of emergent behaviors in LMs?\n\n4. Your proof-of-concept for mitigating prompt injection shows promising results. Have you explored how this technique might scale to more diverse types of adversarial inputs or larger language models? What challenges do you anticipate in applying this method to real-world deployment scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of ORION provides a valuable resource for studying LM behavior across a wide range of retrieval tasks. By creating a structured dataset with consistent abstract representations, the authors enable systematic comparisons across different models and task types. This data-centric approach is particularly useful as it allows for scalable analysis of LM behavior.\n\n2. The discovery of a universal emergent mechanism for retrieval tasks across different model sizes and architectures is a decent contribution. The authors' finding that middle layers process the request while late layers perform retrieval suggests a fundamental organizing principle in LMs that was previously unknown. This insight could have far-reaching implications for our understanding of LM capabilities and limitations.\n\n3. The paper demonstrates a compelling link between theoretical understanding and practical application. By leveraging their insights into the internal workings of LMs, the authors develop a proof-of-concept for mitigating prompt injection attacks. This application showcases the potential real-world impact of mechanistic interpretability research.\n\n4. The study's comprehensive scope, covering 18 different models across various scales, adds robustness to its findings. This breadth allows the authors to identify patterns that persist across different architectures and training regimes, strengthening the generalizability of their conclusions.", "weaknesses": "1. While the paper identifies the modular decomposition of retrieval tasks, it lacks a deep exploration of why this decomposition emerges. The authors could strengthen their analysis by investigating how this modularity relates to the training process or architectural choices in LMs. For instance, they could examine whether the observed layerwise specialization correlates with specific attention patterns or activation distributions.\n\n2. The study focuses primarily on retrieval tasks with single-attribute requests. This limitation raises questions about how well the findings generalize to more complex retrieval scenarios involving multiple attributes or nested queries. Expanding the analysis to include such tasks could provide a more comprehensive understanding of LM retrieval capabilities.\n\n3. The paper's case study on Pythia-2.8b reveals inconsistencies between macro-level modularity and micro-level component behavior. This discrepancy is not fully resolved, leaving open questions about how emergent behaviors arise from the interactions of individual components. A more in-depth analysis of this phenomenon could provide valuable insights into the nature of computation in LMs.\n\n4. The authors' proof-of-concept for mitigating prompt injection is limited in scope. The experiment focuses on a narrow range of distractor types and only two model sizes. A more comprehensive evaluation, including a wider variety of adversarial inputs and model architectures, would strengthen the claims about the method's effectiveness and generalizability.\n\n5. The paper does not adequately address the potential limitations of the residual stream patching technique. For instance, it's unclear how sensitive the results are to the choice of patching layers or whether the observed effects could be artifacts of the intervention method itself. A more thorough discussion of these methodological considerations would enhance the robustness of the study's conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)