Venue: NAACL 2025

# Reversed Attention: On The Gradient Descent Of Attention Layers In GPT

 Shahar Katz &Lior Wolf

Blavatnik School of Computer Science, Tel Aviv University

{shaharkatz3@mail1,wolf6cs}.tau.ac.il

###### Abstract

The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention". We visualized Reversed Attention and examine its properties, demonstrating its ability to elucidate the models' behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights, using a novel method called "attention patching". In addition to enhancing the comprehension of how LMs configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass. Our code is available at: [https://github.com/shacharK2/Reversed-Attention](https://github.com/shacharK2/Reversed-Attention).

## 1 Introduction

The widespread use of automatic gradient technologies such as AutoGrad [18, 4] to obtain the gradients may cause the explicit derivations of these gradients to be overlooked. In this work we explore the equations that govern the backpropagation of the popular Transformer-based [50] GPT [7] architecture.

This explicit analysis leads to several surprising discoveries. First, while during the forward pass the attention mechanism explicitly creates triangular attention matrices, from multiplying the query and key vectors, during the backward pass it implicitly creates triangular matrices that determine the gradients of the queries and keys. Due

Figure 1: In this paper we examine the attention maps obtained from the backward pass, which we named “Reversed Attention” (RA). This example present the forward and backward pass of a single attention head of GPT2-xl when prompt with “I like Italy and France, I visited the city of”. After the model answer “Florence”, a city in Italy, we apply a backward pass with “Florence” as the target for the loss and produce the RA maps. Between all the 1200 attention heads this model has, the presented head has the highest RA map’s norm. Compared to the forward attention map, the RA map is more sparse and interpretable. This RA demonstrates how the backpropagation attempts to amplify the information from the token “Italy” (red) while reducing the influence of “France” (blue).



the similarity of these triangular matrices to the forward attention, we refer to them as the Reversed Attention (RA) matrices, Figure 1. Secondly, we study the effect induced by GD and how it tries to increase or decrease attention scores between the queries and keys of each attention head.

Based on these discoveries, we explore the use of RA in LM explainability. While the forward pass attention, which fails in providing a clear explanation of the model's behavior Jain and Wallace (2019), we demonstrate RA's ability to achieve competitive results in concert with methods such as causal mediation Vig et al. (2020); Meng et al. (2022) in perturbation analysis. Furthermore, one can view the RA as a correction term to the attention, given the loss of the backward pass. As an application, we inject the RA scores directly into the forward pass of attention, in order to modify the model's predictions. This novel method, which we call "attention patching", does not involve any parameter updates and offers a new perspective on how interventions can be performed on LMs.

Our main contributions are as follows: (i) We provide a mathematical walk-through and interpretation of the gradients and Vector-Jacobian Products (VJPs) governing the backpropagation of GPT. (ii) We identify the attention-softmax derivative as an implicit attention map, which we term Reversed Attention (RA). (iii) We visualize and qualitatively explore the interpretability of RA. (iv) We conduct a perturbation test to quantify the explainability of RA. (v) We demonstrate a novel patching method that uses RA to edit LM's predictions.

## 2 Related Work

All leading deep learning models are trained using variants of Gradient Descent (GD), an implementation of the backpropagation algorithm. While much research examines the impact of GD on GPTs' performance, the internal computations of this process often remain a black box Radford et al. (2018, 2019); Gururangan et al. (2020). Some studies simplify the Transformer architecture to understand GD, for instance, reducing multi-head attention to a single head or linear attention in toy models Tian et al. (2023); Tarzanagh et al. (2023); Mahankali et al. (2023); Dai et al. (2023). Notably, the literature on the backward pass of full multi-head attention is limited. Our work addresses this gap by examining assumption-free full GPT models, with a specific emphasis on detailing the mathematical computation of gradients.

Previous investigations into weight updates via GD have focused primarily on the MLP layers or the data it was trained on Gueta et al. (2023). Recently, Katz et al. (2024) has revealed that gradients can be interpreted as tokens' embeddings. Specifically, the Vector Jacobian Products (VJPs) that are passed by the residual from the last layer's loss to earlier layers can be seen as an intermediate state toward the model as it adjusts its weights.

Gradients have been leveraged to localize the source of specific model behavior for a given input Simonyan et al. (2014); Ancona et al. (2018) or specific components of the models Barkan et al. (2021); Ma et al. (2023). In our study, we explore the use of RA as a means to quantify the relative influence of the various components of a model. We compare our method with the current leading technique for such localization, Causal Mediation (CM) Vig et al. (2020); Meng et al. (2022). CM involves probing the effect of altering components during the forward pass, which necessitates significantly more computation compared to the RA approach.

Recent works explore the patching technique, where one model's intermediate state is integrated into the forward pass of another Zhang and Nanda (2023); Elhage et al. (2021); Wang et al. (2023); Todd et al. (2023). While they focus on activation patching, we propose a novel approach: directly injecting attention scoring maps of each attention head, without parameter updates.

As far as we can ascertain, we are the first to identify, visualize, and explore the dynamics of the backward pass using RA maps.

## 3 Background

This section provides the necessary background and establishes the notation used.

Generative Pre-trained Transformer (GPT) is an auto-regressive architecture of multiple transformer blocks connected via a residual stream. As input, a GPT model receives a sequence of \(n\) tokens (a prompt) and predicts a single token. An embedding matrix at the start of the model embeds the token into vectors \(X=[x^{1},\cdots,x^{n}]\in\mathbb{R}^{n\times d}\) where \(d\) is an embedding dimension. At the end of the model, the embedded predictions are projected back into tokens using a decoding matrix. Each transformer block consists of two sub-blocks: multi-head attention (Attn) and a Multi-Layer Perceptron (MLP), 

interconnected by a residual stream.

The attention mechanism is executed using matrices \(W_{q},W_{k},W_{v},W_{o}\in\mathbb{R}^{d\times d}\), named query, key, value, and output, respectively. This calculation is performed after splitting the matrices vertically into \(h\) non-overlapping parts, called heads. We denote the attention matrices for the \(l\) head in \(\hat{W}^{l}\), hence \(\hat{W}^{l}_{q},\hat{W}^{l}_{v},\hat{W}^{l\top}_{o}\in\mathbb{R}^{d\times \frac{d}{h}}\), and for example \(W_{q}=[\hat{W}^{1}_{q},\cdots,\hat{W}^{h}_{q}]\). The first three matrices are used to project the input into queries, keys, and values:

\[Q^{l}=X\hat{W}^{l}_{q},~{}K^{l}=X\hat{W}^{l}_{k},~{}V^{l}=X\hat{ W}^{l}_{v}\in\mathbb{R}^{d\times\frac{d}{h}} \tag{1}\]

Together, the queries and keys are used to calculate the forward attention scores:

\[A^{l}=\text{softmax}\left(\frac{Q^{l}K^{l\top}}{\sqrt{d/h}}+M \right)\in\mathbb{R}^{n\times n} \tag{2}\] \[M_{ij}=\begin{cases}0&\text{if $i\geq j$}\\ -\infty&\text{otherwise}\end{cases} \tag{3}\]

Where \(M\in\mathbb{R}^{n\times n}\) is a masking matrix that zeroes all scores but the one representing the connection of an earlier token to further ones: . The output of each head is calculated by multiplying the attention scores with the values and projecting the result back using the output matrix. The attention block output is the sum of the output of all heads: Attn\((X)=\sum_{l=1}^{h}\text{head}^{l}\), where head\({}^{l}=A^{l}V^{l}\hat{W}^{l}_{o}\) .

The MLP block is a pair of fully connected matrices \(FF_{1}\), \(FF_{2}^{\top}\in\mathbb{R}^{d\times d_{m}}\) and an activation function \(f\). The output of this block is: MLP\((X)=f(XFF_{1})FF_{2}^{T}\). Lastly, the forward pass of the \(i\)-th transformer block on its input hidden state, \(X^{i}\), is: \(X^{i+1}=X^{i}+\text{Attn}(X^{i})+\text{MLP}(\text{Attn}(X^{i})+X^{i})\).

Note that GPT models also include Layer Norms. For simplicity and due to the relatively low contribution to the gradients and inconsistency when they are placed within different architectures, we omit them from this explanation.

**Gradient decent, backward pass and VJPs** GD's backward pass is the execution of Backpropagation Le Cun (1988), the process of applying the chain rule to compute a model's gradients. A backward pass is initiated after the model executes a forward pass; it computes a loss score \(L\), comparing the model's output with a desired target. This loss score is propagated back through the model's layers as an error signal, in the reverse order of the forward pass. The error signal can be represented as a vector that is used as an intermediate state of the backward pass, similar to the hidden state in the forward pass. Given a model's parameter \(W\), which is used to compute \(z=xW\), where \(x\in\mathbb{R}^{d_{1}},z\in\mathbb{R}^{d_{2}}\), the error signal is the loss with respect to the layer's output, \(\delta=\frac{\partial L}{\partial z}\in\mathbb{R}^{d_{2}}\). This vector is known as the Vector-Jacobian Product (VJP) of \(z\). At the last layer of the model, the VJP is calculated directly by the loss function. For earlier layers, the VJP is calculated using the backward step, where the VJP of the next layer is used to compute the VJP of those that precede it. For instance, the output of a sequential layer \(l\) is the input of \(l+1\), meaning \(z^{l}=x^{l+1}\). Given those layers are weight matrices, the VJP of the \(l\) layer is computed by the following step:

\[\delta^{l}=\frac{\partial L}{\partial z^{l}}=\frac{\partial L}{ \partial x^{l+1}}=\delta^{l+1}(W^{l+1})^{\top} \tag{4}\]

Finally, the gradient of each weight matrix \(W\) is the outer product of the layer's input \(x\) and the VJP \(\delta\), which updates the weights using a learning rate \(\eta\in\mathbb{R}\):

\[\frac{\partial L}{\partial W}=\frac{\partial z}{\partial W}\frac {\partial L}{\partial z}=x^{\top}\times\delta\in\mathbb{R}^{d_{2}\times d_{1}} \tag{5}\] \[W\gets W-\eta\frac{\partial L}{\partial W}^{\top}\in \mathbb{R}^{d_{1}\times d_{2}} \tag{6}\]

In models such as GPTs, each forward pass includes a sequence of inputs \(X=x^{1},\cdots,x^{n}\in\mathbb{R}^{n\times d_{1}}\). In this case, each input has its own VJP, \(\delta^{i}\in\mathbb{R}^{d_{2}}\), and the full matrix's gradient is the sum of the outer products of each input and its VJP:

\[\frac{\partial L}{\partial W}=\sum_{i=1}^{n}{x^{i}}^{\top}\times \delta^{i}\in\mathbb{R}^{d_{2}\times d_{1}} \tag{7}\]

## 4 Attention Layers Gradients

In this section, we examine the VJPs and gradient matrices for each of the attention layer matrices. The purpose of this examination is to reveal properties of the gradient matrices, as well as to fill a missing gap in the literature. In this mathematical walk-through, we examine the submatrices of each attention head, \(\hat{W}^{l}_{q},\hat{W}^{l}_{k},\hat{W}^{l}_{v},\hat{W}^{l}_{o}\), dropping the head index \(l\) for brevity. The gradients of the full matrices are the concatenation of the heads' gradients (same as the concatenation of weight matrices) 

Throughout this section, we denote the forward pass input and output vectors of each submatrix \(\hat{W}\in\mathbb{R}^{d\times\frac{d}{h}}\) by \(x\in\mathbb{R}^{d}\) and \(z\in\mathbb{R}_{s}^{\frac{d}{h}}\), i.e., \(z=xN\). The only exception is \(W_{o}\), where the dimensions are swapped, \(\hat{W}_{o}\in\mathbb{R}^{\frac{d}{h}\times d}\). \(A_{i,j}\) denotes the forward pass attention score from the \(i\)-th token to the \(j\)-th token. Lastly, considering the model's input as a sequence of \(n\) tokens, instead of explicitly writing the gradient of every matrix and token, we will focus on determining the VJP for a single token \(j\in\{1,\cdots,n\}\), annotating its input and VJP with \(x^{j},\delta^{j}\). Given the inputs and VIPs, the gradients are the outer product of the two Equation 7.

The output projection matrix \(\hat{W}_{o}\)The VIPs of \(\hat{W}_{o}\), denoted as \(\delta^{j}_{o}\), are obtained directly from the residual stream at the end of the attention block of \(\hat{W}_{o}\), denoted by Attn(\(x\)) in section 3. Viewing GD as an application of the chain rule, we consider \(\delta^{j}_{o}\) as the layer's intermediate editing target.

Only for this matrix, we will demonstrate that when we are given its VIPs, we can infer its gradient and the effect of GD updating. This explanation holds for all further matrices, too. The gradient updates introduced by only the \(j\)-th token (left equation) or by all \(n\) tokens (right equation) are:

\[\hat{W}_{o}\leftarrow\hat{W}_{o}-\eta\delta^{j}_{o}\times x^{j \top}_{o} \tag{8}\] \[\hat{W}_{o}\leftarrow\hat{W}_{o}-\eta\sum_{i=1}^{n}\delta^{i}_{ o}\times x^{i\top}_{o}, \tag{9}\]

where \(\eta\) is the update's learning rate. In general, if we simplify the full update and consider only the change introduced by a single token (left equation) to examine a future forward pass with the same input \(x^{j}_{o}\), then the original layer's output \(z^{j}_{o}=x^{j}_{o}\hat{W}_{o}\), is shifted by the direction of \(\delta^{j}_{o}\) with the magnitude of \(-\eta\|x^{j}_{0}\|_{2}^{2}\in\mathbb{R}\):

\[x^{j}_{o}\hat{W}_{GGD}=x^{j}_{o}(\hat{W}_{o}-\eta\delta^{j}_{o} \times x^{j\top}_{o})= \tag{10}\] \[z^{j}_{o}-\eta\|x^{j}_{o}\|_{2}^{2}\delta^{j}_{o} \tag{11}\]

The value projection matrix \(\hat{W}_{v}\)In the forward pass, the value vector associated with the \(j\)-th token is given by \(v^{j}=x^{j}_{v}\hat{W}_{v}\in\mathbb{R}^{\frac{d}{h}}\). To calculate the VJP for the \(j\) token, we consider, due to causality, the subsequent tokens \(l\geq j\). For every such token, the backward pass propagates its own VIP from \(\hat{W}_{o}\) by computing the error signal from the \(l\)-th token to token \(j\):

\[e^{l}=\delta^{l}_{o}\hat{W}^{\top}_{o}\in\mathbb{R}^{\frac{d}{h}}\,, \tag{12}\]

where \(\delta^{l}_{o}\) is the VJP of \(\hat{W}_{o}\) computed for the \(l\)-th forward pass (the \(l\)-th token) of the autoregressive process. The VJP of \(\delta^{j}_{v}\) is a weighted sum of the error signal and the forward attention scores \(A\):

\[\delta^{j}_{v}=\sum_{l=j}^{n}A_{l,j}e^{l}\in\mathbb{R}^{\frac{d}{ h}} \tag{13}\]

Softmax derivativeAlthough the softmax function is not a weight parameter of the model, comprehending its role in the backward pass is crucial for understanding the editing dynamics. In the forward pass, the attention scores for the \(j\)-th forward pass, \(A_{j}\in\mathbb{R}^{1\times n}\) (the \(j\)-th row of the forward attention) are generated by applying the softmax function to the product of the query and key vectors, with a scaling factor of \(\sqrt{\frac{d}{h}}\). These scores are then multiplied by the attention values, \(V\in\mathbb{R}^{n\times\frac{d}{h}}\). The backward pass performs a reverse operation: each token's error signal from \(\hat{W}_{o}\) is multiplied by the attention values and scaled as follows:

\[\tilde{e}^{j} =\delta^{j}_{o}\hat{W}^{\top}_{o}V^{\top}=e^{j}V\in\mathbb{R}^{n} \tag{14}\] \[r^{j} =A_{j}\odot(\tilde{e}^{j}-\tilde{e}^{j}A_{j}\cdot\mathbf{1}^{n}) \sqrt{\frac{h}{d}}\in\mathbb{R}^{n}, \tag{15}\]

where \(\odot\) is the element-wise product of two vectors (Hadamard product) and \(\tilde{e}^{j}A_{j}\cdot\mathbf{1}^{n}\) is the scalar \(\tilde{e}^{j}A_{j}\) assigned to an \(\mathbb{R}^{n}\) vector.

**Rewriting as a batch of all tokens:** If we rewrite Equation 14 for all tokens together, by concatenating their VJP into a matrix \(\Delta=[\delta^{1}_{o},\cdots,\delta^{n}_{o}]\in\mathbb{R}^{n\times d}\) we get:

\[\tilde{E} =\Delta\hat{W}^{\top}_{o}V^{\top}\in\mathbb{R}^{n\times n} \tag{16}\] \[R =A\odot\Big{(}\tilde{E}^{\top}-\textit{diag}(A\tilde{E}^{\top}) \Big{)}^{\top}\sqrt{\frac{h}{d}}\in\mathbb{R}^{n\times n}, \tag{17}\]

hence \(R_{j}=r^{j},\tilde{E}_{j}=\tilde{e}^{j}\), where \({}_{j}\) is the \(j\)-th row of each matrix. We further use the derivative \(R\) to compute the subsequent gradients of \(\hat{W}_{q},\hat{W}_{k}\). In section 5 we investigate what \(R\) represents.

The query projection matrix \(\hat{W}_{q}\)During the \(j\)-th forward pass, the \(j\)-th query is generated by computing \(q^{j}=x^{j}_{q}\hat{W}_{q}\in\mathbb{R}^{\frac{d}{h}}\). The query is then multiplied by all key vectors, \(K=[k^{1},\cdots,k^{n}]\in\mathbb{R}^{n\times\frac{d}{h}}\) (and masking the ones that followed it), to obtain the raw attention scores (logits). Therefore, the backward pass calculates the VJP of the query by calculating:

\[\delta^{j}_{q}=r^{j}K=R_{j}K\in\mathbb{R}^{\frac{d}{h}}\] (18) 

The key projection matrix \(\hat{W}_{k}\)The gradients of \(\hat{W}_{k}\) are computed similarly to those of \(\hat{W}_{q}\), except that for the \(j\)-th forward pass, we utilize the queries \(Q=[q^{1},\cdots,q^{n}]\in\mathbb{R}^{n\times\frac{d}{h}}\) from each subsequent forward pass after \(j\). The VJP and gradients of \(\hat{W}_{k}\) are then given by:

\[\delta_{k}^{j}=\sum_{l=j}^{n}r_{j}^{l}x_{q}^{l}\hat{W}_{q}=\sum_{l=j}^{n}r_{j}^ {l}q^{l}=R_{j}^{\top}Q\in\mathbb{R}^{\frac{d}{h}}, \tag{19}\]

where \(R_{j}^{\top}\) is the \(j\)-th column in \(R\), which is the error signal from the \(l\)-forward pass to the \(j\)-th one.

## 5 Reversed Attention

In Equation 16 we defined \(R\) as the softmax derivative. \(R\) shares many properties with \(A\), the forward attention (FA):

* \(R\) is computed from the multiplication of the forward pass values \(V\) and the error signal \(\Delta\hat{W}_{o}\). We denote this intermediate result as \(\tilde{E}\). The \(i,j\) entry in \(\tilde{E}\) is a score between the \(i\)-th error signal and the \(j\)-th forward pass value. This resembles the calculation of raw attention scores (logits, before softmax) where we multiply queries and keys to obtain scores between every pair of tokens.
* After computing \(\tilde{E}\), we derive \(R\) from it by row-wise normalization, which involves scaling with the FA \(A\), see Equation 14. In the forward pass, row-wise normalization is accomplished by applying the softmax function.
* Since the normalization from \(\tilde{E}\) to \(R\) includes element-wise multiplication with the FA \(A\), which is a lower triangular matrix, \(R\) is also a lower triangular matrix.
* Just as the FA is used to multiply the attention values \(V\), we use \(R\) to multiply the forward pass' queries \(Q\) and keys \(K\) to obtain the VIPs of one another.

Figure 3: RA model editing dynamics. (a) The query matrix \(\hat{W}_{q}\) will be updated with a VIP directed towards the forward pass key of “tomato”, while the key matrix \(\hat{W}_{k}\) will be updated with a VIP directed towards the query from the token “of”. (b) The latent space of the queries and keys. The circles represent a forward pass query and a key. If their Reversed Attention score is a relatively low negative number, the directions they are moving towards after GD are actually towards one another.

Figure 2: (a) The norms of the attention maps per head and per layer. (b) Forward and (c) Reversed Attention of the same head from GPT2-small (layer 11, head index 2). This is the attention head with the second highest Reversed Attention norm and we can see it focused on editing the query of “of” (row) and the key of “tomato” (column).

 

These shared properties between the FA and the softmax derivative suggest that the softmax derivative serves as an implicit attention matrix. We called \(R\) the "Reversed Attention" (RA). In the following sections, we delve into some properties of the RA and explore its potential uses in explaining and controlling GPTs.

### A qualitative examination of Reversed Attention

We observe the behavior of RA through a single example, using GPT2-small Radford et al. (2019). We prompt the model with the sentence "Cherry tomato is a type of', and using "tomato" as an editing target. Since this prompt already contains the token answer, the behavior of the model can be readily understood.

Figure 2(a) depicts, for each attention head and each layer, the norm of the FA and the RA. While RA displays a very sparse pattern, FA (which is normalized by the softmax) displays a larger number of heads with high values. There does not seem to be a correlation between the two maps, and this is further supported by the perturbation experiments in subsection 5.2.

Our mathematical analysis in Equation 5, section 4 implies that close to zero scores in the RA will produce close to zero VJP vectors and gradients, hence focusing on the attention heads with the highest RA norm is informative enough to update the GD steps.

Next, we consider one of the FA maps with the highest RA norm, which is provided in Figure 2(b). The key of the first token, "n", receives the highest attention scores, which is a well-known phenomenon Xiao et al. (2023).

On the other hand, the RA in Figure 2(c) is sparse and shows that the row corresponding to the last token "of" stands out as much more dominant than the others. We recall that the VJPs of the query matrix \(\hat{W}_{q}\) are the multiplication of the RA with the forward pass keys \(K\). Similarly, the VJPs of the keys matrix \(\hat{W}_{k}\) are the multiplication of the RA with the forward pass queries \(Q\). Since GD updates are performed with a negative learning rate, positive scores in the RA shift the model's weights towards the queries/keys that produced them (and vice versa). Hence, the VJP of the query for "of" is mostly directed towards the key of "tomato" while directed away from the key that belongs to "u". Similarly, the main update is to the key of "tomato", which attempts to shift the model's weights towards the query for "of". This dynamic is illustrated in Figure 3. This example does not necessarily elucidate the function served by this attention head, but it demonstrates how GD attempts to repurpose this head to recall information from the token "tomato".

This example demonstrates the potential of Reversed Attention to provide insights into the model's behavior and editing dynamics. Additional examples are provided in Appendix A.

### Reversed Attention and the importance of each attention head

In subsection 5.1, we demonstrate how we can interpret the effect of RA on the editing dynamics of the model's parameters. This explanation is based on the assumption that high RA scores correspond to important components (model parameters) in the forward computation graphs. Identifying these key parameters is one of the objectives of mechanistic interpretability research Sharkey et al. (2025), which seeks to uncover meaningful sub-networks by selectively pruning model parameters. This approach differs from methods that prune models" inputs to assess the effect of individual tokens while leaving the internal mechanisms intact.

To verify RA's ability to identify the importance of parameters in attention layers, we conduct a perturbation test. This test compares different orders (rankings) of the attention heads, each produced by a method that aims to determine the relative importance between heads. This experiment begins by zeroing out (masking) all heads, resulting in poor performance. Gradually we unmask the heads according to a given order. The performances can be quantified by the Area Under the Curve (AUC) of the graph that displays the model's accuracy as a function of the percentage of unmasked heads.

For RA, we rank the heads according to the RA norms (from highest to lowest). Additionally, we use a training set (omitted examples) to perform separate backward passes and average the norms of each head. The main method we compare to is Causal Mediation (CM), due to its extensive usage in interpreting LM Meng et al. (2022); Mueller et al. (2024). This method examines causal indirect effects by patches the forward pass attention heads' outputs and examines the disparity between the altered and the original model's outputs. In addition, we include the forward pass attention (FA).

 

Similar to RA, FA ranks attention heads according to the the norm of the attention maps. While previous works already established that FA is not sufficient in providing similar models' explanation (Serrano and Smith, 2019; Jain and Wallace, 2019), we include FA for completeness.

Our tests are conducted on 21 tasks by Hernandez et al. (2024); Todd et al. (2023), each consisting of pairs of sentences that follow some relation. For instance, one task involves pairs of countries and their capital cities. In 6 of these tasks, the model is prompted with natural language templates such as "_The capital city of <country name> is_". In the other 15 tasks, in-context-learning (ICL) templates are employed. In this scenario, instead of explicitly stating the relation between the pair (e.g., "the capital city is"), the model must infer it from \(n\)-shots of labeled pairs, such as "an Q: Spain A: Madrid u Q: Italy A: Rome u <country name> A:".

The results are shown for the ICL tasks and the natural questions, respectively, in Table 1 for GPT2-xl and Table 2 for both GPT2-xl and Llama2-7B (Touvron et al., 2023). To illustrate how the AUC scores are calculated, Figure 4 displays graphs for a single task that measures GPT2-xl's accuracy as a function of the percentage of pruned attention heads. The full implementation details and results, including ones for GPT-j (Wang and Komatsuzaki, 2021) and OPT (Zhang et al., 2022), can be found in Appendix B.

As can be seen, RA is competitive with CM when the examined LM can successfully address the task. In the natural language tasks, we observed that RA outperforms CM with larger models that originally had high accuracy. In the ICL tasks, we found that CM achieves good results with a very low number of shots when the LM fails to provide the correct answer. However, when we prompt these failed tasks with a few shots, we see that RA achieves better results. Overallly, even when RA falls behind CM, it still achieves non-trivial results. Hence, we find RA to reflect the importance of each attention head in producing a given prediction. We note that RA is much faster, as it only requires a forward and a backward pass for each example, while CM requires a forward pass for each head.

In conclusion, these results demonstrate that RA produces more interpretable attention maps compared to FA. The comparison with CM shows that RA can achieve comparable results on specific

Figure 4: The accuracy results of GPT2-xl, as a function of the amount of pruned attention heads, on the antonym task with 5-shot ICL.

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{GPT2-xl} & \multicolumn{4}{c}{LIama2-7B} \\ \cline{3-10} Task & Example & Rand & CM & FA & RA & Rand & CM & FA & RA \\ \hline country-capital & The capital city of Sierra Leone is [Freetown] & 0.02 & **0.61** & 0.06 & 0.07 & 0.31 & 0.05 & 0.07 & **0.43** \\ person-plays-pro-sort & Scottie Pippen plays the sport of [basketball] & 0.32 & **0.59** & 0.09 & 0.57 & 0.21 & 0.39 & 0.04 & **0.48** \\ product-by-company & Blogger was created by [Google] & 0.19 & **0.46** & 0.08 & 0.31 & 0.15 & **0.51** & 0.06 & 0.31 \\ \hline \hline \end{tabular}
\end{table}
Table 2: GPT2-xl and Llama2-7B perturbation tests on natural questions, measuring the AUC for Reversed Attention (RA), Forward Attention (FA), Causal Mediation (CM) and random ordering.

 

tasks and models while serving as a complementary method to CM, given RA's ease of visualization and lower computational requirements.

### Attention patching using Reversed Attention

Modifying the forward attention maps of LM to improve them, while doing so in an explainable manner, is one of the goals of interpretability research. After demonstrating that RA produces more interpretable attention maps compared to the forward attention subsection 5.2, this section shows how RA can be used to modify attention maps and alter the predictions of the models.

Since LMs can have thousands of attention heads, each depicting different relationships, a solution that directly edits the attention maps has not been studied extensively. Recent efforts in interpretability research have explored activation patching, demonstrating how injecting different hidden states from one model into another affects its performance. In this section, we explore both the idea of directly modifying attention maps and activation patching through a novel method termed "attention patching". This approach is based on the observation that RA produces attention maps that can be seen as the desired relationships the model attempts to maintain in order to perform a given task.

Given a GPT model and a predefined set of training and test examples, all with the same length and format, attention patching performs the following steps: (i) calculate the RA of each training example (applying forward and backward passes with each example but without modifying the model's parameters). (ii) average the RA scores for each attention head. (iii) For each test example and for each attention head, we add (inject) the RA map to the forward pass attention map, using a learning rate as a scaling factor, Figure 5 illustrates this process. Additionally, to establish a baseline, we applied the same process with the forward attention, averaging and injecting the attention maps collected from forward passes.

The requirement for examples to have the same length and format simplifies the injection process, ensuring that all attention score matrices are of consistent size. This makes tasks such as ICL and short trivia-like questions ideal candidates for this method, as they are templated and easily framed by their length. Additionally, the ability of LMs to answer ICL tasks is usually associated with the attention layer Dong et al. (2022), which serves as another reason to examine attention patching in this context.

We evaluate attention patching on the datasets of Todd et al. (2023); Hernandez et al. (2024), comparing its performance to ICL prompting. The results are displayed in Table 3, with additional details provided in Appendix C. Our findings indicate that attention patching achieves similar results to ICL prompting and outperforms the average of the forward pass attention scores, which does not consistently improve the original model performances. Note that similarly to other patching methods, this test is not meant to demonstrate the robustness of the approach, as the current implementation is frame-specific and full GD outperforms it. Instead, it serves to validate that Reversed Attention indeed reflects the model's desired attention.

## 6 Conclusions

The self-attention component of transformers is perhaps the most distinctive part of this architecture. Its role when performing inference has been extensively studied and shown to provide insights into the inner workings of transformers. Here, we explore a dual entity we call Reversed Attention (RA), which plays a role when transformers learn. We present qualitative samples of the way learning occurs and show that RA can help identify the most influential heads at inference time. Finally,

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{GPT2-xl} & \multicolumn{3}{c}{OPT-1.3B} \\ \cline{3-7} Task & N & original & FA & RA & original & FA & RA \\ \hline \multirow{8}{*}{antonym} & 0 & 0.00 & 0.01 & **0.08** & 0.02 & 0.01 & **0.24** \\  & 1 & 0.18 & 0.43 & **0.56** & 0.20 & 0.26 & **0.57** \\  & 5 & 0.53 & 0.57 & **0.62** & 0.42 & 0.44 & **0.59** \\  & 10 & 0.57 & 0.57 & **0.62** & 0.42 & 0.43 & **0.54** \\ \hline \multirow{8}{*}{capitalize} & 0 & 0.00 & 0.00 & **0.94** & 0.01 & 0.00 & **0.78** \\  & 1 & 0.44 & 0.50 & **1.00** & 0.01 & 0.01 & **0.90** \\  & 5 & 0.98 & **1.00** & **1.00** & **1.00** & 0.99 & **1.00** \\  & 10 & 0.99 & **1.00** & **1.00** & **1.00** & 0.99 & **1.00** \\ \hline \multirow{8}{*}{choose-middle-of-3} & 0 & 0.46 & 0.30 & **1.00** & 0.11 & 0.03 & **1.00** \\  & 1 & 0.81 & 0.76 & **1.00** & 0.57 & 0.54 & **0.76** \\  & 5 & 0.92 & 0.68 & **1.00** & 0.95 & 0.84 & **1.00** \\  & 10 & 0.97 & 0.00 & **1.00** & 0.86 & 0.65 & **1.00** \\ \hline \multirow{8}{*}{next-item} & 0 & 0.03 & 0.00 & **0.16** & 0.09 & 0.00 & **0.53** \\  & 1 & 0.28 & 0.66 & **0.72** & 0.50 & 0.47 & **0.84** \\ \cline{1-1}  & 5 & 0.69 & 0.84 & **0.88** & 0.69 & 0.66 & **0.88** \\ \cline{1-1}  & 10 & 0.88 & 0.88 & **0.91** & 0.75 & 0.81 & **0.84** \\ \hline \hline \end{tabular}
\end{table}
Table 3: GPT2-xl and OPT-1.3B accuracy on ICL tasks of the original models and with forward attention (FA) and Reversed Attention (RA) patching. N = the number of ICL samples.

 

we show how plugging an average RA value can direct the model toward performing a specific task.

The focus of this work was to introduce RA and prove that it is interpretable rather than merely an artifact of the backward pass. Beyond enriching our understanding of how LMs work, we hope our conclusions will serve future research on dynamically editing attention layers as well as explaining how information is stored in them.

## 7 Limitations

In this work, we provide a detailed mathematical exposition of the derivative of the attention mechanism in GPTs. To keep this explanation clear, we focus on decoder-only models with Multi-Head Attention mechanisms and without additional components such as RoPE Su et al. (2024) or sparse attention Brown et al. (2020). LLMs come in various sizes and configurations, and a general mathematical explanation that fits all is not feasible. Our choice to demonstrate RA using GPT2 and OPT aligns with previous work that examines the interpretability of Transformers through the lens of these models Geva et al. (2022); Meng et al. (2022); Voita et al. (2023); Katz and Belinkov (2023). Our use of Llama2 and GPT-j, although they use RoPE, comes to show how our use of RA can be applied to a wide variety of LM.

The 21 tasks we sourced from Todd et al. (2023); Hernandez et al. (2024) consist of relatively simple and limited tasks. Therefore, our results in subsection 5.2 and subsection 5.3 serve as a proof of concept rather than as a definitive assessment of RA's robustness in its ability to identify critical components of models or to be used for editing using patching.

Causal Mediation (CM) can come in variety of implementations, each patch the activation differently. In 5.2 we examined a basic implementation of CM. In Appendix B we include additional implementation provided by Todd et al. (2023) to make our comparison to RA comprehensive.

The perturbation experiments compare RA with a small number of alternative methods. We acknowledge that other methods might achieve similar or even better results, particularly those based on gradients. The purpose of these experiments is not to discover a new component-localization method but rather to provide a proof that RA maps correspond to existing LM explainability methods. Therefore, we focused on comparing RA to CM, a widely used method in similar works.

### Ethics Statement

This paper aims to advance our understanding of how language models learn and the dynamics behind the backward pass. Future work might leverage our findings to edit or train language models effectively. However, we are concerned about the potential dangers associated with manipulating these systems. For instance, editing LMs could amplify existing biases or be exploited for unethical purposes. Our primary commitment is to advance research that prioritizes safety and fairness. We hope that future studies will use our findings to further contribute to the creation of better and more aligned models, rather than facilitating the production of harmful content or exploiting the knowledge stored within these models.

## Acknowledgements

This work was supported by the Tel Aviv University Center for AI and Data Science (TAD). The contribution of the first author is part of a Ph.D. thesis research conducted at Tel Aviv University.

## References

* Ancona et al. (2018) Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for deep neural networks. In _6th International Conference

Figure 5: Attention patching using Reversed Attention (RA): first we collect the RA maps of the model without applying any model editing (without changing its weights). Later, for each attention head, we add its corresponding RA map to the forward pass attention.

 

on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30-May 3, 2018, Conference Track Proceedings_. OpenReview. net.
* Barkan et al. (2021) Oren Barkan, Edan Hauou, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. 2021. Grad-sam: Explaining transformers via gradient self-attention maps. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 2882-2887.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Dai et al. (2023) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
* Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. _arXiv preprint arXiv:2301.00234_.
* Elhage et al. (2021) N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. 2021. A mathematical framework for transformer circuits.
* Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 30-45.
* Gueta et al. (2023) Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023. Knowledge is a region in weight space for fine-tuned language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 1350-1370, Singapore. Association for Computational Linguistics.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayambtta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8342-8360.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _International Conference on Learning Representations_.
* Hernandez et al. (2024) Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of relation decoding in transformer language models. In _Proceedings of the 2024 International Conference on Learning Representations_.
* Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3543-3556.
* Katz and Belinkov (2023) Shahar Katz and Yonatan Belinkov. 2023. Visit: Visualizing and interpreting the semantic information flow of transformers. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 14094-14113.
* Katz et al. (2024) Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024. Backward lens: Projecting language model gradients into the vocabulary space. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 2390-2422, Miami, Florida, USA. Association for Computational Linguistics.
* Kim and Schuster (2023) Najoung Kim and Sebastian Schuster. 2023. Entity tracking in language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3835-3855.
* Lample et al. (2018) Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2018. Word translation without parallel data. In _International Conference on Learning Representations_.
* Le Cun (1988) Y Le Cun. 1988. A theoretical framework for backpropagation. In _Proceedings of the 1988 Connectionist Models Summer School_.
* Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. _Advances in neural information processing systems_, 36:21702-21720.
* Maclaurin (2016) Douglas Maclaurin. 2016. _Modeling, inference and optimization with composable differentiable procedures_. Ph.D. thesis.
* Mahankali et al. (2023) Arvind Mahankali, Tatsumori B Hashimoto, and Tengyu Ma. 2023. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. _arXiv preprint arXiv:2307.03576_.
* Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. _Advances in Neural Information Processing Systems_, 36.
* Mueller et al. (2024) Aaron Mueller, Jamnik Brinkmann, Millicent Li, Samuel Marks, Koyama Pal, Nikhil Priakash, Can Rager, Aruna Sankararanayanan, Arnab Sen Sharma, Jiuding Sun, et al. 2024. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. _arXiv preprint arXiv:2408.01416_ 

Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Distinguishing antonyms and synonyms in a pattern-based neural network. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. _Advances in neural information processing systems_, 32.
* Prakash et al. (2024) Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. 2024. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In _The Twelfth International Conference on Learning Representations_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog.
* Serrano and Smith (2019) Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2931-2951.
* Sharkey et al. (2025) Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimesheim, Alejandro Ortega, Joseph Bloom, et al. 2025. Open problems in mechanistic interpretability. _arXiv preprint arXiv:2501.16496_.
* Simonyan et al. (2014) K Simonyan, A Vedaldi, and A Zisserman. 2014. Deep inside convolutional networks: visualising image classification models and saliency maps. In _Proceedings of the International Conference on Learning Representations (ICLR)_. ICLR.
* Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. _Neurocomputing_, 568:127063.
* Tarzanagh et al. (2023) Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. 2023. Transformers as support vector machines. _arXiv preprint arXiv:2308.16898_.
* Tian et al. (2023) Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. 2023. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. _arXiv preprint arXiv:2305.16380_.
* Todd et al. (2023) Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function vectors in large language models. _arXiv preprint arXiv:2310.15213_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhoisale, et al. 2023. Alama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shiieber. 2020. Investigating gender bias in language models using causal mediation analysis. _Advances in Neural Information Processing Systems_, 33:12388-12401.
* Voita et al. (2023) Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2023. Neurons in large language models: Dead, n-gram, positional. _arXiv preprint arXiv:2309.04827_.
* Wang and Komatsuzaki (2021) Ben Wang and Arran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingofloliz/mesh-transformer-jax](https://github.com/kingofloliz/mesh-transformer-jax).
* Wong et al. (2023) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In _The Eleventh International Conference on Learning Representations_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Xiao et al. (2023) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. In _The Twelfth International Conference on Learning Representations_.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4791-4800.
* Zhang and Nanda (2023) Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. _arXiv preprint arXiv:2309.16042_.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_ 



## Appendix A Additional Reversed Attention Examples

In this section, we extend the qualitative examination of the Reversed Attention (RA) discussed in subsection 5.1. We provide examples for GPT2-small, GPT2-xl and OPT-350m. RA offers simple-to-read insights into how the attention matrices are modified by GD, revealing which attention queries \(Q\) and keys \(K\) are adjusted to bring specific tokens closer in the embedding space. However, while this explanation sheds light on the changes made by GD, it does not necessarily elucidate the function served by each attention head. In this section, we aim to establish how we interpret (read) the patterns observed in RA maps. Further investigation into the functional roles of individual attention heads is left for future work.

Gpt2-smallThe qualitative example in subsection 5.1 presents GPT2-small RA while using the prompt "The cherry tomato is a type of" with the editing target "tomato". The pattern given by one of the highest RA norms shows that GD focuses on editing the query belonging to the token "of" and the key of the token "tomato". The following example examines what happens if we change the editing target to "cherry", which, like "tomato", could refer to the prompt itself. Similarly to the previous example, this head is still one of the heads with highest RA norms, but unlike the previous example, GD focuses on editing the key of the token "cherry", as presented in Figure 6. This pattern, identifies the token most similar to the editing target and comes to illustrate RA's ability to localize relevant information in the input tokens regarding the editing target ("tomato" or "cherry"), through the attention heads.

Gpt2-xlWe prompt the model with the following sentence: "I like Italy and France. I visited the city of", expecting the model to complete it with a city from either Italy or France. In the case of GPT2-xl, it returns "Florence". We extract RA when given "Paris" as the editing target and examine the top by norm RA heads, head 8 at layer 30. The forward attention of this head assigns relatively high attention to "Italy" and "France".

Note that this is the same example from Figure 1. Thus, these examples also examined how different targets affect the produced RA maps.

Looking at the RA, the last row, corresponding to the last ("of") token's \(W_{q}\) VJP, we see it assigns a positive score to "Italy" and a negative score to "France". According to section 4, subsection 5.1, this pattern suggests that GD tries to bridge the query of the last token with the key of "France", while doing the opposite with "Italy". Hence, if we isolate the possible outcomes of editing other heads and only editing this one, its output would be more correlated with the attention value from the token "Paris". We assume this behavior arises from GD's attempt to enhance connection to information that is more relevant to the editing target, "Paris".

We repeat the same experiment with the editing target "Rome". This time, the RA of the same head assigns a negative score to the token that follows "Paris" and a positive score to the token "Rome". Using the same analogy, given the target that is more related to Italy, the model tries to amplify the connection between the last token's query and the

Figure 6: The forward and Reversed Attention (RA) maps of an attention head from GPT2-small (layer 11, head index 2), given the editing target “cherry” with the prompt “Cherry tomato is a type of ”. The pattern presented by the RA map attempts to increase the forward pass attention between the query belonging to “of” and the key of “Cherry”, encouraging the model to answer “cherry”.



key of "Italy". This suggests that the RA method dynamically adjusts attention scores to strengthen the associations relevant to the given editing target.

OptWe use OPT-350m with the prompt "I like Italy and France. I visited the city of". This model responds with "Rome". We keep the target token "Paris" and compute the loss to obtain the RA maps, presented in Figure 9.

This model has 384 attention heads. We sort the heads according to their RA maps' norms and present the one with the highest norm in Figure 10, revealing a pattern similar to that of GPT2-xl in Figure 8. To emphasize that heads with low RA norm are barely updated, we maintain the same coloring scale from the head with the highest norm and present the 11-th highest by norm head. This head's RA map appears empty, indicating all scores are close to zero. It is also evident that the forward pass attention map of this head only attends to the first token of the sentence, a phenomenon known as "attention sink" Xiao et al. (2023). This seems to be a default pattern when the function that the head describes is not activated.

### Additional Prompts

We present selected examples in which we used RA to better understand the editing dynamics of prompts from previous works in the area of mechanistic interpretability. The following two examples offer qualitative hypotheses on how backpropagation attempts to correct specific attention heads.

Indirect Object IdentificationWang et al. (2023) identify circuits, subgraphs of a model's parameters, in GPT2-small by examining its response to the prompt "When Mary and John went to the store, John gave a drink to", which the model is expected to complete with "Mary". They pinpoint key attention heads that contribute to the correct response and classify the roles of each head in that process. For instance, they distinguish between heads that support the correct prediction and those that decrease the final probability of predicting "Mary".

We use the same prompt with GPT2-small, using "Mary" as the target token to create RA maps. The top three attention heads, ranked by the norm of their RA, are presented in Figure Figure 11. The first head, head 2 in layer 11, has the highest RA norm and produces a negative RA score for "Mary", suggesting that backpropagation is working to adjust this head to allocate more attention to that token, as its forward pass barely attends to "Mary".

In contrast, the other two heads, head 7 in layer 10 and head 10 in layer 11, exhibit a positive RA score for "Mary", meaning they tend to reduce the attention given to this token. According toWang et al. (2023), these two heads negatively impact the final prediction, and ablating them increases the probability of "Mary". Consequently, it may be possible that when editing the model using backpropagation, these attention layers could be modified to neutralize their effect on this particular prompt. In this context, RA offers complementary insights and provides simple visualizations that can contribute to further research.

Entity TrackingKim and Schuster (2023) study the ability of LMs to track the states of entities. In their work, they examine whether language models can correctly answer prompts such as "The orange is in Box X, the book is in Box A, the apple is in Box S ,the game is in Box E, the bill is in Box M, the cross is in Box K, the map is in Box D. Box A contains the" (with the correct answer "book"). The work of Prakash et al. (2024) examine how such task can be used to study the roles of different attention heads of LMs before and after fine-tuning.

We used GPT2-xl with selected prompts in the format of items in boxes and produced their RA. Figure 12 illustrate two examples: one is the prompt from the previous example, and the second is "The magnet is in Box B, the pot is in Box K, the document is in Box M, the apple is in Box H, the bill is in Box C, the cross is in Box S, the orange is in Box A. Box C contains the". In both examples, we found that one of the top \(10\) heads according to RA norm is head 14 from layer 26. Examining its forward pass attention does not reveal a clear pattern to explain why this head ranks so highly by RA. However, both of the RA maps show highly sparse patterns, with only the target token of each prompt, "book" and "bill", are given attention. This pattern suggests that, in order to improve the model's probability to answer these prompts, the model need to suppress its forward attention to those tokens.

In the context of Prakash et al. (2024), RA helps visualize attention patterns and demonstrates how backpropagation tries to alter such patterns. Future research can leverage RA to investigate editing methods for specific attention layers that exhibit different RA patterns 

Figure 8: Attention maps for the prompt “I like Italy and France. I visited the city of” (head 8, layer 30). While the forward attention is the same for both cases (a), the changes of the editing target produce two different Reversed Attention maps. Each reversed map shows that GD tries to change the attention block’s matrices in order to amplify the attention scores of the tokens that are more correlated to the editing target: by amplifying “France” for the target “Paris” (b), or by enhancing “Italy” when the target is “Rome” (c).

Figure 7: The forward and Reversed Attention maps of GPT2-xl, given the editing target “Paris” and the prompt “I like Italy and France. I visited the city of”.

 

Figure 10: Attention maps of OPT-350m for the prompt “I like Italy and France. I visited the city of”. The head with the highest Reversed Attention (RA) norm is head 5 at layer 22 (L22H5). Its RA map (b) shows how it amplifies the attention score for “France” while reducing that of “Italy”. Head 1 at layer 20 is the 11-th highest head by RA norm. We present its RA map (d) using the same color bar scale as that of L22H5, which highlights that this head barely undergoes any update.

Figure 9: The forward and Reversed Attention (RA) of OPT-350m, given the editing target “Paris” and the prompt “I like Italy and France. I visited the city of”. The head with the highest RA norm is highlighted in red as well as the 11-th highest.

 

Figure 11: The GPT2-small attention maps for the prompt “When Mary and John went to the store, John gave a drink to” with the target token “Mary”. The displayed heads are the top three ranked by the norm of their Reversed Attention (RA). The first by its RA norm, head 2 at layer 11, exhibits an RA pattern (b) that directs the editing process to increase the attention given to the input token “Mary”. In contrast, the other two heads, (d) and (f), show patterns that decrease the attention score for “Mary”, suggesting that these heads have a negative direct effect on predicting the token “Mary”.

 


* [126] A. K. M. C. Goss, J. R. D. Lee, and S. W. G. Wilson, "The spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent spin-dependent  

Perturbation Test

In this section, we present the full results of the perturbation analysis outlined in subsection 5.2. This analysis can be viewed as a sparsity test, wherein we mask (zero-out) all attention heads and gradually unmask them according to different methods.

Throughout this section, we examine the ordering provided by Reversed Attention (RA) norms and by Causal Mediation (CM). As a baseline, we also consider the ordering provided by the norm of the average forward pass attention scores ("Forward"), random order ("Random"), and the order of the attention heads in the model ("Index"). Additionally, we evaluate each method by the order it provides and by the reverse order, denoted by the suffix "\([\leftarrow]\)". For example, RA orders the heads from ones with the largest norm to the one with the smallest, while "RA\([\leftarrow]\)" orders them from the smallest to the largest. This reversal in orders represents a negative perturbation test, whereas the original order represents a positive test. A successful order in such a case should yield high Area Under the Curve (AUC) in the positive test and low AUC in the negative test, for the graph that measures the model's accuracy as we unmask the heads in the order of the method.

Causal Mediation implementationCM is applied separately for each task, returning the average indirect effect (AIE) for each attention head. For all tasks, we exclude the AIE when applying CM on the prompts' last tokens (the target token), as it is the token that the model should edit. One implementation we use (CM1) zeros out each head separately.

Another implementation we examined (CM2) is the implementation provided by Todd et al. (2023). This implementation instead of zeroing-out the heads, uses the average activation of each head, as collected from a few left-out examples. An example for the CM2 we produced, compare to the RA norms, is shown in Figure 13.

DatasetsThe ICL tasks we used are sourced from Todd et al. (2023). Each task consists of a set of pairs with a common relation in the format "~question>,<answer>", where the model is tasked with completing the answer given the question. For constructing the prompt for each task and pair, we used the following format: "Q: <question> VA: <answer> W". For example, a 2-shot prompt is in the form of: "Q: <questionI> VA: <answerI> QQ: <question2> VA: <answer2> VQ: <question> VA:".

The natural language questions we utilized are from Hernandez et al. (2024). This dataset comprises tasks, each with a set of pairs in the format "<question>,<answer>", along with a natural language format for constructing the prompt. For instance, in the task concerning the relationship between persons that plays pro sports, the format is "<question> plays the sport of <answer>". We observed that two tasks, country-capital and present-past, had too few examples but shared the same relation as tasks in the ICL dataset by Todd et al. (2023).1 We employed the format from Hernandez et al. (2024) along with the pairs from Todd et al. (2023).

Footnote 1: The Antonym and Synonym tasks used by Todd et al. (2023) are from Nguyen et al. (2017) and the English-French task is from Lample et al. (2018).

In practice, we additionally examined the natural language tasks by prompting the tasks' question with few-shots labeled example. Hence, the 0-shots represent the original tasks, while the n-shots results are provided for comprehensive examination. We adopt this practice from NLP benchmark like HuggingFace leaderboard.2 For example, a

Figure 13: GPT2-xl causal mediation and Reversed Attention maps for the ICL capitalize task with 5-shots.

 

common practice is to evaluate HellaSwag (Zellers et al., 2019) with 10-shots and MMLU (Hendrycks et al., 2021) with 5-shots.

The full list of tasks is provided in Table 4. To summarize, we have ICL tasks with a uniform format of prompt, where the model is required to infer the relation between the question and the answer from given examples. Additionally, we have natural language questions (with and without given examples) where the relation between the question and the answer is explicitly stated in the prompt. In all experiments, and for each task separately, we used a split of \(1/3\) from all available examples to extract the test set. This set of examples is used to report our results and is not included in the creation of any method (i.e. RA, CM).

### Perturbation results

We conducted the perturbation test on GPT2-xl (Radford et al., 2019), OPT-1.3B (Zhang et al., 2022), GPT-j (Wang and Komatsuzaki, 2021) and Llama2-7B (Touvron et al., 2023) models. For each task and method, we provided 25 examples to extract the order of the attention heads. We quantified the performance of each method by the AUC of the graph that measures the model's accuracy as we unmask the heads. Figure 14 provides an example of such a graph and the AUC results extracted from it.

The full ICL results are displayed in Table 567 and 8. We notice that for all models, RA achieves the best results in the majority of the tasks when prompted with 5 or 10 shots, only falling behind CM without any shots. It is also evident that RA shows more dominance across tasks with relatively larger models, like LLaMA2-7B.

The natural language task results are provided in Table 91011 and 12. When considering the 0-shot setting, which is equivalent to examining the perturbation task on the original natural language tasks, we find that when the models' original ability to answer the asked question is high, RA achieves better results compare to CM. In all other cases CM is better only under the 0-shot setup. With few-shot prompting, RA is the preferred method.

In summary, the perturbation test reveals that RA has the potential to localize the model's behavior similar to existing methods and in particular in ICL tasks.

 



\begin{table}
\begin{tabular}{c c l} \hline \hline Task & Task source and type & Examples \\ \hline adjective-v-verb-3 & Q: uplifting, approve, decide A: [uplifting] \\ alphabetically-first-3 & Q: blissful, rat, dingo A: [blissful] \\ animal-v-object-3 & Q: bicycle, skunk, egg A: [skunk] \\ antonym & Q: expire A: [renew] \\ capitalize & Q: cow A: [Cow] \\ capitalize-first-letter & Q: bunny A: [B] \\ choose-middle-of-3 & Q: white, house, wallet A: [house] \\ country-capital & Q: Cabo Verde A: [Praia] \\ english-french & Q: careful A: [prudent] \\ next-item & ICL (Todd et al., 2023) & Q: 12 A: [13] \\ person-sport & Q: Lou Gehrig A: [baseball] \\ present-past & Q: justify Ar: [justified] \\ prev-item & Q: 13 A: [12] \\ singular-plural & Q: glue A: [glues] \\ synonym & Q: missing A: [lost] \\ \hline company-hq & & EMI is headquartered in the city of [London] \\ landmark-in-country & Natural question & route 75 is in the country of [Australia] \\ person-plays-pro-sport & (Hernandez et al., 2024) & Lou Gehrig plays the sport of [baseball] \\ product-by-company & & Digital Negative was created by [Adobe] \\ \hline country-capital & Natural question & The capital city of Cabo Verde is [Praia] \\ present-past & (Hernandez et al., 2024), & The past tense of justify is [justified] \\  & (Todd et al., 2023) & \\ \hline \hline \end{tabular}
\end{table}
Table 4: List of tasks

Figure 14: Perturbation test results visualized for Llama2-7B on the capitalize ICL task. With no shots, when the original model achieves only 0.4 accuracy, causal mediation (CM) achieve the highest AUC score. With few-shots prompting, Reversed Attention (RA) outperforms all other methods.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r} \hline Task & n-shots & Random & Index [\(\leftarrow\)] & Index & CM2 [\(\leftarrow\)] & CM2 & CM1 [\(\leftarrow\)] & CM1 [\(\leftarrow\)] & CA1 [\(\leftarrow\)] & FA [\(\leftarrow\)] & RA & R \\ \hline adjective-v-verb- & 0 & 0.03 & 0.04 & 0.05 & 0.09 & 0.01 & **0.44** & 0.01 & 0.03 & 0.03 & 0.01 & 0.05 \\
3 & 1 & 0.21 & 0.18 & 0.09 & 0.27 & 0.03 & **0.63** & 0.02 & 0.1 & 0.12 & 0.03 & 0.3 \\ v-verb- & 5 & 0.23 & 0.22 & 0.08 & 0.21 & 0.05 & 0.15 & 0.03 & 0.18 & 0.14 & 0.03 & **0.35** \\  & 10 & 0.25 & 0.24 & 0.09 & 0.12 & 0.1 & **0.4** & 0.04 & 0.2 & 0.11 & 0.03 & 0.36 \\ alphabetic- & 0 & 0.02 & 0.03 & 0.05 & 0.15 & 0.01 & **0.28** & 0.01 & 0.02 & 0.04 & 0.02 & 0.04 \\ ally- & 1 & 0.12 & 0.13 & 0.06 & 0.07 & 0.11 & **0.22** & 0.02 & 0.07 & 0.08 & 0.02 & 0.2 \\ first-3 & 5 & 0.15 & 0.15 & 0.05 & 0.08 & 0.11 & 0.19 & 0.05 & 0.1 & 0.08 & 0.02 & **0.23** \\  & 10 & 0.14 & 0.13 & 0.05 & 0.11 & 0.13 & **0.23** & 0.05 & 0.1 & 0.06 & 0.02 & 0.22 \\ animal- & 0 & 0.09 & 0.03 & 0.06 & 0.11 & 0.01 & **0.28** & 0.01 & 0.03 & 0.04 & 0.02 & 0.08 \\ v- & 1 & 0.2 & 0.16 & 0.08 & 0.3 & 0.04 & **0.39** & 0.03 & 0.13 & 0.11 & 0.02 & 0.29 \\ object- & 5 & 0.28 & 0.27 & 0.1 & 0.18 & 0.13 & **0.44** & 0.04 & 0.24 & 0.16 & 0.04 & 0.42 \\
3 & 10 & 0.3 & 0.29 & 0.1 & 0.23 & 0.1 & 0.3 & 0.05 & 0.27 & 0.15 & 0.05 & **0.49** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.01** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ antonym & 1 & 0.02 & 0.07 & 0.02 & 0.01 & 0.01 & **0.15** & 0.01 & 0.02 & 0.02 & 0.01 & 0.07 \\  & 5 & 0.09 & 0.24 & 0.06 & 0.03 & 0.03 & 0.27 & 0.03 & 0.15 & 0.05 & 0.02 & **0.32** \\  & 10 & 0.11 & 0.26 & 0.07 & 0.03 & 0.03 & 0.3 & 0.03 & 0.17 & 0.05 & 0.03 & **0.34** \\  & 0 & 0.01 & 0.0 & 0.0 & 0.08 & 0.0 & **0.23** & 0.0 & 0.0 & 0.0 & 0.03 & 0.0 \\ capitalize & 1 & 0.08 & 0.19 & 0.05 & 0.12 & 0.05 & **0.64** & 0.01 & 0.03 & 0.07 & 0.03 & 0.1 \\  & 5 & 0.31 & 0.43 & 0.14 & 0.1 & 0.27 & **0.52** & 0.05 & 0.26 & 0.17 & 0.05 & 0.5 \\  & 10 & 0.36 & 0.45 & 0.14 & 0.08 & 0.27 & **0.59** & 0.05 & 0.32 & 0.16 & 0.05 & **0.59** \\ choose- & 0 & 0.02 & 0.02 & 0.05 & 0.01 & **0.24** & 0.01 & 0.01 & 0.02 & 0.01 & 0.02 \\ middle- & 1 & 0.1 & 0.16 & 0.06 & 0.1 & 0.05 & **0.35** & 0.03 & 0.05 & 0.08 & 0.02 & 0.19 \\ of-3 & 5 & 0.17 & 0.23 & 0.07 & 0.04 & 0.11 & 0.21 & 0.05 & 0.13 & 0.1 & 0.03 & **0.3** \\  & 10 & 0.2 & 0.25 & 0.07 & 0.05 & 0.25 & 0.2 & 0.05 & 0.14 & 0.08 & 0.04 & **0.35** \\  & 0 & 0.0 & 0.0 & 0.01 & 0.02 & 0.0 & **0.04** & 0.0 & 0.0 & 0.02 & 0.0 & 0.0 \\ country- & 1 & 0.07 & 0.18 & 0.08 & 0.08 & 0.02 & 0.22 & 0.02 & 0.08 & 0.09 & 0.03 & **0.29** \\ capital & 5 & 0.19 & 0.3 & 0.1 & 0.05 & 0.05 & 0.4 & 0.04 & 0.24 & 0.07 & 0.04 & **0.42** \\  & 10 & 0.19 & 0.29 & 0.1 & 0.07 & 0.08 & **0.55** & 0.03 & 0.24 & 0.07 & 0.04 & 0.4 \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.01** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ english- & 1 & 0.01 & 0.03 & 0.01 & 0.01 & 0.01 & 0.02 & 0.0 & 0.02 & 0.01 & 0.01 & **0.04** \\ french & 5 & 0.06 & 0.08 & 0.02 & 0.02 & 0.03 & 0.08 & 0.01 & 0.06 & 0.03 & 0.01 & **0.12** \\  & 10 & 0.07 & 0.09 & 0.03 & 0.02 & 0.04 & 0.06 & 0.02 & 0.07 & 0.03 & 0.01 & **0.13** \\ lowercase- & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ last- & 1 & 0.02 & 0.03 & 0.01 & 0.03 & 0.02 & **0.08** & 0.0 & 0.01 & 0.02 & 0.01 & 0.06 \\ letter & 5 & 0.05 & 0.06 & 0.02 & 0.06 & 0.04 & **0.14** & 0.01 & 0.04 & 0.02 & 0.01 & 0.1 \\  & 10 & 0.06 & 0.08 & 0.02 & 0.07 & 0.04 & **0.11** & 0.03 & 0.05 & 0.03 & 0.01 & 0.1 \\  & 0 & 0.02 & 0.01 & 0.03 & 0.06 & 0.0 & **0.22** & 0.0 & 0.02 & 0.01 & 0.01 & 0.03 \\ next- & 1 & 0.06 & 0.1 & 0.04 & 0.07 & 0.02 & **0.25** & 0.01 & 0.05 & 0.04 & 0.03 & 0.14 \\ item & 5 & 0.17 & 0.26 & 0.08 & 0.1 & 0.05 & 0.31 & 0.03 & 0.2 & 0.07 & 0.03 & **0.34** \\  & 10 & 0.22 & 0.27 & 0.09 & 0.12 & 0.09 & 0.3 & 0.04 & 0.21 & 0.08 & 0.04 & **0.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r} \hline Task & n-shots & Random & Index [\(\leftarrow\)] & Index & CM2 [\(\leftarrow\)] & CM2 & CM1 [\(\leftarrow\)] & CM1 & FA [\(\leftarrow\)] & FA & RA [\(\leftarrow\)] & RA \\ \hline adjective-v-verb-3 & 0 & 0.01 & 0.01 & 0.03 & 0.03 & 0.0 & **0.29** & 0.0 & 0.01 & 0.06 & 0.01 & 0.02 \\  & 1 & 0.18 & 0.1 & 0.04 & 0.24 & 0.02 & **0.42** & 0.02 & 0.12 & 0.15 & 0.02 & 0.39 \\ v-verb-3 & 5 & 0.24 & 0.11 & 0.05 & 0.08 & 0.08 & 0.11 & 0.03 & 0.2 & 0.13 & 0.02 & **0.45** \\  & 10 & 0.24 & 0.11 & 0.03 & 0.02 & 0.11 & 0.03 & 0.06 & 0.17 & 0.11 & 0.02 & **0.34** \\ alphabetic & 0 & 0.01 & 0.01 & 0.03 & 0.1 & 0.01 & **0.23** & 0.01 & 0.02 & 0.06 & 0.01 & 0.05 \\ ally-strs & 1 & 0.09 & 0.08 & 0.03 & 0.04 & 0.07 & 0.16 & 0.02 & 0.09 & 0.1 & 0.02 & **0.24** \\ infst-3 & 5 & 0.12 & 0.08 & 0.03 & 0.05 & 0.12 & 0.06 & 0.07 & 0.13 & 0.08 & 0.02 & **0.25** \\ animal-v- & 10 & 0.12 & 0.08 & 0.02 & 0.03 & 0.09 & **0.22** & 0.01 & 0.11 & 0.06 & 0.02 & 0.2 \\ animal- & 0 & 0.02 & 0.01 & 0.03 & 0.01 & 0.01 & **0.31** & 0.01 & 0.03 & 0.06 & 0.01 & 0.06 \\ v- & 1 & 0.16 & 0.06 & 0.04 & 0.1 & 0.06 & **0.55** & 0.02 & 0.11 & 0.11 & 0.02 & 0.29 \\ object- & 5 & 0.21 & 0.1 & 0.04 & 0.06 & 0.07 & **0.39** & 0.03 & 0.18 & 0.14 & 0.03 & 0.36 \\
3 & 10 & 0.19 & 0.1 & 0.02 & 0.06 & 0.1 & 0.09 & 0.07 & 0.18 & 0.12 & 0.03 & **0.31** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.16** & 0.0 & 0.0 & 0.0 & 0.0 & 0.01 \\ antonym & 1 & 0.03 & 0.04 & 0.01 & 0.02 & 0.01 & **0.19** & 0.01 & 0.05 & 0.02 & 0.01 & 0.09 \\  & 5 & 0.07 & 0.09 & 0.02 & 0.03 & 0.02 & 0.1 & 0.02 & 0.12 & 0.07 & 0.02 & **0.21** \\  & 10 & 0.08 & 0.1 & 0.02 & 0.02 & 0.02 & 0.05 & 0.02 & 0.12 & 0.06 & 0.02 & **0.2** \\  & 0 & 0.02 & 0.0 & 0.0 & 0.04 & 0.0 & **0.29** & 0.0 & 0.02 & 0.01 & 0.01 & 0.02 \\ capitalize & 1 & 0.03 & 0.02 & 0.01 & 0.01 & 0.0 & **0.19** & 0.0 & 0.01 & 0.01 & 0.0 & 0.05 \\  & 5 & 0.35 & 0.24 & 0.09 & 0.08 & 0.23 & 0.52 & 0.05 & 0.38 & 0.19 & 0.05 & **0.67** \\  & 10 & 0.36 & 0.24 & 0.06 & 0.07 & 0.23 & 0.47 & 0.05 & 0.39 & 0.18 & 0.05 & **0.6** \\ choose- & 0 & 0.01 & 0.0 & 0.0 & 0.02 & 0.0 & **0.14** & 0.0 & 0.01 & 0.02 & 0.0 & 0.01 \\ middle- & 1 & 0.05 & 0.05 & 0.02 & 0.02 & 0.04 & 0.14 & 0.01 & 0.07 & 0.07 & 0.01 & **0.18** \\ of-3 & 5 & 0.13 & 0.12 & 0.04 & 0.04 & 0.13 & 0.08 & 0.04 & 0.19 & 0.11 & 0.03 & **0.31** \\  & 10 & 0.14 & 0.11 & 0.03 & 0.03 & 0.1 & 0.11 & 0.04 & 0.16 & 0.07 & 0.03 & **0.25** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.04** & 0.0 & 0.01 & 0.0 & 0.0 & 0.02 \\ country-capital & 1 & 0.15 & 0.14 & 0.05 & 0.04 & 0.12 & **0.52** & 0.04 & 0.22 & 0.11 & 0.03 & 0.43 \\ capitalizer & 5 & 0.22 & 0.19 & 0.07 & 0.06 & 0.05 & 0.55 & 0.05 & 0.28 & 0.19 & 0.04 & **0.56** \\  & 10 & 0.2 & 0.17 & 0.04 & 0.06 & 0.08 & 0.38 & 0.05 & 0.28 & 0.13 & 0.04 & **0.46** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.02** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ english-french & 1 & 0.02 & 0.03 & 0.01 & 0.01 & 0.01 & 0.03 & 0.01 & 0.03 & 0.01 & 0.0 & **0.07** \\ french & 5 & 0.1 & 0.08 & 0.02 & 0.02 & 0.04 & 0.16 & 0.02 & 0.1 & 0.05 & 0.02 & **0.19** \\  & 10 & 0.11 & 0.08 & 0.02 & 0.02 & 0.05 & 0.08 & 0.02 & 0.11 & 0.05 & 0.02 & **0.18** \\ lowercase- & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ last-letter & 1 & 0.02 & 0.02 & 0.01 & 0.01 & 0.01 & **0.07** & 0.01 & 0.01 & 0.01 & 0.0 & **0.07** \\  & 5 & 0.04 & 0.03 & 0.01 & 0.06 & 0.04 & 0.02 & 0.02 & 0.04 & 0.02 & 0.01 & **0.08** \\ letter & 10 & 0.04 & 0.03 & 0.01 & **0.08** & 0.03 & 0.02 & 0.04 & 0.05 & 0.03 & 0.01 & **0.08** \\  & 0 & 0.01 & 0.02 & 0.01 & 0.1 & 0.01 & **0.27** & 0.01 & 0.04 & 0.04 & 0.01 & 0.07 \\ next-item & 1 & 0.08 & 0.08 & 0.02 & 0.09 & 0.02 & **0.35** & 0.02 & 0.09 & 0.08 & 0.02 & 0.25 \\ item & 5 & 0.15 & 0.12 & 0.03 & 0.05 & 0.03 & 0.34 & 0.03 & 0.18 & 0.11 & 0.03 & **0.41** \\  & 10 & 0.14 & 0.14 & 0.03 & 0.06 & 0.04 & 0.2 & 0.03 & 0



\begin{table}
\begin{tabular}{l c r r r r r r r r r r r} \hline Task & n-shots & Random & Index [\(\leftarrow\)] & Index & CM2 [\(\leftarrow\)] & CM2 & CM1 [\(\leftarrow\)] & CM1 [\(\leftarrow\)] & CA1 & FA [\(\leftarrow\)] & FA & RA [\(\leftarrow\)] & RA \\ \hline adjective-v-verb-3 & 0 & 0.06 & 0.03 & 0.03 & 0.15 & 0.01 & **0.32** & 0.01 & 0.03 & 0.05 & 0.01 & 0.17 \\  & 1 & 0.07 & 0.13 & 0.03 & 0.2 & 0.02 & **0.47** & 0.02 & 0.06 & 0.09 & 0.02 & 0.31 \\ v-verb-3 & 5 & 0.11 & 0.2 & 0.03 & 0.07 & 0.07 & **0.48** & 0.04 & 0.22 & 0.12 & 0.03 & 0.45 \\  & 10 & 0.12 & 0.24 & 0.03 & 0.06 & 0.08 & 0.24 & 0.03 & 0.27 & 0.14 & 0.03 & **0.51** \\ alphabetic- ally- first-3 & 0 & 0.2 & 0.04 & 0.04 & 0.25 & 0.01 & **0.32** & 0.01 & 0.04 & 0.05 & 0.02 & 0.18 \\  & 1 & 0.11 & 0.12 & 0.02 & 0.09 & 0.08 & **0.24** & 0.02 & 0.07 & 0.07 & 0.02 & 0.23 \\ animal- first-3 & 5 & 0.17 & 0.14 & 0.02 & 0.05 & 0.07 & 0.1 & 0.04 & 0.14 & 0.1 & 0.03 & **0.25** \\  & 10 & 0.16 & 0.14 & 0.02 & 0.05 & 0.08 & 0.22 & 0.02 & 0.16 & 0.08 & 0.03 & **0.25** \\ animal- u- animal- v- & 0 & 0.1 & 0.04 & 0.04 & 0.26 & 0.01 & **0.32** & 0.01 & 0.04 & 0.05 & 0.01 & 0.24 \\ oviec- viec- object- & 1 & 0.23 & 0.12 & 0.03 & 0.27 & 0.02 & **0.46** & 0.02 & 0.09 & 0.07 & 0.03 & 0.3 \\  & 5 & 0.27 & 0.15 & 0.02 & 0.14 & 0.07 & 0.28 & 0.03 & 0.21 & 0.13 & 0.03 & **0.41** \\
3 & 10 & 0.27 & 0.19 & 0.03 & 0.1 & 0.08 & 0.36 & 0.04 & 0.27 & 0.13 & 0.04 & **0.46** \\  & 0 & 0.01 & 0.0 & 0.0 & 0.03 & 0.0 & **0.07** & 0.0 & 0.0 & 0.0 & 0.02 \\ antonym & 1 & 0.08 & 0.1 & 0.04 & 0.03 & 0.02 & **0.25** & 0.01 & 0.06 & 0.05 & 0.01 & 0.19 \\  & 5 & 0.14 & 0.26 & 0.03 & 0.04 & 0.04 & 0.21 & 0.03 & 0.26 & 0.1 & 0.03 & **0.43** \\  & 10 & 0.15 & 0.28 & 0.03 & 0.04 & 0.04 & 0.41 & 0.03 & 0.26 & 0.1 & 0.03 & **0.46** \\  & 0 & 0.0 & 0.0 & 0.13 & 0.0 & **0.17** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ capitalize & 1 & 0.07 & 0.23 & 0.04 & 0.03 & 0.02 & **0.34** & 0.02 & 0.08 & 0.07 & 0.02 & 0.28 \\  & 5 & 0.2 & 0.46 & 0.06 & 0.08 & 0.3 & **0.77** & 0.05 & 0.45 & 0.19 & 0.05 & 0.68 \\  & 10 & 0.21 & 0.46 & 0.05 & 0.07 & 0.29 & **0.74** & 0.05 & 0.47 & 0.18 & 0.05 & 0.7 \\ choose- middle- of & 0 & 0.01 & 0.01 & 0.0 & **0.14** & 0.0 & 0.12 & 0.0 & 0.01 & 0.0 & 0.0 & 0.02 \\  & 1 & 0.09 & 0.16 & 0.02 & 0.03 & 0.07 & **0.53** & 0.02 & 0.05 & 0.06 & 0.02 & 0.29 \\ middle- of & 5 & 0.13 & 0.25 & 0.03 & 0.04 & 0.12 & **0.38** & 0.03 & 0.21 & 0.12 & 0.04 & 0.36 \\  & 10 & 0.15 & 0.31 & 0.04 & 0.05 & 0.11 & **0.51** & 0.04 & 0.32 & 0.13 & 0.06 & 0.47 \\  & 0 & 0.01 & 0.01 & 0.01 & **0.05** & 0.0 & **0.05** & 0.0 & 0.01 & 0.01 & 0.0 & 0.03 \\ country- capitalized & 1 & 0.2 & 0.3 & 0.06 & 0.05 & 0.1 & 0.19 & 0.04 & 0.27 & 0.1 & 0.04 & **0.48** \\  & 5 & 0.28 & 0.42 & 0.07 & 0.05 & 0.15 & 0.33 & 0.05 & 0.4 & 0.13 & 0.06 & **0.68** \\  & 10 & 0.27 & 0.44 & 0.06 & 0.06 & 0.11 & 0.1 & 0.05 & 0.44 & 0.14 & 0.07 & **0.68** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.01 & 0.0 & **0.02** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ english- fresh & 1 & 0.11 & 0.23 & 0.03 & 0.03 & 0.06 & 0.13 & 0.03 & 0.13 & 0.05 & 0.03 & **0.27** \\ french & 5 & 0.19 & 0.37 & 0.04 & 0.06 & 0.11 & 0.44 & 0.04 & 0.33 & 0.09 & 0.04 & **0.52** \\  & 10 & 0.2 & 0.38 & 0.04 & 0.08 & 0.11 & 0.28 & 0.04 & 0.37 & 0.09 & 0.04 & **0.55** \\ lowercase- last- letter & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\  & 1 & 0.03 & 0.05 & 0.01 & 0.04 & 0.01 & **0.11** & 0.01 & 0.04 & 0.03 & 0.01 & 0.1 \\  & 5 & 0.05 & 0.1 & 0.01 & 0.02 & 0.09 & 0.13 & 0.02 & 0.08 & 0.04 & 0.01 & **0.16** \\ letter & 10 & 0.05 & 0.09 & 0.01 & 0.12 & 0.04 & 0.04 & 0.08 & 0.13 & 0.04 & 0.01 & **0.15** \\  & 0 & 0.05 & 0.06 & 0.02 & 0.12 & 0.01 & **0.17** & 0.01 & 0.06 & 0.01 & 0.02 & 0.13 \\ next- item & 1 & 0.09 & 0.25 & 0.03 & 0.08 & 0.03 & **0.42** & 0.02 & 0.16 & 0.07 & 0.02 & 0.38 \\  & 5 & 0.19 & 0.38 & 0.04 & 0.09 & 0.05 & 0.42 & 0.03 & 0.33 & 0.13 & 0.04 & **0.58** \\  & 10 & 0.24 & 0.41 & 0.04 & 0.05 & 0.09 & 0.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r} \hline Task & n-shots & Random & Index [\(\leftarrow\)] & Index & CM2 [\(\leftarrow\)] & CM2 & CM1 [\(\leftarrow\)] & CA1 & FA [\(\leftarrow\)] & FA & RA [\(\leftarrow\)] & RA \\ \hline adjective-v-verb-3 & 0 & 0.1 & 0.04 & 0.03 & 0.17 & 0.01 & **0.35** & 0.01 & 0.14 & 0.04 & 0.01 & 0.15 \\  & 1 & 0.16 & 0.2 & 0.04 & 0.17 & 0.03 & **0.4** & 0.03 & 0.28 & 0.05 & 0.03 & 0.37 \\ v-verb-3 & 5 & 0.25 & 0.21 & 0.04 & 0.08 & 0.06 & 0.11 & 0.03 & 0.35 & 0.07 & 0.03 & **0.45** \\  & 10 & 0.29 & 0.26 & 0.04 & 0.09 & 0.09 & 0.12 & 0.04 & 0.36 & 0.05 & 0.04 & **0.48** \\ alphabetic & 0 & 0.03 & 0.04 & 0.03 & 0.14 & 0.01 & **0.17** & 0.01 & 0.12 & 0.04 & 0.01 & 0.11 \\ ally- first-3 & 1 & 0.08 & 0.14 & 0.03 & 0.1 & 0.06 & 0.13 & 0.02 & 0.19 & 0.04 & 0.02 & **0.22** \\ animal- & 5 & 0.08 & 0.14 & 0.03 & 0.08 & 0.04 & 0.08 & 0.02 & 0.19 & 0.04 & 0.02 & **0.24** \\ iv- & 10 & 0.07 & 0.13 & 0.02 & 0.06 & 0.06 & 0.07 & 0.02 & 0.19 & 0.02 & 0.02 & **0.26** \\ animal- & 0 & 0.08 & 0.07 & 0.03 & 0.12 & 0.02 & 0.09 & 0.01 & **0.15** & 0.04 & 0.02 & **0.15** \\ v- & 1 & 0.12 & 0.13 & 0.03 & 0.16 & 0.05 & 0.17 & 0.03 & 0.22 & 0.05 & 0.02 & **0.27** \\ object- & 5 & 0.22 & 0.18 & 0.03 & 0.09 & 0.04 & 0.11 & 0.03 & 0.28 & 0.06 & 0.03 & **0.39** \\
3 & 10 & 0.27 & 0.22 & 0.04 & 0.08 & 0.06 & 0.04 & 0.06 & 0.34 & 0.05 & 0.03 & **0.43** \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.01** \\ antonym & 1 & 0.07 & 0.1 & 0.02 & 0.06 & 0.02 & 0.12 & 0.02 & 0.15 & 0.02 & 0.02 & **0.21** \\  & 5 & 0.17 & 0.25 & 0.04 & 0.07 & 0.05 & 0.08 & 0.03 & 0.29 & 0.04 & 0.03 & **0.34** \\  & 10 & 0.19 & 0.28 & 0.04 & 0.05 & 0.07 & 0.1 & 0.03 & 0.19 & 0.04 & 0.03 & **0.33** \\  & 0 & 0.05 & 0.08 & 0.02 & 0.08 & 0.02 & **0.52** & 0.02 & 0.14 & 0.02 & 0.02 & 0.16 \\ capitalize & 1 & 0.19 & 0.36 & 0.04 & 0.07 & 0.03 & 0.07 & 0.03 & 0.41 & 0.05 & 0.03 & **0.58** \\  & 5 & 0.35 & 0.45 & 0.05 & 0.07 & 0.15 & 0.14 & 0.05 & 0.65 & 0.06 & 0.05 & **0.76** \\  & 10 & 0.36 & 0.46 & 0.05 & 0.1 & 0.14 & 0.33 & 0.05 & 0.61 & 0.05 & 0.05 & **0.76** \\ choose- & 0 & 0.01 & 0.01 & 0.0 & 0.05 & 0.0 & **0.16** & 0.0 & 0.02 & 0.0 & 0.01 & 0.02 \\ middle- & 1 & 0.09 & 0.19 & 0.03 & 0.05 & 0.1 & **0.38** & 0.03 & 0.22 & 0.04 & 0.03 & 0.3 \\ of-3 & 5 & 0.16 & 0.17 & 0.03 & 0.04 & 0.03 & 0.22 & 0.02 & 0.21 & 0.05 & 0.03 & **0.36** \\  & 10 & 0.14 & 0.2 & 0.03 & 0.07 & 0.05 & 0.06 & 0.03 & 0.24 & 0.04 & 0.03 & **0.43** \\  & 0 & 0.02 & 0.0 & 0.0 & 0.0 & 0.02 & 0.0 & 0.0 & **0.04** & 0.01 & 0.0 & 0.02 \\ country- & 1 & 0.17 & 0.39 & 0.04 & 0.21 & 0.09 & 0.17 & 0.04 & 0.5 & 0.05 & 0.05 & **0.52** \\ capital & 5 & 0.22 & 0.43 & 0.05 & 0.15 & 0.08 & 0.15 & 0.05 & **0.58** & 0.07 & 0.05 & 0.56 \\  & 10 & 0.24 & 0.42 & 0.05 & 0.12 & 0.22 & 0.14 & 0.05 & 0.48 & 0.05 & 0.05 & **0.57** \\  & 0 & **0.02** & 0.0 & 0.0 & **0.02** & 0.0 & **0.02** & 0.0 & 0.01 & 0.0 & 0.0 & 0.01 \\ english- & 1 & 0.08 & 0.24 & 0.03 & 0.05 & 0.07 & 0.06 & 0.03 & **0.31** & 0.03 & 0.03 & **0.31** \\ french & 5 & 0.13 & 0.36 & 0.04 & 0.07 & 0.1 & 0.14 & 0.04 & 0.44 & 0.04 & 0.04 & **0.47** \\  & 10 & 0.14 & 0.37 & 0.04 & 0.07 & 0.06 & 0.17 & 0.04 & 0.38 & 0.04 & 0.04 & **0.45** \\ lowercase- & 0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ last- & 1 & 0.02 & 0.06 & 0.01 & 0.01 & 0.01 & 0.03 & 0.01 & **0.09** & 0.01 & 0.01 & **0.09** \\ letter & 5 & 0.05 & 0.11 & 0.02 & 0.02 & 0.06 & 0.02 & 0.02 & 0.15 & 0.02 & 0.01 & **0.16** \\  & 10 & 0.05 & 0.11 & 0.02 & 0.07 & 0.03 & 0.09 & 0.02 & 0.12 & 0.02 & 0.01 & **0.16** \\ next- & 0 & 0.11 & 0.08 & 0.06 & 0.11 & 0.05 & 0.14 & 0.01 & 0.1 & 0.07 & 0.04 & **0.15** \\ item & 1 & 0.1 & 0.19 & 0.06 & 0.12 & 0.07 & 0.12 & 0.07 & 0.26 & 0.07 & 0.05 & **0.35** \\ item & 5 & 0.19 & 0.39 & 0.08 & 0.13 & 0.1 & 0.18 & 0.05 & 0.47 & 0.09 & 0.09 & **0.66** \\  & 10 & 0.2 & 0.44 & 0.07 & 0.09 & 0.07 & 0.2



\begin{table}
\begin{tabular}{l c r r r r r r r r r r r} \hline \hline Task & n-shots & Random & Index [\(\leftarrow\)–] & Index & CM2 [\(\leftarrow\)–] & CM2 & CM1 [\(\leftarrow\)–] & CM1 & FA [\(\leftarrow\)–] & FA & RA [\(\leftarrow\)–] & RA \\ \hline  & 0 & 0.16 & 0.05 & 0.04 & 0.18 & 0.02 & **0.27** & 0.02 & 0.2 & 0.05 & 0.03 & 0.23 \\ company- & 1 & 0.11 & 0.08 & 0.04 & 0.19 & 0.04 & **0.27** & 0.02 & 0.17 & 0.07 & 0.02 & 0.23 \\ hq & 5 & 0.15 & 0.12 & 0.04 & 0.16 & 0.02 & 0.21 & 0.04 & 0.2 & 0.07 & 0.03 & **0.25** \\  & 10 & 0.16 & 0.14 & 0.03 & 0.08 & 0.06 & 0.21 & 0.04 & 0.2 & 0.08 & 0.04 & **0.25** \\  & 0 & 0.04 & 0.01 & 0.03 & 0.03 & 0.01 & **0.56** & 0.01 & 0.15 & 0.04 & 0.01 & 0.2 \\ country- & 1 & 0.33 & 0.16 & 0.12 & 0.1 & 0.34 & **0.67** & 0.05 & 0.34 & 0.22 & 0.05 & 0.62 \\ capital & 5 & 0.34 & 0.17 & 0.12 & 0.15 & 0.3 & 0.44 & 0.12 & 0.31 & 0.22 & 0.05 & **0.56** \\  & 10 & 0.31 & 0.17 & 0.11 & 0.11 & 0.28 & 0.26 & 0.17 & 0.29 & 0.19 & 0.05 & **0.5** \\ landmark- & 0 & 0.2 & 0.07 & 0.04 & 0.22 & 0.02 & **0.32** & 0.02 & 0.26 & 0.05 & 0.02 & 0.29 \\ in- & 1 & 0.2 & 0.11 & 0.06 & 0.17 & 0.06 & 0.21 & 0.05 & 0.21 & 0.1 & 0.04 & **0.37** \\ country & 5 & 0.21 & 0.11 & 0.06 & 0.17 & 0.06 & 0.26 & 0.04 & 0.28 & 0.11 & 0.04 & **0.37** \\  & 10 & 0.21 & 0.11 & 0.04 & 0.1 & 0.07 & 0.19 & 0.05 & 0.26 & 0.1 & 0.03 & **0.35** \\ person- & 0 & 0.33 & 0.22 & 0.08 & 0.32 & 0.04 & 0.47 & 0.04 & 0.44 & 0.1 & 0.07 & **0.51** \\ plays- & 1 & 0.3 & 0.23 & 0.11 & 0.22 & 0.24 & 0.42 & 0.15 & 0.34 & 0.17 & 0.11 & **0.45** \\ pro- & 5 & 0.34 & 0.26 & 0.08 & 0.32 & 0.13 & 0.39 & 0.09 & 0.39 & 0.13 & 0.08 & **0.44** \\ sport & 10 & 0.35 & 0.25 & 0.07 & 0.29 & 0.15 & **0.47** & 0.12 & 0.38 & 0.13 & 0.09 & 0.46 \\  & 0 & 0.0 & 0.0 & 0.0 & 0.0 & **0.02** & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ present- & 1 & 0.1 & 0.15 & 0.06 & 0.02 & 0.05 & **0.64** & 0.02 & 0.1 & 0.1 & 0.02 & 0.27 \\ past & 5 & 0.28 & 0.23 & 0.12 & 0.06 & 0.19 & **0.62** & 0.04 & 0.4 & 0.26 & 0.04 & 0.57 \\  & 10 & 0.3 & 0.24 & 0.13 & 0.06 & 0.23 & 0.44 & 0.05 & 0.4 & 0.22 & 0.05 & **0.61** \\ product- & 0 & 0.26 & 0.1 & 0.09 & 0.09 & 0.3 & **0.46** & 0.02 & 0.3 & 0.1 & 0.03 & 0.41 \\ by- & 1 & 0.32 & 0.17 & 0.08 & 0.22 & 0.11 & **0.52** & 0.06 & 0.32 & 0.16 & 0.1 & 0.46 \\ company & 5 & 0.31 & 0.19 & 0.08 & 0.21 & 0.18 & 0.48 & 0.08 & 0.41 & 0.16 & 0.08 & **0.53** \\
10 & 0.29 & 0.2 & 0.08 & 0.23 & 0.16 & 0.29 & 0.11 & 0.38 & 0.14 & 0.07 & **0.48** \\ \hline \hline \end{tabular}
\end{table}
Table 10: OPT-1.3B perturbation test on tasks of natural questions.

\begin{table}
\begin{tabular}{l l r r r r r r r r r r} \hline \hline Task & n-shots & Random & Index [\(\leftarrow\)–] & Index & CM2 [\(\leftarrow\)–] & CM2 & CM1 [\(\leftarrow\)–] & CM1 & FA [\(\leftarrow\)–] & FA & RA [\(\leftarrow\)–] & RA \\ \hline  & 0 & 0.15 & 0.09 & 0.05 & 0.13 & 0.09 & **0.19** & 0.05 & 0.15 & 0.06 & 0.04 & 0.18 \\ company- & 1 & 0.07 & 0.09 & 0.04 & 0.12 & 0.04 & **0.19** & 0.03 & 0.12 & 0.05 & 0.02 & 0.16 \\ hq & 5 & 0.14 & 0.14 & 0.05 & 0.11 & 0.04 & **0.21** & 0.03 & 0.14 & 0.07 & 0.03 & 0.2 \\  & 10 & 0.14 & 0.13 & 0.05 & 0.12 & 0.03 & 0.19 & 0.05 & 0.13 & 0.08 & 0.04 & **0.2** \\  & 0 & 0.02 & 0.03 & 0.05 & 0.36 & 0.01 & **0.61** & 0.01 & 0.08 & 0.06 & 0.01 & 0.07 \\ country- & 1 & 0.27 & 0.31 & 0.13 & 0.21 & 0.13 & **0.58** & 0.04 & 0.32 & 0.14 & 0.04 & 0.57 \\ capital & 5 & 0.43 & 0.34 & 0.12 & 0.19 & 0.15 & 0.52 & 0.1 & 0.32 & 0.13 & 0.07 & **0.57** \\  & 10 & 0.41 & 0.33 & 0.12 & 0.24 & 0.1 & 0.36 & 0.07 & 0.28 & 0.17 & 0.07 & **0.51** \\ landmark- & 0 & 0.06 & 0.05 & 0.03 & 0.08 & 0.01 & **0.35** & 0.01 & 0.12 & 0.05 & 0.01 & 0.18 \\ in- & 1 & 0.14 & 0.14 & 0.06 & 0.11 & 0.04 & **0.29** & 0.04 & 0.11 & 0.08 & 0.03 & 0.28 \\ country & 5 & 0.18 & 0.17 & 0.07 & 0.07 & 0.07 & 0.23 & 0.02 & 0.2 & 0.09 & 0.04 & **0.33** \\  & 10 & 0.2 & 0.18 & 0.07 & 0.1 & 0.08 & 0.2 & 0.02 & 0.2 & 0.1 & 0.03 & **0.33** \\ person- & 0 & 0.32 & 0.24 & 0.08 & 0.36 & 0.05 & **0.59** & 0.03 & 0.48 & 0.09 & 0.07 & 0.57 \\ plays- & 1 & 0.32 & 0.27 & 0.1 & 0.24 & 0.25 & 0.42 & 0.07 &



\begin{table}
\begin{tabular}{l c r r r r r r r r r r r} \hline \hline Task & n-shots & Random & Index [\(\leftarrow\)—] & Index & CM2 [\(\leftarrow\)—] & CM2 & CM1 [\(\leftarrow\)–] & CM1 & FA [\(\leftarrow\)—] & FA & RA [\(\leftarrow\)—] & RA \\ \hline company- & 0 & 0.15 & 0.13 & 0.05 & 0.12 & 0.02 & 0.25 & 0.02 & 0.24 & 0.04 & 0.03 & **0.3** \\ company- & 1 & 0.11 & 0.13 & 0.04 & 0.16 & 0.02 & **0.26** & 0.02 & 0.2 & 0.03 & 0.02 & 0.2 \\ hq & 5 & 0.18 & 0.23 & 0.05 & 0.15 & 0.05 & 0.23 & 0.03 & 0.32 & 0.06 & 0.04 & **0.38** \\  & 10 & 0.21 & 0.25 & 0.05 & 0.13 & 0.05 & 0.23 & 0.03 & 0.33 & 0.06 & 0.04 & **0.36** \\ country- & 0 & 0.31 & 0.28 & 0.07 & 0.48 & 0.04 & **0.72** & 0.04 & 0.48 & 0.13 & 0.05 & 0.53 \\ country- & 1 & 0.44 & 0.44 & 0.1 & 0.35 & 0.19 & **0.67** & 0.05 & 0.53 & 0.16 & 0.12 & **0.67** \\ capital & 5 & 0.43 & 0.43 & 0.09 & 0.13 & 0.23 & 0.31 & 0.06 & 0.51 & 0.18 & 0.14 & **0.67** \\ landmark- & 10 & 0.43 & 0.44 & 0.09 & 0.13 & 0.25 & 0.47 & 0.07 & 0.47 & 0.16 & 0.14 & **0.68** \\ landmark- & 0 & 0.21 & 0.19 & 0.02 & 0.28 & 0.02 & **0.43** & 0.02 & 0.26 & 0.02 & 0.02 & 0.35 \\ in-country & 1 & 0.23 & 0.33 & 0.04 & 0.19 & 0.06 & **0.51** & 0.03 & 0.42 & 0.06 & 0.04 & 0.45 \\ country & 5 & 0.26 & 0.35 & 0.05 & 0.12 & 0.06 & 0.5 & 0.04 & 0.45 & 0.09 & 0.04 & **0.51** \\
10 & 0.26 & 0.37 & 0.05 & 0.21 & 0.06 & 0.5 & 0.04 & 0.49 & 0.09 & 0.04 & **0.53** \\ person- & 0 & 0.22 & 0.25 & 0.03 & 0.27 & 0.03 & **0.54** & 0.03 & 0.39 & 0.03 & 0.43 \\ plays- & 1 & 0.31 & 0.36 & 0.08 & 0.39 & 0.09 & 0.37 & 0.07 & **0.49** & 0.09 & 0.08 & **0.49** \\ pro- & 5 & 0.32 & 0.41 & 0.09 & 0.25 & 0.18 & **0.63** & 0.05 & 0.52 & 0.12 & 0.1 & 0.57 \\ sport & 10 & 0.34 & 0.45 & 0.08 & 0.35 & 0.22 & 0.56 & 0.07 & **0.62** & 0.14 & 0.1 & 0.61 \\
0 & 0.06 & 0.23 & 0.03 & 0.12 & 0.04 & **0.61** & 0.03 & 0.3 & 0.03 & 0.35 \\ present- & 1 & 0.27 & 0.41 & 0.05 & 0.1 & 0.15 & **0.74** & 0.05 & 0.34 & 0.18 & 0.05 & 0.56 \\ past & 5 & 0.41 & 0.46 & 0.05 & 0.1 & 0.39 & **0.69** & 0.05 & 0.48 & 0.23 & 0.05 & 0.61 \\
10 & 0.42 & 0.47 & 0.05 & 0.07 & 0.39 & 0.65 & 0.05 & 0.51 & 0.15 & 0.05 & **0.67** \\ product- & 0 & 0.29 & 0.23 & 0.1 & **0.43** & 0.02 & 0.42 & 0.02 & 0.37 & 0.1 & 0.05 & 0.42 \\ by- & 1 & 0.3 & 0.33 & 0.12 & 0.35 & 0.13 & 0.46 & 0.04 & 0.44 & 0.1 & 0.09 & **0.5** \\ company & 5 & 0.34 & 0.36 & 0.15 & 0.38 & 0.12 & 0.47 & 0.08 & 0.49 & 0.15 & 0.1 & **0.56** \\
10 & 0.35 & 0.38 & 0.15 & 0.41 & 0.12 & **0.55** & 0.08 & 0.49 & 0.17 & 0.08 & 0.54 \\ \hline \hline \end{tabular}
\end{table}
Table 11: GPT-j perturbation test on tasks of natural questions.

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline Task & n-shots & Random & Index [\(\leftarrow\)—] & Index & CM2 [\(\leftarrow\)—] & CM2 & CM1 [\(\leftarrow\)–] & CM1 & FA [\(\leftarrow\)—] & FA & RA [\(\leftarrow\)—] & RA \\ \hline company- & 0 & 0.17 & 0.23 & 0.04 & 0.2 & 0.08 & 0.03 & 0.03 & 0.3 & 0.05 & 0.03 & **0.32** \\ company- & 1 & 0.11 & 0.26 & 0.04 & 0.19 & 0.05 & 0.24 & 0.03 & 0.28 & 0.04 & 0.03 & **0.33** \\ lq & 5 & 0.2 & 0.3 & 0.05 & 0.24 & 0.04 & 0.12 & 0.03 & 0.31 & 0.04 & 0.04 & **0.37** \\  & 10 & 0.19 & 0.32 & 0.05 & 0.26 & 0.05 & 0.11 & 0.04 & 0.3 & 0.04 & 0.04 & **0.39** \\  & 0 & 0.31 & 0.23 & 0.05 & 0.22 & 0.05 & 0.05 & 0.05 & **0.58** & 0.07 & 0.08 & 0.43 \\ country- & 1 & 0.39 & 0.42 & 0.08 & 0.34 & 0.1 & **0.71** & 0.05 & 0.62 & 0.11 & 0.08 & 0.52 \\ capital & 5 & 0.41 & 0.43 & 0.07 & 0.32 & 0.1 & 0.33 & 0.06 & **0.61** & 0.11 & 0.07 & 0.57 \\  & 10 & 0.41 & 0.43 & 0.06 & 0.28 & 0.23 & 0.34 & 0.05 & 0.53 & 0.09 & 0.07 & **0.57** \\ landmark- & 0 & 0.12 & 0.33 & 0.04 & 0.19 & 0.05 & 0.44 & 0.04 & 0.48 & 0.06 & 0.05 & **0.51** \\ in-country & 1 & 0.13 & 0.38 & 0.05 & 0.19 & 0.06 & 0.33 & 0.04 & 0.48 & 0.07 & 0.04 & **0.54** \\ country & 5 & 0.15 & 0.42 & 0.06 & 0.24 & 0.06 & 0.12 & 0.05 & 0.5 & 0.08 & 0.05 & **0.66** \\  & 10 & 0.16 & 0.44 & 0.06 & 0.22 & 0.09 & 0.34 & 0.05 & 0.43 & 0.06 & 0.05 & **0.57** \\ person- & 0 & 0.21 & 0.33 & 0.04 & 0.27 & 0.04 & 0.39 & 0.04 & 0.43 & 0.04 & 0.04 & **0.48** \\ play- & 1 & 0.28 & 0.37 & 0.09 & 0.24 & 0

Attention Patching

In this section, we provide additional information about the implementation of attention patching.

Learning rateAttention patching involves adding (injecting) attention maps into the forward pass of a given model. The attention maps that are injected can be taken from the forward pass or from the RA, each time after being multiplied by a learning rate. Figure15 demonstrates the effect of different learning rates on the success of attention patching. If we inject forward pass maps, a successful learning rate is a positive number. For reversed attention, a negative learning rate would improve the model. In our experiments, we used a learning rate of \(1\) for the forward pass attention, and for the reversed attention, we used a learning rate of \(-30\) for all models. Although we found that different learning rates might affect the models' performances, we decided not to tune the learning rate per task or model since our goal is to show proof of concept rather than optimizing the model accuracy in our experiments.

TasksThe bottleneck of attention patching is the need to have the same length and format of examples. We use the ICL tasks from section B to construct sets of examples with the same length. For each task, we randomly pick one of those sets and use it in our experiments. Following the perturbation test AppendixB, we used 25 examples to average the forward or reversed attention maps for each attention head.

Additional results and discussionThe full patching results are presented in Table13.

The results show that RA patching can achieve similar results to ICL prompting, without the need to provide examples for the model to learn from. Surprisingly, forward attention also shows improvement over the model's original accuracy, but not a consistent one across all tasks. One possible reason for this is the ICL setup, in which the model does not identify the task it needs to perform, and hence returns irrelevant answers. When averaging the forward pass attention scores, it might aggregate multiple different forward passes, amplifying common patterns and reducing noise attribute to the lack of context in the model's prompt. In this sense, forward attention patching can be seen as a form of regularization or a compression of multiple examples' attention scores into one.

To the best of our knowledge, intervention on the attention maps has yet to be explored directly as our implementation. We used RA as a method to inject the attention maps we suspect would improve the model's performance. The injection of the forward pass, despite its surprisingly partial success, was added as a baseline to the RA patching. Future work might explore more complex methods of injecting attention maps, potentially overcoming the need for a constant size of attention maps during the patching process.

## Appendix D Additional Implementation Details

ComputeAll experiments were conducted on standard Nvidia-A40 series GPU.

Models:All LM models were accessed via HuggingFace transformers stack Wolf et al. (2019).

 



\begin{table}
\begin{tabular}{l c r r r r|r l r