Overthinking the Truth: Understanding how Language Models Process False Demonstrations
Authors: Danny Halawi
Jean-Stanislas Denain
Jacob Steinhardt
Venue: ICLR 2024 spotlight (2024)
Keywords: Mechanistic Interpretability
AI Safety
Interpretability
Science of ML
few-shot learning
Large Language Models
Primary area: visualization or interpretation of learned representations

Abstract: Modern language models can imitate complex patterns through few-shot learning, enabling them to complete challenging tasks without fine-tuning. However, imitation can also lead models to reproduce inaccuracies or harmful content if present in the context. We study harmful imitation through the lens of a model’s internal representations, and identify two related phenomena: overthinking and false induction heads. The first phenomenon, overthinking, appears when we decode predictions from intermediate layers, given correct vs. incorrect few-shot demonstrations. At early layers, both demonstrations induce similar model behavior, but the behavior diverges sharply at some “critical layer”, after which the accuracy given incorrect demonstrations progressively decreases. The second phenomenon, false induction heads, are a possible mechanistic cause of overthinking: these are heads in late layers that attend to and copy false information from previous demonstrations, and whose ablation reduces overthinking. Beyond scientific understanding, our results suggest that studying intermediate model computations could be a promising avenue for understanding and guarding against harmful model behaviors.

Published as a conference paper at ICLR 2024
OVERTHINKING THE TRUTH: UNDERSTANDING HOW LANGUAGE MODELS PROCESS FALSE DEMONSTRA- TIONS
Danny Jean-ss, Denain", and Jacob Steinhardt
UC Berkeley
[dhalawi, _s__ennan, jsteinhardt @@erreley.edu
ABSTRACT
Modern language models can imitate complex patterns through few-shot learning, enabling them complete challenging tasks without fine-tuning. However, imita- tion can also lead models to reproduce inaccuracies or harmful content if present in the context. We study harmful imitation through the lens of model's internal rep- resentations, and identify two related phenomena: overthinking and false induction heads. The first phenomenon, overthinking, appears when we decode predictions from intermediate layers, given correct incorrect few-shot demonstrations, At early layers, both demonstrations induce similar model behavior, but the behavior diverges sharply at some "critical layer", after which the accuracy given incorrect demonstrations progressively decreases. The second phenomenon, false induction heads, a possible mechanistic cause of overthinking: these are heads in late layers that attend to and copy false information from previous demonstrations, and whose ablation reduces overthinking. Beyond scientific understanding, our results suggest that studying intermediate model computations could be promising avenue for understanding and guarding against harmful model behaviors."
INTRODUCTION
key behavior of modern language models is context-following: large-scale transformer models are able to infer and imitate the patterns in their prompt (Brown et al., 2020). At its best, this allows language models to perform well on benchmarks without the need for fine-tuning (Rae al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022; Srivastava et al., 2022). This has led researchers to study how context affects few-shot performance (Min et al., 2022; Kim et al., 2022; Xie et al., 2021; Zhao et al., 2021) well as the internal mechanisms that produce it (Olsson et al., 2022).
However, context-following can also lead to incorrect, toxic, or unsafe model outputs (Rong, 2021). For example, if inexperienced programmer prompts Codex with poorly written vulnerable code, the model is more likely to produce poorly written or vulnerable code completions (Jones & Steinhardt, 2022; Perry et al., 2022). Intuitively, the issue is that context-following learns too much-in addition to inferring the overall intent of the in-context task (what code a user is trying to write), it also learns the pattern of user errors and reproduces it, similar how gradient-based learning algorithms reproduce label errors in their predictions (Sambasivan et al., 2021).
In this work, we seek better understand harmful context-following. Since models often perform well zero-shot, we conjecture that when presented with a harmful context, the model knows the right answer, but imitates and says the wrong answer (Meng et al., 2022a). This lead us study how incorrect imitations emerge over the course of the model's processing, and to look for the model components that cause them.
To investigate this, we set up contrast task, where models are provided either correct or incorrect labels for few-shot classification (Figure 1, left). We study the difference between these two settings
Equual contribution
All code needed to reproduce our results can be found at https http://ttthub...mcmmoomm over/ overth nki ng the_ttttttte thuh truth

---

Published as a conference paper at ICLR 2024
CaCorrett Demes Incorrect Demos
Dimonstrazimna
0.9
Caras Demus
Iuumeci
soaring.
Fositive.
Caliboreed Accurncy
Calibmted Accuracy
Text: CEO fined
Nepsiva
0.75
Randen Haseline
0.75
Tot:: Profiteble! Scntiment:
GF7 Positive..
0.6
Inswrredt Demonsiirnioos
0.45
0.15
Text: Stocks svaring.
Sentiment: Negativn
Tex: CEO fired team.
Sentimett: Positicn.
0.3
0.3
Teat: Profiruble! Sentiment:
GF? Negative.
0.15
0.15
5
10
15 Layer
20
25
No
Random
Our
Intervention
Heads
Given prompt of incorrect demonstrations, language models are more likely to output incorrect labels. Center: When demonstrations are incorrect, zeroing out the later layers increases the classification accuracy, here on heads and remove them from the model: this reduces the effect of incorrect demonstrations by 32.6% on Ffnancial-PPrasebante Phrasebank, without decreasing the accuracy given correct demonstrations.
Figure Left:
Right: We identify attention
by decoding from successively later layers of the residual stream (Nostalgebraist, 2020) (Figure 1, center). Intuitively, this allows to decode the model's intermediate predictions as iteratively builds its final output, and to determine which stages of computation propagate the incorrect labels.
We find that correct and incorrect demonstrations yield similar accuracy at early stages of computation, until some "critical layer" at which they sharply diverge. After the critical layer, performance improves given correct demonstrations but drops given incorrect demonstrations. particular, when demonstrations are incorrect, the neural network "overthinks" (Kaya al., 2018); stopping the model early increases its accuracy.
We localize overthinking to specific attention heads that attend to and reproduce previous incorrect demonstrations, analogous to the "induction heads" identified in Olsson et al. (2022). These heads concentrated in the later layers of the model (after the critical layer), perhaps because they attend to complex features (the correctness of an example) that are not present in earlier layers. Removing 5 such heads (1% of heads) reduced the accuracy gap between correct and incorrect prompts by average of 38. 38.9% over 14 datasets, with negligible effects on the performance given correct prompts (Figure 1, right).
In summary, we found that harmful context-following only appears late in a model's computation, and identified specific attention heads that contribute these incorrect imitations. More generally, our findings suggest that benign and harmful model behaviors are often processed differently. Indeed, follow-up work (Belrose al., 2023) has used and extended our insights detect prompt injection attacks (Perez & Ribeiro, 2022). To proactively understand and reduce harmful model behaviors, researchers should continue build tools to understand their intermediate computations.
2 RELATED WORK
Our work is related Min al. (2022), Kim al. (2022), and Wei al. (2023), who examine the role of inaccurate demonstrations on model accuracy. Min et al. (2022, figure 4) find that for the pre-trained model GPT-J, the correctness of demonstrations has a large effect on classification accuracy. These works measure the input-output behavior of models on misleading prompts, whereas our work investigates model internals: early-exiting allows to study how the model builds its representations, and our ablations make it possible to understand the role of specific attention heads.
This high-level perspective matches that of recent work in mechanistic interpretability (Cammarata et al., 2021: Geiger et al., 2021; Elhage et al., 2021), which analyzes model internals to reverse engi- neer the algorithms learned by the network. Mechanistic interpretability techniques have previously been used to study behaviors such modular arithmetic (Nanda et al., 2023), or factual recall (Meng et al., 2022a;b). However, we take a less "bottom-up" approach than most mechanistic interpretability work: focus the role of layers and attention heads, rather than lower-level components such as individual neurons or key, query and value vectors. Moreover, mechanistic interpretability techniques typically applied to small scale, synthetic tasks, such indirect object identification (Wang
2

---

Published as a conference paper at ICLR 2024
4x
Acc(Correc)) Acc(Incorrect)
0.9
Average Dutusets
3.5
Average over Datasets
0.75
Permuted Score
0.6
2.5
0.45
0.3
1.5
0.15
0.5
0
5
10
15
25
30
35
10
15
20
30
35
40
Number of Demos
Number of False Demos
Figure 2: GPT-J behavior in the permuted labels setting (3.1). Left: The difference in accuracy between correct and incorrect prompts increases with the number of demonstrations. Right: As the number of false demonstrations increases, the model chooses the permuted label (<clas((x:) more often than the other labels, rather than making random errors.
et al., 2022). In contrast, we study model behavior across variety of more realistic tasks, including sentiment analysis, natural language inference, and topic classification.
The literature on early-exiting and overthinking (Kaya et al., 2018; Panda et al., 2015; Teerapittayanon al., 3017; Figurnov et al., 2017; Hou et al., 2020; Liu et al., 2020; Xin et al., 2020; Zhou et al., 2020; Zhu, 2021; 1; Schuster et al., 2022) also investigates decoding from intermediate layers. These works focus using early-exiting improve inference speed, although Mehra al. (2022) also study the accuracy under distribution shift. contrast, we use early exiting to scientifically understand the intermediate steps of the model's computation. Moreover, most early exiting methods modify the training process to allow for early exit, or train additional probes to decode intermediate states. In contrast, the logit lens (Nostalgehraist, 2020), which does not require any extra training decode answers from internal representations.
3 PRELIMINARIES: FEW-SHOT LEARNING WITH FALSE DEMONSTRATIONS
We begin by introducing the setting study: few-shot learning for classification, given demon- strations with correct or incorrect labels. Incorrect demonstrations consistently reduce classification performance, which is the phenomenon that aim study in this work.
Few-shot learning. We consider autoregressive transformer language models, which produce a conditional probability distribution p(ln+1 (1,...; (n) over the next token 4,+1 given previous tokens. We focus on few-shot learning (Brown et al., 2020) for classification tasks: given task instruction u. we sample & demonstrations (input-label pairs) from the task dataset, denoted (æ1, y1), ..., (2x, yk). To query the model on new input æ, we use the predictive distribution p(y 21, 1/1 .... Zk, 3k Z))
classification datasets: SST-2 (Socher et al., 2013), Poem Sentiment (Sheng & Uthus, 2020), Financial Phrasebank (Malo et al., 2014), Ethos (Mollas al., 2020), TweetEval-Hate -Atheism, and -Feminist (Barbieri al., 2020). Medical Questions Pairs (McCreery et al., 2020), MRPC (Wang et al., 2019), SICK (Marelli et al., 2014), RTE (Wang al., 2019), AGNews (Zhang et al., 2015), TREC (Voorhees Tice, 2000), and DBpedia (Zhang al., 2015). We used the same prompt formats in Min et al. (2022) and Zhao et al. (2021) (Table 6, 5). For SST-2 we use the 15 prompt formats in Zhao et al. (Table 7). We also considered a toy dataset, Unnatural, that extends task in Rong (2021). In Unnatural, demonstrations are of the form "[object]: [label]" and the labels are "plant/vegetabll"" "sport", and "animal".
Datasets and models. We consider fourteen
We evaluated 8 pretrained autoregressive language models: GPT-J-6B (Wang & Komatsuzaki,, 2021), XPT2-XL-1.555 (Radford et al., 2019), GPT-NeoX- (Black et al., 2022); Pythia models with 410M, 2.8B, 6.9B, and 12B parameters (Biderman et al., 2023); and Llama2-7B (Touvron al.,

---

Published as a conference paper at ICLR 2024
Conrect Labels Permuted Labels
Zaro Shot
Baseline
Calibrated Accuracy
00.8
0.5
0.6
0.8
0.6
0.5
0.5
0.4
0.1
0.4
0.2
20
25
0.2
10
11.3
0
27
26
0.3
10
20
25
(a) SST-2
(b)
(c) TweetEval-Hate
(d) MRPC
Calibrated Accuracy
1F
0.8
0.6-
0.5
0.8
0.4
0.1
0.4
0.1
0.2
0.2
0.2
0.2
0.
10
01
20
25
10
15
20
20
25
10
Layer
Layer
Layer
Layer
(c) SICK
(f) AGNews
(g) DBPedia
(h) Unnatural
Figure 3: GPT-J early-exit classification accuracies across 6 task categories, given accurate and inaccurate demonstrations (here in the permuted labels setting). Plots are grouped by task type: senti- ment analysis (a-b), hate speech detection (c), paraphrase detection (d), natural language inference (c), topic classification (f-g), and a toy task (h). Given incorrect demonstrations, zeroing out all transformer blocks after layer 16 outperforms running the entire model.
2023). We also evaluated instruction-tuned versions of GPT-2-XL (Gallego, 2023), GPT-J-6B (Cloud, 2023) and GPT-NeoX-20RB (Clive, 2023).
Evaluation metrics. Given our focus on classification tasks, we are interested how often the model assigns higher probability to the true label than any other label. However, model predictions can be very unstable with respect to small prompt perturbations (Gao et al., 2021). To mitigate this variability, we measure the calibrated classification accuracy (Zhao et al., 2021). Concretely, for 2-class classification task, we measure how often the correct label has a higher probability than its median probability over the dataset. Assuming the dataset is balanced, which we enforce by sampling demonstration labels with equal probability, this step has been shown to improve performance and reduce variability across prompts. Calibration for multi-class tasks follows similar procedure, detailed in Appendix A.3.
3.1 FALSE DEMONSTRATION LABELS DECREASE ACCURACY
We first set up our contrast task and confirm that the models we study exhibit false context-following behavior. Concretely, we compare the performance of models when the demonstration labels are all correct, i.c. = class(z;), and when they are all incorrect, i.c. permutation a over the set of classes (Figure 1, left). In particular, inputs from the same class are always assigned the same (possibly incorrect) label within each prompt. Because all few-shot labels permutation of the classes, we call this the permuted labels setting.
(cclass(z;)) ), for a cyclic
are chosen according
For each model and dataset, we sample 1000 sequences each containing k demonstrations and evaluate the model's ss calibrated accuracy. We sample different demonstrations (ar;, ys) and label permutations for every sequence, and vary l from 0 to 40 (from 0 to 20 for GPT2-XL, due to its smaller context size).
Figure 2 (left) shows the difference between GPT-J's calibrated accuracy given correct and incorrect prompts as the number demonstrations increases. As expected, incorrect demonstrations lead to worse performance, and the accuracy gap tends increase with & for most datasets. These results are in agreement with Min et al. (2022), who found that incorrect demonstrations decreased GPT-J's performance classification tasks (Min et al., Figure 4).
4

---

Published as a conference paper at ICLR 2024
0.7
0.7
Calibrated Accuracy
Permured Lubels
Rancons Labels
0.6
Hall Conect Label
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
25
Layer
Layer
Layer
(a) GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 4: Average calibrated accuracy across 14 tasks for GPT2-XL (a), GPT-J (b), and GPT-NeoX (c). Early-exiting outperforms running the entire model when the demonstrations contain permuted, random, or half correct labels.
Models could lose accuracy by copying the incorrect label, or by becoming confused and choosing random labels. To confirm it is the former, also measure which labels the model chooses for tasks with more than labels. Specifically, we measure the permuted score: how often the model chooses the permuted label a (xlass(x:) over the other labels. For each dataset, random classifier would have a permuted score of #labels To make the results comparable across datasets, we divide the permuted scores by this random baseline. Figure 2 (right) shows these normalized permuted scores for GPT-J on the 9 multi-class datasets in our collection, as well as the average across datasets. The permuted score increases steadily and reaches twice its initial value after 40 demonstrations.
3.2 RANDOM AND PARTIALLY CORRECT LABELS LEAD TO LOWER ACCURACY THAN CORRECT LABELS
In the previous subsection, presented particular kind of misleading prompt, in which all demon- stration labels are chosen according to permutation of the classes. To study other kinds of misleading prompts, consider variations this setup: prompts in which half the demonstrations have correct labels and half have permuted labels (half correct labels), and prompts where each demonstration label chosen random (random labels). These prompts also lead worse classification accuracy compared to true demonstrations: the accuracy gap for GPT-J at k = 40 is 0.15 for random labels and 0.12 for half correct labels, which is around half the value for permuted labels (0.28).
ZEROING LATER LAYERS IMPROVES ACCURACY
In this section, to study false followin.. we decode model predictions directly from intermedi- ate layers. This allows us evaluate the model's performance midway through processing the inputs. incorrect demonstrations, we find that the model performs better midway through processing, especially for GPT-J, and investigate this phenomenon in detail.
Intermediate layer predictions: the logit lens. Given an autoregressive transformer language model with L layers, we decode next-token probabilities for each intermediate layer, using the "logit lens" method (Nostalgebraist, 2020). Intuitively, these intermediate distributions represent model predictions after é {1, L} layers of processing.
In more detail, let 1,69) E Rdd denote the hidden state of token I, layer E, i.e. sum of everything up to layer in the residual stream. For sequence of tokens (1, In E V, the logits of the full model's s predictive distribution p(tx+1 fi...., tx) are given by
logit .... logit We LayerNormhh (")))
where LayerNorm is the the pre-unembedding layer normalization, and Wy R/V[xd is the embedding matrix, The logit lens mimics this operation, but replaces hy with an intermediate

---

Published as a conference paper at ICLR 2024
state he. This yields the intermediate layer distribution pe(fnta [t,,..,,)) fn), defined as 1o1 = We LayerNorm(h (*)).
This provides measurement of what predictions the model represents at layer L, without the need to train a new decoding matrix. It can therefore be interpreted as a form of early exiting (Panda et al., 2015; Teerapittayanon et al., 2017; Figurnov et al., 2017).
We compute the intermediate layer distributions pe for our 11 language models, and measure the corresponding calibrated accuracies on the fifteen datasets from Section 3. Figure 4 shows the average accuracy of 3 of our 11 models over the fourteen non-toy datasets as a function of f, given demonstrations with correct labels, permuted labels, random labels, half correct labels, well as no demonstrations (we show these results for other models in 7).
Accurate and incorrect demonstrations sharply diverge at "critical layers". Given correct demonstrations, the accuracy tends to increase with layer depth. With permuted or random labels, the accuracy follows a similar trend early layers, but then diverges and decreases at the later layers. This trend is consistent across individual datasets (Figures 3, Figures 9-19).
Moreover, for each model, the accuracies for correct and incorrect prompts diverge at the same layers across almost all datasets: we call these the critical layers. For example, for GPT-J, the accuracies diverge between layers 13 and 14 for all but two datasets (Figure 10)². We observe similar results for the other 10 models: for example, for Pythia-6.9D with layer 9, for Llama-2-7B with layers 13 to 17, and for GPT-NeoX-288 ---------B--nnntee with layers 10 to 13 (Figures 14, 16, and 19).
Early-exiting improves classification performance given incorrect demonstrations. Given incor- rect demonstrations, we observe "overthinking". decoding from earlier layers performs better than decoding from the final layer. For example, for GPT-J, using P16 (the first 16 layers) achieves a better accuracy than the full model on all but one dataset (Figures 3, 4b). Early-cxiting also outperforms the full model for our 10 other models: in particular, overthinking does not seem to be affected by model size (Figure 7 (a-d)) or instruction tuning (Figure 7 (e-g)). Furthermore, overthinking is not a result of undertraining. In contrast to our other models, Llama2-7B was trained using the scaling laws from (Hoffmann et al., 2022). yet P1o outperforms the full model for all 14 datasets. Finally, early exiting also helps for other misleading prompts: our results were qualitatively similar given random labels and half correct labels (see Figure 4 and 7).
Ablating attention heads only improves accuracy further. We hypothesize that correct and incor- rect demonstrations diverge at the critical layers because the correctnessness of each demonstration is only encoded after these layers. This would imply that overthinking is caused by the late attention layers, which attend back to the late layers of previous demonstrations. To test this, we zero out only the attention heads (and not the MLPs) in late layers. When overthinking is most pronounced (e.g. for GPT-J), we find that ablating just the attention heads has similar effect to ablating the entire layer, whereas ablating just MLPs has a much smaller effect (Table 2). Since removing only late attention heads recovers almost the full effect of carly-exiting, conclude that these late heads, more than MIPs, are responsible for overthinking. This motivates understanding the attention heads in detail, which we turn to next.
5 ZOOMING INTO ATTENTION HEADS
Previously, we found that the gap between true and false demonstrations is predominantly due to attention heads in the later layers of the model. This suggests that false contexi-following due to heads attending to complex features in previous demonstrations. In this section, we look for particular heads that are responsible for this context- followinf behavior.
Drawing from Olsson et al. (2022), we hypothesize that there are are false induction heads that attend to false labels in similar past demonstrations, and make the model more likely to output them. For example, for the input "beet" in Figure 5, the right-most head attends consistently to the previous incorrect demonstrations of the token "sport".
More formally, we introduce three properties that make a head a false induction head. First, it should be (1) label-attending, i.e. concentrate its attention on labels in the previous demonstrations. Second,
We formalize this by measuring the layer at which the accuracy gap first reaches half of its final value.

---

Published as a conference paper at ICLR 2024
tomato
sport
tomato
sport
tomato
sport
garlic:
sport
garlic: sport
garlic sport
llama:
plant
able
llama plant
ve get able
llama plant
ve get able
kale:
sport
kale
sport
kale sport
rugby?
animal
rugby:
animal
rugby: animal
beet
sport
beet
sport
beet
|
sport
Figure 5: Examples of attention patterns on incorrect demonstrations from the toy Unnatural dataset, for heads that are label-attending but not class-sensitive (Left), heads that are class-sensitive but not label-attending (Center), and heads that are both label-attending and class-sensitive (Right).
it should be (2) class-sensitive, meaning attends specifically to labels that follow inputs from the same class (e.g "tomato", "garlic" and "kale" in Figure 5). Finally, it should be (3) label-promoting, meaning it increases the probability of the labels it attends to.
To identify false induction heads, we define a score that quantifies how label-attending and class- sensitive attention head is (we will return the label-promoting property at the end of this section). For sequence of demonstrations (zt, y:) and final input æ, the prefix-matching score (PM") of head h is:
PM Σ Att (x. Vi)
Att (6)
# #labels
This score is high when the head attends strongly to the labels following inputs from class(z) (first term), and low when the head attends to the labels following other imputs (second term). We compute the wrefix-matchinn
score of each head by averaging over incorrect prompts on the Unnatural dataset, and plot the distribution of PM scores across each layer (Figure 6). For all models, the scores remain low at early layers, then increase around the critical layers that we identified Section 4. This lends correlational support to hypothesis that false induction heads cause false context-following.
Ablating false induction heads. However, we are interested in causal evidence. Therefore, we check whether removing false induction heads reduces false context-following We select the 5 heads from GPT-J with the highest PM scores, and ablate them by setting their values to zero. We evaluate the resulting lesioned model on all 14 datasets, comparing its layerwise performance to the original model's. s. As a control baseline, we also perform the same analysis for heads selected at random.
Our ablations significantly increase accuracy given incorrect demonstrations: they reduce the gap between correct and incorrect prompts by an average of 38.9%, with only a small loss in accuracy for correct demonstrations (Table 1). In contrast, ablating random heads barely improves the accuracy given false demonstrations, and sometimes even increases the size of the accuracy gap. These results suggest that false induction heads cause significant fraction of the false context-following behavior. In addition, since false induction heads were identified using only the toy Unnatural dataset but affect context-followee foolowin on all datasets, this implies their behavior generalizes across tasks.
Verifying that our heads are abbel-rromoting So far, we have identified label-attending and dass-sensitive heads and shown that they contribute to false context-following behavior. To test our initial hypothesis, we next check that they are also label-promoting, i.e. that they increase the probability of the false labels they attend to. We therefore study the outputs of our heads to understand how they affect the residual stream, focusing here on the Unnatural dataset.
We follow the methodology in Wang al. (2022) to apply the logit lens to each head individually, by applying layer normalization followed by the unembedding matrix to its outputs. This tells us how much the head increases or decreases the intermediate logits of each token. For every head, we define its false label promoting score the difference between the logit increases of the permuted and correct labels. A high score means that the head greatly increase the probability of the permuted label, whereas score of zero means that it promotes the correct and permuted labels equally.
7

---

Published as a conference paper at ICLR 2024
5-
3.2g
22.
Sum PM Gup
Sum PM Scores
0.75
2.4
16.5
0.75
of Final Gap
2.5
0.5
1.6
0.5
0.5
1.25-
0.25
0.8
0.25
5.3
0.25
5
10 15
20
25 30 Layer
35
40
45
10
15 Layer
20
25
28
10
15
20
25 30 Layer
35
40
GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 6: Sum of prefix-matching scores for GPT2-XL. (a), GPT-J (b), and GPT-NeoX (c) on the toy Unnatural dataset. The prefix-matchhe scores increase where the accuracy gap (averaged over tasks)
nessstaaee and doorgee S www.eurooroooooooorrrooa mor of 6.5: they increase the permuted label logit by 6.5 more than the correct label on average. In contrast, when sampling 100 sets of 5 random heads, we find an average score of -0.04, with a standard deviation of 0.41. These results confirm that our label-attending and class- heads are indeed false induction heads.
In summary, our results validate our hypothesis at the beginning of this section: we found a small number of false induction heads in the later layers that contribute to false context-following. by attending to false labels in past demonstrations, and increasing their probability.
6 DISCUSSION
In this paper, we studied why language models imitate incorrect demonstrations in their context. By extracting predictions from intermediate model layers, we showed that models overthink: given incorrect prompts. the final layers hurt its performance. We then identified small number of false induction heads that attend and reproduce false information from past demonstrations, and showed via a lesion study that they contribute to incorrect imitation.
How does the logit lens compare to probing? Our work, especially Section 4, relies heavily on the "logit lens" (Nostalgebraist, 2020). We find it useful to think of this method in comparison to probing.
If a layer has a high probing accuracy, this means that the correct answer can be decoded from the hidden states. However, this is often a low bar to clear, especially when the classification task is easy and the hidden states are high-dimensional (Hewitt Liang, 2019). In contrast, if a layer has a high logit lens accuracy, this shows that it encodes correct answers along direction in the residual stream that the model subsequently decodes from, which is more meaningful. In particular, it implics a high probing accuracy, but the reverse is not necessarily truc.
One intermediate between probing and zeroing out later layers is the tuned lens (Belrose et al., 2023): instead of training new probe for each classification task directly using the final layer's decoding matrix, Belrose et al. train a single universal "translator matrix" for each layer on a language modelling dataset such the Pile (Gao al., 2020). Inspired by our work, Belrose al. applied the tuned lens to our setup, observing overthinking for additional models such as BLOOM-560M).
Semantically unrelated labels.
One hypothesis about the permuted labels setting is that the model simply learns a relabelling of the classes, and is not sensitive to the substance of the incorrect labels. If this were true, we would observe the same logit lens predictions for permuted labels and for semantically unrelated labels (Wei et al., 2023), i.e. labels that have no relation to the task. However, this is not the case: for SST-2, we tried replacing the demonstration labels "Positive" and "Negative" by "A" and "B", and measured the logit lens accuracies in this new setting given incorrect demonstrations (see Figure 10o). While we observe overthinking for related as well as unrelated labels, aarly-exiting achieves higher than random accuracy for SST-2, but not for its variant. This shows that the ground-truth of demonstration labels is an important factor in our results.
Realism of setting. While find consistent results across 14 datasets, our experiments are restricted to a specific setting: text classification with a large number of incorrect few-shot examples. Nevertheless, we believe that the permuted labels setting captures important properties of
8

---

Published as a conference paper at ICLR 2024
Table 1: Ablating false induction heads recovers significant fraction of the accuracy gap between correct and incorrect prompts, without hurting performance given correct demonstrations. We show the percent reduction in the accuracy gap ("Gap") and absolute change in correct prompt performance ("TP") when ablating the 5 false induction heads chosen using the Unnatural dataset ("top") 5 random heads ("random"), Subscript numbers denote standard error. We bold gap reductions when they are greater for our heads than for the random heads. We show results for one dataset in each task category; full results are Table 3.
Dataset
Heads
Permuted Labels
Half Permuted Labels
Random Labels
A (P) (t)
A Gap (t)
A (1)
A Gap (t)
A TP (t)
A Gap (t)
top
1.67p.01
30.76g.20
2.430.02
66.36o.11
1.63p.02
38.97o.15
Poem-Sen Beet-Sentiment random
1.470.01
4.68p.05
1.270.01
17.40p.07
0.37p.01
-17.08p.12
top 6.00p.07
28.61o.0
4.20p.06
5.210.04
3.200.04
- -1.190.01
Ethos
random 33.00..u4
5.97u.os
0.60p.u1
7.290.04
1.40u.02
-2.88p.u1
top 89.02 89.02o. -5.70..2 19
1.200.01
7.690.01
0.01o.01
115. 756.99000
MRPC
random
-3.50..01
23.170.05
- -1.00..01
- -38.460.05
0.60p.o1
47.37
top -3.630.03
15. 15.290.1r
-9.43.07
19.680.14
-6.20..01
10.97o.08
SICK
random
2.270.02
-2.820.01
- -1800.02
-10.99..08
0.130.01
-0.5l..01
top random
2.400.06
32.34o. 20
0.80p.02
46.59..12
1.300.03
3..77
AGNews
2.700.06 - 111.06..11
1..10..03
9.09o.01
1.500.04
6.490.04
top random
- -1.260.02
38 38.980.16
-1.360.01
15.14.
-1.880.01
31.47o.07
Average
0.790.02
3.97o.03
-0.40..01
- -16.50..01
0.37p.01
- -18.740.01
realistic failure modes. Indeed, humans often err consistent, systematic ways. For example, an inexperienced coder might consistently use the wrong method name, thereby permuting the method names in their prompts to a code completion model.
Moreover, findings provide valuable information to understand misleading prompts beyond the permuted labels setting. Indeed, Belrose et al. (2023) drew inspiration from our work to detect another failure of large models: "prompt injection" (Branch et al., 2022). We ran preliminary analysis of the intermediate predictions in this setting, and found that injected prompts, like incorrect demonstrations, exhibit overthinking (see Figure 26).
Ablations on true prefix. Surprisingly, we find that even with correct demonstrations, models have a tendency to overthink. When removing late layers and late attention in GPT2-XI, observed a benefit in performance. Furthermore, early exiting at the critical layer improves performance majority of datasets across all models. This signifies potential misalignment between the pretraining objective and the downstream few-shot task, which is an interesting direction for future study.
Limitations and future work. Our head ablations do not fully remove the accuracy gap between correct and incorrect demonstrations. This could be because we did not identify some of the model components that cause false context-follood followt-foloowing However, there is another possibility: an attention head's outputs are average far from zero, zeroing out that head takes the intermediate states off-distribution, which can decrease overall performance. Thus, one promising future direction would be to replace head outputs by their average value, as in Nanda et al. (2023).
Our work relates to mechanistic interpretability, which seeks to reverse engineering model behaviors from a bottom-up understanding of low-level components. In contrast, embrace more top-down strategy, extracting predictions from entire layers. This shift not only enhances efficiency, compute, and time, but also allows to scrutinize model behavior more realistic tasks. Our results suggest that aberrant and normal model behaviors are often processed differently, so more comprchensively measuring model internals could help us understand and fix broad variety of unwanted behaviors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Thanks Erik Jones, Collin Burns, Nora Belrose, Lisa Dunlap, Alex Pan and our anonymous reviewers for helpful comments and feedback. DH was supported by an award from the C3.ai Digital Transformation Institute. JSD is supported by the NSF Division of Mathematical Sciences Grant No. 2031899. JS was supported by the National Science Foundation SaTC CORE Award No. 1804794 and the Simons Foundation.
REFERENCES
Francesco Barbieri, Jose Camacho-Collados. Luis Espinosa Anke, and Leonardo Neves. Tweet- Eval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, nov 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2020 10.18655511122200mddmmnn-mm Aindings-emnl..45. 148. URL https: oor.laannntoooooooor/200000 wrt///20..0000 findings emn 148.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20BB An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https: /awxivaiiiiv///aa//aaa/aaaaaalla org / 0674400775.
Hezekiah J. Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility pre- trained language models via handerafted adversarial examples, 2022. URL https: //axxiv.. org/ /rg/abs// 2209.22.22 02128.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. doi: circuits.
https://disilllpubb//000oicruii/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Jordan Clive.
Instruction tuned gpt-neox-20b, 2023, URL https://uuggingface
NLP Cloud. Instruct--pt-j-ppp6, 2023. URL http::///ogggngffeec..///////l
instruct-gpt-j---p1..
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http::/transformee-ccccaiii http://ciciiiiioo/////aa//oooonn html.
10

---

Published as a conference paper at ICLR 2024
Michael Figurnov, Artem Sobolev, and Dmitry Vetrov. Probabilistic adaptive computation time, 2017.
Victor Gallego. Gpt2 finetuned on the open-instruct--l dataset, 2023. URL https: :
huggingface. co wtw.cccoovvillaeeooooooccccc.caa hatpaalleeept---oe---ssrrrrooe instrutttstttt---
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 101000007 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computationl Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 3816 38300, Online, aug 2021. Association for Computational Linguistics. doi: (0.118633/1/2221lacllooong 295.
URL htwps: http://s///aaaattthloooooo
org arl-llnclllonno
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks, 2021.
J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. In Empirical Methods in Natural Language Processing (EMNLP), 2019. URL https: httpt::t//pp..ssttettttee http://pppsaaaaddddeeeee edu pubs/ hewitt22 9control.pdd
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth, 2020.
Erik Jones and Jacob Steinhardt. Capturing failures of large language models via human cognitive biases, 2022.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking, 2018.
Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Wor Lee, Sang goo Lee, Kang Min Yoo, and Tacuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations, 2022.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and QI JU. Fastbert: a self- distilling bert with adaptive inference time. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 110663//1/2200.almmmia... http:// /ttt:///x... www..oor......o..........oo.o...
URL
18653/86/// 20155/3v12220000 acl-main. 537.
Malo, A. Sinha, P. Korhonen, Wallenius, and Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216 116-223, Reykjavik, Iceland, may 2014. European Language Resources Associ- ation (ELRA). URL http:htth//... /www. rec- f org wttg/rooceee/eeooooooooo proceed ngs/ rec201 4/2220114//////// 363 2ap__aaper___ f.
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Effective transfer learning for identifying similar questions: Matching user questions to covid-19 faqs, 2020.
Akshay Mehra, Skyler Seto. Navdeep Jaitly, and Barry-John Theobald. Understanding the robustness of multi-exit models under common comruttions, 2022.

---

Published as a conference paper at ICLR 2024
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint arXi::2002.05262, 2022a.
Kevin Meng, Arna Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022.
Ioannis Mollas, 7oe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. Ethos: an online
hate speech detection dataset, 2020.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023.
Nostalgebraist. Interpreting gpt: the logit lens, 2020. URL https:tt/////ec.c. http///wwww.wwwo......o com/ /esstt//ccttBBooppaaaanbbntneeee
gpt tapt-the--e the loneeloogt--aa lens.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse. Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. -ecurnngg-and-inuuuee
Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy- efficient and enhanced pattern recognition, 2015.
Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models, 2022.
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with ai assistants?, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 2021.
Frieda Rong. Extrapolating to unnatural language processing with gpt-3's in-context learning: The good, the bad, and the mysterious, 2021. URL htttps://alt http:///a.saaooddd.oeooomo hww.//oooddddd//oogoggggo in wer-coottxt-leermnmee
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyone wants do the model work, not the data work": Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380966 doi: 10. 00.1145/3411764.44555 8. URL https:
ot1/10..0111.. 145/3411715 111434111..44555111. 3445518.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling, 2022.
Emily Sheng and David Uthus. Investigating societal biases in poetry composition system, 2020.
URL https: wawiiviiivv.////ii//aaaa/aaaaavv org/ abs 220/121111. 02686.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL hwtptttts///. / / wnwhclanntllllooooo wow.olllyyoooooooooooeo -
12

---

Published as a conference paper at ICLR 2024
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Gariga-Alona et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 2:Xiv:2206.04615. 2022.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks, 2017.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ellen M. Voorhees and Dawn M. Tice. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC Athens, Greece, May 2000. European Language Resources Association (ELRA). URL Utthttp:
//www, www.1rec-com wwwpe/ccooooroopooeeeeeooeo wtt.o//ccooeedgggee.eo.... rtc200e/pdf /t/
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: / com/kingoflolz /ewsh-eeeeeeeee mmnehhtranseee transformer jax, May 2021.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL
https::/arrxn org/ abs// 2211.1111 00593.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang. Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently. 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference, 2021.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2015. URL https::t///t http//ixvvvv..////////.aa org/ 1ot.b.///1..... 0266.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit, 2020.
Wei Zhu. Leebert: Learned early exit for bert with cross-level optimization. In ACL, 2021.
13

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 LOGIT LENS RESULTS FOR OTHER MODELS
Figure 7, we have plotted the average Logit Lens results for our other models across tasks.
Carnel abbls
Penmuted abbels
Half Corratl I Labek Zero Shot
Rander Labele
Biseline
Calibrated Accuracy
0.6-
0.5
1.6
0.8
0.5
0.5
0.5
0.5
00.3
0
10
0.3
20
25
11.3
15
30
0.3
(a) Pythia-410M
(b) Pythia-2.88
(c) Pythia-6.9B
(d) Pythia- 12B
Calibrated Accuracy
0.8
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.4
0.4
0.4
0.4
00.3
0
15
20
30
0.3
12
0.3
10
15 Layer
20
25
0.3
10
20
30
Layer
Layer
Layer
(e) Llama2-7B
(I) Instructt
(g) GPT-J-Instruct
(h) GPT-NeoX-200 Instr.
Figure 7: Average calibrated accuracy across 14 tasks for Pythia models of different sizes (a-d), Llama2-7B (c), and instruction-tuned versions of GPT2-XL, GPT-J, and GPT-NeoX-200 (f-h), Early- exiting outperforms running the entire model when the demonstrations contain permuted, random, or half correct labels.
14

---

Published as a conference paper at ICLR 2024
Table Average calibrated accuracy on correct and incorrect labels when running the full model, zeroing out late layers, zeroing out late attention heads (but not MLPs), and zeroing out late MLPs (but not attention heads). We ablate after layer 16 for GPT-J, 30 for GPT2-XL, and 32 for GPT-NeoX. The best and second best ablated accuracy are bolded and underlined respectively. Subscript numbers denote standard error. We find that ablating late attention heads and ablating late layers have similar performance: this suggests that late attention heads play especially important role in overthinking.
Model
Permuted Labels
Correct Labels
Full Model Late Heads Late MLP Late Layers
Full Model
Late Hcads
Late MLP
Late Layers
GPT2-XL 41.971.49
46.09..0
42.881.48
44, 44.631.50
54.191.51
54.091.48
52.471.51
53.681.88
GPT-J 37.421.47
47.581.47
17.97
47.72 47.721146 46
65.541.42
64.461.39 333
65. 65.841.40 40
64.00 .11
GPT-NeoX 45.191.47
14.441.47
44
46.061.49
61.681.41
60.86,.4
56.781.94
62.151.42
A.2 ABLATING ONLY ATTENTION HEADS, OR ONLY MLPS
15

---

Published as a conference paper at ICLR 2024
A.3 CALIBRATION
For k-way tasks, we measure how often the correct label has a higher probabilitiy than the R-1- quantile of its probability over the dataset. In figure 20, show the logit lens accuracies of GPÎ-J over the 16 datasets: although the uncalibrated accuracies at earlier layers are much noisier and occasionally indistinguishable from the bascline accuracy, we also find overthinking on majority of datasets.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
P(True)
Figure 8: The probability of the label "True" for 30 random test inputs in MRPC. Inputs from the "True" class are marked with green dots, and inputs from the "False" class are marked with red dots. As observed in Zhao al. (2021), the model can biased towards one the labels: here the model tends to assign higher probability to the "True" label than to the "False" label, for inputs from both classes.
16

---

Published as a conference paper at ICLR 2024
A.4 LOGIT LENS RESULTS FOR OTHER MODELS ACROSS TASKS
We plot the Logit Lens results across all tasks for all models: GPT2-XL (Figure 9), GPT-J (Figure 10), GPT-NeoX-20B (Figure 11), Pythia-410M (Figure 12), Pythia-2.9B (Figure 13), Pythia-6.9B (Fig- ure 14), Pythia-122 (Figure 15), Llama2-7B (Figure 16), GPT22Instruct (Figure 17), GPT-J-Instruct
(Figure 18), and GPT-NeoX-200 20B-Instruct (Figure 19).
Carnect Labals
Zero Shot
Permuted Labels
Baseline
Calibrated Accuracy
0.8
0.5
188
0.8
0.4
0.4
0.4
0.4
10
20
30
10
10
20
30
10
10
20
30
(a) SST-2
(h) Poem-Sentiment
(c) Emancial-Phrasebaan
(d) Ethos
Calibrated Accuracy
0.6
0.6
0.45
0.45
1.45
0.45
00.3
0.3
0.3
0.3
0.15
40
0.15
30
0.15
10
0.15
(e) TweetEval-Haa
(I) SweetEvalAAtheisee
TweetEval TweetEval-Feminnst Femmnnss
(h) MedQ-Pairs
Calibrated Accuracy
0.8
0.6
0.6
11.6
0.6
0.45
0.15
0.1
0.3
0.2
0.2
0.15
40
0.15
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.5
11.8
0.8
0.6
0.6
0.4-
0.4
0.4
0.2
0
10
20
30
40
10
20
30
10
10
20
30
20
30
Layer
Layer
Layer
T. Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 9: GPT2-XL carly-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations. Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 36 outperforms running the entire model on 13 out of 16 datasets.
17

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
1-
00.8
0.5
18.
0.5
0.6
0.4
0.1
0.4
0.3
0.2
0.2
0.2
0.2
20
25
10
15
20
10
25
(a) SST-2
(h) mnem-Sentiment
(c) Smancial-Phrasebban
(d) Ethos
Calibrated Accuracy
0.45
0.45
1.45
0.45
0033
0.3
0.3
0.3
0.15
25
10
20
0.15
25
5
10
(e) TweetEval-Haa
(I) TweetBEal-Athhisee
TweetBval-Feminee
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
0.8
0.8
0.45
0.15
11.6
0.6
0.4
0.3
0.15 11
0.15
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
1
0.8
0.5
0.8
0.6
0.6
0.4
0.4
0.2
0.2
1:2
0.2
0-
02
0-
20
26
10
15
20
26
0
10
16 Layer
20
26
10
20
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 10: GPT-J early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer outperforms running the entire model on 15 out of 16 datasets.
18

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Bascline
Calibrated Accuracy
1-
00.8
0.5
11.
0.8
0.6
0.4
0.1
0.4
0.3
0.2
0.2
0.2
0.2
20
30
20
10
10
20
30
10
10
(a) SST-2
(h) meem-Sentiment
(c)
(d) Ethos
Calibrated Accuracy
0.45
0.45
1.45
0.45
0333
0.3
0.3
0.3
0.15
30
0.15
10
20
40
0.15
30
10
(e) TweetEval-Hae Hate
(I) TweetEval- -ll-tthssm
TweetEvvl--Feming
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
0.8
0.8
0.45
0.15
11.6
0.6
0.1
0.3
0.2
0.15
0.15
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.5
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0:2
0.2
02
0
20
30
-0
10
20
40
10
20
30
10
201
30
40
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 11: GPT-NeoX-20E carly-exit classification accuracies across 16 tasks, given correct and
incorrect demonstrations. Plots grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n). and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 27 outperforms running the entire model on 14 out of 16 datasets.
19

---

Published as a conference paper at ICLR 2024
Carrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
0..8
0.5
18.
0.8
004
0.4
104
0.4
10
15
20
25
20
10
15
20
(a) SST-2
(h) Tnem-Sentiment
(c)
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0.33
0.3
0.3
0.3
0.15
10
0.15
15
20
0.15
10
20
26
(e) TweetEval- Hate
(If) SireetEvll-Ahhiise
Eereeiiall-emmiisttt
(h) MedQ-PPirs
Calibrated Accuracy
0.6
0.6
11.
0.6
0.45
0.15
0.1
0.3
0.2
0.2
0.15
0.15
(i) MRPC
(j) RTE
(k) SICK
(I) AGNews
Calibrated Accuracy
0.8
0.8
11.
0.8
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0:
01
0
10
16
20
15
20
10 16
20
10
Layer
Layer
Layer
T Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 12: Pythia-410M carly-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer outperforms running the entire model on 11 out of 16 datasets.
20

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
0.8
0.5
12.
0.8
004
0.4
184
0.4
15
25
30
10
20
25
15
20
30
10
25
(a) SST-2
(h) Poem-Sentiment
(c)
(d) Ethos
Calibrated Accuracy
0.45
0.45
1.45
0.45
0333
0.3
0.3
0.3
0.15
15
30
0.15
10
20
25
0.15
30
10
25
(e) TweetEval- Hate-
(I) TweetEval-Athein
(g) TweetEval EeeetEEal-FFmmiisst
(h) MedQ-PPirs
Calibrated Accuracy
0.6
0.6
11.6
0.6
0.45
0.15
0.4
0.3
0.2
0.2
0.15
0.15
25
30
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
118
0.8
0.6
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0:
01
0
15
20
26
30
10
16
20
20
30
0
15 20
26
30
10
16
20
20
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 13: Pythia-2p88 early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 13 outperforms running the entire model on 12 out of 16 datasets.
21

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
0.5
12.
0.8
0.8
0.4
184
0.4
0:2
15
20
25
80
10
20
25
15
20
26
30
10
25
(a) SST-2
(h) meem-Sentiment
(c) Financil-Phrasehhee
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0333
0.3
0.3
0.3
0.15
15
30
0.15
10
20
25
0.15
30
10
25
(e) TweetEval-Haan Hate
(I) TweetEval-Athein
(g) TweetEval EeeetEval-FFmmiistt
(h) MedQ-Pairs
Calibrated Accuracy
00.6
0.6
11.6
0.6
0.45
0.15
0.3
0.2
0.2
0.15
0.15
25
30
0-
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
11.8
0.8
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0:
01
0
15
20
26
30
10
16
20
20
30
0
15 20
26
30
10
15
20
20
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 14: Pythia-6p9B early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 5 outperforms running the entire model on 11 out of 16 datasets.
22

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
00.8
0.5
18.
0.8
00.
0.4
18.
0.4
15
20
25
80
5
10
20
25
20
30
10
25
(a) SST-2
(h) Poem-Sentiment
(c)
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0333
0.3
0.3
0.3
0.15
15
30
0.15
10
20
25
0.15
30
10
25
(e) TweetEval-Haa
(I) TweetEval-AAthein
(g) TweetEval Eeeetiial--Fmmiistt
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
11.6
0.6
0.45
0.15
0.1
0.3
0.2
0.2
0.15
0.15
25
30
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
11.8
0.8
0.6
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0.
01
0
15
20
26
30
10
16
20
20
30
0
10 15
20 Layer
26
30
10
16
20
20
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 15: Pythia-122 12B early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer outperforms running the entire model on 11 out of 16 datasets.
23

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
0.5
18.
0.8
0.8
0.4
184
0.4
15
20
25
80
10
20
25
15
20
26
30
10
(a) SST-2
(b) Poem-Sentiment
(c) Financil-PPrrssbaane
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0033
0.3
0.3
0.3
0.15
15
30
0.15
10
20
25
0.15
30
10
15
25
(e) TweetEval- Hate
(I) TweetEval-Atheiie
(g) TweetEval Feeiinisl--ee
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
11.6
0.6
0.1
0.2
0.2
0.2
0.2
0
30
(i) MRPC
(i) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.88
0.8
11.
0.8
0.6
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0:
0p
0
15
20
26
30
10
16
20
20
30
0
15 20
26
30
10
16
20
20
Layer
Layer
Layer
T
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 16: Llama2-7B early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 19 outperforms running the entire model on 16 out of 16 datasets.
24

---

Published as a conference paper at ICLR 2024
Correct Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
00.8
0.5
188
0.8
00.
0.4
184
0.4
0.2
10
12
10
2
12
(a) SST-2
(b) Poem-Sentiment
(c) Financial-Phrasehee
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0.33
0.3
0.3
0.3
0.15
12
0.15
10
8
12
(e) TweetEval-Haar
(I) TweetEval-Atheise
(g) TweetBval-Femingg
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
1.6
0.6
0.45
0.45
0.3
0.2
0.2
0.15
0.15
(i) MRPC
(i) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
118
0.8
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0-
00
0
0
10
12
12
10
12
8
10
12
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 17: GPT2-Instruct early-exit classification accuracies across 16 tasks, given correct and
incorrect demonstrations, Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n). and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer outperforms running the entire model on 12 out of 16 datasets.
25

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Baseline
Calibrated Accuracy
0..
0.5
18.
0.8
00.
0.4
184
0.4
20
25
10
20
21
20
10
20
25
(a) SST-2
(h) meet-Sentiment
(c) Bamancial-Phrsebbare
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
0033
0.3
0.3
0.3
0.15
25
10
20
0.15
25
5
10
(e) TweetEval-
(I) SrretEval-AAthiine
(g) TweetEval-Feminn
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
11.6
0.6
0.45
0.15
0.3
0.2
0.2
0.15
01
0.15
0
(i) MRPC
(i) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
11.8
0.8
0.6
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0-
01
0-
20 26
10
15
20
26
0
10
16
20
26
10
16
20
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 18: GPT-J-Instruct carly-exit classification accuracies across 16 tasks, given correct and
incorrect demonstrations. Plots grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n). and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 17 outperforms running the entire model on 13 out of 16 datasets.
26

---

Published as a conference paper at ICLR 2024
Conrect Labels
Zaro Shot
Permuted Labels
Bascline
Calibrated Accuracy
0..8
0.5
11.
0.8
00.
0.4
184
0.4
20
30
20
10
10
20
30
10
10
(a) SST-2
(h) Prem-Sentiment
(c) Amnancia--Phraebbane
(d) Ethos
Calibrated Accuracy
0.45
0.45
1145
0.45
033
0.3
0.3
0.3
0.15
30
0.15
20
40
0.15
30
10
(e) TweetEval- Hate
(I) AwcetEval-Athhisse
TweetEval- FereeiEvll-Feminsttt
(h) MedQ-Pairs
Calibrated Accuracy
0.6
0.6
11.6
0.6
0.45
0.45
0.1
0.3
0.2
0.2
0.15
0.15
0
(i) MRPC
(j) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
0.8
0.8
11.8
0.8
0.6
0.4-
0.4
0.4
0.4
0.2
0.2
0g
0:
0
20
30
-0
10
20
40
10
20
30
10
201
30
40
Layer
Layer
Layer
1 Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 19: GPT-NcoX-2001 Instruct carly-exit classification accuracies across 16 tasks, given correct
and incorrect demonstrations. Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d-g), paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n). and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 32 outperforms running the entire model on 11 out of 16 datasets.
27

---

Published as a conference paper at ICLR 2024
A.5
LOGIT LENS RESULTS FOR GPT-J WITHOUT CALIBRATION
Correct
Permuied Labels
Baseline
Calibrated Accuracy
0.8
0.5
888
0.8
0.6
0.6
0.6
0.6
0.3
0.2
0.2
0.2
0.2
(a) SST-2
(h) Paem-Sentiment
(c) Phariiial-Phrabbaakkk
(d) Ethos
Calibrated Accuracy
0.6
0.6
0.6
0.3
0.3
0.3
0.3
0.15
0
10
15
20
25
0.15
5
10
15
20
26
0.15
0
10
15
20
25
0.15
5
10
13
20
25
(e)
TweetEval-
(I) TweetEval-Athen
(g) TweetEval-Femint
(h) MedQ-Pairs
Calibrated Accuracy
0.8
0.45
0.15
11.6
0.6
0.4
0.4
0.3
0.3
0.2
0.15
25
10
15
20
25
10
(i) MRPC
(j) RTE
(k) SICK
(I) AGNews
Calibrated Accuracy
1--
0.8-
0.5
0.8
0.8
0.6
0.6
0.0
0.6
0.4
0.4
0.2
02
0
16
20
25
26
20
26
25
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 20: GPT-J carly-exit uncalibrated classification accuracies across 16 tasks, given correct and incorrect demonstrations. The lack of calibration brings early layer performance to baseline for some datasets, but early-exiting still frequently outperforms running the full model.
28

---

Published as a conference paper at ICLR 2024
A.6
LOGIT LENS RESULTS FOR EACH SST-2 PROMPT FORMAT
Calibrated Accuracy
0.8
0.8
0.8
Permuted Labels
Zero Shot
0.6
Random Baseline
0.2-
0.2
0
15
20
25
18
20
25
10
15
20
25
(a)
Id:
(b) Id: 2
(c) Id:
Calibrated Accuracy
0.6
0.6
0.4-
0.4
0.4
0.4
0:2
0.2
0:2
0.2
0-
20
20
10
10
20
20
26
10
15
20
26
(d)
Id: 4
Id:
(l) Id: 6
Id:
Calibrated Accuracy
0.8
0.5
11.8
0.8
0.6-
0.6
0.6
0.8
0.4
0.4
0.4
(h) Id: 8
(i) Id: 9
(i) Id: 10
(k) Td: 11
Calibrated Accuracy
0.8
0.8
0.8
0.8
0.4
0.2-
0.2
0.2
11
01
15 20
25
10
15
30
0
20
25
10 15
20
Layer
Layer
Layer
Layer
(1) Id: 12
(m) Id: 13
(n) Id: 14
(0) Id: 15
Figure 21: Calibrated Accuracy for all 15 prompt formats for SST-2 (from Zhao et al. (2021)). Given incorrect demonstrations, prompt formats 3, 4, 5, 7, 8, 9, 10, and 13 experience an increase in performance before experiencing decline. Prompt formats 6, 12, 14, and 15, on the other hand, do not exhibit this effect. Prompt format 11 produces poor performance, given both correct and incorrect demonstrations. See Table 7 for prompt format details.
29

---

Published as a conference paper at ICLR 2024
A.7
LOGIT LENS RESULTS FOR OTHER METRICS
Correct Labels
Zero Shot
Permuted Labels
Baseline
Uncalibrated Accuracy
0.75
0.75
0.75
0.6
0.6
0.6
0.45
0.45
0.45
0.3
0.3
0.3
10
20
30
40
5
10
15
20
25
10
20
30
40
0.6
0.6
0.6
Top-1 Accuracy
0.4
0.4
0.4
0.2
0.2
0.2
15
20
25
10
30
Permuted Score
1.5
1.5
0.5
0.5
0.5
10
20
30
40
0
10
15
20
25
10
20
30
40
1
Label Space Probs
0.75
0.75
0.75
0.5
0.5
0.5
0.25
0.25
0.25
0
0
0
10
20
30
40
0
5
10
15
20
25
10
20
30
40
Layer
Layer
Layer
(a) GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 22: Uncalibrated Accuracy (row 1), Top-1 Accuracy (row 2), Permuted Score (row 3), and Label Space Probabilities (row 4) averaged over 14 tasks (9 multi-class tasks for the permuted score). As the label space learned, we observe the emergence and ensuing increase the gap in the other metrics.
30

---

Published as a conference paper at ICLR 2024
A.8
ACCURACY GAP AS FUNCTION OF FOR OTHER MODELS
0.9
0.9
Acc(Correct) Acc(Incorrec)
Average Datascts
Ace(Correct) Acc(Incorrec)
Average over Datasets
0.75
0.75
0.6
0.6
0.45
0.45
0.3
0.3
0.15
0.15
0
10
15
20
25
30
35
40
0
5
10
15
20
Number of Demos
Number of Demos
Figure 23: GPT-NeoX (left) and GPT2-XL (right) behavior in the Permuted Labels setting (3.1). The difference accuracy between accurate and inaccurate prompts increases with the number of demonstrations.
31

---

Published as a conference paper at ICLR 2024
A.9 FALSE INDUCTION HEAD ABLATION RESULTS ON ALL TASKS
Table Ablating false prefi--matching heads recovers a large fraction of the accuracy gap between true and false prefixes, without hurting performance given prefixes. We show the percentage reduction of the accuracy gap and percentage change in true prefix performance when ablating the false prefix-matching heads chosen using the Unnatural dataset ("top") or 5 random heads ("random")) We bold gap reductions when they are greater for our heads than for the random heads. Subscript numbers denote standard error.
Dataset
Heads
Permuted Labels
Half Permuted Labels
Random Labels
A TP (f) A Gap (
(T)
A Gap (t)
A (T)
A Gap
Sentiment Analysis
top
5.86p.18
54.56p.42
3.61o.08
88.710.48
4.71g.13
100.62g 100.620.33
SST-2
random
5.460.16
-7.940.01
2.01o.02
23.949.61
3.71p.08
21.43p.59
top random
1.670.01
30.76o.2r
2.43p.02
66.36o.1
1.63p.02
38.97o.15
Beem-Sentiment
1.470.01
4.68p.05
1.27p.01
17.40p.07
0.37
-17.08..12
top
2. 2.30p.0s
32.67u.30
14.72u.10 2.600..
1.43g.03
25
25.340.14
Financial-Phreen Phraseeank
random
2.33p.05
5.89p.os
- 1.93p.04
-1.170.01
2.030.04
4.89g.04
Hate Speech Detection
top 6.00p.07
28.61
28.610.06
4.200.06
5.210.04
3.200.04
1.190.01
Ethos
random 3.00p.04
5.97p.08
0.60p.01
7.290.04
1.40..02
2.38a.01
top 1..00.03
10.63o.0s
4.400.04
35.22
7.200.05
27.
Taett
random 1.50v.01
1.970.02
2.200.02
36.21o.19
3.000.03
15.250.07
TWeeeEEal-Atheism
top
99.00..01
34.030.25
-7.03..01
-11.24p.07
-3.57
9.62o.07
random
- 1.570.00
9.63p.11
0.40p.01
3.35p.02
2.270.01
13 13.21o.08
top 0.43p.01
34.53p.15
0.63p.01
28.533.07
0.770.01
2.68p.01
Feeeinval-Feminist
random 0.03p.o1
5.86p.04
0.50p.01
14.120.04
1.930.01
29.177.14
Paraphrase Detection
MedQ-Pairs
lop random
0.30p.01
36. 36.610.08
-4.90..01
1.85p.01
-1.70..01
28.855.06
2.90p.01
9.82p.03
-0.10..01
5.56p.01
3.10p.02
-1.92
MRPC
top random
5. 70.020
89.02o.19
1.20p.01
7.690.01
0.01g.01
115.790.02
3.50p.01
23.17o.05
1.000.01
38.46p.05
0.60p.01
47.370.03
Natural Language Inference
top random
-3.66 3.63p.a3
15. 15.299.17
-9.43p.07
-19.68..14
-6.20p.04
10.97o.08
SICK
2.27u.02
-2.82p.04
-1.80o.02
110.99..08
0.13p.01
-0.51g.01
top 1.90p.01
95.16p.or
2.00o.01
36.36p.or
3.70p.03
141.670.02
RTE
random
0.50p.01
4.84p.01
2.400.01
218.18p.50
0.70p.o1
291.676.45
Topic Classification
top 2.40p.0s
32.34p.20
- 0.800.02
46.59p.1z
- 1.330.03
33.777.17
AGNews
random
2.70p.06
-11.06p.11
1.100.03
9.090.01
-1.50..4
6.49g.01
top -5.90..02
19.65p.05
-7 -7.00.00
- 28.885..20
8.40g.03
3.73,.oz
TREC
random
2.10p.01
6.74p.11
-0.80..01
-10.26p.06
-0.30..01
0.75g.01
top 2.10p.11
31.83p.ea
1.90p.0a
22.35a.16
-1 -1.30o.07
14.60o.21
DBPedia
random
1.80p.09
-1.130.02
1.900.08
3.530.03
- -1.00..05
1.46g.02
top
1.26p.02
38. 38.98p.16
2.36p.01
15.140.03
1.58..01
31.470.07
Average
random
0.79p.02
3.97o.03
0..00.01
-16.50p.01
0.37
-18.74p.01
32

---

Published as a conference paper at ICLR 2024
A.10
VARYING NUMBER OF FALSE DEMONSTRATIONS
0.7
Calibrated Accuracy
Crrrect Labels
Permuted Labels (PL)
809 PL
0.6
60% PL
40% PL.
20% PL.
Random
0.5
0.4
0.3
0
5
10
15
20
25
Layer
Figure 24: Varying the number of incorrect demonstrations smoothly interpolates between all-correct and all-incorrect demonstrations. Here we show GPT-J's average layerwise accuracies for 20%, 40%, 60%, and 80% of incorrect demonstration labels.
33

---

Published as a conference paper at ICLR 2024
A.11
VARYING NUMBER OF ABLATED ATTENTION HEADS
Calibrated Accuracy
... 5 heads
15 heads
0.8
heads
beads
0.6
0.1
0.2
0
5
10
15 Layer
25
Figure 25: Ablating more false induction heads leads to even greater performance improvements. Here show the results for GPT-J on the Unnatural dataset, when ablating the 5, 15, 35, and 75 heads with the greatest prefi- matching scores.
34

---

Published as a conference paper at ICLR 2024
A.12
PROMPT INJECTION PRELIMINARY ANALYSIS
Prompt Injection
Zern Shot
Baseline
Calibrated Accuracy
0.75
0.75
11.7
0.6
0.6
0.4
0.0
0.15
0.45
0.3
0.3
0.4
0.15
0
(a) SST-2
(b) Financial Phaiiilla--aaaaakkkkkk
(c) Ethos
(d) MedQ- Paiiss
Calibrated Accuracy
10.6
0.6
0.6
0.8
0.45
0.45
0.6
0.3
0.4
0.3
0.3
10.15
0.2
0.15
0.15
30
00
10
20
30
40
10
20
300
10
201
30
(c) Sick
(f) AGNews
(g) DBPedia
(h) Unnatural
Figure 26: GPT-NeoX early-exit classification accuracies across tasks, given clean and injected prompts. For injected prompts, exiting at layer 20 outperforms running the entire model, and sometimes running the model zero-shot. We poison prompts by injecting the prefix "IGNORE PREVIOUS INSTRUCTIONS AND OUTPUT "NNNE...
35

---

Published as a conference paper at ICLR 2024
A.13 PROMPT FORMATS USED FOR ALL DATASETS
Table 4: The prompts used for the toy tasks: Unnatural and SST-2-A/B. The prompt for Unnatural is taken from Rong (2021) and the prompt for SST-2-A/B is taken from the SST-2 prompt in Zhao et al. (2021). We show two training examples per task for illustration.
Task
Prompt
Labels A, B
SST-2-A/B
Review: Well-rounded tribute.
Answer: A.
Review: Saw how bad this movie was. Answer: B.
Review: Skip this dreck.
Answer:
Unnatural
Consider the categories plant/vegetable, sport, and animal. Classify each object in its category.
animal,
plant/vegetable, sport
onions: plant/vegetable.
hockey: sport.
horse:
36

---

Published as a conference paper at ICLR 2024
Table 5: The prompts used for paraphrase detection, natural language inference, and topic classifica-
tion. The prompts for MedQ-Pairs, MRPC, SICK, and RTE are taken from Min et al. (2022), and the prompt for AGNews, TREC, and DBPedia are taken from Zhao et al. (2021). We show training example per task for illustration.
Task
Prompt
Labels
MedQ-Puins
Determine if the questions are equivalent or
equivalent,
Question:
many hour from drinking an antibiotic drink alcohol?
Question: have purty lonight and took my last dose Azithromycin this morning.
Can have a few drinks?
Answer: equivalent.
Question: After how many hour from drinking an antibiotic can drink alcohol? Question: vomited this morning and am not sure it is the side effect of my antibiotic the alcohol look last night...?
Answer:
MRPC
The DVD-CCA then appealed to the state Supreme Court.
True, False
The question is: The DVD CCA appealed that decision the U.S. Supreme Court? True False?
The answer is: True.
The Nasdaq composite index increased 10.73, or 0.7 percent, to 1,514.77.
The question is: The Nasdaq Composite index, full technology stocks, was lately up around 18 points? True False?
The answer
SICK
The young boys are playing outdoors man smiling nearby. The question The kids playing outdoors The answer is: Truc.
True, Falsc,
with smile? True False?
Not
Two people are kickboxing und spectatios watching. The question is: Two people are kickhoxing and spectators are watching? True or False?
The
RTE
The Ameed Forces Press Committee (COPREFA) admitted that the government troops True, False sustained casualties these clashes, adding that they inflicted three casoalties the rebels.
The question is: Three rebels were killed by government troops? True False?
The answer is: True.
Gastrointestinal bleeding can huppen adverse effect of non-stervidal anti-inflammatory drugs such as aspirin or ibuprofen.
The question is: Aspirin prevents gastrointestinal bleeding.
The answer
AGNews
Article: Bush, Republicans Outpoll Kerry, Democrats TV (Reuters) Reuters Although the election
World, Sports, Business, Science
until.
Answer: World.
Article: Baseball Today (AP) AP Chicago Montreal (7:05 p.m. EDT). Greg Maddux (12-8) starts for the Cubs.
Answer:
TREC
Classify the questions based whether their type Number, Location, Person, Description. Entity, Abbreviation.
Description, Entity, Abbreviation. Person, Number, Location
Question: What liver enzymes? Answer Type: Description.
Question: What considered the costliest disaster the insurance industry has ever faced?
Answer Type:
DBPedia
Classify the documents bused on whether they about Company, School, Artist, Athlete. Politician, Transportation, Building. Nature, Village, Animal, Plant, Album, Film, Book.
Compuny, School, Artist. Athlete. Politician. Transportation, Building, Nature, Village. Animal. Plant, Alhum, Film, Book
Article: CIB Bank is the second- second-biggest commercial bank Hungary after the January 2008 merger with Eutér- Bank. This follows 2007 menger respective Italian parent companies Banca Intesa and Sanpaolo IMI form Intesa Sanpaolo.
Answer: Company.
Atticle: Adarsh Vidya Kendra a school in India
37

---

Published as a conference paper at ICLR 2024
Table 6: The prompts used for sentiment analysis and hate speech detection. The prompt used for SST-2 is taken from Zhao et al. (2021), and the prompts used for Poem-Sentiment, Financial- Phrasebank, Ethos, TweetEval-Hate, -Weettyaa-Attheism, and TWeetEval- taken from Min et al. (2022). We show one training example per task for illustration.
Task
Prompt
Labels
SST-2
Review:
Well-rounded tribute.
Positive, Negative
Answer: Positive.
Saw how bad this movic was. Answer:
Review:
Poem-Sentiment
shines superior throne of gold:
positive, negative, no impact
The sentiment is: positive.
while rude winds blow off each shadowy crown:
The sentiment
Financial- Sitancial-Phraseee Phrasebank
Text: Operating profit for the month period increased from EUR13 66m, while net sales increased from EUR394 .7m, as compared to the corresponding period 2005. Sentiment: Positive.
Positive, Negative, Neutral
Text:
According to Gran, the company has no plans to move all production Russia, although that where the company growing. Sentiment:
Ethos
Is the following hate specch? Answer yes
Text: You look like Russian and speak like Indian. Both are disgusting go kill yourself. Answer: yes.
Text:
Civil rights movement for transgendered people Answer:
TweetEval-Hate
Tweet: Hundreds Syrian refugees home from Lebanon ABC favor, against News.
Sentiment: favor.
And another flood of immigrants coming our way. #BuildThatWall
Tweet:
Sentiment:
TweetFval-Athess
Determine if the text supports atheism. Answer with yes, no, neither. yes, no, neither
Tweet: It's Ask Atheist Day! Have question? #askanatheist #SemST
Answer: yes.
Tweet: Oh Jesus, We write songs praise you. #Songwriters #hewrite #Songs #Praiscyou #ScmST Answer:
TwectEval-Feminst
Determine if the supports feminism. Answer with yes, no, neither.
yes, no, neither
Tweet: FINALLY WOMEN RUNNING FOR PRESIDENT #SemST Answer: yes.
Tweet: Australia even has a fucking Minister for women for fuckk sake! IsAwful #SemST
Answer:
38

---

Published as a conference paper at ICLR 2024
Table 7: The different prompt formats used for SST-2 from Zhao al. (2021). We show one training example for illustration.
Format ID Prompt
Labels Positive, Negative
Review; Well-rounded tribute.
Answer: Positive.
Review:
Saw bad movie was.
Answer:
Review: Well-rounded tribute.
good.
Answer: good.
Review: Saw how movie was.
Answer:
My review for last night's 's film: Well-rounded tribute. The critics agreed that this movie was good.
good,
My review night's film: Saw how movie was. The critics agreed that this movie was
Here what our crities think for this month's films.
positive, negative
One of our crities wrote "Well-roumded tribute." Her sentiment towards the film was positive.
One of our critics wrote "Saw how had this movie was." Her sentiment towards the film was
Critical reception edit
good, had
contemporary review. Roger Ebert wrote "Well rounded tribute." Entertainment Weekly agreed, and the overall critical reception of the film was good.
contemporary review, Roger Ebert "Saw how bad this movie Entertainment Weekly agreed. und the overall critical reception of the film was
Review: rounded tribute.
Yes, No
Positive Review? Yes.
Review: Saw how bad movic Positive Review?
Review: rounded tribute.
Positive, Negative
Question: the sentiment the above review Positive Negutive?
Answer: Positive.
Review: Saw how bad this movie was.
Question:
Is the sentiment of the above review Positive or Negutive?
Answer:
8
Review: Well rounded tribute.
good, had
Question: the author think that movic was good or bad?
Answer: good.
movie was. Question: Did the author think that the movic was good bad?
Review: Saw how
Answer:
Question: Did the author of the following think that the movie good bud?
good, bad
Tweet: Well rounded tribute.
Answer: good.
Question: Did the author of the following tweet think that the movie was good or bad?
Tweet: how bad this movie was.
Answer:
Well rounded trihute. My overall feeling was that the movie was good.
good,
Saw how bad this movie was. My overall feeling was that the movie was
11
Well rounded tribute. liked the movie.
liked, hated
Saw how this movie was.
12 Well rounded tribute. My friend asked me if would give the movie 0 or 5 stars, said 5.
0,
Saw how bad this movie was. My friend asked me I would give the movie or 5 stars, said
Input: Well rounded tribute.
Positive, Negative
Sentiment: Positive.
Input:
Saw how bad this movic
Sentiment:
Review: rounded tribute.
True, False
Positive: True.
Review:
Saw how bad movic
Positive:
Review; rounded tribute.
Stars:
Review:
Saw how bad movic
Stars:
39


Reviews:
[{"review_id": "6fvkGKfrX6", "reviewer": "Reviewer_qL6Z", "summary": "The paper investigates how language models can learn to generate harmful outputs from few-shot prompts containing inaccuracies or biases. \n\nThe key findings are:\n\nModels tend to \"overthink\" and decrease in accuracy on incorrect prompts after a certain \"critical layer\", while accuracy keeps improving on correct prompts. This suggests incorrect information mainly affects later processing stages.\nAttention heads in later layers preferentially attend to and reproduce incorrect labels from the prompt demonstrations. Ablating a small number of these \"false induction heads\" significantly reduces the accuracy gap between correct and incorrect prompts.", "questions": "- If the prompt contained factual inaccuracies or harmful content, would you expect to see similar overthinking and false induction effects?\n- Could you train with additional regularization to discourage attention to incorrect prompt content?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper:\n- Provides novel insights into the internals of in-context learning through layerwise decoding and attention analysis. Links model behavior to specific components.\n\n- Extensive experiments across models, datasets and prompt variations. Head ablation results generalize across tasks.\n\n- Connects to related ideas like overthinking and induction heads. Builds understanding of model internals.", "weaknesses": "The paper lacks in the following ways:\n- Focuses on a simplified setting of permutations of class labels. \n- Does not cover more subtle inaccuracies or biases.\n- Only studies text classification tasks. Unclear if findings apply to more open-ended generative tasks.\n- No modification of model training process itself to mitigate issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvYkaq3a7U", "reviewer": "Reviewer_bUf7", "summary": "The paper focuses on understanding how language models process correct versus incorrect inputs in few-shot learning contexts, emphasizing harmful imitation. Two phenomena are identified: overthinking and false induction heads. Overthinking occurs when models, given incorrect demonstrations, tend to have reduced accuracy when predictions are decoded from later layers; stopping the model early, before all layers have processed the input, can increase accuracy. False induction heads are specific components in the model that contribute to overthinking by attending to and replicating incorrect information from previous demonstrations. The study's findings advocate for a deeper examination of intermediate model computations to understand and mitigate harmful behaviors in language models​.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper explores new concepts like \"overthinking\" (previously introduced) and \"false induction heads\" (novel), offering a fresh lens to study transformers, particularly in handling harmful imitation.\n\n- Solid research execution is evident. Specific model components, like attention heads causing errors, are identified and well analyzed.\n\n- The paper is well-structured and very readable. Concepts are clearly and consistently articulated.\n\n- The methodologies provided comprise a strong set of interpretability methods directed specifically at in-context learning and the role of the examples included.\n  - For instance, interpreting outputs from intermediate layers in such ways may prove useful for understanding what the optimal sets of example to provide to the LLM is, in addition to what is attempted in this paper, which is to understand how bad examples affect overall accuracy.\n\n- The paper studies the important issue of harmful content (which, as outlined above, may extend eventually to non-useful or even non-optimal content within the context). Overall, the findings are practically significant and offer insights for controlling model outputs against harmful examples and wrongful imitation.", "weaknesses": "- The paper details the phenomena of overthinking and false induction heads, but it doesn’t fully articulate how these insights could be applied practically to improve model behavior.\n  - For instance, while head ablations are discussed as a method to reduce overthinking, the practical impact of these modifications on model functionality and broader applicability is not thoroughly explored in real-world scenarios​ i.e. how does one know when the input examples are likely to be false, and that heads ought to be ablated?\n- The methodology involves decoding from intermediate layers and identifying false induction heads, but the paper could provide a more explicit explanation of these processes i.e. how predictions were extracted and analyzed at each layer, and how the heads contributing to overthinking were identified and analyzed.\n- The paper could better acknowledge that the 'incorrectness' of permuted labels can vary based on the dataset and what each label signifies. Some permuted labels might be clearly wrong, while others could be more ambiguous, and providing specific examples for each dataset could clarify this aspect and help to understand the results.\n  - It's not clear in the example from Figure 1 that demonstrations are \"incorrect\", and that all \"imitation\" is wrongful- there is still an argument that context is important, and in this example (at least in the example where all labels are permuted), could the LLM not understand the label permutation and simply interpret that negative sentiments require a positive classification, and vice versa? \n- The reference and justification around the use of the logit lens near the start of the paper could be strengthened, though the authors do come back to discuss it in more detail towards the end.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GTDQOzzQCb", "reviewer": "Reviewer_Lx2y", "summary": "This paper studies inner mechanism of LLMs when facing false demonstations.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like this paper. This work provides interesting findings.", "weaknesses": "1. Since the authors conduct experiments under in-context learning setting, we may not be sure these findings are from models themself or implicit tuning (in-context learning). Could the authors provide explanations?\n\n2. Can we expand tasks to include generation-based tasks like QA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)