Overthinking the Truth: Understanding how Language Models process False Demonstrations
Authors: Danny Halawi
Jean-Stanislas Denain
Jacob Steinhardt
Venue: Submitted to ICLR 2023 (2023)
Keywords: Large Language Models
Interpretability
Safety
Mechanistic Interpretability
Science of ML

Abstract: Through few-shot learning or chain-of-thought prompting, modern language models 
can detect and imitate complex patterns in their prompt. 
This behavior allows language models to complete challenging tasks without fine-tuning, 
but can be at odds with completion quality: if the context is inaccurate or harmful, then the model 
may reproduce these defects in its completions.
In this work, we show that this {harmful context-following} appears late in a model's 
computation--in particular, given an inaccurate context, models perform \emph{better} after zeroing out later layers. 
More concretely, at early layers models have similar performance given either accurate and inaccurate few-shot prompts, but a gap appears at later layers (e.g.~layers 10-14 for GPT-J).
This gap appears at a consistent depth across datasets, and coincides with the appearance of “induction heads” that attend to previous answers in the prompt. 
We restore the performance for inaccurate contexts by ablating a subset of these heads, reducing the gap by 28\% on average across 8 datasets.
Our results suggest that studying early stages of computation could be a promising strategy to prevent misleading outputs, and that understanding and editing internal
mechanisms can help correct unwanted model behavior.

Under review as a conference paper at ICLR 2023
OVERTHINKING THE TRUTH: UNDERSTANDING HOW LANGUAGE MODELS PROCESS FALSE DEMONSTRATIONS
Anonymous authors Paper under double-bling review
ABSTRACT
Through few-shot learning or chain-of-thougle prompting, modern language mod- els can detect and imitate complex patterns their prompt. This behavior allows language models to complete challenging tasks without fine-tuning, but can be at odds with completion quality: the context inaccurate or harmful, then the model may reproduce these defects in its completions. In this work, we show that this harmful context-folle monexx--foloowig appears late in a model's computation- partic- ular, given inaccurate context, models perform better after zeroing out later layers. More concretely, at early layers models have similar performance given either accurate and inaccurate few-shot prompts, but gap appears at later lay- ers (e.g. layers 13-14 for GPT-J). This gap appears at a consistent depth across datasets, and coincides with the appearance of "induction heads" that attend previous answers in the prompt. We restore the performance for inaccurate con- texts by ablating a small subset of these heads, reducing the gap by 23.2% on average across 14 datasets. Our results suggest that studying early stages of com- putation could be a promising strategy to prevent misleading outputs, and that understanding and editing internal mechanisms can help correct unwanted model behavior.
INTRODUCTION
key behavior of modern language models is context-following: neural networks like GPT-3 are able to infer and imitate the patterns in their prompt. At its best, this allows language models to perform well on benchmarks without the need for fine-tuning (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Chowdhery al., 2022; Srivastava al., 2022). This has led researchers to study how properties of the context affect few-shot performance (Min et al., 2022b; Kim et al., 2022; Xie al., 2021; Zhao al., 2021), and what internal mechanisms underlie context-following (Olsson et al., 2022).
However, context-following also lead to incorrect, toxic or unsafe model outputs (Rong, 2021). For example, if an inexperienced programmer prompts Codex (Chen et al., 2021) with poorly written or vulnerable code, the model is likely to produce poorly written or vulnerable code completions. Similarly, in this work we study few-shot learning for classification tasks: prompting the model with inaccurate demonstrations reduces model accuracy (Figure 1, left), because the model learns to reproduce the false demonstrations. We thus ask: Can we attribute this "false context-following behavior to specific model components, and can we mitigate it by intervening on these components?
We show that, perhaps surprisingly, false context-following in text classification is primarily a prop- erty of late stages of computation. In particular, stopping the model early-by zeroing out the later layers (Nostalgebraisst, 2020)-actuall] improves performance (Figure 1, center). Moreover, true and false contexts yield similar accuracy until some "critical layer" at which they sharply diverge. This demonstrates that even with false demonstrations, the model often "knows" the correct answer (it can be easily decoded from the latent states) but later replaces it with an incorrect answer that is more likely given the context.
To identify the underlying mechanism for false context- follexxt-ollowing. we turn to Olsson et al. (2022), who identify "induction heads" that attend to and reproduce previous patterns the input. Motivated

---

Under review as a conference paper at ICLR 2023
by this, searched for heads that consistently attend to previous examples that have the same (true) answer as the current prompt. We found many such heads, primarily concentrated later layers of the model (after the critical layer). By removing 10 of these heads, are able to reduce the accuracy gap between accurate and inaccurate prompts by an average of 23.2% over 14 datasets, with negligible effects the performance given true prefixes (Figure 1, right).
CnCooeect Demos inTorreect Demos
Curnel Demotaizutions
0.9
Review: Best fil of 2022. Sealiment:
Positive
0.9
Calibrated Accuracy
0.75
Callbbared Accurncy
0.75
Revisw: Rntermely boring. Sentiment: Negation
Revioe: Amazing! Sentiment: Positive
/
0.6
0.6
0.45
Reviow: Best film of 2022. Sentiment:
Negatirn
Cornect
0.3
Incorrect Demos
0.3
Buseline
Review: Entremely boring, Sentiment:
Positive
0.15
0.15
Rerbow: Amazing! Seatimeat:
Negnttrex
10
20
25
No
Random
Our Heads
Layer
Intervention
Heads
Figure/ 1: Left: Given a prompt of inaccurate demonstrations, language models are more likely to output incorrect labels. Center: When demonstrations are incorrect, zeroing out the later layers increases the classification accuracy, here on SST-2. Right: We identify 10 attention heads and remove them from the model: this reduces the effect of incorrect demonstrations by 36.7% on SST- 2, averaged over 15 prompt formats, without decreasing the accuracy given correct demonstrations.
Our findings show how analyzing and editing model internals can help practictioners understand and mitigate model failures. Indeed, one intuition for why early-exiting succeeds is that the attention heads we identified cannot in general occur at the earliest layers. This is because these heads must recognize which inputs belong to the same class, which likely requires multiple layers of processing. Thus, early exiting might be generally promising strategy detect dishonest behavior in models.
2 PRELIMINARIES, FEW-SHOT LEARNING WITH FALSE DEMONSTRATIONS
We begin by introducing the setting we study: few-shot learning for classification, given demon- strations with correct or incorrect labels. Incorrect demonstrations consistently reduce classification performance, which is the phenomenon that we aim to study and mitigate in this work.
Few-shot learning. We consider autoregressive transformer language models, which produce a conditional probability distribution p(t,+1 (1,,.., (n) over the next token fx+1 given previous tokens. We focus on the few-shot learning setting (Brown et al., 2020) for classification tasks: we sample & demonstrations (input-label pairs) from the task dataset, denoted (1), (Ix, yk.) To query the model on a new input x, we use the predictive distribution p(y Ø1, 1/1 Vi....,.. Ik. yk x).
Datasets and models. We consider fourteen text classification datasets: SST-2 (Socher al., 2013), Poem Sentiment (Sheng & Uthus, 2020), Financial Phrasebank (Malo et al., 2014), Ethos (Mollas al., 2020), -Wee (Barbieri et al., 2020), TWeetffall-Ahheim (Barbieri al., 2020), TweetEvvl--emiiiss (Barbieri et al., 2020), Medical Questions Pairs (McCreery et al., 2020), MRPC (Wang et al., 2019), SICK (Marelli et al., 2014), RTE (Wang et al., 2019), AGNews (Zhang al., 2015), TREC (Voorhees & Tice, 2000), and DBpedia (Zhang et al., 2015). We used the same prompt formats in Min et al. (2022b) and Zhao al. (2021) (Table 2, 3). For SST-2 we use the first of the 15 prompt formats in Zhao et al. (Table 5). We evaluated 3 autoregressive language models: GPT-J (Wang & Komatsuzaki, 2021), GPT2-XL (Radford al., 2019), and GPT-NeoX 2-2BBB---BBB-- (Black al., 2022).
Evaluation metries. Given our focus on classification tasks, we are interested in how often the model assigns higher probability to the true label than to all other labels. However, model predictions can be very unstable with respect to small prompt perturbations (Gao et al., 2021). To mitigate this variability, we measure the calibrated classification accuracy (Zhao et al., 2021). Concretely, for 2-class classification task, we measure how often the correct label has a higher probability than its median probability over the dataset. Assuming the dataset is balanced (which is true for us), this step has been shown to improve performance and reduce variability across prompts. Calibration for multi-class tasks follows similar procedure, detailed in appendix A.
2

---

Under review as a conference paper at ICLR 2023
0.9
Average over Datasets
35
Average Dutasets
Acc(True) Acc(False)
0.73
Permuted Score
0.6
2.5
0.45
2
0.3
1.5
0.15
0.5
0
1x
10
15
20
25
30
35
40
5
10
15
20
25
30
35
40
Number of Demonstrations
Number of False Demonstrations
Figure 2: Left: The difference in accuracy between accurate and inaccurate prompts increases with the number of demonstrations, Right: As the number false demonstrations increases, the model chooses the permuted label ar (class(x)) more often than the other labels, rather than making random error.
2.1 COMPARING TRUE AND FALSE DEMONSTRATIONS.
We first confirm that the models we study exhibit false context-follwwing behavior. To do so, we compare the performance of models when the demonstration labels are all correct, i.e. Vi class(x;), ) and when they are all incorrect, i.c. Y = (class(z.)). for a cyclic permutation a over the set of labels (Figure 1, left). In particular, inputs from the same class always assigned the same (possibly false) label within each prompt.
For each model and dataset, sample 1000 sequences each containing demonstrations and eval- uate the model's 's calibrated accuracy. We sample different demonstrations (I, ys) and label permu- tations for every sequence, and vary & from 0 40 (from to 20 for GPT2-XI, due to its smaller context size).
Figure 2 (left) shows the difference between GPT-J's calibrated accuracy given accurate and inac- curate prompts as the number of demonstrations increases. As expected, false demonstrations lead worse performance, and the gap tends to increase with & for most datasets. These results in agreement with Min et al. (2022b), who found that incorrect demonstrations decreased GPT-J's performance on classification tasks (see Figure in Min al.).
Models could lose accuracy by copying the incorrect label, or by becoming confused and choosing random labels. To confirm it is the former, we also measure which labels the model chooses for multi-class tasks. Specifically, we measure the permuted score: how often the model chooses the permuted label ar (class(x)) over the other labels. For each dataset, a random classifier would have permuted score of #labols To make the results comparable across datasets, we divide the permuted scores by this random baseline. Figure (right) shows these reweighted permuted scores for GPT- J on the multi-class datasets in our collection, as well as their average over the datasets. The permuted score increases steadily with the number of demonstrations and reaches twice its bascline value after 40 demonstrations.
THE LOGIT LENS: ZEROING OUT LATER LAYERS IMPROVES ACCURACY
In this section, we decode model predictions directly from intermediate layers. This allows us to evaluate the model's 's performance midway through processing the inputs. On false prefixes, find that the model performs better midway through processing, and investigate this phenomenon in detail.
Intermediate layer predictions: the logit lens. Given an autoregressive transformer language model, we will decode probability distribution over the next token from each intermediate layer,
3

---

Under review as a conference paper at ICLR 2023
Calibrated Accuracy
Correct Demes Rumdom
0.8
0.8
Carly-
0.6
0.6
0.4
0.2
0.2
0
10
20
30
40
0
10
15
20
25
10
20
30
40
Layer
Layer
Layer
(a) GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 3: Average performance across 14 tasks for GPT2-XL, GPT-J, and GPT-NeoX, y-axis (left):
Calibrated accuracy given correct and incorrect demonstrations, denoted by full lines. y-axis (right):
Percentage of tasks where zeroing out all succeeding transformer blocks is superior than full model evaluation, denoted by dashed lines. Early-exiting effective given false demonstrations, and per- haps more surprisingly, also effective given correct demonstrations.
using the "logit lens" method (Nostalgebraist, 2020). Intuitively, these intermediate distributions represent model predictions after E (1...., [.} layers of processing.
In more detail, let ( E Fd denote the hidden state of token &; layer e, i.e. the sum of everything up to layer É in the residual stream. For sequence of tokens 41,..., in V, the logits of the predictive distribution p(1x+1 (1: (n) are given by
logit,, vil = Wo LayerNorm, (1)
where LayerNorm, is the the pre-unembedding layer normalization, and We R Vixd is the unembedding matrix. The logit lens applies the same unembedding operation to the earlier hidden ty):
states (, yielding intermediate layer distribution pe(tn++1
Dogit logit( = Wu Ley-LaynNoomm (h("))
This provides a measurement of what predictions the model represents at layer L, without the need to train a new decoding matrix. It can therefore be interpreted 2015; Teerapitaaynoon et al., 2017; Figurnov et al., 2017).
form of early exiting (Panda et al.,
Early exiting improves classification performance. We measure the calibrated accuracies of the intermediate layer distributions pe for the three models and fourteen datasets from Section 2, using context lengths of 40 demonstrations (20 demonstrations for GPT2-XL). We also measure the lay- erwise accuracies for two toy datasets: "SST-2-A/B", modification of SST-2 (Socher et al., 2013), and "Unnatural". that extends task in Rong (2021; section 4). In "SST-2-A/B", labels (e.g. 'Positive" and 'Negative') with letters 'A' and 'B'. In "Unnatural". demonstrations are of the form "[object]: [label]" and the labels are "plant/vegetable" "sport", and "animal".
replace the
Figure displays results for GPT-J, with corresponding plots for GPT2-XL and GPT-NcoX in Fig- ures and in the Appendix. For GPT-J with correct demonstrations, accuracy tends increase with layer depth, and starts to stagnate grow more slowly around layer 15. The accuracy for incorrect demonstrations follows similar trend at the early layers, but then diverges and decreases at the later layers.
For incorrect demonstrations, decoding from earlier layers performs better than decoding from the final layer. For GPT-J, using Pie (the first 16 layers) achieves better accuracy than the full model for 12 out of the 14 datasets, by an average of 8.6 percentage points. This gain on false prefixes comes with comparatively small cost for prefixes: 1.6 percentage points (see Table ??). Similarly, for GPT2-XL and GPT-NeoX, the intermediate predictions Pso and P27 outperform using the full model for 11 out of 14 datasets, however the magnitude of the effect is smaller: average of 3 percentage points for GPT2-XL and 0.7 percentage points for GPT-NeoX.
4

---

Under review as a conference paper at ICLR 2023
Correct Demos
Incorrect
Random Baelline
Calibrated Accuracy
0.8
0.5
0.8
0.89
0.6-
0.6
0.8
0.4
0.4
0.2
0.2
01
0
15
20
25
10
15
20
25
0
21
25
5
10
15
(a) SST-2
(b) Poem-Sentiment
(c) Financial- Phrasebank
(d) Ethos
Calibrated Accuracy
0.8
0.6
0.4
0.4
0.4
0.4
0.2-
0.2
0:2
0.2
10
15
21
25
10
15
20
25
0
10
15
21
25
5
30
15
20
25
(c) TweetEval-HHate
TweetEval-Atheisn
(g) TwectEFal-Feminnt
(h) MedQ-Pairs
Calibrated Accuracy
0.5
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
2
26
(i) MRPC
(i) RTE
(k) SICK
()) AGNews
Calibrated Accuracy
12
11
11
0.8
0.8
0.8
0.8
0.6-
0.6
0.6
0.2
0.2
0
0
15
20
25
10
15
20
25
0
21 25
10
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 4: GPT-J early-exit classification accuracies across 16 tasks, given correct and incorrect demonstrations. Plots are grouped by task type: sentiment analysis (a-c), hate speech detection (d- g). paraphrase detection (h-i), natural language inference (j-k), topic classification (I-n), and toy tasks (o-p). Given incorrect demonstrations, zeroing out all transformer blocks after layer 16 outperforms running the entire model, across 14 out of the 16 tasks.
For the toy "Unnatural" dataset, these effects are particularly pronounced (see Figure 4p). At layer 16, the accuracy GPT-J for incorrect demonstrations reaches 0.91, which is 92% of the final layer accuracy given an accurate prompt. In contrast, at the final layer, GPT-J's accuracy given false demonstrations reaches its lowest value, 0.07.
True and false prefixes sharply diverge at "critical layers". For each model, the accuracies for correct incorrect demonstrations diverge the same layers across all datasets. For example, for GPT-J, the accuracy gap between accurate and inaccurate prompts first exceeds 50% (and 45%, and 55%) its final layer value between layers 13 and 14 for 12 out the 14 datasets. We obtain similar results for GPT-NeoX with layers 10 12 and for GPT2-XL with layers 21 to 23. In summary, zeroing out later layers leads to better classification accuracies given incorrect demonstrations, and the accuracy gap between correct and incorrect demonstrations emerges at consistent set of layers across datasets.
5

---

Under review as a conference paper at ICLR 2023
tomato
sport
tomato
sport
tomato sport
garlic
sport
garlic sport
garlic sport
llama
plant
vc get able
llama plant / ve get able
llama plant
ve get able
kale
sport
kale
sport
kale sport
rugby:
animal
rugby animal
rugby animal
beet
sport
beet sport
beet
| sport
Figure 5: Examples of attention patterns incorrect demonstrations from the "unnatural" dataset, for heads that are label-attending but not class-sensitive (Left), heads that are class-sensitive but not label-attending (Center), and heads that are both label-attending and class-sensitive (Right).
4 ZOOMING INTO ATTENTION HEADS
We found that for all datasets, the gap between true and false demonstrations appears small set of transformer blocks. We would like to know whether specific attention heads are responsible for this behavior.
(Olsson al., 2022) introduce induction heads: attention heads that attend to previous occurences of the present token, and increase the probability of the outputs that follow them. Inspired by this work, we investigate the hypothesis that small number of "induction heads" play key role in false context-following, by attending to the labels in previous similar demonstrations and making the model more likely to output them.
For example, in Figure 5, we know that the model assigns a high probability to the mistaken label "sport". According to the hypothesis, this is because of heads that attend the previous occurences of "sport" in this context, and increase the probability of that token. The previous occurences of "sport" share two properties: (1) they are labels in the previous demonstrations, and (2) they follow inputs with the same class as "beet": "tomato" and "garlic".
Therefore, we look for heads that satisfy two conditions when they attend to inaccurate prompts. First, they should be label-altending, i.e. concentrate their attention on labels in the previous demon- strations. Second, they should be flass-sensitive, meaning they should attend specifically to those labels that follow inputs in the same class the latest input. We call heads that are both label- attending and class class- class-sensitive sensitive given incorrect demonstrations false prefi-matching heads.
We define score to identify false prefix-matching heads. For sequence of demonstrations (I,, (:) and final input x, the prefix-matching score (PMS") of head is:
n Att' (xx B4)
PMS* Σ Aut" (x, y:) =dass() #labels
Prefix-mathing heads should have a high PMS scores. We therefore plot the distribution of prefix- matching scores across layers for these heads. Figure 6 shows these results for the "Unnatural" dataset. For each model, the scores remain low the early layers, but then increase around the "critical layers" that we identified in the previous section. This lends correlational support to our hypothesis that false prefix-matching heads cause false-context following behavior.
Ablating false prefix-matchin heads. However, we are interested in causal evidence. Therefore, we check whether removing false prefix-matching heads reduces false context-following. We select the 10 heads from GPT-J with the highest trefix-matching scores given incorrect demonstrations on the "Unnatural" dataset. We ablate these heads by setting their keys, queries and values to We then evaluate the resulting lesioned model on all 14 datasets, and compare its layerwise performance to the original model's. As a control baseline, we also perform the same analysis for 10 heads selected at random.
The ablations considerably increase the accuracy given false demonstrations: they reduce the gap in accuracy between accurate and inaccurate prompts by an average of 23.2% for k 40 and 39.3%

---

Under review as a conference paper at ICLR 2023
6
3.2
22
Sum PMS
Stop
Sum of PMS
0.75
0.75
16.5
0.75
Final of
2.5
0.5
1.
0.5
11
0.5
0.25
0.25
dee
0
10 15 20 25.00 35 40 4549
10 15
10 15
25 Layer
35.30
35
45
Layer
Layer
(a) GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 6: Sum of prefix-matching scores given true and false demonstrations, for GPT2-XL (a), GPT-J (b), and GPT-NeoX (c) on the toy Unnatural dataset. The prefix-matching scores increase around the layers where the accuracy gap (averaged over tasks) between true and false demonstra- tions emerges.
for kt = 10 (see Table 1). In contrast, ablating random heads reduces the gap by 3.6% for = 40 and 13.4% for & = 10. While they greatly improve the accuracy given false prefix, our ablations have a comparatively small effect on the accuracy given correct demonstrations: ablating the false prefix-matching heads decreases the accuracy given true demonstrations by 2.2% for = 40 and by 1.3% for = 10. These results show that the false trefix-matchin heads cause a large fraction of the false context-following behavior.
Analysing the outputs of false prefix-matching heads. We identified false prefix-matching heads based only their attention patterns. However, postulated mechanism also depends the heads' outputs: they must increase the probability of the labels they attend to. We therefore study the outputs of these heads to understand how they affect the residual stream.
We apply the logit lens to each head individually, by applying layer normalization followed by the unembedding matrix to its outputs. This tells us how much the head increases or decreases the intermediate logits of each token. For every head, we measure the difference between the logit increases of the permuted and correct labels on the Unnatural dataset (following the methodology in (Wang et al., 2022)). Our 10 false prefix-matching heads have average score of 1.2, which shows that they increase the logits of the permuted label more than those of the correct label. In contrast, when sampling 100 sets of 10 random heads, we find average score of 0.25, with standard deviation of 0.4. Therefore, false prefix-machhing heads directly increase the probability of the permuted labels relative to the correct labels more often than random heads.
Calibrated Accuracy
0.8
Nose
0.8
0.8
Dan
75 hrads
0.6
0.6
0.6
0.6
0.4
0.1
0.4
0.4
0.2
0.2
1.2
25
25
10
15
20
(a) 10-shot Average
(b) 40-shot Average
(c) AGNews
(d) Unnatural
Figure Ablating false prefix- matching heads increases accuracy across multiple layers. (a), (b): Average accuracy at each layer before and after ablating false-prefix matching or random heads, given correct and incorrect demonstrations. (c), (d): Accuracy at each layer for incorrect demonstrations on AGNews and Unnatural, after ablating the k most class sans-sensitive heads, for E (5, 15, 35, 75}.
5 DISCUSSION AND RELATED WORK
In this paper, we showed how stopping language models early by zeroing out their later layers im- proves classification performance given inaccurate contexts, without requiring any additional train-
7

---

Under review as a conference paper at ICLR 2023
Ablating false prefix-matching heads recovers large fraction of the accuracy gap between true and false prefixes, without hurting performance given true prefixes. We show the percentage re- duction of the accuracy gap and percentage change in true prefix performance when ablating the 10 false prefix-matching heads chosen using the Unnatural dataset ("top") or 10 random heads ("ran- dom"). We bold gap reductions when they are greater for our heads than for the random heads.
Table 1:
Dataset
Heads
10-shot
40-shot
True Prefix
Gap Reduction
True Prefix %A(t)
Gap Reduction %(1)
%a(1)
%(t)
Sentiment Analysis
top 0.490.01
33.66g.29
0.099.05
36.69
SST-2
random 2.340.08
7.150.05
-2.51o.11
11.18p.04
top 2.510.07
27.05g.13
0.460.01
33.59p.23
Poem-Sentiment
random 5.410.16
14.21o.11
-5.28p.16
13. 13.19p.08
top 2.780.0s
30.04..11
-3.01a.11
24.96
random -2.83.009
9.08g.01
14.22..15
9.34g.09
Hate Speech Detection
top 7.360.21
43.16g.01
4.86g. 15
21.50p.05
Ethos
random 1.340.01
8,425.13
-2866..0
10.5p.01
top 3.430.10
10.20
-3.42u.10
15.49p.or
TweetEval-Hate
random 4.050.12
11.22o.08
-0.31a.01
9.15p.06
top 2.590.07
19.770.01
-5.30p.14
14.43p.02
-WeetEval-Athhiim
random -3.070.09
25.21o.01
-2.49..0
7.66p.01
top 2.35p.07
4.15p.07
1.09g.03
6. 6.160.07
TweetEval-Feminist
random 0.30g.01
2.860.02
3.600.
16. 16.100.03
Paraphrase Detection
top 3..70..11
56.67o.01
10.83p.30
56.45p.or
MedQ-Pairs
random 2.220.06
46.67..01
44.69..14
38.71a.01
top -0.38..01
99.50g.01
0..55.001
41.670.0
MRPC
random 1..14.03
22.22o.01
-1.74p.05
25.00a.o1
Natural Language Inference
top 12.82
38.40q.11
10.34..30
29.51
SICK
random 2.770.07
11.130.01
-7.55..22
16.41o.08
top 8.770.27
81.81
4.20
-25.00.
RTE
random 1.75p.05
45.450.02
0.350.01
99.999.01
Topic Classification
top 1.620.05
21.37o.11
0.64g.0
32.20
AGNews
random 0.59g.02
18.800.12
2.57
5.260.08
top 0.16p.01
49.03g.00
1.00g 03
19.22
TREC
random 4.30o.13
7.696.13
0..88.m1
5.21
top 1.21g.06
28.70g.17
0.63g.05
18 18.32u.c
DBPedia
random 1.320.07
5.21
0.84
6.62
top 1.310.49
39.26 39.266.76
22.66.79
23.23
Average
random
-2.140.92
13.38p.79
- -2.781.24
3.62o.59
ing. We also identified attention heads that contribute to the effect of misleading prompts, and showed that ablating these heads mitigates this effect.
Related work. Our work closely related to Min et al. (2022b) and Kim et al. (2022), who ex- amine the role of false demonstration model accuracy. Min al. (2022b, figure, 4) find that for classification by a pre-trained model (GPT-J), the ground truth of demonstrations has a large effect the accuracy. However, they do not find such effect with meta-tuned model (Min
8

---

Under review as a conference paper at ICLR 2023
et al., 2022a). Therefore. meta-tuning could serve as "negative control" to test our hypothesis that false prefix-matte matching heads cause false context-following: an interesting future direction would be to check whether meta-tuning reduces the number of false
heads.
The literature on carly-exiting and overthinking (Kaya et al., 2018; Panda et al., 2015; Tecrapit- tayanon et al., 2017; Figurnov et al., 2017; Hou et al., 2020; Liu et al., 2020; Xin et al., 2020; Zhou et al., 2020; Zhu, 2021; Schuster et al., 2022) also highlights how decoding from intermediate lay- ers can save compute and sometimes produce better results. One major difference that most of these methods rely either on modifying the training process to allow for early-exit, or on training additional probes to decode intermediate states, In contrast, the logit lens does not require any extra training to decode answers from internal representations.
How does the logit lens compare to probing? the "logit lens" (Nostalgebraist, 2020). We find it useful to think of this method in comparison to probing.
Our work, especially Section relies heavily on
If a layer has a high probing accuracy, this means that the correct answer can be decoded from the hidden states. However, this is often a low bar to clear, especially when the classification task is easy and the hidden states are high-dimensional (Hewitt & Liang, 2019). In contrast, if a layer has high logit lens accuracy, this shows that encodes correct answers along a direction in the residual stream that the model subsequently decodes from, which is much more informative. On the other hand, a low logit lens performance at a layer does not imply that the correct answers cannot be decoded from that layer.
One intermediate between probing and zeroing out later layers is the "tuned lens" (Ostrovsky et al., 2022): instead of training probes on each classification task or directly using the final layer's 's decod- ing matrix, for each layer the authors train a new square adapter matrix between the residual stream and the unembedding matrix would be interesting to run our experiments with this alternative decoding method.
language modelling dataset such as the Pile (Gao et al., 2020). It
Future work. While we find consistent results across 14 datasets, our experiments are restricted to a specific setting: text classification with large number of incorrect few-shot examples. We also studied variations on our setting, with qualitatively similar results (Figure 14): when the demon- stration labels are selected at random rather than according to permutation, and when only half of the demonstration labels are incorrect. In the future, researchers could use the logit lens to study diverse real-world failures of large models, such as "prompt injection" (Branch et al., 2022) or vul- nerable code completions (Pearce et al., 2022). In both of these cases, the model outputs inaccurate or harmful completions even though it is capable of producing correct ones given better prompt.
In addition, our head ablations do not recover the entirety of the accuracy gap between accurate and inaccurate prompts. This could be because we did not identify some of the model components that cause false context-following. However, there is another possibility: if an attention head's 's outputs on average far from zero, zeroing out that head takes the intermediate states off-distribution, which can decrease overall model performance. Thus, one promising future direction would be to replace head outputs by their value on different inputs, in (Meng et al., 2022).
Relatedly, while we identified induction heads that increase the probability of the wrong answers, we still do not have full mechanistic understanding of false context molexx-followwig behavior. For example, we do not know which heads in the earlier layers compose to form these induction heads. Future work could build on our methodology reverse-engineer circuits (Cammarata et al., 2020) in GPT models that implement false context- context-following
In this work we showed how studying the early stages of computation can mitigate the effects of mislcading prompts. We hope this will spur further work on auditing network internals to detect dishonest behavior in models.
REFERENCES
Francesco Barbieri, Jose Manacho-Colladoor Luis Espinosa Anke, and Leonardo Neves. Tweet- Eval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, nov 2020. Association for Computational Linguistics. doi: 12000/33/.12222000mnnnnnng-onm
URL https:

---

Under review as a conference paper at ICLR 2023
/rcllnntoology.orr/2200 findings-emnip.
Sid Black, Stella Biderman, Eric Hallahan. Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Work- shop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https: //arxiv. org. abs 2204. 06745.
Hezekiah J. Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. Evaluating the susceptibility of pre- trained language models via handcrafted adversarial examples, 2022. URL https://arxte org/ /agg/bb// 2209. ..0000.000 02128.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 10.2391 1/distillil1 https://issill.ppb/200/ciririii
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Michael Figurnov, Artem Sobolev, and Dmitry Vetrov. Probabilistic adaptive computation time, 2017.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, aug 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/202 acl acl-long.28 .95.
16653//1/002..aclloon....2. lcI-
URL https: / /acl anthe ogy org/2021.
J. Hewitt and P. Liang. Designing and interpreting probes with control tasks. Empirical Methods in Natural Language Processing (EMNLP), 2019. URL https:/ /nlp. stanford. .eduf pubs/ httph//ttttwoooororelr.mm.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-opine optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth, 2020.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking, 2018
Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang Woo Lee, Sang goo Lee, Kang Min Yoo, and Tacuk Kim. Ground-truth labels matter: A deeper look into input-label demonstrations, 2022.
10

---

Under review as a conference paper at ICLR 2023
Weijie Liu, Peng Zhou, Zhiruo Wang. Zhe Zhao, Haotang Deng, and QIJU. Fastbert: self bert with adaptive inference time. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10. 11/88865/1111202oo/ acl-main.537 URL http://dh
doi.org oogg/org/110 0. /elrg//1.116533/0. 18653/v1ll... /2020. ac ma a 537.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 216-223, Reykjavik, Iceland, may 2014. European Language Resources Associ- ation (ELRA). URL http: www. 1rec- conf. o wttgg//ooeeeiigogoeoooooooo wttpeeeddding//eeee11eee 1rec2014 / 363 Pap.aapp.ppp.
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. Ef- fective transfer learning for identifying similar questions: Matching questions to covid-19 faqs, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint arXiv:2202.05222, 2022.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaiel: Learning to learn in context. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022a. doi: 1118116/1//22222
ma0 URL http: /ttp:://dxx./ doi. ozg /tc//.... 1ww.111155///22222 2022222222222 http:////w/..222mmmmammammmaaaaa nwacl-main. 201.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022b.
Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. hate speech detection dataset, 2020.
Ethos: an online
Nostalgebraist. Interpreting gpt: the logit lens, 2020. URL httpt: http::///wwwwwwwwseeeeeee wwwwwwong. com/ Beestts/AccRBBBppaavvvoooo http::///e..epeeeeooooo----e-ooa
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-oodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. www.aaammnnnnaanddnnninnoonneeee htmm..
Igor Ostrovsky, Ram Bharadwaj Aryasomayajula, Dakota Mahan, and Stella Biderman. Towards 20/20 vision: Interpreting transformers with tuned lenses, forthcoming (private communication), 2022.
Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. Conditional deep learning for energy- efficient and enhanced pattern recognition, 2015.
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot's code contributions. In 2022 IEEE Symposiun on Security and Privacy (SP), pp. 754-768. IEEE, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Jack W Rae. Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12. 2021.
11

---

Under review as a conference paper at ICLR 2023
Frieda Rong. Extrapolating to unnatural language processing with gpt-3's in-context learning:
The good, the bad, and the mysterious, 2021. URL petttps:/aa..stannti edw.boog. nn-coontext-Learnian
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling, 2022.
Emily Sheng and David Uthus. Investigating societal biases in a poetry composition system, 2020.
URL https: /ttpvv/avvvv///v///av//...aaaaav org/ abs/ 20111 02686.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1644 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: : htt///aaaaaatoooooo.ooooeo /
and Christopher Potts. Recursive deep models for semantic compositionality
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 14X1022206.04655, 2022.
Surat Teerapittayanon, Bradley McDanel, and H. Kung. Branchynet: Fast inference via early exiting from deep neural networks, 2017.
Ellen M. Voorhees and Dawn M. Tice. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC'00), Athens, Greece, May 2000. European Language Resources Association (ELRA). URL http:// /www. pdf.
lrec-conf. org/ http://ooooo.nnooeeeeoooo///////
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 Billion Parameter Autoregressive Language Model. https: / thub. com/ / ingoflgllloo oll z/ teass-tanns r- May 2021.
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL
htttp:://axiiv http://///axioooo/a////////s//// 2211 00593.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference, 2021.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification, 2015. URL https: arxiv. org/ /igg/abb
01 626.
Tony 7. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improv- ing sew-shot performance of language models, 2021.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses pa- tience: Fast and robust inference with early exit, 2020.
Wei Zhu. Leebert: Learned early exit for bert with cross-level optimization. In ACL, 2021.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 CALIBRATION
For k-way tasks, measure how often the correct label has a higher probabilitiy than the 1=1- quantile of its probability over the dataset. In figure 12, we show the logit lens accuracies of GPTT-J over the 16 datasets, and confirm that they are similar the accuracies with calibration, albeit bit noisier.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
P(True)
Figure 8: The probability of the label "True" for random test inputs in MRPC. The "True" class is marked with green dots and the "False" class is marked with red dots. As observed in Zhao et al. (2021), the model can be biased towards one of the labels.
13

---

Under review as a conference paper at ICLR 2023
A.2 LOGIT LENS RESULTS FOR THE OTHER MODELS
We plot the Logit Lens results for GPT2-XL and GPT-NeoX in Figure and Figure 10.
Correct Demos
Incorrect Demos
Random Baseline
Calibrated Accuracy
0.8
0.8
0.6
0.5
1.6
0.6
0.4
0.4
0.4
01.4
0.2-
0.2
0.2
10
40
10
20
30
10
(a) SST-2
(b) Poem-Sentiment
(c) Financial-Phrasbbank
(d) Ethos
Calibrated Accuracy
0.8
0.5
088
0.8
0.6
0.6
0.G
0.6
0.4
0.2
02
0
10
30
(e) TweetEval-Hatt
(f) TweetEval-Athenn
(g) TweetEval-Femiine
(h) MedQ-PPirs
Calibrated Accuracy
0.5
0.8
0.6
0.4-
0.4
1.4
0.4
0.2
0.2
0.2
0.2
0
10
20
30
40
10
20
30
40
10
20
30
40
10
20
30
40
(i) GLUE-MRPC
(i) GLUE-RTE
(k) SICK
(I) AGNews
Calibrated Accuracy
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.1
0.1
0.3
0.2
0.2
0.2
10
20
10
20
30
10
10
20
40
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 9: GPT2-XL early-exit classification accuracies across 16 datasets, given correct and incor- rect demonstrations. Given incorrect demonstrations, zeroing out all transformer blocks after layer 30 outperforms running the entire model on 13 out of 16 datasets. all datasets, running the entire model is never superior than the max performance over the preceding layers.
14

---

Under review as a conference paper at ICLR 2023
Correct Demos
Incorrect
Random Baeline
Calibrated Accuracy
0.8
0.5
0.8
0.89
0.6-
0.8
0.8
0.4
0.2
0.2
01
01
30
20
40
30
10
(a) SST-2
(b) Poem-Sentiment
(c) Phrassebak
(d) Ethos
Calibrated Accuracy
0.8
0.6
V
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
10
20
30
20
10
21
30
10
10
20
90
10
20
30
10
(c) TweetEval-Hai -att
EvectEval-Atheisee
(g) FemmnnsvllFemmmist
(h) MedQ-Pairs
Calibrated Accuracy
0.5
8.8
0.5
0.6
0.6
0.6
0.8
0.4
0.4
0.4
0.4
0.2
0.2
10
(i) GLUE-MRPC
(i) BLUB-RTE
(k) SICK
()) AGNews
Calibrated Accuracy
1-
18
18
0.8
0.5
01.
0.8
0.6
0.6
0.4
0.2
0.2
0
20
30
211
40
10
20
30
10
20
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 10: GPT-NeoX early-exit classification accuracies across 16 datasets, given correct and in- correct demonstrations. Given incorrect demonstrations, zeroing out all transformer blocks after layer 27 outperforms running the entire model on 13 out of 16 datasets. In all datasets, running the entire model never superior than the max performance over the preceding layers.
15

---

Under review as a conference paper at ICLR 2023
A.3
LOGIT LENS RESULTS FOR EACH SST-2 PROMPT FORMAT
Calibrated Accuracy
0.8
0.8
0.8
0.6
Correct Demos
0.2
Incorrect Demos
0
15
21
25
11
18
20
25
10
15
20
25
(a) Id:
(b) Id: 2
(c)
Id:
Calibrated Accuracy
0.6
0.6
0.4
10.4
0.4
0:2
0.2
0:2
0.2
0-
20
20
10
16
20
10
20
20
10
16
20
(d) Id: 4
Id:
(l) Id: 6
Id:
Calibrated Accuracy
0.8
0.5
11.
0.8
0.6-
0.6
0.6
0.8
0.4
0.4
0.4
0.2
25
(h) Id: 8
(i) Id: 9
(i) Id: 10
(k) Td: 11
Calibrated Accuracy
0.8
0.8
0.8
0.8
0.6
0.6
0.4
0.2-
0.2
0.2
01
01
15 20
25
10
15
30
0
20
25
10
15
20
Layer
Layer
Layer
Layer
(1) Id: 12
(m) Id: 13
(n) Id: 14
(0) Id: 15
Figure 11: Calibrated Accuracy for all 15 prompt formats for SST-2 (from Zhao et al. (2021)). Given incorrect demonstrations, prompt formats 1, 3, 4, 5, 7, 8, 9, 10, and experience an increase in performance before experiencing decline. Prompt formats 6, 12, 14, and 15, on the other hand, do not exhibit this effect. Prompt format 11 produces poor performance, given both correct and incorrect demonstrations. See Table 5 for prompt format details.
16

---

Under review as a conference paper at ICLR 2023
A.4
LOGIT LENS RESULTS FOR GPT-J WITHOUT CALIBRATION
Correct Demos
Incorrect Demos
Random Baeeline
Calibrated Accuracy
0.5
0.8
0.5
0.6
0.6
0.6
0.8
0.4
0.4
0.4
0.3
0:2
0.2
0.2
25
(a) SST-2
(b) Poem- Deet-mentiment
(c) Financial-Prrasebante
(d) Ethos
Calibrated Accuracy
0.8
0.5
1.8
0.5
0.6
0.6
0.2
0.2
0.2
0
0
0
15
20
25
10
15
30
0
27
25
5
10
15
20
(c) TweetEval- -ate
(f) EvcetEval-Athcism
(g) TweetEval-Feming
(h) MedQ-Pairs
Calibrrted Accuracy
0.6
0.6
0.4
0.1
11.4
0.3
0.2
0.2
0.2
0.2
08
0
20 20
10
16
20
20
(i) MRPC
RTE
(k) SICK
(I) AGNews
Calibrated Accuracy
M
0.8
0.5
11.
0.8
0.6-
0.6
0.6
0.6
0.4
0.4
25
Layer
Layer
Layer
Layer
(m) TREC
(n) DBPedia
(o) SST2-AB
(p) Unnatural
Figure 12: GPT- early-exit uncalibrated classification accuracies across 16 tasks, given correct and incorrect demonstrations. The lack of calibration makes the results noisier especially at early layers, but early-exit still generally outperforms running the full model.
17

---

Under review as a conference paper at ICLR 2023
A.5
LOGIT LENS RESULTS AS MEASURED BY OTHER METRICS
Correct Demos
Incorrect Demos
Random Baseline
Uncalibrated Accurccy
1
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
0
10
20
30
40
0
5
10
15
20
25
10
20
30
40
Top-1 Accuracy
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0
10
20
30
40
0
5
10
15
20
25
10
20
30
40
2.5
2.5
2.5
Permuted Score
2
2
1.5
1.5
1.5
1
0.5
0.5
0.5
0
15
20
10
30
Label Space Probs
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
20
10
20
30
Layer
Layer
Layer
(a) GPT2-XL
(b) GPT-J
(c) GPT-NeoX
Figure 13: Uncalibrated Accuracy (row 1), Top-1 Accuracy (row 2), Permuted Score (row 3), and Label Space Probabilities (row 4) averaged over tasks (9 multi-class tasks for the permuted score). As the label space is learned, we observe the emergence and ensuing increase in the gap in the other metrics.
18

---

Under review as a conference paper at ICLR 2023
A.6
LOGIT LENS RESULTS FOR VARIANTS OF OUR SETUP
Calibrated Accuracy
Correct Incorrect
Noae
0.8
Random
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
01
5
10
15
20
25
10
15
20
15
20
25
(a) Half Correct Half Permuted
(b) Random Incorrect
(c) Random
Figure 14: Average GPT-J layerwise accuracies for the original model, the null ablation, and ablation variants of our setup: when half of the 40 demonstrations are true and half have permuted labels (a), when each demonstration' label is chosen random among the incorrect labels (b), and when each demonstration's label is chosen at random among all the labels (c). We find qualitatively similar results in these different settings.
19

---

Under review as a conference paper at ICLR 2023
A.7 PROMPT FORMATS USED FOR ALL DATASETS
Table 2: The prompts used for sentiment analysis and hate speech detection. The prompt used for SST-2 is taken from Zhao al. (2021), and the prompts used for Poem-Sentiment, Financial- Phrasebank, Ethos, TweetEval-Hate. TweetEval-Atheism, and TweetEval-Feminis are taken from Min et al. (2022b). We show one training example per task for illustration.
Task
Prompt
Labels
SST-2
Review: Well-rounded tribute.
Positive, Negative
Answer: Positive.
Review:
Saw how bad this movie was.
Answer:
it shines superior a throne of gold:
positive, negative, no impact
The sentiment is: positive.
while rude winds blow off each shadowy crown: The sentiment is:
Financial-Phrese Phrasebank
Text: Operating profit for mine-month period increased from EURI3 66m, while sales increased from EUR394 .7m, compared the corresponding period 2005.
Positive, Negative, Neutral
Sentiment: Positive.
Text: According to Gran, the company has no plans to move all production to Russia, although that is where the company is growing.
Sentiment:
Ethos
Is the following hate speech? Answer yes or no.
yes, no
Text: You look like Russian and speak like Indian. Both are disgusting go kill yourself.
Answer: yes.
Text:
Civil rights movement for transgendered people Answer:
TweetEval-Hate
Tweet: Hundreds of Syrian refugees return home from Lebanon ABC favor, against News.
Sentiment: favor.
Tweet:
And now another flood of immigrants coming our way.
#BuildThatWall
Sentiment:
TweerEval-Atheire
Determine if the supports atheism. Answer with yes, no, or neither. yes, no, neither
Tweet: It's Ask Atheist Day! Have question? #uskamatheist #SemST
Answer: yes.
Tweet: Oh Jesus, We write songs to praise you. #Songwriters #wewrite #Songs #Praiscyou #SemST
Answer:
TweetEval-FFe FFemiiisstt
Determine if the text supports feminism. Answer with yes, no, yes, no, neither
neither.
Tweet: FINALLY A WOMEN RUNNING FOR PRESIDENT #SemST
Answer: yes.
Tweet: Australia even has a fucking Minister for women fucks
sake! IsAwful #SemST
Answer:
20

---

Under review as a conference paper at ICLR 2023
The prompts used for paraphrase detection, natural language inference, and topic classifi- cation. The prompis for MedQ-Pairs, MRPC, SICK, and RTE are taken from Min et al. (2022b), and the prompt for AGNews, TREC, and DBPedia are taken from Zhao et al. (2021). We show one training example per task for illustration.
Table 3:
Task
Prompt
Labels
MedQ-Pairs
Determine if the two questions are equivalent or not.
equivalent, not
Question: After how many hour from drinking an antibiotic can drink alcohol? Question: have party tonight and took my last dose of Azithromycin this morning Can have few drinks?
Answer: equivalent.
Question: After many hour drinking an antibiotic can drink alcohol? Question: vomited this morning and am not if the side effect my antibiotic the alcohol I took last night...? Answer:
MRPC
The DVD-CCA then appealed to the state Supreme Court.
True, False
The question is: The DVD CCA appealed that decision to the U.S. Supreme Court? True False?
The is: True.
The Nusdaq composite index increased 10.73, or 0.7 percent, to 1,514.77.
The question is: The Nasday Composite index, full of technology stocks, was lately up around points? True or False? The answer
SICK
The young boys are playing outdoors and the man is smiling nearby.
Truc, False,
The question is: The kids are playing outdoors near a man with a smile? True or False?
Not sure
The answer is: True.
Two people kickhoxing and spectators are not watching. The question is: Two people are kickboxing and spectators watching? True or False?
The answer is:
RTE
The Armed Forces Press Committce (COPREFA) admitted that the govermment troops sustained 11 casualties these clashes, adding that they inflicted three casualties on the rebels.
Truc, False
The question is: Three rebels were killed by government troops? True or False?
The answer is: True.
Gastrointestinal bleeding happen adverse effect of non-steroidal anti-inflammatorr drugs such aspirin or ibuprofen.
The question is: Aspirin prevents gastrointestinal bleeding. True or False? The answer is:
AGNews
Article: Bush, Republicans Outpoll Kerry, Democrats on TV (Reuters) Reuters Although the election is until.
World, Sports,
Business, Science
Answer: World.
Article: Baseball Today (AP) AP- APChiicago at Montreal (7:05 p.m. EDT)) Greg Maddux (12-8) starts for the Cubs.
Answer:
TREC
Classify the questions based whether their answer type is Number, Location, Person, Description, Entity, Abbreviation.
Description, Entity.
Abbreviation,
Question: What are liver enzymes? Answer Type: Description.
Person, Number,
Location
Question: What is considered the costliest disaster the insurance industry has ever faced?
Answer Type:
DBPedia
Classify the documents based on whether they about Company, School, Artist, Athlete. Politician, Transportation, Building. Nature, Village. Animal, Plant, Album, Film, or Book.
Company, School, Artist. Athlete. Politician, Transportation, Building. Nature, Village, Animal, Plant, Album, Film, Book
Article: CTB Bank the second-biggest commercial bank in Hungary after January 2008 merger with Inter-Furípa Bank. This follows the 2007 merger of their respective Italian parent companies Banca Intesa and Sanpaolo IMI to form Intesa Sanpaolo.
Answer: Company.
Article: Adarsh Vidya Kendra is school in India
Answer:
21

---

Under review as a conference paper at ICLR 2023
Table The prompts used for the toy tasks: Unnatural and SST-2-A/B. The prompt for Unnatural is taken from Rong (2021) and the prompt for SST-2-A/B is taken from the SST-2 prompt in Zhao al. (2021). We show two training examples per task for illustration.
Task
Prompt
Labels
SST-2-A/B
Review: Well-rounded tribute.
A, B
Answer: A.
Review: Saw how bad this movie was.
Answer: B.
Review: Skip this dreck.
Answer:
Unnatural
Consider the categories plant/vegetable, sport, and animal. Classify each object in its category.
animal, plant/vegetable, sport
onions: plant/vegetable.
hockey: sport.
horse:
22

---

Under review as a conference paper at ICLR 2023
Table The different prompt formats used for SST-2 from Zhao al. (2021). We show one training example for illustration.
Format ID Prompt
Labels
Review; Well-rounded tribute.
Positive, Negative
Answer: Positive.
Review:
Saw bad movie was.
Answer:
Review: Well-rounded tribute.
good.
Answer: good.
Review: Saw how movie was.
Answer:
My review for last night's film: Well-rounded tribute. The critics agreed that this movie was good.
good,
My review for night's film: how bad this movie The critics agreed that this movie
Here what our crities think for monthss films.
positive, negutive
One of our crities wrote "Well-roumded tribute." Her sentiment towards the film was positive.
One of our critics wrote "Saw how had this movie was." Her sentiment towards the film was Critical reception edit
good, had
contemporary review, Roger Ebert wrote "Well rounded tribute." Entertainmeet Weekly agreed, and the overall critical reception of the film good.
contemporary review, Roger Ebert "Saw how bad this movie Entertainment Weekly agreed, and the overall critical reception of the film was
Review: rounded tribute.
Yes, No
Positive Review? Yes.
Review: Saw how bad movic Positive Review?
Review: rounded tribute.
Positive, Negative
Question: the sentiment the above review Positive Negutive?
Answer: Positive.
Review: Saw how bad this movie was.
Question:
Is the sentiment of the above review Positive or Negutive?
Answer:
8
Review: Well rounded tribute.
good, had
Question: Did the author think that movic was good or bad?
Answer: good.
movie was. Question: Did the author think that the movic good bad?
Review: Saw how
Answer:
Question: Did the author of the following think that the movie good bud? Tweet:: Well rounded tribate.
good, bad
Answer: good.
Question: Did the author of the following tweet think that the movie was good or bad?
bud this movie was.
Answer:
Well rounded trihute. My overall feeling was that the movie was good.
good,
Saw how bad this movie was. My overall feeling was that the movie was
11
Well rounded tribute. liked the movie.
liked, hated
Saw how
movie was.
12 Well rounded tribute. My friend asked me if would give the movie O or 5 stars, said 5.
0,
Saw how bad this movie was. My friend asked me if would give the movie or stars, said
Input: rounded tribute. Sentiment: Positive.
Positive, Negative
Input:
Saw how bad this movie
Sentiment:
Review: rounded tribute.
True, False
Positive: True.
Review:
Saw how bad movic
Positive:
Review; rounded tribute.
Stars:
Review:
Saw how bad movic
Stars:
23


Reviews:
[{"review_id": "DSwi3WUHH5S", "reviewer": "Reviewer_TBuw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates what happens if LLMs are prompted with false demonstrations: the LLMs will output false answers. Authors further investigate the underlying mechanism at a neuron level for this phenomenon and discovered that early exiting would reduce wrong labels. Furthermore, authors identified induction heads that are responsible for outputting wrong labels, showing empirical evidence that LLMs are “post-processing” the truth at later layers, mainly with a few(~15) induction heads.   ", "summary_of_the_review": "I recommend accept as I think this paper provided significant empirical insights on a significant problem, and the presentation of this paper is clear, and the logical flow is coherent.", "strength_and_weaknesses": "Strength: Clarity, quality (see next section)\nWeakness: technical novelty (see next section)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DI-Fx3cb4J", "reviewer": "Reviewer_PZwM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on in-context learning where the model performs a classification task without gradient updates by reading a few labeled examples as part of the input (demonstrations). Specifically, the paper consists of three parts: (1) false demonstrations, where each label in the demonstrations is re-assigned based on the permutation of labels (e.g. positive->negative, negative->positive), leads to significant performance drop, due to the model choosing the permuted true label instead of the true label – which is not desired, because we want our model to be robust to false demonstrations. (2) This is due to the presence of “false prefix-matching heads”, a subset of attention heads that assign high attention score to labels in the demonstrations paired with the input that share the true label as the test input, and they mostly present in the later layers of the Transformer. (3) Zero-ing out those attention heads significantly reduces the gap in performance between true demonstrations and false demonstrations.\n", "summary_of_the_review": "In summary, I think this paper has a set of very interesting findings about in-context learning and novel analysis on the connection between attention heads and in-context learning. It is well-written and is very easy to follow. However, I have a critical concern in the most important underlying assumption in the paper – that the “false demonstrations” in the paper is not actually the false demonstrations but is rather a re-definition of the task, and it is unclear why the goal is to make the model predict the original, true label. If authors/other reviewers agree with this, this means the paper has to be almost completely rewritten: for instance, the authors can re-run the same experiments/analysis with random labels (instead of permuted labels) and likely the same findings will still hold; or alternatively, the paper can be re-written to be about how the model is able to perform the task when the task is redefined with permuted labels whose semantic meaning doesn’t match with pre-training, attribute this to a few attention heads, and prove it is true by showing zero-ing out such attention heads significantly degrades performance again redefined (permuted) true labels.\n", "strength_and_weaknesses": "### Strength\n* The topic is interesting and timely. It has a set of interesting analyses in in-context learning.\n* It includes detailed analysis on how attention heads contribute to in-context learning by attending to labels in the demonstrations that share the same true label as the test input. In fact, as far as I know, this is the first work that shows how in-context learning works with respect to attention heads (while Olsson et al identified such induction heads, they focus on language modeling rather than in-context learning with real NLP datasets). This analysis is also supported by experiments that measure the impact of zero-ing those attention heads. It is also quite clever to use synthetic dataset (called “unnatural dataset” in the paper) to find false prefix-matching heads and use them for NLP datasets that are real tasks.\n\n\n### Weaknesses\nThe entire paper is based on the assumption that the model has to be robust to false demonstrations, which is not convincing. First of all, the demonstrations that are called “false demonstrations” in the paper are hardly “false” in my opinion. They still preserve the valid mapping between inputs and (an abstract notion of) labels, but the surface form of the labels are permuted. For instance, false demonstrations in the binary sentiment classification (positive->negative, negative->positive) can be seen as true demonstrations whose defined task is “map a positive review to the word ‘negative’ and a negative review to the word ‘positive’”. Therefore, it is not undesirable that the model predicts a permuted label instead of a true label (e.g., predicts ‘negative’ to a positive review), since this is the valid task defined by the demonstrations. In fact, some analysis in the paper (e.g. Figure 2b) shows that the model performance against original true labels degrades only because the model accurately predicts the permuted true label. I would be more convinced by the idea if the false demonstrations were demonstrations with random labels, because then the task defined by the demonstrations is not the valid task.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B-J_Jt6Tjq-", "reviewer": "Reviewer_4Jp2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs an extensive analysis on how language models suffer from incorrect demonstrations, and which layers/heads contribute to such performance degradations. The authors further propose a few ways for mitigating this problem, e.g., zeroing-out after critical layers, and ablating the false prefix-matching heads, and show they can effectively reduce the performance gap between correct and incorrect demos.", "summary_of_the_review": "Overall I think this paper presents some quite interesting findings, in terms of under incorrect demonstrations, what's the contribution of each layer and how the gap between incorrect demos and correct demos can be minimized via zeroing-out certain heads. I do have some doubts in terms of the robustness of the analysis and its generalizability across different/unseen tasks, so not sure if the method can be applied generally to any task, which requires further study. Thus I recommend weak rejection for now.", "strength_and_weaknesses": "Strengths:\n- The findings that later layers contribute more to the errors under incorrect demonstrations are quite novel.\n- The proposed method of zeroing-out later layers or certain heads seem to be effective in reducing the gaps between correct and incorrect demonstrations.\n\nWeaknesses:\n- I think the overall analysis presents quite interesting findings but the *robustness* and *generalizability* of the analysis is questionable. There are many arbitrary choices in performing the analysis/mitigation, e.g.,\n\n1)  to ablate the heads in Section 4, for the head choice (Appendix A.1 has the details), the authors \"first consider the 25 heads with the highest label-attending score, then select the 10 heads with the highest class-sensitivity score\"; further some bias was added \"towards heads in the later layers, like selecting the 5 heads of 25 that belong to layers 20 and later\". How are those choices made? They look very arbitrary to me and I'm not sure if it can generalize to other models or other tasks.\n\n2) the early-exit strategy in Figure 3, it's true that the overall trend shows later layers contribute more to the enlarged gaps, but the gap happens at different layers for different tasks. Zeroing out all layers after 16 might lead to an overall better performance, but for many tasks the performance under incorrect demos have already started degradation long before layer 16. In addition, this layer choice is based on the overall pattern analysis on these 8 datasets, would this conclusion generalize to a new task/dataset? \n\n3) how would people apply the method in practice? From Figure 3, the accuracy under correct demos are highly sensitive to the #layers. Would early-existing sometimes cause significant performance drop if not applied carefully after the critical layer? In addition, it seems like people need to perform extensive analysis for each model/task combination to identify the critical layers / false prefix-matching heads (as they all seem different across models/tasks), how can one apply the method in a scalable way for any new/unseen tasks?\n\n- In Table 1, any analysis to explain why the ablation doesn't work for natural language inference (SICK)? Are there more tasks in this category analyzed to show is it a single-task failure or the method doesn't work for the entire category?\n\n- As an analysis paper, some of the key ablations are missing from the study, e.g., the false demos permute the label space for the entire demonstration. What if some of the labels remain correct and some of them are wrong? Would that change the conclusion significantly?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Kx2DeXV4ZY", "reviewer": "Reviewer_gm7Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how pretrained language models might be affected by **incorrect** in-context examples at different layers. By using the \"logit lens\" approach and analyzing the attention heads, they found such the copying or generation of these incorrect labels emerge in the later layers and some specific attention heads. After removing these components, they demonstrate the models are less affected by the incorrect labels.", "summary_of_the_review": "This paper provides a deeper analysis of how PLMs perform at different layers and attention heads when presented with incorrect labels, and they find that generation of incorrect labels is mostly related to the later layers and specific heads. So, they propose to remove these components and show that this mitigates the effect of incorrect labels. Although it's interesting to see such an analysis of in-context learning, the findings are expected overall, and the proposed method cannot really compete with the performance of full language models. ", "strength_and_weaknesses": "Strengths:\n* This paper takes a closer look at how the model will perform at different layers & attention heads when presented with incorrect labels. The findings generally confirm that PLMs are still largely impacted by incorrect labels, which is a bit contradictory to the messages in Min et al., 2022* but aligns with Yoo et al., 2022*. The experiments look convincing.\n* It's also interesting that the method of \"logit lens\" can work with the hidden representation at different layers without further training. After zeroing out the later layers, the final output layer can still work.\n\nWeaknesses:\n* My biggest concern is on the underlying assumption of this paper---the model should not be impacted by the incorrect labels. It's controversial here because when we say in-context \"learning\", we actually want the model to be truthful to the provided demonstration examples. I am willing to hear the author's thoughts on this.\n* Despite that this work might be the first to show that the copy of incorrect labels emerges in the later layers, this finding is not that surprising considering the early layers are mostly responsible for fusing information, and the later layers are mainly for generating the final output (see the references).\n* Regarding the proposed method, although removing the particular components can mitigate the negative impact of incorrect labels, it's also questionable how this will affect the generation capability. It would be helpful if the author can show how much this will change the model's performance when the labels are correct.\n\nReferences:\n\n\\* Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?\n\n\\* Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Taeuk Kim. Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations \n\n\\* Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations\n\n\\* Jesse Vig, Yonatan Belinkov. Analyzing the Structure of Attention in a Transformer Language Model", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject