Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how pretrained language models might be affected by **incorrect** in-context examples at different layers. By using the "logit lens" approach and analyzing the attention heads, they found such the copying or generation of these incorrect labels emerge in the later layers and some specific attention heads. After removing these components, they demonstrate the models are less affected by the incorrect labels.
strength_and_weaknesses: Strengths:
* This paper takes a closer look at how the model will perform at different layers & attention heads when presented with incorrect labels. The findings generally confirm that PLMs are still largely impacted by incorrect labels, which is a bit contradictory to the messages in Min et al., 2022* but aligns with Yoo et al., 2022*. The experiments look convincing.
* It's also interesting that the method of "logit lens" can work with the hidden representation at different layers without further training. After zeroing out the later layers, the final output layer can still work.

Weaknesses:
* My biggest concern is on the underlying assumption of this paper---the model should not be impacted by the incorrect labels. It's controversial here because when we say in-context "learning", we actually want the model to be truthful to the provided demonstration examples. I am willing to hear the author's thoughts on this.
* Despite that this work might be the first to show that the copy of incorrect labels emerges in the later layers, this finding is not that surprising considering the early layers are mostly responsible for fusing information, and the later layers are mainly for generating the final output (see the references).
* Regarding the proposed method, although removing the particular components can mitigate the negative impact of incorrect labels, it's also questionable how this will affect the generation capability. It would be helpful if the author can show how much this will change the model's performance when the labels are correct.

References:

\* Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work?

\* Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Taeuk Kim. Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations 

\* Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations

\* Jesse Vig, Yonatan Belinkov. Analyzing the Structure of Attention in a Transformer Language Model
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The methods mostly follow the existing "logit lens" and "induction heads" approaches, but apply them into their analysis. The findings from the analysis are generally useful but might not be novel enough. 
summary_of_the_review: This paper provides a deeper analysis of how PLMs perform at different layers and attention heads when presented with incorrect labels, and they find that generation of incorrect labels is mostly related to the later layers and specific heads. So, they propose to remove these components and show that this mitigates the effect of incorrect labels. Although it's interesting to see such an analysis of in-context learning, the findings are expected overall, and the proposed method cannot really compete with the performance of full language models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper performs an extensive analysis on how language models suffer from incorrect demonstrations, and which layers/heads contribute to such performance degradations. The authors further propose a few ways for mitigating this problem, e.g., zeroing-out after critical layers, and ablating the false prefix-matching heads, and show they can effectively reduce the performance gap between correct and incorrect demos.
strength_and_weaknesses: Strengths:
- The findings that later layers contribute more to the errors under incorrect demonstrations are quite novel.
- The proposed method of zeroing-out later layers or certain heads seem to be effective in reducing the gaps between correct and incorrect demonstrations.

Weaknesses:
- I think the overall analysis presents quite interesting findings but the *robustness* and *generalizability* of the analysis is questionable. There are many arbitrary choices in performing the analysis/mitigation, e.g.,

1)  to ablate the heads in Section 4, for the head choice (Appendix A.1 has the details), the authors "first consider the 25 heads with the highest label-attending score, then select the 10 heads with the highest class-sensitivity score"; further some bias was added "towards heads in the later layers, like selecting the 5 heads of 25 that belong to layers 20 and later". How are those choices made? They look very arbitrary to me and I'm not sure if it can generalize to other models or other tasks.

2) the early-exit strategy in Figure 3, it's true that the overall trend shows later layers contribute more to the enlarged gaps, but the gap happens at different layers for different tasks. Zeroing out all layers after 16 might lead to an overall better performance, but for many tasks the performance under incorrect demos have already started degradation long before layer 16. In addition, this layer choice is based on the overall pattern analysis on these 8 datasets, would this conclusion generalize to a new task/dataset? 

3) how would people apply the method in practice? From Figure 3, the accuracy under correct demos are highly sensitive to the #layers. Would early-existing sometimes cause significant performance drop if not applied carefully after the critical layer? In addition, it seems like people need to perform extensive analysis for each model/task combination to identify the critical layers / false prefix-matching heads (as they all seem different across models/tasks), how can one apply the method in a scalable way for any new/unseen tasks?

- In Table 1, any analysis to explain why the ablation doesn't work for natural language inference (SICK)? Are there more tasks in this category analyzed to show is it a single-task failure or the method doesn't work for the entire category?

- As an analysis paper, some of the key ablations are missing from the study, e.g., the false demos permute the label space for the entire demonstration. What if some of the labels remain correct and some of them are wrong? Would that change the conclusion significantly?
clarity,_quality,_novelty_and_reproducibility: Clarify: the paper is very clear.

Quality: the paper presents some interesting findings.

Novelty: the finding that later layers contributed more to wrong predictions under incorrect demonstrations is quite novel.

Reproducibility: no code is provided, not sure if the analysis in the paper can be reproduced.
summary_of_the_review: Overall I think this paper presents some quite interesting findings, in terms of under incorrect demonstrations, what's the contribution of each layer and how the gap between incorrect demos and correct demos can be minimized via zeroing-out certain heads. I do have some doubts in terms of the robustness of the analysis and its generalizability across different/unseen tasks, so not sure if the method can be applied generally to any task, which requires further study. Thus I recommend weak rejection for now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates what happens if LLMs are prompted with false demonstrations: the LLMs will output false answers. Authors further investigate the underlying mechanism at a neuron level for this phenomenon and discovered that early exiting would reduce wrong labels. Furthermore, authors identified induction heads that are responsible for outputting wrong labels, showing empirical evidence that LLMs are “post-processing” the truth at later layers, mainly with a few(~15) induction heads.   
strength_and_weaknesses: Strength: Clarity, quality (see next section)
Weakness: technical novelty (see next section)

clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is written clearly, reading it was enjoyable. 
Quality: This paper is of high quality, the empirical evidence provided is very strong, the logical flow of the experiments and presentation is coherent and consistent. 
Novelty: The problem itself is an obvious next step for AI Safety and alignment research, authors’ approach was mainly based on prior work, but the combination of prior methods with this problem is novel. This makes this paper somewhat novel but not significantly novel since there is little algorithmic innovation. But given the significance of the problem itself and the insights obtained, I still think this paper is above the ICLR standard.

summary_of_the_review: I recommend accept as I think this paper provided significant empirical insights on a significant problem, and the presentation of this paper is clear, and the logical flow is coherent.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on in-context learning where the model performs a classification task without gradient updates by reading a few labeled examples as part of the input (demonstrations). Specifically, the paper consists of three parts: (1) false demonstrations, where each label in the demonstrations is re-assigned based on the permutation of labels (e.g. positive->negative, negative->positive), leads to significant performance drop, due to the model choosing the permuted true label instead of the true label – which is not desired, because we want our model to be robust to false demonstrations. (2) This is due to the presence of “false prefix-matching heads”, a subset of attention heads that assign high attention score to labels in the demonstrations paired with the input that share the true label as the test input, and they mostly present in the later layers of the Transformer. (3) Zero-ing out those attention heads significantly reduces the gap in performance between true demonstrations and false demonstrations.

strength_and_weaknesses: ### Strength
* The topic is interesting and timely. It has a set of interesting analyses in in-context learning.
* It includes detailed analysis on how attention heads contribute to in-context learning by attending to labels in the demonstrations that share the same true label as the test input. In fact, as far as I know, this is the first work that shows how in-context learning works with respect to attention heads (while Olsson et al identified such induction heads, they focus on language modeling rather than in-context learning with real NLP datasets). This analysis is also supported by experiments that measure the impact of zero-ing those attention heads. It is also quite clever to use synthetic dataset (called “unnatural dataset” in the paper) to find false prefix-matching heads and use them for NLP datasets that are real tasks.


### Weaknesses
The entire paper is based on the assumption that the model has to be robust to false demonstrations, which is not convincing. First of all, the demonstrations that are called “false demonstrations” in the paper are hardly “false” in my opinion. They still preserve the valid mapping between inputs and (an abstract notion of) labels, but the surface form of the labels are permuted. For instance, false demonstrations in the binary sentiment classification (positive->negative, negative->positive) can be seen as true demonstrations whose defined task is “map a positive review to the word ‘negative’ and a negative review to the word ‘positive’”. Therefore, it is not undesirable that the model predicts a permuted label instead of a true label (e.g., predicts ‘negative’ to a positive review), since this is the valid task defined by the demonstrations. In fact, some analysis in the paper (e.g. Figure 2b) shows that the model performance against original true labels degrades only because the model accurately predicts the permuted true label. I would be more convinced by the idea if the false demonstrations were demonstrations with random labels, because then the task defined by the demonstrations is not the valid task.


clarity,_quality,_novelty_and_reproducibility: * The clarity, quality, novelty and reproducibility of the paper are all very good in general.
* “False demonstrations”: As I briefly mentioned earlier, I think the term “false demonstrations” might be too abstract and broad to describe what the paper actually does. It would be better to say “demonstrations with permuted labels”.

summary_of_the_review: In summary, I think this paper has a set of very interesting findings about in-context learning and novel analysis on the connection between attention heads and in-context learning. It is well-written and is very easy to follow. However, I have a critical concern in the most important underlying assumption in the paper – that the “false demonstrations” in the paper is not actually the false demonstrations but is rather a re-definition of the task, and it is unclear why the goal is to make the model predict the original, true label. If authors/other reviewers agree with this, this means the paper has to be almost completely rewritten: for instance, the authors can re-run the same experiments/analysis with random labels (instead of permuted labels) and likely the same findings will still hold; or alternatively, the paper can be re-written to be about how the model is able to perform the task when the task is redefined with permuted labels whose semantic meaning doesn’t match with pre-training, attribute this to a few attention heads, and prove it is true by showing zero-ing out such attention heads significantly degrades performance again redefined (permuted) true labels.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper explores the impact of providing false labels during few shot learning. It demonstrates that false labels can cause incorrect predictions from a language model, and shows that the effect can be mitigated by zero-ing certain heads in later layers that attend to the incorrect answers. 

The paper is a borderline case. Reviewers disagreed somewhat with how relevant and interesting they find this experimental set up, how surprising the results are, different intuitions on what the model 'should' do on this task, and how well they expect the analysis to generalize to other datasets. The revision has partially addressed these concerns - however, the consensus in discussion was that the paper needs a bit more work before acceptance.

justification_for_why_not_higher_score: In the online meeting, all reviewers stated that they were happy with a reject decision.
justification_for_why_not_lower_score: n/a
Decision: Reject