Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell
Authors: Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, Daniel Khashabi


Reviewer paper summaries: This paper investigates a scenario where a large language model (LLM) may fail to generate the correct answer from a list of documents (values) containing a 'gold' document, due to the position of the gold document. The authors demonstrate that although the final output may not incorporate information from the gold document, the model's intermediate representations still correlate with it. This is illustrated by training linear classifiers to predict the index of the gold document using these intermediate representations as features.
Summary of strengths: 1. The exploration of whether intermediate representations retain information about the gold document is intriguing and could significantly enhance understanding of positional bias within the community.
2. The paper is articulate and well-structured, making it a commendable read.
Summary of weaknesses: The experimental design exhibits a notable flaw: comparing the accuracy of linear classifiers, which have access to the index of the gold document, with the accuracy of the generated output is inherently biased. The classifiers' foreknowledge of the gold document's index gives them an undue advantage, skewing the comparison.

A more equitable approach would involve dividing the dataset into training and test splits and evaluating the discrepancy in performance across these splits to ensure a fair assessment of the model's capabilities.
Comments, suggestions and typos: Please see Weaknesses.
Overall assessment scores: 3