A Realistic Evaluation of LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3
Authors: Gaspard Michel, Elena V. Epure, Romain Hennequin, Christophe Cerisara


Reviewer paper summaries: The paper investigates the capabilities of Large Language Models (LLMs), specifically focusing on the instruction fine-tuned version of Llama3, in performing quotation attribution tasks within novels. The authors explore the issue of memorization and data contamination in LLMs, which can skew the assessment of their validity. They introduce a realistic evaluation methodology that considers the degree of book memorization and annotation contamination. Using Llama3 as a case study, they demonstrate that while the model's performance is positively correlated with the level of book memorization, it still performs impressively on books it hasn't memorized or seen before. The paper contributes a structured evaluation approach that could be extended to other literary tasks, offering insights into Llama3's quotation attribution abilities and its generalization to unseen novels.
Summary of strengths: The investigation of Llama3's performance on unseen docs demonstrates its generalization capability, which is crucial for evaluating the robustness of LLMs beyond memorization-based success.
Summary of weaknesses: Although the paper aims to mitigate the effects of annotation contamination, it does not provide a definitive method for detecting such issues.

While the paper acknowledges the importance of reasoning capabilities, there might be an overemphasis on memorization metrics, which could lead to the ignoration of the true reasoning skills of LLMs in handling unseen data.

Related work has explored LLM memorization a lot. There is a lack of novel findings and aspects of this paper regarding the book memorization and data contamination task.
Comments, suggestions and typos: N/A
Overall assessment scores: 2.5