Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yapei Chang
authors: Kyle Lo
authors: Tanya Goyal
authors: Mohit Iyyer
reviews: 
Comments: The paper investigates book-length summarization. It proposes an automatic, LLM-based, reference- and source-free metric to evaluate the coherence of a summary. Using this metric, two different book-length summarization techniques are evaluated, and the evaluation is compared to human evaluation.
Strengths: - The proposed metric is reference- and source-free, and thus has broad applicability. It also follows recently proposed best practices.
- To reduce the likelihood that LLMs have seen this data before, the paper used an evaluation dataset based on recently published books.
- The authors promise to release code and annotations.
Weaknesses: - The evaluation of human annotations focuses only on precision and does not investigate recall. However, the authors are open about this limitation.
- Some of the examples of types of errors shown in Table 1 seem to be of questionable quality; for example, the first example about the "mysterious man" might as well be unanswered in the book (although I do not know the contents of the book in question). This may be a side effect of offering a monetary reward based in part on the number of annotations (cf. Appendix G).
- The sentence-level score may disproportionately favor summaries that contain a large number of (short) sentences. Furthermore, the evaluation of different models does not further investigate whether some of the score differences can be explained by the different length of the summaries (e.g., a shorter summary may be more prone to omissions).
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper tackles the problem of summarization of books. This is an ambitious task (that would have been considered unattainable a few years ago), as well as very expensive (because any annotation or even evaluation is very cognitive intense and long). Very little research is done in that area so far, and most of it is based on automatically scraped material.
Strengths: There is much to like in this paper. It is a great idea, and has been very, very well executed.

- The authors obtain 100 recent books from the last year - while reviews of those books exist those are not summaries. I would be less certain that those books or their summaries are _impossible_ to be part of the training corpus. Modern LLM go well beyond only crawling Internet data and providers have deals with many different publishers. Still, this is arguably one of the most rigorous attempts to create a test set that avoids dataset contamination.

- The evaluation focuses on precision. The standard approach of evaluation summaries is to rely on human-written summaries and the compare those with system-generated. This is very expensive, and creates lots of pitfalls as it depends on the similarity metric used to compare summaries (even without using n-gram overlap metrics, learnt embedding metrics like BERTScore have their own set of biases). Instead, this work focuses on assessing a given summary by asking questions about it.

- It compares the two standard ways of summarizing very long documents: hierarchical (summarizing parts and combining those) and incremental (left to right) with interesting insights  (eg, incremental is preferred for level of detail, but not for other aspects)


This work will certainly be considered a fundamental paper in the coming months and years, and be a required reference and reading for summarization research. It jumps right into what is still difficult to do with modern LLMs
Weaknesses: I have very little concerns about this paper. It is however unfortunate that the evaluation framework did not consider faithfulness. This could have been done by asking annotators to assess each fact and search for it inside the book to verify its factuality.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper addresses the problem of book-length summarization evaluation, which refers to documents with 100k+ tokens. It identifies the following main challenges: 1) the problem of data contamination, that is, most books are already in the pre-training data of large language models; 2) the lack reliable automatic metrics. To solve the first issue, the authors collect a dataset set of 100 recently published books. To address the second issue, they generate summaries for each book with GPT-4, and then ask humans to identify and classify coherence errors such as "entity omission", "causal omission", and "duplication". Then, to automate this process, they prompt GPT-4 to classify sentences in summaries according to the same coherence error taxonomy, and they find the precision of GPT-4 is similar to the human annotators. The coherence annotation by GPT-4 is the basis of BooooKScore, a reference-free automatic metric counts the fraction of sentences in summaries that are free from coherence issues. Finally, the paper evaluates summaries generated by GPT-4, Claude 2, and LLama-2-7B using two techniques: hierarchical merging and incremental updating. They find that GPT-4 and Claude 2 summaries are more coherent and that hierarchical merging also results in more coherent summaries, but at a cost of detail.
Strengths: The paper addresses an important problem in summarization, proposing solutions to scale the evaluation of very long documents that are longer (for now) than current context sizes of large language models. The effort applied in validating a reference-free automatic metric with newly collected books and human annotation are relevant contributions for future work in summarization.
Weaknesses: One point I miss in the experiments is a baseline without hierarchical merging or incremental updating. The reason is that a significant fraction of books have length around 100k tokens or less (if we observe the statistics of BookSum, for instance), and it would be interesting to see if the hierarchical merging or incremental updating introduce (or not) a high quantity of coherence issues compared to a vanilla LLM approach. Even with some level of truncation, it should still be possible to assess coherence issues.

Minor issue: you mention in section 3 that "we did not find summaries online for any books in our dataset, which means LLM memorization of these summaries is impossible." Not finding results online by no means imply that memorization is impossible. In fact, we have no guarantee that closed-source models such as GPT-4 are trained just on publicly available data.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: This paper provides a systematic empirical survey of the coherence problem in book-length summarization with LLMs. The contributions include a novel evaluation protocol, an automatic metric for coherence assessment, and a systematic evaluation of different LLMs.
Strengths: The paper presents a very systematic survey, with solid and extensive experiments. The writing is clear and the narrative is easy to follow, facilitating understanding of complex concepts. The creation of the BOOOOKSCORE metric is particularly innovative, offering a decent solution to the cost and time constraints of human evaluation.
Weaknesses: A potential weakness lies in the design of the BOOOOKSCORE metric. The approach of weighing all sentences equally may overlook the varying importance of different parts of the text for overall coherence. Additionally, there is an absence of a consistency check for the evaluation metric.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent