LCFO: Long Context and Long Form Output Dataset and Benchmarking
Authors: Marta R. Costa-jussà, Pierre Andrews, Joy Chen, Joe Chuang, Mariano Coria Meglioli, David Dale, Alexandre Mourachko, Christophe Ropers, Eduardo Sánchez, Holger Schwenk, Tuan Tran, Arina Turkatenko, Carleigh Wood
Accepted at: acl25

LCFO:
Long Context and Long Form Output Dataset and Benchmarking

Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, Tuan Tran, Arina Turkatenko, Carleigh Wood
FAIR, Meta
{costajussa,mortimer,mfcoria,joyqchen
joe.chuang,daviddale,chrisropers,alexmourachko
eduardosanchez,schwenk,tuantran,arinatur,carleighwood}@meta.com

Abstract

This paper presents the Long Context and Form Output (LCFO) benchmark, a novel evaluation framework for assessing gradual summarization and summary expansion capabilities across diverse domains. LCFO consists of long input documents (5k words average length), each of which comes with three summaries of different lengths (20%, 10%, and 5% of the input text), as well as approximately 15 questions and answers (QA) related to the input content. Notably, LCFO also provides alignments between specific QA pairs and corresponding summaries in 7 domains.

The primary motivation behind providing summaries of different lengths is to establish a controllable framework for generating long texts from shorter inputs, i.e. summary expansion. To establish an evaluation metric framework for summarization and summary expansion, we provide human evaluation scores for human-generated outputs, as well as results from various state-of-the-art large language models (LLMs).

GPT-4o-mini achieves best human scores among automatic systems in both summarization and summary expansion tasks (\( \approx +10\% \) and \( +20\% \), respectively). It even surpasses human output quality in the case of short summaries (\( \approx +7\% \)). Overall automatic metrics achieve low correlations with human evaluation scores (\( \approx 0.4 \)) but moderate correlation on specific evaluation aspects such as fluency and attribution (\( \approx 0.6 \)).

1 Introduction

Robust long text generation capabilities are required to meet user demand for extensive content creation, including story writing and essay composition (Xie and Riedl, 2024), which is why recent models such as GPT-4 (et al., 2024b) are expanding the output lengths from 4k tokens in GPT-4o to 64k in the latest versions.

However, evaluating the performance of Large Language Models (LLMs) on summarization and summary expansion tasks (see definitions in Section 2.1) is particularly challenging, especially when it comes to summarizing very long input documents and generating either long summaries or long summary expansions. Although there is a lot of work and interest in studying summarization evaluation (e.g. Zhang et al., 2024a), evaluation of long text outputs is an emerging area (Que et al., 2024). Indeed, long-context input processing tasks (such as summarization or comprehension question answering applied to long documents) and long-form output production both involve high cognitive loads for humans. This may be why evaluation work in these areas is not as mature as in others.

To complement the summarization task, we include an “inverted” task of summary expansion: generating a longform text based on its shorter sketch. It requires more creativity than summarization (which can be approached extractively) but imposes more constraints than open-ended language modeling. A practical application for this task could be with a writing assistant expanding the text sketch created by the human writer into a longer output that the human subsequently revises. Another potential use is “back-translation” for summarization, with a model training to summarize the synthetic expanded documents into their original short forms.

The LCFO dataset that we are presenting in this paper is an exclusively human annotated benchmark and challenge dataset involving natural language understanding and generation across multiple domains in the aforementioned tasks. Our dataset is carefully manually crafted with human revision from the selection of the documents to the final annotation, without relying on LLMs at any point. We provide detailed linguistic guidelines and abstractive QA. Furthermore, we provide another set of linguistic guidelines to evaluate the tasks of
Figure 1: LCFO annotation and evaluation framework for the gradual summarization (top) and summary expansion (bottom) tasks. The resulting benchmark can also be used for other tasks such as reading comprehension and automatic evaluation metric development. In gradual summarization, long documents are split into paragraphs and annotated by humans to generate 3 level of summaries, plus a questions-answers set. Additionally, different LLMs were used to construct automated summaries at corresponding levels. In summary expansion, different human summaries are prompted to LLMs to generate the output targeting the same length as the source document. The components are described in different sections (marked in parentheses)

summarization and summary expansion.

Together, we present the LCFO benchmark. Given a source of long structured documents, we generate multiple long outputs and associated QA pairs, and we evaluate human and model outputs with human annotations. The main contributions of this work are the following (see Figure 1 for a schematic representation of the dataset and tasks):

• Dataset creation with structured inputs and alternative references; each input document is associated with 3 summaries of different lengths (20%, 10%, 5%). Gradual summarization is useful in that it provides both long and short summaries references. Moreover, the availability of summaries of various length is expected to improve the development of summary expansion approaches, which allows us to provide a more controllable summary expansion framework.
• A set of QA pairs for each input document aligned with each of the different length summaries. Our QA pairs are in free-form short-answer format (i.e. not multiple choice) and are of the abstractive type (i.e. they are not copies of parts of the source document). This QA can potentially be used to evaluate the model outputs, based on the appropriateness of the responses, similarly to previous proposals (Wang et al., 2022).
• Selection of automatic metrics. Most of these metrics can be used to evaluate at the paragraph level, and more widely can be used to evaluate summary expansion or long-form generation in multiple tasks and languages.
• Evaluation of several LLMs on our dataset both automatically and manually; and evaluation of automatic metrics on summarization and summary expansion.

2 The LCFO Creation Framework: Principles and Methods

This section describes the detailed principles and methods that underlie LCFO (Figure 1 (right)).

2.1 Definitions

Long context/form. We define long text as text that exceeds 5k words. For reference, the attention
span for reading and taking notes has been considered 10 to 15 minutes since a 1978 seminal paper (Hartley and Davies, 1978). More recent research cited in a survey paper (Bradbury, 2016) shows that attention paid to lectures declines significantly after 20 minutes. The tasks we describe here are closer to note taking than lecture attendance; therefore, we should keep the 10–15 minute reference. If we base ourselves on the reported reading average time for first-language, secondary-education level readers (125 words per minute on average), we can consider that we will reach high cognitive load at around 1.5k words, and peak cognitive load at around 2.5k words. The quality of cognitive processing then starts to decrease significantly after.

Structured/hierarchical. The input and output documents are partitioned into sections and, if necessary, nested sub-sections. Their structure is determined by task- and domain-specific guidelines.

Gradual summarization (GS). The input is a long document (defined here as 5k words or more). The summarization task can be described as the act of generating a much shorter corresponding document that includes the essential information contained in the long document and the same logical structure linking the various pieces of essential information. The summarization task presented here consists in taking long documents as inputs and generating three corresponding summaries of length that represent 20%, 10%, or 5% of the input document.1

Summary expansion (SE). The input is a short and concise document that has similar properties to those of a summary (that is, it is mainly a standalone document that abstracts from details). The summary expansion task can be described as the act of generating a much longer document that preserves the essential elements found in the corresponding short document as well as the logical structure that connects such elements. More specifically, the task presented here consists in taking summaries as inputs and generating 3 long documents of different lengths. Each of the 3 lengths is set such that an input summary represents either 5%, 10%, or 20% of its respective expanded documents. As this is a more freely generative task, an additional requirement to be taken into consideration is that of coherence (for example, the detailed information included in one generated sentence should not contradict that included in another sentence). We prompt the model only with one single summary and specify the length of the output we want (5x, 10x...). We do not include in the prompt longer summaries or the whole document, since they may be a possible output.

2.2 Data selection and preprocessing

Selection. We select input documents that cover different domains, which meet the desired average length and the structure requirement.
• We cover 7 domains including politics, news, Wikipedia, scientific, literature, conversational, and legal documents, with the format of documents and conversations. We source from 10 datasets: LexGLUE (Chalkidis et al., 2022), BookSum (Kryscinski et al., 2022), SQuality (Wang et al., 2022), FacetSum (Meng et al., 2021) JRC-Acquis (Steinberger et al., 2006), MultiUN, Wikipedia, GovReport (Huang et al., 2021), Summscreen (Chen et al., 2022), and Seahorse (Clark et al., 2023). The correspondence between the source data sets and the domains is shown in Table 1.
• The source documents are selected to be on average 5k words / document. We prefer documents with a hierarchical structure and containing relatively few numbers2, which are better suited for summarization and summary expansion tasks.
• We prioritize recent documents when the domains allow (e.g., Wikipedia, where articles have a significant amount of new information since 2024). We preprocess documents in structured domains to provide a flattened structure while keeping hierarchical markers that are readable to annotators and models. It also ensures a consistent format across datasets.
• We filter out documents that contain toxicity using the ETOX package (Costa-jussà et al., 2023) and add manual verification to ensure the high integrity of the selected documents.

1We did not explicitly instruct the human annotators to generate the shorter summaries hierarchically, by compressing the longer ones, but this is what they apparently did most of the time. Hence the term “gradual”: our data allows gradually interpolating between various levels of details in a text.

2We de-prioritize documents with many numbers (such as Wikipedia pages with large tables of various statistics), because our focus is less on structured data and more on natural language, and also because comparing large sets of numbers in the source and in the summary would be a difficult task for human annotators.
Preprocessing. To reduce the cognitive load for human annotation, we split paragraphs automatically (APS). Details on this paragraph splitting differ from corpus to corpus and are reported in the Appendix A.

2.3 Human summary and QA-pair generation

To obtain human-written summaries of long-form texts, detailed guidelines are developed (Appendix E). All summary writers must be native English speakers and have writing or editing experience.

These writers receive 252 long form documents (each around 5k words), and they are asked to read each document in its entirety and write three summaries for each document: the first summary representing around 20% of the length of the source document; the second and the third summaries representing further summarization — around 10% and 5% of the source document, respectively. When writing these summaries, the writers are tasked with compressing and retaining all the core ideas of the source. Giving a definition of a “core” or “main” idea of a text presented one of the challenges of our work with the writers. Each summary is supposed to be a cohesive standalone text that could be read and understood on its own.

The fact that the source documents represent different domains poses another challenge for the writers: they need to possess some knowledge and expertise in each of these. The documents are split into sections and paragraphs, and the writers are asked to keep the flow of the section/paragraph structure of the source text, trying to summarize it from top to bottom. However, we emphasize that the sentence by sentence summarization is exactly what we do not need, and the summaries need to be abstractive rather than extractive, which means copying the source text is strongly discouraged.

In addition, the writers are asked to provide a set of questions and answers for each long document. They need to compose at least 13–15 questions per 5,000 words. The answers are supposed to cover the points reflected in the summaries. We instruct the writers to produce open-ended, complex questions, which provide a good baseline for testing reasoning. For tracking and alignment purposes, each paragraph in each source document is given a number. The writers are asked to specify in which paragraph each answer can be found. They are also asked to indicate which of the summaries provides the answer to each question.

Besides the general guidance, we discovered that working with conversational content needed additional clarifications, so we prepared an additional document for working specifically with long-form text that contains conversations (such as plays or screenplays).

2.4 Automatic output and postprocessing

We want to understand how current state-of-the-art models perform on our new benchmark, both on the capability of comprehension a very long context and on the generation of long outputs. We conduct the automatic abstractive summarization for the former and summary expansion for the latter. We give details on the tasks below.

Gradual Summarization. We prompt the models with the human guidelines with a slight adaptation to be LLM friendly. We input the entire document without paragraph splitting. To give a fair evaluation, we prompt all LLMs in the zero-shot setting. To control the length of the LLM output, we have added additional instructions with the upper and lower bounds of the permissible words. For example, to ask the model to generate a summary of the R% length of the source text, the prompt contains "Make sure the summary has {y} words or less....Please write at least {x} words""", where x and y are determined per document with respect to the length and ratio R. In practice, we see that enforcing the length of the document right before and after the content block in the prompt gives consistent results. We give details of our summary prompts in the Appendix I.

Summary expansion. We customize prompts for each domain, plus the model-specific prompt templates. Similarly to the summarization task, the prompt contains instructions on the desired range of the generated text length. In addition, each prompt has instructions to guide the model in generating content of a certain quality (consistency, coherence, and keeping the main ideas in the summary). We prompt the model with specific formats for different domains as reported in the appendix I.

Automatic Paragraph Alignment (APA). We add this step to help human annotators evaluate the outputs that we are creating (as detailed in the next section 2.5). The task of comparing long inputs and outputs creates a high cognitive load on human evaluators. To reduce it, we provide an approximate alignment between the input paragraphs and the segments of the output, taking advantage of
the assumption that a summary usually follows the structure of the source document. First, we use dynamic programming to find a monotonic alignment path between input and output sentences that would maximize the sum of cosine similarities of the SONAR embeddings (Duquenne et al., 2023) of the two sentences. An output sentence could be aligned with multiple consecutive input sentences, potentially from different paragraphs, but we assign it to a single input paragraph with which it is aligned the most frequently. Thus, each input paragraph gets aligned to a contiguous output segment (potentially empty) in a monotonic way. This alignment helps the annotators navigate the input and output documents jointly.

2.5 Human Evaluation

To perform human evaluation on previously generated output, we design human evaluation guidelines inspired by previous works (Clark et al., 2023; Krishna et al., 2023; Que et al., 2024) and fully reported in Appendices G and H.

Human evaluation on gradual summaries. Before starting the evaluation, annotators are allowed to reject a task if the output text is gibberish or obviously of low quality.

The generated summaries are evaluated in two tasks. In Task 1, the annotators first read the source document and the three summaries and then rate the generated text in four aspects, including attribution, coverage of the main ideas, conciseness and readability (similar to the 'checklist' in HelloBench by Que et al. (2024)). The annotators rate the summary on a 0-4 Likert scale and finally give an overall rating on a 0-10 Likert scale. Each summary receives its own separate set of scores.

In Task 2, the annotators validate the QA sets that were previously created by human writers. For each question in the QA sets (13–15 questions and answers), the annotators are required to determine whether the content of the summary contains enough information to answer the question (i.e., the answer is directly stated, heavily implied or logically entailed in the summary). The annotators give a YES or NO to each QA pair. For each summary, the annotators validate the whole set of QA once.

The whole evaluation is referenceless, which means that the human written summaries are not shown to annotators, and that they only see a single set of summaries from one anonymous model output each time.

Both tasks 1 and 2 involve human judgment, and to reduce the bias, 3 sets of rating from random annotators are required for each generated output. The same guidance should be used for all different domains. Detailed evaluation guidelines are included in the Appendix G.

Human evaluation on summary expansion. We use the same format as the previous summarization evaluation tasks and integrate some of the questions from Story Plot Generator (Zhu et al., 2023) and HelloBench (Que et al., 2024). For task 1, the annotators read the source summary and the generated long-form output, rate the output on 6 aspects, including the coverage of main core ideas, cohesion, richness in details, creativity, non-repetitiveness, and interest, and give an overall rating at the end. In task 2, they validate the QA set with the generated long-form text. Each output is evaluated separately without reference, and three sets of random annotation ratings are required. Detailed evaluation guidelines are included in the Appendix H.

Evaluation statistics. Summaries and summary expansions are each evaluated separately. For the evaluation of generated summaries, 252 documents from all domains are used as the source to generate the summaries (with 2 documents being excluded during the process). The summaries are generated using three different models (as reported in Section 3 and chosen to represent close and open models of different sizes): GPT-4o-mini-64k (et al., 2024b), LLAMA 3.1-70B, and LLAMA 3.1-8B (et al., 2024a). This results in 756 outputs and, along with 252 sets of human-written summaries, creates a dataset of 1,000 document-summary pairs for evaluation. A vendor sources 287 annotators, who are required (1) to be native speakers of English and (2) to hold a language-related degree. These annotators are selected from a pool that is different from that of the summary writers, ensuring that they have no prior knowledge of the source documents or the written summaries. Tasks are randomly assigned to annotators until every set of generated output receives three complete annotations. A limit of 10 evaluations for each model is set per annotator to mitigate biases in the results.

For evaluation of generated summary expansions, only a subset of data is selected,3 including

3We exclude domains with high density of factual information, because we believe they are less appropriate for the summary expansion task. The problem here is potential hallucination of factual information (e.g. Berlin is the capital of
SummScreen, BookSum, SQuality and FacetSum (102 source documents in total). The expansions are generated with the same models as previously (GPT-4o-mini, LLAMA 3.1-70B, and LLAMA 3.1-8B), resulting in 306 long-form outputs. Ten experienced data analysts are selected to conduct the evaluation. Similarly to the evaluation of summaries, the tasks are assigned randomly until every long form output receives 3 complete annotations.

2.6 Automatic evaluation

The summarization outputs are typically evaluated by computing ROUGE scores (Lin, 2004) with respect to a reference. However, this approach is not sufficient for at least three reasons (Schluter, 2017): it depends too much on the reference, it offers only a comparison at the surface level, and it does not explain why a summary is good or bad. Thus, we compute several other reference-free metrics, each targeting a specific aspect of summarization quality (over the 6 aspects introduced in the SEAHORSE dataset by Clark et al. (2023)). For each aspect, we tried to pick a metric that is focused on this specific aspect, preferring ones that are computationally transparent and that have already been implemented elsewhere:

1. Repetitiveness: how much the summary repeats the same phrases. We report the count of all the word n-grams (\( n \in \{1, 2, 3\} \)) in the summary, divided by the count of such unique n-grams (REP-3) (Welleck et al., 2019).

2. Fluency: how grammatical the text is. We report the average probability of a summary sentence being grammatical (or linguistically acceptable in the Chomskyan sense of the term) computed with a CoLA classifier (Krishna et al., 2020).

3. Coherence: how similar are the sentences in the generated texts to each other. COH-2 averages similarity of the neighboring-over-one sentences in the embedding space (Parola et al., 2023).

4. Attribution: how much of the summary is directly attributable to the source (something like “precision” of the ideas in the summary). The average score of SEAHORSE Q4 (SH-4) model evaluates attribution (Clark et al., 2023).

5. Coverage of the source: how much of the source is reflected in the summary. The average SEAHORSE Q5 (SH-5) score reports this aspect (Clark et al., 2023).

6. Overall In order to evaluate the overall quality of the text, we use two metrics:
(1) For summarization, we aggregate a score (AVG) from the above metrics, namely, averaging \( -REP-3, CoLA, COH-2, SH-4, SH-5 \).
For summary expansion, the aggregated score (AVG) is the average score of \( REP-3, CoLA, COH-2 \). Note that \( REP-3 \) is negated to make the score monotonic. Also, for the summary expansion, \( REP-3 \) increases the value over the length of output, so the factor 0.2 is empirically set to normalize the value on the summary expansion task 20%.
(2) We use HelloEval (HE) score (Que et al., 2024): an LLM-as-judge model with various checklists trained w.r.t. human evaluation.

Selection of these metrics from a larger candidate set was partially motivated by their correlations with human annotations from Clark et al. (2023), described in Appendix D. Table 5 in Appendix C summarizes the list of metrics.

For the SEAHORSE scores, we had to feed the whole source text to a transformer model, which was neither feasible computationally with long context inputs nor made sense given the relatively short-form training data of those models. To bypass this problem, we segment sources and summaries into aligned fragments (using a modification of the alignment algorithm in Section 2.4) with at most 50 sentences on the source side and compute model-based metrics for the fragment pairs.

2.7 Data Statistics

LCFO covers 7 domains sourced from 10 datasets with an average document length of 5k words. Table 1 contains the distribution of the LCFO dataset in subsets and domains, as well as the average word length of the documents. More details are reported in the Appendix B.

3 Experiments

Settings. We experimented with closed and open LLMs. We chose GPT-4o-mini-64k\(^4\) for the closed model and LLAMA 3.1-70B (Dubey et al., 2024) for the open-source one. For summarization, we

\footnotetext{4https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/}

France), which can be detrimental for real-life use cases but is out of scope of our evaluation that does not check for extrinsic factuality but only for faithfulness to the source text.
<table>
  <tr>
    <th>DATASET</th>
    <th>N</th>
    <th>DOMAIN</th>
    <th>LEN</th>
  </tr>
  <tr>
    <td>LexGLUE</td>
    <td>25</td>
    <td>Legal: supreme court opinions</td>
    <td>4953</td>
  </tr>
  <tr>
    <td>BookSum</td>
    <td>27</td>
    <td>Literary: books, novel, act</td>
    <td>4114</td>
  </tr>
  <tr>
    <td>SQuality</td>
    <td>25</td>
    <td>Literature: stories</td>
    <td>4856</td>
  </tr>
  <tr>
    <td>FacetSum</td>
    <td>25</td>
    <td>Scientific: journal articles on various domains</td>
    <td>4904</td>
  </tr>
  <tr>
    <td>JRC-Acquis</td>
    <td>25</td>
    <td>Legal: legislative text of the European Union</td>
    <td>4825</td>
  </tr>
  <tr>
    <td>MultiUN</td>
    <td>25</td>
    <td>Political: UN docs</td>
    <td>4539</td>
  </tr>
  <tr>
    <td>Wikipedia</td>
    <td>25</td>
    <td>Wikipedia: 22 docs on biomedicine</td>
    <td>5266</td>
  </tr>
  <tr>
    <td>GovReport</td>
    <td>25</td>
    <td>Political: Congressional Research Service and US Government Accountability Office</td>
    <td>5078</td>
  </tr>
  <tr>
    <td>Summscreen</td>
    <td>25</td>
    <td>Conversational: TV series transcript</td>
    <td>5030</td>
  </tr>
  <tr>
    <td>Seahorse</td>
    <td>25</td>
    <td>News: English BBC news</td>
    <td>4576</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>252</th>
    <th>Average word count</th>
    <th>4814</th>
  </tr>
</table>

Table 1: LCFO Summary: domains and statistics (number of documents N and average length in words LEN)

ran the model with all length ratios (5%, 10%, 20%), while for summary expansion, we only expanded the summaries 20% to the full document. We also performed a postprocessing step to filter the templated response such as “**Summary**”, “Here is the summary:”, etc.

Summarization results. Table 2 shows the general results of the selected models at different levels of gradual summarization. Results broken down by domains are reported in the Appendix J. Note that LLMs tend to perform similarly regardless of the length of the output in terms of human scores. This is not the case for humans that show to lag behind when performing short summaries. The best results are consistently achieved with GPT-4o-mini and are consistent with previous research findings (Que et al., 2024). This model even surpasses human-level quality in short summaries. This may be explained by humans tending to perform worse when summarizing short documents and better when summarizing long ones.

Summary expansion results. Table 3 shows the overall results on the summary expansion task by a factor of 5, giving the 20% summary input. The performance of models is not coherent across metrics that look only at the output (i.e.%WC, REP-3, CoLA, COH-2, and AVG). In terms of HE and coherently with the human evaluation results, GPT-4o-mini is the best performing model. Additionally, we report the results of other combinations, for example, expanding by larger factors (10 and 20) giving 10% summary input and giving 5% summary input, respectively, in Appendix J.

When comparing across tasks (i.e. summarizing to 20% or doing summary expansion from 20% by a factor of 5), the results show better performance in the former. It is expected that summary expansion is a more challenging task across domains and all models. Current models struggle with this task. If we compare HE, the deltas in the same model vary from 6% for GPT-4o-mini to \( \approx 30\% \) for LLAMA 3.1-70B. When comparing output-based metrics, there are discrepancies in conclusions (i.e., LLAMA 3.1-8B better than 70B model). However, HE is still worse for the 8B model. This may indicate that selected output-based quality metrics are less reliable than the HE score (see the analysis below for metrics evaluation).

Metrics evaluation. In our study, we consider human evaluation, conducted according to the guidelines outlined in Appendices G and H, as the definitive measure of the overall quality score, as well as the scores for individual quality aspects such as coverage and attribution. To mitigate potential biases among the annotators, we calculate the average of three annotations for each task. Table 4 presents the Spearman correlation coefficients for various aspects and overall scores, comparing automatic metrics and human evaluations for summarization and summary expansion, respectively. A higher Spearman correlation coefficient signifies a stronger correlation between the automatic metrics and human annotation. The metric that shows the highest correlation with human annotation corresponds to SH-4, which measures attribution. When comparing metrics that measure overall performance, we observe that R-L is not very good at correlation, but it may also be due to the fact that our task is not the best suited to use human references. HE is the one with the highest correlation. AVG low-correlation (both in S and SE) may be explained by the fact that individual averaged metrics are not very good or they cover more specific aspects which may not end capturing the overall performance. This low correlation for R-L and AVG can explain the discrepancy observed in the model ranking (specially between LLAMA 3.1-70B and 8B in Tables 2 and 3. Correlations are low in all cases, which shows the difficulty of the evaluation. Beyond the challenge of automatizing it, we should add the fact that humans struggle in generating short summaries, which may imply that humans also struggle in evaluating them.
<table>
  <tr>
    <th rowspan="2">Output</th>
    <th colspan="10">LCFO.5%</th>
  </tr>
  <tr>
    <th>R-L(↑)</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>SH-4↑</th>
    <th>SH-5↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>Hum↑</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>n/a</td>
    <td><b>0.308</b></td>
    <td>0.941</td>
    <td>0.809</td>
    <td>0.644</td>
    <td>0.387</td>
    <td>0.494</td>
    <td>52.195</td>
    <td>6.61</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.331</td>
    <td>0.328</td>
    <td>0.968</td>
    <td>0.719</td>
    <td><b>0.635</b></td>
    <td><b>0.487</b></td>
    <td><b>0.496</b></td>
    <td><b>76.917</b></td>
    <td><b>7.25</b></td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.384</td>
    <td>0.383</td>
    <td>0.965</td>
    <td>0.861</td>
    <td>0.622</td>
    <td>0.377</td>
    <td>0.488</td>
    <td>72.468</td>
    <td>6.27</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.377</td>
    <td>0.411</td>
    <td><b>0.969</b></td>
    <td><b>0.865</b></td>
    <td>0.618</td>
    <td>0.372</td>
    <td>0.482</td>
    <td>63.894</td>
    <td>6.32</td>
  </tr>
  <tr>
    <th colspan="11">LCFO.10%</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>n/a</td>
    <td><b>0.395</b></td>
    <td>0.945</td>
    <td>0.816</td>
    <td><b>0.661</b></td>
    <td>0.416</td>
    <td><b>0.489</b></td>
    <td>64.688</td>
    <td>7.44</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.385</td>
    <td>0.404</td>
    <td><b>0.964</b></td>
    <td>0.695</td>
    <td>0.621</td>
    <td><b>0.471</b></td>
    <td>0.469</td>
    <td><b>77.863</b></td>
    <td><b>7.50</b></td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.434</td>
    <td>0.515</td>
    <td>0.944</td>
    <td><b>0.860</b></td>
    <td>0.614</td>
    <td>0.369</td>
    <td>0.454</td>
    <td>72.497</td>
    <td>6.42</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.429</td>
    <td>0.534</td>
    <td>0.963</td>
    <td>0.858</td>
    <td>0.612</td>
    <td>0.366</td>
    <td>0.453</td>
    <td>59.385</td>
    <td>6.63</td>
  </tr>
  <tr>
    <th colspan="11">LCFO.20%</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>n/a</td>
    <td><b>0.244</b></td>
    <td>0.938</td>
    <td>0.805</td>
    <td>0.615</td>
    <td>0.357</td>
    <td><b>0.494</b></td>
    <td>69.745</td>
    <td><b>7.78</b></td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.445</td>
    <td>0.497</td>
    <td><b>0.961</b></td>
    <td>0.673</td>
    <td><b>0.616</b></td>
    <td><b>0.464</b></td>
    <td>0.443</td>
    <td><b>76.706</b></td>
    <td>7.52</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.467</td>
    <td>0.631</td>
    <td>0.928</td>
    <td>0.860</td>
    <td>0.596</td>
    <td>0.357</td>
    <td>0.422</td>
    <td>71.603</td>
    <td>6.32</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.469</td>
    <td>0.647</td>
    <td>0.956</td>
    <td><b>0.861</b></td>
    <td>0.594</td>
    <td>0.370</td>
    <td>0.427</td>
    <td>51.015</td>
    <td>6.60</td>
  </tr>
</table>

Table 2: Performance on the summarization task

<table>
  <tr>
    <th>Output</th>
    <th>%WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>Hum↑</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>1.931</b></td>
    <td>0.707</td>
    <td><b>0.913</b></td>
    <td>0.609</td>
    <td>0.460</td>
    <td><b>70.896</b></td>
    <td><b>6.431</b></td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>1.058</td>
    <td><b>0.680</b></td>
    <td>0.877</td>
    <td>0.750</td>
    <td>0.497</td>
    <td>39.199</td>
    <td>4.469</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>1.187</td>
    <td>0.809</td>
    <td>0.903</td>
    <td><b>0.779</b></td>
    <td><b>0.507</b></td>
    <td>38.416</td>
    <td>4.801</td>
  </tr>
</table>

Table 3: Performance on the summary expansion task by a factor of 5, giving the 20% summary input.

<table>
  <tr>
    <th>R-L</th>
    <th>CoLA</th>
    <th>SH-4</th>
    <th>SH-5</th>
    <th>AVG</th>
    <th>HE</th>
  </tr>
  <tr>
    <td>S</td>
    <td>0.196 (0.065)</td>
    <td>0.595 (6.337e-10)</td>
    <td>0.616 (1.005e-10)</td>
    <td>0.445 (1.105e-5)</td>
    <td>0.159 (0.135)</td>
    <td>0.428 (2.591e-05)</td>
  </tr>
  <tr>
    <td>SE</td>
    <td>n/a</td>
    <td>n/a</td>
    <td>n/a</td>
    <td>n/a</td>
    <td>0.285 (3.646e-05)</td>
    <td>0.405 (1.957e-09)</td>
  </tr>
</table>

Table 4: Spearman correlation coefficients (and p-value) for various aspects and overall scores between automatic metrics and human evaluation for summarization (S) and summary expansion (SE). For the former, we show correlations between CoLA and Human evaluation Q2d; SH-4 and Human evaluation Q2a; SH-5 and Human evaluation Q2b and R-L/AVG/HE and Human evaluation Q3 (appendix G). For the latter, we show correlations between AVG/HE and Human evaluation Q3 (appendix H).

4 Related Work

Related work on long context and long form output comes in many flavors. We cover a summary on long context and long-form output datasets.

Long-context datasets. Infinite length datasets such as NIAH, RULER (Hsieh et al., 2024) work with distracting information. Finite-length nondistractive-based datasets include: MultiLexSum (Shen et al., 2022) is a collection of 9,280 expert-authored summaries drawn from single domain (Civil Rights Litigation Clearinghouse) writing the length of the source documents often exceeds two hundred pages per case and summaries are presented in two-sizes: a short and longer version; Longbench (Bai et al., 2024) and Marathon (Zhang et al., 2024b) that includes tasks with 5–25k context and, more recently, (Kwan et al., 2024) build a dataset up to 8k tokens context length to evaluate LLMs’ long-context understanding across five key abilities: understanding of single or multiple relevant spans in long contexts based on explicit or semantic hints, and global context understanding. Loong is a multi-document QA dataset up to 200k context to assess RAG abilities. HelloBench (Que et al., 2024) includes summarization of a selection of long-input documents (3k to 6k word length).

Long-form output datasets. There is a lack of reference-based datasets on long form output. However, there are datasets that study prompting of different long-form generation; e.g., StoryGen (Zhu
et al., 2023) includes prompts to generate stories, and HelloBench is one of the most diverse long form generation benchmarks including stories, screenplays, keyword writing.

Our contribution on datasets involve the manual collection of 3-length summaries from long input documents. This collection also includes abstractive QA (non-multiple choice) to test comprehension. Our contribution on metrics involves new human evaluation protocols on summarization and summary expansion, as well as annotations to develop supervised metrics on long-form outputs.

5 Conclusions

LCFO provides gradual summaries references from 5k input documents with QA pairs for each of the documents and summaries. Additionally, we provide human evaluation of human and model-generated summaries and model-generated summary expansions. Overall, LCFO enables the evaluation of several tasks and metrics in the setting of long-context input documents and long-form output. While the main contribution of this paper is to present the freely available LCFO dataset\footnote{Available at https://huggingface.co/datasets/facebook/LCFO}, we also evaluate model and human outputs, showing that LLMs are capable of surpassing human results when producing short summaries. Current evaluation results question the usefulness of manually generating human references for short summarization of long documents. To confirm this, as further work, we plan to exploit the capabilities of LCFO by using QA as part of automatic evaluation (i.e. scoring how many questions are correctly answered in model-generated summaries).

Limitations and Ethical considerations

Data contamination. Source documents may exist in the training data of the models, therefore, generation may be at risk. To mitigate this, we prioritize recent documents, since this is not enough, we annotate the correspondence of sections in summarization versions, so that we can generate only portions of the document. Therefore, if the model uses internal knowledge, we can quantify by spotting details from other sections.

Experiments. The experimental options that LCFO offers are much larger than the ones we explore in this paper. Also, the dataset can be easily expanded to have more summary references by matching with existing summaries in some of the domains. QA pairs have not been used in the paper but this is designed (but not limited) to serve for doing reading comprehension and/or for creating an evaluation metric.

Metrics. Summarization is a generative task with very diverse aspects of quality, and no single automatic evaluation metric captures them all adequately. To compensate for this, we report multiple evaluation metrics, but still, some of them are not well established; for example, there is no single metric of longform text coherence that the summarization community agrees upon. By providing the results of the human evaluation, we hope to help the community develop and validate better automatic evaluation metrics in the future.

Computing In terms of computing, evaluating LLMs on LCFO benchmark require larger memory due to both its big context size and the long-form output (should the models be capable to it). In case of LLAMA, we used 1 NVIDIA GPU A100 80 GB for the 8B model, and 8 GPUs for the 70B model. The resource was shared with the loading of scoring models (SH, CoLA) as well. Each evaluation run over 10 domains takes 90 minutes, including the computation of all the scores except HelloEval (where the computing time depends on the external availability of GPT-4 endpoint deployment).

Annotations Annotators were paid a fair rate. Each of the annotators signed a consent form agreeing on the dataset and its usage that they were participating in.

Acknowledgements

This paper is part of the LCM project\footnote{https://github.com/facebookresearch/large_concept_models} and authors would like to thank the entire LCM team for the fruitful discussions.

References

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand. Association for Computational Linguistics.
N. Bradbury. 2016. Attention span during lectures: 8 seconds, 10 minutes, or more? Advances in Physiology Education, 40(4):509–513.

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.

Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for abstractive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602–8615, Dublin, Ireland. Association for Computational Linguistics.

Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roei Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. 2023. SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9397–9413, Singapore. Association for Computational Linguistics.

Marta Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Carlos Escolano. 2023. Toxicity in multilingual machine translation at scale. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9570–9586, Singapore. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. SONAR: sentence-level multi-modal and language-agnostic representations. arXiv preprint.

Abhimanyu Dubey et al. 2024a. The LLaMA 3 herd of models. Preprint, arXiv:2407.21783.

OpenAI et al. 2024b. Gpt-4 technical report. Preprint, arXiv:2303.08774.

James Hartley and Ivor K. Davies. 1978. Note-taking: A critical review. Programmed learning and educational technology, 15(3):207–224.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? Preprint, arXiv:2404.06654.

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online. Association for Computational Linguistics.

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1650–1669, Dubrovnik, Croatia. Association for Computational Linguistics.

Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Empirical Methods in Natural Language Processing.

Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6536–6558, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Yuxin Jiang, Lifeng Shang, Qun Liu, and Kam-Fai Wong. 2024. M4LE: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15568–15592, Bangkok, Thailand. Association for Computational Linguistics.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: a faceted summarization dataset for long scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 1080–1089, Online. Association for Computational Linguistics.

Alberto Parola, Jessica Mary Lin, Arndis Simonsen, Vibeke Blikted, Yuan Zhou, Huiling Wang, Lana Inoue, Katja Koelkebeck, and Riccardo Fusaroli. 2023. Speech disturbances in schizophrenia: Assessing cross-linguistic generalizability of nlp automated measures of coherence. Schizophrenia Research, 259:59–70.

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong,
Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. 2024. Hellobench: Evaluating long text generation capabilities of large language models. Preprint, arXiv:2409.16191.

Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics.

Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlangen, and Doug Downey. 2022. Multilexsum: real-world summaries of civil rights lawsuits at multiple granularities. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.

Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Erjavec, Dan Tufts, and Dániel Varga. 2006. The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).

Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. 2022. SQuALITY: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319.

Kaige Xie and Mark Riedl. 2024. Creating suspenseful stories: Iterative planning with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2391–2407, St. Julian’s, Malta. Association for Computational Linguistics.

Haopeng Zhang, Philip S. Yu, and Jiawei Zhang. 2024a. A systematic survey of text summarization: From statistical methods to large language models. Preprint, arXiv:2406.11289.

Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024b. Marathon: A race through the realm of long context with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5201–5217, Bangkok, Thailand. Association for Computational Linguistics.

Hanlin Zhu, Andrew Cohen, Danqing Wang, Kevin Yang, Xiaomeng Yang, Jiantao Jiao, and Yuandong Tian. 2023. End-to-end story plot generator. Preprint, arXiv:2310.08796.

A Automatic Paragraph Splitting Details

As part of the guidelines for summarization, annotators are instructed to read long documents from different domains and mentally distill key points into paragraphs and then into a cohesive document summary. This process requires the source text to be logically segmented into well-structured paragraphs that facilitate comprehension and synthesis.

During pilot studies, it became evident that the quality of the initial paragraph segmentation significantly impacted annotation outcomes. Poorly segmented paragraphs increased cognitive load and risked misinterpretation, while cohesive and logically structured paragraphs improved annotation consistency and efficiency.

Given the variability in formats and structures across datasets, a uniform approach to paragraph splitting was not feasible. Some datasets provided explicit structural markers (e.g., new lines, section headers), while others required more algorithmic intervention, such as employing the Segment Anything Text (SaT-I3) model. Furthermore, the SaT-I3 model’s performance varied across text types, necessitating dataset-specific thresholds and post-processing techniques to optimize paragraph segmentation.

This section outlines the tailored methodologies applied to each dataset in the LCFO corpus, highlighting how their unique characteristics were addressed to produce high-quality, preprocessed documents for annotation.

LexGLUE Paragraphs were split using double newlines as separators, preserving the inherent paragraph structure in the dataset.

BookSum and SQuality Lines were joined with a blank space to create continuous text blocks. Sentences and paragraphs were split using the SaT-I3 model with a threshold of 0.8, producing lists of sentences grouped into paragraphs. Paragraphs exceeding 3,000 characters were further split using the SaT-I3 model with a stricter threshold of 0.4. Consecutive short paragraphs (fewer than 2 sentences or under 400 characters) were merged to ensure coherence, especially for dialogue-heavy sections.
JRC-Acquis  Lines were joined with a blank space to preserve the flow of text. Sentences and paragraphs were split using the SaT-I3 model with a standard threshold of 0.5. Consecutive paragraphs containing fewer than 2 sentences were merged. Sections and subsections were extracted from paragraph beginnings using the dataset’s consistent numbered format (e.g., 1.1.2), serving as structural indicators.

MultiUN  Lines were joined using blank spaces to form initial text blocks. Sentences and paragraphs were split using the SaT-I3 model with a threshold of 0.5. Short consecutive paragraphs (fewer than 2 sentences each, and up to 20 sentences total) were merged to improve readability and flow.

Wikipedia  Original paragraphs were identified using empty lines (meaning double newline in the original text), which appeared as blank lines or in the CSV format. Long paragraphs (over 500 tokens) were split further using the SaT-I3 model with a threshold of 0.5 to improve segmentation accuracy for longer text units.

GovReport  Same as LexGLUE, paragraphs were split using double newlines as separators.

Summscreen  Initial paragraph segmentation was based on scene indicators ((SCENE-BREAK)) in the transcripts. However, this often resulted in excessively long paragraphs, with some documents containing only one or two paragraphs. Text formatting issues, such as double spaces in punctuation (e.g., ", ", ") were corrected to align with the SaT-I3 model’s sensitivity. Long paragraphs exceeding 3,000 characters were re-segmented using the SaT-I3 model with a threshold of 0.9. Short consecutive paragraphs containing only one sentence were merged to form cohesive segments.

B  Data Details

Our data collection

• 100% human annotated (no LLM pre-selection)

• 7 domains (political, wikipedia, scientific, literature, conversational, legal

• 252 Source Documents ( 5k)

• 4 lengths of the same Source Document (≈5k, ≈1k, ≈500, ≈250 words)

• 13-15 QA on each Long Context Source Document

• Annotation on the presence of these QA on each of the summaries

• Human evaluation of automatic and manual summaries

• Human evaluation of summary expansion

C  Summary evaluation

Table 5 summarises the metrics used to evaluate.

D  Selection of automatic metrics

At an early stage of our work (before collecting human annotations), we came up with a list of candidate metrics of summarization quality. We provide it in Table 6.

To select a single metric per aspect, we analyzed their Spearman correlations with human annotations of the English subset of SEAHORSE (Clark et al., 2023), focusing on Questions 2 to 6: absence of repetitions, grammatical correctness, attribution to the source, coverage of the source, and conciseness (which also reflects overall quality). We did not work with Question 1 (comprehensibility), because our metrics focus on inter-sentential comprehensibility, whereas 66% of SEAHORSE summaries consist of a single sentence.

We report the resulting correlations, grouped by the three English SEAHORSE subsets, in Table 7. One can see that the metrics we selected for summarization correlate reasonably with their corresponding aspects across all 3 subsets most of the time7, which justifies our choice of them.

E  Summarization Guidelines

Annotator proficiency requirements

• Native speaker of English

• Editor / writer / domain expert

Task  You will receive document(s) that are approximately 5,000 words or longer from the following domains:

• Political (GovReports, MultiUN)

• News (Seahorse)

7An exception is the grammaticality/fluency aspect (Question 2) on the Wikihow subset, where no metric demonstrated adequate correlation with human annotations.
<table>
  <tr>
    <th>Task</th>
    <th>Area</th>
    <th>Metric</th>
    <th>Description</th>
    <th>Reference</th>
  </tr>
  <tr>
    <td>Sum</td>
    <td>Target similarity</td>
    <td>R-L</td>
    <td>ROUGE-L (longest common subsequence)</td>
    <td>Lin (2004)</td>
  </tr>
  <tr>
    <td>Sum/SumExp</td>
    <td>Grammaticality</td>
    <td>REP-3</td>
    <td>Portion of duplicated N-grams (N=4)</td>
    <td>Welleck et al. (2019)</td>
  </tr>
  <tr>
    <td>Sum/SumExp</td>
    <td>Fluency</td>
    <td>CoLA</td>
    <td>Sentence fluency classifier score</td>
    <td>Krishna et al. (2020)</td>
  </tr>
  <tr>
    <td>Sum/SumExp</td>
    <td>Coherence</td>
    <td>COH-2</td>
    <td>2nd-order word-level coherence score</td>
    <td>Parola et al. (2023)</td>
  </tr>
  <tr>
    <td>Sum</td>
    <td>Attribution</td>
    <td>SH-4</td>
    <td>Seahorse-Large-Q4 score</td>
    <td>Clark et al. (2023)</td>
  </tr>
  <tr>
    <td>Sum</td>
    <td>Semantic coverage</td>
    <td>SH-5</td>
    <td>Seahorse-Large-Q5 coverage score</td>
    <td>Clark et al. (2023)</td>
  </tr>
  <tr>
    <td>SumExp</td>
    <td>Word Count</td>
    <td>WC</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Sum/SumExp</td>
    <td>Overall</td>
    <td>AVG</td>
    <td>Empirical average of metrics</td>
    <td></td>
  </tr>
  <tr>
    <td>Sum/SumExp</td>
    <td>Overall</td>
    <td>HE</td>
    <td>HelloEval score</td>
    <td>Que et al. (2024)</td>
  </tr>
</table>

Table 5: Summary of automatic metrics used in different tasks.

<table>
  <tr>
    <th>Metric</th>
    <th>Short name</th>
    <th>Implementation</th>
  </tr>
  <tr>
    <td>ref_rouge_fmeasure</td>
    <td>R-L</td>
    <td>ROUGE-L F1 score, computed with the rouge-score Python package (in the rougetsum mode)</td>
  </tr>
  <tr>
    <td>ref_rouge_recall</td>
    <td></td>
    <td>ROUGE-L recall (see above)</td>
  </tr>
  <tr>
    <td>ref_rouge_precision</td>
    <td></td>
    <td>ROUGE-L precision (see above)</td>
  </tr>
  <tr>
    <td>char_len_ratio</td>
    <td></td>
    <td>Summary-to-source lengths ratio, in characters</td>
  </tr>
  <tr>
    <td>sent_len_ratio</td>
    <td></td>
    <td>Summary-to-source lengths ratio, in sentences</td>
  </tr>
  <tr>
    <td>word_len_ratio</td>
    <td></td>
    <td>Summary-to-source lengths ratio, in words</td>
  </tr>
  <tr>
    <td>td_coherence_1st</td>
    <td></td>
    <td>1st order coherence, computed with the textdescriptives Python package</td>
  </tr>
  <tr>
    <td>td_coherence_2nd</td>
    <td>COH-2</td>
    <td>2nd order coherence, computed with the textdescriptives Python package</td>
  </tr>
  <tr>
    <td>td_flesch_kincaid_grade</td>
    <td></td>
    <td>Flesch-Kincaid grade readability score (textdescriptives)</td>
  </tr>
  <tr>
    <td>td_gunning_fog</td>
    <td></td>
    <td>Gunning-Fog readability score (textdescriptives)</td>
  </tr>
  <tr>
    <td>td_dependency_distance_mean</td>
    <td></td>
    <td>Mean distance of syntactic dependencies (textdescriptives)</td>
  </tr>
  <tr>
    <td>word_ngram_src_overlap</td>
    <td></td>
    <td>Fraction of source word n-grams (1 to 3) that appear in the summary</td>
  </tr>
  <tr>
    <td>word_ngram_repetition_rate</td>
    <td>REP-3</td>
    <td>Fraction of summary word n-grams (1 to 3) that appear more than once</td>
  </tr>
  <tr>
    <td>src_rouge_recall</td>
    <td></td>
    <td>ROUGE-2 recall of the summary w.r.t. the source</td>
  </tr>
  <tr>
    <td>src_rouge_f1</td>
    <td></td>
    <td>ROUGE-2 F1 score of the summary w.r.t. the source</td>
  </tr>
  <tr>
    <td>src_rouge_precision</td>
    <td></td>
    <td>ROUGE-2 precision of the summary w.r.t. the source</td>
  </tr>
  <tr>
    <td>tgt_fluency</td>
    <td>CoLA</td>
    <td>Mean predicted probability of the sentence being linguistically acceptable, as per the model from Krishna et al. (2020)</td>
  </tr>
  <tr>
    <td>src_fluency_diff</td>
    <td></td>
    <td>Difference of mean predicted probabilities of being acceptable in the summary and in the source</td>
  </tr>
  <tr>
    <td>mean_self_sonar_sim</td>
    <td></td>
    <td>Mean cosine similarity SONAR embeddings of each summary sentence to its most similar summary sentence</td>
  </tr>
  <tr>
    <td>mean_src_sonar_sim</td>
    <td></td>
    <td>Mean cosine similarity SONAR embeddings of each summary sentence to its most similar source sentence</td>
  </tr>
  <tr>
    <td>mean_src_coverage_sonar_sim</td>
    <td></td>
    <td>Mean cosine similarity SONAR embeddings of each source sentence to its most similar summary sentence</td>
  </tr>
  <tr>
    <td>mean_monotonic_src_sonar_sim</td>
    <td></td>
    <td>Mean cosine similarity SONAR embeddings of each summary sentence to its monotonically aligned source sentence</td>
  </tr>
  <tr>
    <td>sonar_sim_monotonicity</td>
    <td></td>
    <td>Ratio of mean_monotonic_src_sonar_sim to mean_src_sonar_sim</td>
  </tr>
  <tr>
    <td>p_entail_full</td>
    <td></td>
    <td>Probability of the summary being entailed by the source, predicted with tasksource/deberta-small-long-nli model</td>
  </tr>
  <tr>
    <td>p_noncontradict_full</td>
    <td></td>
    <td>Probability of the summary not contradicting the source, predicted with tasksource/deberta-small-long-nli model</td>
  </tr>
  <tr>
    <td>smolm_reconstruct_loss</td>
    <td></td>
    <td>Cross-entropy loss of the source given the summary, computed with the huggingface/TB/SmoLM-360M-Instruct model</td>
  </tr>
  <tr>
    <td>seahorse_q4</td>
    <td>SH-4</td>
    <td>Probability of the “Yes” response extracted from the SEAHORSE Q4 model (google/seahorse-large-q4)</td>
  </tr>
  <tr>
    <td>seahorse_q5</td>
    <td>SH-5</td>
    <td>Probability of the “Yes” response extracted from the SEAHORSE Q5 model</td>
  </tr>
  <tr>
    <td>seahorse_q6</td>
    <td></td>
    <td>Probability of the “Yes” response extracted from the SEAHORSE Q6 model</td>
  </tr>
</table>

Table 6: The automatic metrics we considered as candidates, along with their implementation details.

• Wikipedia (Wikipedia)

• Scientific/Technical (FacetSum)

• Literature (BookSum, SQuality)

• Conversational (Summscreen)

• Legal (LexGlue, JRCAcquis)

The documents will contain sections/chapters. You will need to summarize them retaining the section alignment. For certain domains, there will be additional guidance in the form of special guidelines (legal, medical etc.)

You will need to create 3 summaries:

• Summary 1: around 20% of the source text (1,000 words if total length is 5000)

• Summary 2: around 10% of the source text (500 words)

• Summary 3: around 5% of the source text (250 words)

After finishing summarizing, you will need to write a minimum of 15 questions with corresponding answers (QA) per each 5000 words.

Requirements for Summarization Here’s more information on what that means and how to summarize:

Please read the provided document in its entirety. Consider making notes of the main core ideas while you read. After you have finished reading, please write a short summary of the source text. The summary should:
<table>
  <tr>
    <th rowspan="2">SEAHORSE subset Quality aspect</th>
    <th colspan="6">wiki</th>
    <th colspan="6">xsum</th>
  </tr>
  <tr>
    <th>q2</th><th>q3</th><th>q4</th><th>q5</th><th>q6</th>
    <th>q2</th><th>q3</th><th>q4</th><th>q5</th><th>q6</th>
  </tr>
  <tr>
    <td>ref_rouge_fmeasure (R-L)</td>
    <td>0.33</td><td>0.11</td><td>0.25</td><td>0.27</td><td>0.17</td><td>0.12</td><td>0.23</td><td>0.15</td><td>0.11</td><td>0.13</td><td>0.16</td><td>0.15</td>
  </tr>
  <tr>
    <td>ref_rouge_recall</td>
    <td>0.31</td><td>0.02</td><td>0.15</td><td>0.26</td><td>0.25</td><td>0.16</td><td>0.11</td><td>0.10</td><td>0.23</td><td>0.18</td><td>0.11</td><td>0.09</td><td>0.12</td><td>0.17</td><td>0.14</td>
  </tr>
  <tr>
    <td>ref_rouge_precision</td>
    <td>0.29</td><td>0.02</td><td>0.03</td><td>0.16</td><td>0.20</td><td>0.17</td><td>0.11</td><td>0.14</td><td>0.20</td><td>0.18</td><td>0.12</td><td>0.10</td><td>0.14</td><td>0.14</td><td>0.15</td>
  </tr>
  <tr>
    <td>char_len_ratio</td>
    <td>0.03</td><td>0.08</td><td>-0.04</td><td>-0.04</td><td>0.06</td><td>0.03</td><td>0.09</td><td>0.20</td><td>0.13</td><td>0.05</td><td>-0.01</td><td>-0.06</td><td>0.12</td><td>0.03</td>
  </tr>
  <tr>
    <td>sent_len_ratio</td>
    <td>-0.02</td><td>0.14</td><td>-0.06</td><td>-0.05</td><td>-0.07</td><td>-0.04</td><td>-0.02</td><td>0.08</td><td>0.13</td><td>0.08</td><td>-0.02</td><td>-0.05</td><td>0.05</td><td>0.01</td>
  </tr>
  <tr>
    <td>word_len_ratio</td>
    <td>0.03</td><td>0.08</td><td>-0.05</td><td>-0.05</td><td>-0.07</td><td>-0.02</td><td>0.03</td><td>0.07</td><td>0.18</td><td>0.11</td><td>0.06</td><td>-0.02</td><td>-0.07</td><td>0.11</td><td>0.02</td>
  </tr>
  <tr>
    <td>td_flesch_kincaid_grade</td>
    <td>0.08</td><td>0.01</td><td>0.11</td><td>0.11</td><td>0.06</td><td>0.00</td><td>0.01</td><td>0.11</td><td>0.06</td><td>0.00</td><td>0.01</td><td>0.00</td><td>0.01</td><td>0.12</td><td>0.02</td>
  </tr>
  <tr>
    <td>td_gunning_fog</td>
    <td>0.14</td><td>0.13</td><td>0.12</td><td>0.12</td><td>0.09</td><td>0.09</td><td>0.07</td><td>0.05</td><td>0.05</td><td>0.03</td><td>0.10</td><td>0.13</td><td>0.12</td><td>0.12</td><td>0.12</td>
  </tr>
  <tr>
    <td>td_dependencies_distance_mean</td>
    <td>0.07</td><td>-0.05</td><td>-0.01</td><td>-0.01</td><td>-0.04</td><td>0.09</td><td>0.03</td><td>-0.02</td><td>0.09</td><td>0.04</td><td>0.06</td><td>-0.01</td><td>0.03</td><td>0.07</td><td>0.05</td>
  </tr>
  <tr>
    <td>word_ngram_src_overlap</td>
    <td>0.18</td><td>0.09</td><td>0.08</td><td>0.06</td><td>0.05</td><td>0.06</td><td>0.09</td><td>0.14</td><td>0.18</td><td>0.13</td><td>0.03</td><td>-0.01</td><td>0.05</td><td>0.15</td><td>0.08</td>
  </tr>
  <tr>
    <td>word_ngram_repetition_rate (REP-3)</td>
    <td>-0.59</td><td>0.05</td><td>-0.13</td><td>-0.19</td><td>-0.23</td><td>-0.31</td><td>-0.08</td><td>-0.13</td><td>-0.05</td><td>-0.11</td><td>-0.27</td><td>-0.05</td><td>-0.02</td><td>0.04</td><td>-0.01</td>
  </tr>
  <tr>
    <td>src_rouge_recall</td>
    <td>0.06</td><td>0.09</td><td>0.08</td><td>0.06</td><td>0.03</td><td>0.04</td><td>0.05</td><td>0.11</td><td>0.09</td><td>0.11</td><td>-0.05</td><td>-0.01</td><td>0.07</td><td>0.13</td><td>0.08</td>
  </tr>
  <tr>
    <td>src_rouge_f1</td>
    <td>0.07</td><td>0.10</td><td>0.07</td><td>0.04</td><td>0.04</td><td>0.05</td><td>0.11</td><td>0.09</td><td>0.11</td><td>-0.05</td><td>-0.01</td><td>0.07</td><td>0.13</td><td>0.08</td>
  </tr>
  <tr>
    <td>src_rouge_precision</td>
    <td>0.10</td><td>0.07</td><td>0.11</td><td>0.12</td><td>0.09</td><td>0.05</td><td>0.03</td><td>0.09</td><td>0.05</td><td>0.09</td><td>0.00</td><td>0.09</td><td>0.07</td><td>0.06</td>
  </tr>
  <tr>
    <td>tgt_fluency (CoLa)</td>
    <td>0.07</td><td>0.09</td><td>0.08</td><td>0.09</td><td>0.35</td><td>0.25</td><td>0.08</td><td>0.16</td><td>0.23</td><td>0.20</td><td>0.17</td><td>0.18</td><td>0.18</td><td>0.18</td><td>0.18</td>
  </tr>
  <tr>
    <td>src_holistic</td>
    <td>0.09</td><td>0.08</td><td>0.00</td><td>0.00</td><td>0.03</td><td>0.03</td><td>0.05</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.06</td><td>0.08</td>
  </tr>
  <tr>
    <td>mean_self_sonar_sim</td>
    <td>-0.57</td><td>0.09</td><td>-0.13</td><td>-0.14</td><td>-0.16</td><td>-0.12</td><td>0.03</td><td>-0.02</td><td>-0.03</td><td>-0.02</td><td>0.05</td><td>0.02</td><td>0.00</td><td>0.06</td><td>-0.01</td>
  </tr>
  <tr>
    <td>mean_src_sonar_sim</td>
    <td>-0.10</td><td>0.06</td><td>0.16</td><td>0.03</td><td>0.02</td><td>0.01</td><td>0.07</td><td>0.08</td><td>0.00</td><td>0.01</td><td>-0.05</td><td>0.01</td><td>0.09</td><td>0.04</td><td>0.07</td>
  </tr>
  <tr>
    <td>mean_src_coverage_sonar_sim</td>
    <td>0.09</td><td>0.08</td><td>0.10</td><td>0.10</td><td>0.07</td><td>-0.01</td><td>0.09</td><td>0.11</td><td>0.09</td><td>0.06</td><td>-0.04</td><td>0.01</td><td>0.10</td><td>0.06</td><td>0.09</td>
  </tr>
  <tr>
    <td>mean_monotonic_src_sonar_sim</td>
    <td>0.10</td><td>0.03</td><td>0.21</td><td>0.13</td><td>0.11</td><td>0.01</td><td>0.07</td><td>0.08</td><td>0.00</td><td>0.01</td><td>-0.05</td><td>0.01</td><td>0.09</td><td>0.04</td><td>0.07</td>
  </tr>
  <tr>
    <td>sonar_sim_monotonicity</td>
    <td>0.37</td><td>-0.07</td><td>0.13</td><td>0.13</td><td>0.10</td><td>-0.04</td><td>0.00</td><td>0.08</td><td>0.05</td><td>0.06</td><td>-0.02</td><td>-0.05</td><td>-0.04</td><td>-0.07</td><td>0.02</td>
  </tr>
  <tr>
    <td>p_entail_full</td>
    <td>-0.02</td><td>0.02</td><td>0.38</td><td>0.20</td><td>0.04</td><td>0.00</td><td>0.21</td><td>0.17</td><td>0.22</td><td>-0.02</td><td>-0.00</td><td>0.39</td><td>0.26</td><td>0.31</td>
  </tr>
  <tr>
    <td>p_contradict_full</td>
    <td>-0.03</td><td>0.03</td><td>0.04</td><td>0.04</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td><td>0.00</td>
  </tr>
  <tr>
    <td>smaller_construct_loss</td>
    <td>-0.05</td><td>-0.11</td><td>-0.03</td><td>-0.03</td><td>-0.01</td><td>-0.06</td><td>-0.11</td><td>-0.07</td><td>0.05</td><td>0.02</td><td>-0.17</td><td>-0.11</td><td>-0.13</td><td>-0.13</td><td>-0.13</td>
  </tr>
  <tr>
    <td>seahorse_q4 (SH-4)</td>
    <td>0.24</td><td>0.52</td><td>0.37</td><td>0.41</td><td>0.19</td><td>0.08</td><td>0.51</td><td>0.17</td><td>0.27</td><td>0.14</td><td>0.06</td><td>0.54</td><td>0.35</td><td>0.43</td>
  </tr>
  <tr>
    <td>seahorse_q5 (SH-5)</td>
    <td>0.43</td><td>0.03</td><td>0.33</td><td>0.51</td><td>0.47</td><td>0.20</td><td>0.12</td><td>0.25</td><td>0.27</td><td>0.29</td><td>0.16</td><td>0.14</td><td>0.36</td><td>0.30</td><td>0.31</td>
  </tr>
  <tr>
    <td>seahorse_q6</td>
    <td>0.45</td><td>0.01</td><td>0.42</td><td>0.51</td><td>0.52</td><td>0.26</td><td>0.10</td><td>0.31</td><td>0.22</td><td>0.31</td><td>0.20</td><td>0.10</td><td>0.47</td><td>0.35</td><td>0.41</td>
  </tr>
</table>

Table 7: Spearman correlations of candidate automatic metrics with human judgments on the English subset of SEAHORSE (Clark et al., 2023), grouped by subsets of the data.

• Be much shorter than the source text. Please see the information about the length above.

• Convey ALL the main core ideas and information of the source document.

• Have a structure of a standalone cohesive text.

• Follow the flow of the section/paragraph structure of the source text, try to summarize it from top to bottom.

• Each paragraph in each summary should be marked with a number of the source text section/chapter showing where this information is from.

Here’s a checklist which can help you with the task:

• Understand the Main Idea: Read the entire text to grasp the overall theme and the author’s intent. Identify the main idea of the text.

• Highlight Key Points: Mark or note down the essential points and arguments.

• Eliminate Redundancies: Remove any repetitive information or examples that do not add value to the understanding of the main idea.

• Use Your Own Words: Paraphrase the key points in your own words instead of copying verbatim. This helps ensure the summary is concise.

• Keep It Objective: Focus on the information presented in the text without inserting personal opinions or interpretations.

• Structure the Summary: Organize the ideas logically, maintaining the flow of the original text.

• Be Concise: Aim for clarity and brevity; Use simple and direct language to convey the points.

• Review and Revise: Compare the summary to the original text to ensure accuracy and completeness. Edit for coherence, transitions, and readability.

Additional Guidelines for Conversational Text
You may be assigned to work on conversational type of text, such as meeting transcripts, screenplays, and novels. Since the text structure is quite different from documents, this is an additional guideline to help you working on conversational text:

• Skim through the whole document: Try to get a rough idea of the whole plot
• Identify the characters and main core ideas: Identify the main characters and focus on their interaction

• Omit the trivial details: there maybe side plot or supporting characters in the source text, carefully decide if it is related to the main core ideas (plot)

• Group and summarize with respect to main core ideas: There could be plot twists or related hints in the source documents. Remember the summary should be clear and straightforward, the plot outline

• Should be clear in the summary without referencing to the source document

Requirements for Question and Answer sets

• The questions need to be abstractive not extractive: this means they need to be directed at the ideas in the text, not words and sentences as such.

• The questions should be open-ended, not Yes/No questions

• The correct answers should cover the main points in the source text: the questions should roughly correspond to paragraphs/sections in the text

• Thus, the correct answer should cover points reflected in the summary (for your convenience, you can refer to your longest summary, but please mind your shorter summary also need to be able to answer at least some of the questions)

• The answers should not be short (30 words or more)

• The correct answer should be found namely IN THE SUMMARY and not able to be just pulled from general knowledge

• If possible, refrain from factual questions, but try composing questions for reasoning, such as WHY- questions

• You are encouraged to combine information from different sections together

• Avoid only asking questions about the beginning and the end of the section, use the information in the middle as well

• If possible, the questions should not have several answer possibilities.

F Human Summary Sample

This is one sample of 3 human written summaries and QA set of the document 4586 from GovReport.

F.1 Source Document Excerpt

[This is the first and last paragraphs of the source document. The whole document is 5411 words long.]

This report examines technological innovation in payment systems generally and particular policy issues as a result of retail (i.e., point of sale) payment innovation. The report also discusses wholesale payment, clearing, and settlement systems that send payment messages between banks and transfer funds, including the "real-time payments" service being introduced by the Federal Reserve. This report includes an Appendix that describes interbank payment, clearing, and settlement systems related to U.S. payments.

...

To address systemic risk concerns, a private RTP system could be designated as a systemically important Financial Market Utility (FMU) under Title VIII of the Dodd-Frank Act ( P.L. 111-203 ). The Dodd-Frank Act allows the Financial Stability Oversight Council , a council of financial regulators led by the Treasury Secretary, to designate a payment, clearing, or settlement system as systemically important on the grounds that "the failure of or a disruption to the functioning of the FMU could create or increase the risk of significant liquidity or credit problems spreading among financial institutions or markets and thereby threaten the stability of the U.S. financial system." FMUs, currently including the Clearing House Interbank Payments System, are subject to heightened regulation, and the Fed has supervisory and enforcement powers to ensure those standards are met. Policymakers could consider whether systemic risk concerns are better addressed through Fed operation of payment and settlement systems or Fed regulation of private systems.

F.2 Long, Medium and Short Summaries

Each paragraph of the summaries is paired with the source paragraph id (e.g. p1, p2, etc) to indicate the information source.
Long Summary (20%)  To the average consumer, swiping their credit card seems simple, because the complex the infrastructure involved in is 'hidden'. These deceptively "simple" electronic payments are comprised of three main steps. First, the sender makes the payment through an online payment service or an app, which instructs the sender’s bank to make the payment to the recipient. Second, the bank sends a payment message to the recipient’s bank through a payment system or clearing service. Finally, the payment is completed (settled) when the funds are received by the recipient. (p2)

Some of the bank-to-bank (ACH) payment, clearing, and settlement (PCS) systems are operated by the Federal Reserve, and others by private-sector organizations. Recently, the use of electronic payment methods (credit card, debit card, and ACH) has grown, while the use cash and check payments has declined. Electronic payments have been made easier and more convenient with digital wallets and payment apps like Venmo, Cash App, and Zelle - all of which require users to link a bank account, credit card, or debit card. (p4, p5, p6, p7)

There are concerns about whether current regulations are equipped to handle electronic payments. If not, this poses potential risks to cybersecurity, data privacy, industry competition, and consumer access and protection. Current payment regulations depend, in part, on if the service is provided by a bank, who have many strict regulatory requirements. As such, Nonbank payment systems are not subject to existing regulatory enforcement and can only be supervised - as money transmitters at state level and money service businesses at federal level. (p8, p9)

Electronic payment regulatory concerns could be addressed by including nonbank payment companies into the bank regulatory regime. One way could be via the Office of the Comptroller of the Currency (OCC) special purpose national bank charter. And another, through a state-level industrial loan company (ILC) charter with the Federal Deposit Insurance Corporation (FDIC). Both methods could provide nonbank firms access to the Fed wholesale payment systems, which could be advantageous. However, some state regulators have filed lawsuits to block nonbank companies access to these charters, arguing that it allows companies to circumvent state consumer protections. So far, no companies have applied for an OCC charter, likely due to the legal uncertainty surrounding it. (p11, p12)

The main argument against nonbank payment companies filing ILC charters is that it would allow them to own banks - and the FDIC has not approved deposit insurance for a new ILC since 2006. Opponents argue that allowing a company to own a bank could expose the US economy to risks like imprudent underwriting. Proponents assert that these concerns are exaggerated, noting that several other countries allow similar arrangement with no ill effects. So far, Square is the only company with a pending application and two other companies have withdrawn their applications. (p13)

New technology reduces some risks related to payments but creates new ones. The risk of having one’s wallet stolen is reduced, but payment information is subject to more sophisticated risks such as malware attacks. Furthermore, storing payment information on a variety of websites, apps, and devices creates more opportunities for hackers. After recent security breaches which allowed user information to be stolen, several solutions have been proposed. For example, a federal breach notification law could be enacted, to create federal cybersecurity standards or to increase penalties for companies with inadequate security measures. (p15, p16, p17)

Payment systems need to collect detailed information about customer transactions in order to function properly. This data can be used by companies to target ads. Scammers can also use this information for fraudulent purposes. The constantly increasing use of Electronic payments has led to questions about how user data is used and whether consumers are sufficiently informed and given enough control about how their data is used. (p18)

There are some consumer benefits to storing consumer data. It can help them track payments and budget more easily by importing to budgeting apps. They can also share financial information with banks more easily when applying for loans. But, given the benefits and the risks, the question remains: how much access should companies have to individuals’ information? (p19)

Privacy policies are another area of concern with respect to consumer protection and electronic payments. According to the Bureau of Consumer Financial Protection (CFPB), it is difficult to provide disclosures that are clear and easy to understand, partly due to the small screens on phones. Clearer privacy policies and allowing consumers more control over how their data is used could help. (p20)

The Electronic Fund Transfer Act, Regulation
E implemented by the CFPB, is the most relevant law aimed at protecting consumers who are making electronic payments. It mandates consumer disclosures, limits consumer liability for unauthorized payments, and maintains procedures for resolving errors. Further regulations are being considered. (p22)

Consumers could also be protected through financial education, especially for more at-risk older and lower-income groups. This could include learning how to use new payment systems safely and how to protect against financial harm, as well as knowing how to get help if something goes wrong. (p24)

Payment system innovations may affect consumers differently based on income. Consumers who mainly pay with cash, don’t have bank accounts, or don’t have internet or mobile access won’t be able to benefit. Neither will those who are not comfortable using new technology. (p25)

However, surveys reveal that 83% of underbanked, and 50% of unbanked, consumers have smart phone access. So, as costs of these payment services decline, some marginalized groups could experience better access to the the financial system through access to digital currency channels via cash equivalents like pre-paid cards. But, the cost of internet and mobile data plans may limit access to faster payment systems, so this also needs to be considered. (p26, p27, p28)

Faster payment systems may also benefit low-income consumers by allowing them faster access to their paychecks and other fund transfers. But a potential drawback is that withdrawals from their accounts would occur more quickly as well. (p28)

In 2019, the Fed announced that it plans to create a wholesale real-time payment (RTP) system. (p32)

Originally, the Fed’s primary function was to provide bank-to-bank check-clearing services. Private clearing houses were experiencing issues that led to the creation of the Fed. As payment methods have evolved, the Fed has begun providing other types of payment systems. It does this by linking the accounts that all banks keep at the Fed so that it can complete the transfers. The new system that the Fed is developing, called FedNow, would allow payments to occur in real time, rather than later in the day - or even the next day, as is the case currently. (p33, p35)

However, there are some concerns regarding implementation of FedNow. Many worry that it will undermine private sector development of similar systems. Others fear that failing to implement FedNow will lead to a monopoly of a private-sector company, to the detriment of consumers and smaller banks. (p42, p44)

Medium Summary (10%) Electronic payments have three stages. First, the sender makes the payment through an online payment service or an app, which instructs the sender’s bank to make the payment to the recipient. Second, the bank sends a payment message to the recipient’s bank through a payment system or clearing service. Finally, the payment is completed (settled) when the funds are received by the recipient. (p2)

Some of the bank-to-bank (ACH) payment, clearing, and settlement (PCS) systems are operated by the Federal Reserve, and others by private-sector organizations. Recently, the use of electronic payment methods (credit card, debit card, and ACH) has grown, while the use cash and check payments has declined. Electronic payments have been made easier and more convenient with digital wallets and payment apps like Venmo, Cash App, and Zelle - all of which require users to link a bank account, credit card, or debit card. (p4, p5, p6, p7)

There is concern about whether current regulations are equipped to handle these technological advances. If not, they could pose risks to cybersecurity, data privacy, industry competition, and consumer access and protection. (p8)

One way to address these concerns is to add nonbank companies to the bank regulatory regime. Another is via the Office of the Comptroller of the Currency (OCC) special purpose national bank charter. And another, through a state-level industrial loan company (ILC) charter with the Federal Deposit Insurance Corporation (FDIC). Both could provide nonbank firms access to the Fed wholesale payment systems, which could be advantageous. However, some state regulators have tried to block nonbank access to these charters, arguing that it allows companies to circumvent state consumer protections. So far, no companies have applied for an OCC charter, likely due to the legal uncertainty surrounding it. (p11, p12)

The main argument against ILC charters is that it would allow companies to own banks. The FDIC has not approved deposit insurance for a new ILC since 2006. Opponents argue that allowing a retailer to own a bank could expose the US economy to risks such as imprudent underwriting. Proponents assert that these concerns are exaggerated
and that several other countries allow similar arrangement with no ill effects. Currently, Square is the only company with a pending application. (p13)

Privacy policies are another area of concern. According to the Bureau of Consumer Financial Protection (CFPB), it is difficult to provide disclosures that are clear and easy to understand, partly due to the small screens on phones. Clear privacy policies and allowing consumers more control over how their data is used could help. (p20)

The Electronic Fund Transfer Act, Regulation E implemented by the CFPB, is the most relevant law aimed at protecting consumers who are making electronic payments. It mandates consumer disclosures, limits consumer liability for unauthorized payments, and maintains procedures for resolving errors. (p22)

In 2019, the Fed announced that it plans to create a wholesale real-time payment (RTP) system. The proposed system, called FedNow, would allow payments to occur in real time, rather than later in the day - or even the next day, as is the case currently. (p32, p33, p35)

There are some concerns regarding FedNow. Many worry that it will undermine private sector development of similar systems. Others fear that failing to implement FedNow will lead to a monopoly of a private-sector company, to the detriment of consumers and smaller banks. (p42, p44)

Short Summaries (5%) Electronic payments have three stages. First, the sender makes the payment through an online payment service or an app, which instructs the sender’s bank to make the payment to the recipient. Second, the bank sends a payment message to the recipient’s bank through a payment system or clearing service. Finally, the payment is completed (settled) when the funds are received by the recipient. (p2)

Some of the bank-to-bank payment, clearing, and settlement (PCS) systems are operated by the Federal Reserve, and others by private-sector organizations. Recently, the use of electronic payment methods has grown, while the use cash and check payments has declined. (p4)

There is concern about whether current regulations are equipped to handle these technological advances. If not, they could pose risks to cybersecurity, data privacy, industry competition, and consumer access and protection. (p8, p9)

A potential way to resolve concerns regarding regulation is to add certain nonbank payment companies into the bank regulatory regime. One way to accomplish this is through the OCC or Office of the Comptroller of the Currency special purpose national bank charter. The second is through a state-level industrial loan company (ILC) charter with the Federal Deposit Insurance Corporation (FDIC). (p11, p12)

Privacy policies are another area of concern. According to the Bureau of Consumer Financial Protection (CFPB), it is difficult to provide disclosures that are clear and easy to understand, partly due to the small screens on phones. Clear privacy policies and allowing consumers more control over how their data is used could help. (p20)

In 2019, the Fed announced that it plans to create a wholesale real-time payment (RTP) system. The proposed system, called FedNow, would allow payments to occur in real time, rather than later in the day - or even the next day as is the case currently. (p32, p33, p35)

F.3 Question and Answer Set

Question 1: What are the three parts of a payment system?

• Answer: First, there is the sender or the person making the payment through an online payment service or an app, which instructs the sender’s bank to make the payment to the recipient. Second, the bank sends a payment message to the recipient’s bank through a payment system or clearing service. Finally, the payment is completed when the funds are transferred, or settled.

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p2)

Question 2: Who operates bank-to-bank payment, clearing, and settlement systems?

• Answer: Some of these systems are operated by the Federal Reserve and some are operated by private-sector organizations.

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p4)

Question 3: What issues could there be if current regulations are not equipped to handle these payment system innovations?
• Answer: If regulations are inadequate, there could be issues related to cybersecurity, data privacy, industry competition, and consumer access and protection.

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p8)

Question 4: What are two ways to bring nonbank companies into the bank regulatory regime?

• Answer: One way to accomplish this is through the OCC or Office of the Comptroller of the Currency special purpose national bank charter. The second is through a state-level industrial loan company (ILC) charter with the Federal Deposit Insurance Corporation (FDIC).

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p11,12)

Question 5: According to the Bureau of Consumer Financial Protection, what are some of the difficulties with privacy policies?

• Answer: It is difficult to provide disclosures that are clear and easy to understand, partly due to the small screens on phones.

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p20)

Question 6: What is FedNow?

• Answer: In 2019, the Fed announced that it plans to create a wholesale real-time payment (RTP) system. The proposed system, called FedNow, would allow payments to occur in real time, rather than later in the day or even the next day as is the case currently.

• (Information contained in 20% Summary, 10% Summary, 5% Summary.)

• (Source paragraph number: p29)

Question 7: What are some reasons for the increase in electronic payments?

• Answer: Electronic payments have increased because of payment apps such as Venmo, Cash App, and Zelle make it convenient and easy for consumers to send payments. Digital wallets stored on phones are another reason for increased electronic payments due their ease of use and convenience.

• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p6)

Question 8: What is necessary in order for a consumer to be able to use electronic payment services?

• Answer: The consumer must have a debit card, credit card, or bank account linked to an electronic payment system.

• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p7)

Question 9: Why have state regulators filed lawsuits to block the OCC?

• Answer: Regulators feel that the OCC charter would allow companies to avoid state regulations that protect consumers.

• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p11, p12)

Question 10: What is the main argument against the ILC charter?

• Answer: The ILC would allow companies such as retailers to own banks. Opponents are concerned that this could lead to imprudent underwriting and could hurt the US economy by exposing it to risk.

• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p13)

Question 11: What does the Electronic Funds Transfer Act Regulation E do?

• Answer: Regulation E mandates consumer disclosures, limits consumer liability for unauthorized payments, and maintains procedures for resolving errors.
• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p22)

Question 12: How could financial education help consumers use electronic payment systems safely?

• Answer: Consumers could be taught how to use new payment systems safely and how to protect against financial harm, as well as how to get help if something goes wrong.

• (Information contained in 20% Summary.)

• (Source paragraph number: p24)

Question 13: What are is an argument against the FedNow?

• Answer: Many worry that it will undermine private sector development of similar systems.

• (Information contained in 20% Summary, 10% Summary.)

• (Source paragraph number: p42, p44)

Question 14: How can storing more consumer data benefit consumers?

• Answer: It can help consumers track payments and budget more easily using budgeting apps. They can also share financial information with banks more easily when applying for loans.

• (Information contained in 20% Summary.)

• (Source paragraph number: p19)

Question 15: How could faster payment systems affect low-income consumers?

• Answer: Faster payment systems may benefit low-income consumers by allowing them faster access to their paychecks and other fund transfers. But a potential drawback is that withdrawals from their accounts would occur more quickly as well.

• (Information contained in 20% Summary.)

• (Source paragraph number: p28)

G Summarization Human Evaluation Guidelines

Annotator proficiency requirements All annotators must meet ALL of the following requirements:

• Native speaker of English AND

• Language related degree holder or related professionals

Background Information What is a good summary? A good summary should meet the following criteria:

• Conciseness: The summary should only contain the most important information while maintaining readability. Trivial information should not be included, even in the longest summary. Additionally, the summary should be comprehensible on its own, without needing to refer to additional documentation.

• Coverage of main core ideas: The summary should preserve the most important ideas, regardless of its length. In our task, summaries are created by gradually omitting less important information. Therefore, we expect that the core ideas will be retained in all summaries, even the shortest ones. Main core ideas should be the key ideas that help the reader to understand the main topic. Depending on the type of documents, the definition of idea would be slightly different. For example, if the source document is a meeting note, the summary should include the main topic, the discussion, the result / final decision. The trivial details like greetings or small talks should not be included. If the source document is a novel, the summary should focus on main characters and important events rather than trivial description of the character or side events.

• Attribution: the information in the summary can be accurately referred back to the source documents. All the information in the summary should be an abstraction from the source documents. No additional information that can not be found in the source document should be included in the summary.

• Cohesion as a document: Each summary should be an abstraction of the entire source
document. All the information or ideas should be digested from different parts of the source document and combined into a new paragraph. Merely shortening a document paragraph by paragraph will not be considered as a good summary. Similarly, a bulletin-like document jumping from point to point also will not be considered as a good summary.

Annotation Our summarization structure is as follows: The source text which is approximately 5,000 words long gets summarized three times: The first summary is 20% of the original length of the doc. It should retain all the core ideas of the source document. The second summary is 10% of the length of the source document. It should also retain all the core ideas of the source. The third summary is short, it should be 5% of the source length. We understand there will be some information loss, but again, all the core ideas should be present in the summary.

There are two tasks related to evaluating the summary.

Task 1 In this task, you need to rate the overall quality of the summaries regarding several aspects. Here is the detailed workflow:

Step 1 Screening
Please spend no more than 5 minutes skimming through the longest (20%) summary and answer the question below.

• Q1: Is it a cohesive text? Can you fully understand it?
  – If NOT, reject the task completely.
  – If YES, continue with the following steps

Step 2 Read the texts and take notes
Please read the whole source document carefully and take notes in your own way. It could be highlighting the key points or jotting down the ideas in your own words or any means that can help you digest the document. While reading please do not skip any line. After reading and taking note, you should be able to identify several main core ideas or more (You can spot more main core ideas if the text is longer). Please continue to read the summary and identify if the main ideas also exist in the summaries. Now check how many ideas can be found in the summary.
(This procedure is to help rate the summaries more objectively, you are not required to submit the highlights or the notes.)

Step 3 Rate the summaries
Answer all the following questions with a 4-point scale:

• Q2a Check the attribution of the summary. Can all the information in the summary be attributed to the source text?
  – Give 4 points if yes, all the information can be directly attributed to the source text.
  – Give 3 points if mostly yes, only 1 idea seems to not be found in the source text.
  – Give 2 points if not really, more than 1 idea cannot be attributed to the source text.
  – Give 1 point if not, most ideas cannot be found in the source text and seem to be completely new.
  – Give 0 points if not, none of the ideas can be found in the source text.

• Q2b Check the coverage of main core ideas of the source text in the summary. Are all the main core ideas of the source document retained?
  – Give 4 points if yes, all the main core ideas of the source are retained.
  – Give 3 points if mostly yes, only 1 or 2 main core ideas are not found in the summary.
  – Give 2 points if not really, more than 2 main core ideas are not found in the summary.
  – Give 1 point if not, most main core ideas are not found in the summary.
  – Give 0 points if not, none of the ideas can be found in the summary

• Q2c Check the conciseness of the summary. Is the summary short and clear without repetition and redundancy?
  – Give 4 points if yes, the summary is not wordy but clear.
  – Give 3 points if mostly yes, but 1 part is unnecessary.
  – Give 2 points if not really, more than 1 part is unnecessary.
  – Give 1 point if not, the summary is lengthy and most passages can be omitted without losing the core ideas.
– Give 0 points if not, the summary is lengthy and most passages can be omitted without losing the core ideas.

• Q2d Check the readability of the summary. Is the summary fluent and understandable?
  – Give 4 points if yes, the summary is fluent and understandable, and well written.
  – Give 3 points if mostly yes, but there is room for improvement.
  – Give 2 points if not really, not quite fluent and sometimes hard to understand.
  – Give 1 point if not, the summary is hard to read and understand.
  – Give 0 points if not, the summary is impossible to read and understand.

After evaluating all the aspects of the summary, please give an overall score of 0-10 on the quality of the summary.

• Q3 Do you think it is a good summary? On a scale of 0-10, how would you rate the overall quality of the summary?
  – 10: The summary is perfect in every aspect
  – 8-9: The summary is considered good. It contains minor issues in certain aspects but it meets all requirements with room for improvement.
  – 6-7: The summary is moderate, it contains non-critical errors but to help the reader understand the source documents
  – 4-5: The summary is below acceptable level. It contains critical errors that could potentially mislead the reader.
  – 2-3: The summary contains very limited information that is relevant to the source document
  – 0-1: The summary is barely readable and comprehensible or it barely contains relevant information to the source document

Make sure you have answered all the questions for every summary.

Task 2 You will be provided with 15 questions depending on the length of the documents. Please identify if the answer is directly stated, heavily implied, or logically entailed in the summary. You need to answer YES or NO only.

H Summary Expansion Human Evaluation Guidelines

Annotator proficiency requirements All annotators must meet ALL of the following requirements:

• Native speaker of English AND
• Language related degree holder or related professionals

Background Information What is a good summary expansion? A good summary expansion should meet the following criteria:

• Coverage of main core ideas: Main core ideas should be the key ideas covered in the original summary. During each expansion more details will be added, however the main core ideas should remain the same. For example, if the setting of the original summary is an office comedy, the ultimate text should not be an unrelated superhero movie screenplay.

What is a good long form text? On top of coverage of main core ideas, a good summary expansion should also meet the following criteria: Cohesion as a document:

• The ultimate document should be a stand-alone document so that the reader doesn’t need an additional document to understand the text. The ultimate document should be well-structured and formatted. For example, if the ultimate document is a screenplay, it should have clear scene sections, character direction and dialogs of characters.

• Non-repetitive and rich in details: Although the ultimate document is based on the summary, additional information / details is allowed. The ultimate document should not just repeat the core ideas.

• Being Interesting: A good screenplay and novel should be able to capture the reader’s attention and keep them engaged throughout the story / plot. We are not searching for an Oscar winning novel / screenplay, as long as the plot makes sense and the added details serve the purpose of the story, it is considered as an interesting plot. For example, you can check the following questions depending on the story: If it’s a comedy, does it sound funny
to you? If it’s a romance story, does it evoke the proper sentiment? Are the added details aligned with the plot, or do they feel out of place? … etc

Task 1   In this task, you need to rate the overall quality of the summary expansions regarding several aspects. Here is the detailed workflow:

Step 1 Screening
Please spend no more than 5 minutes skimming through the long form text and answer the question below.

• Q1: Is it a cohesive text? Can you fully understand it?
  – If NOT, reject the task completely.
  – If YES, continue with the following steps

Step 2 Read the texts and highlight key points
Please read the original summary carefully and highlight the key points and make notes. Do not skip any line. Continue to read other summaries and long form text, highlighting the key points that are the same as the original summary
(This procedure is to help rating the summaries more objectively, you are not required to submit the highlights or the notes.)

Step 3 Rate the long form text
Answer all the following questions with a 4-point scale separately for the long form text:

• Q2a Check the coverage of main core ideas. Are all the core concepts of the original summary retained?
  – Give 4 points if yes, all the main core ideas are retained.
  – Give 3 points if mostly yes, only 1 or 2 core ideas are lost.
  – Give 2 points if not really, more than 2 ideas are lost.
  – Give 1 point if not, most ideas are lost.
  – Give 0 points if not, none of the ideas can be found in the source text.

• Q2b Check the cohesion of text. Is it well structured? Does it contain all the necessary components? (Scene description, dialog, main characters, etc) Does it flow logically and maintain consistency?
  – Give 4 points if yes, it is a well structure screenplay / novel
  – Give 3 points if mostly yes, but 1 part is missing.
  – Give 2 points if not really, more than 1 part is missing.
  – Give 1 point if text does not follow the structure of a screenplay / novel
  – Give 0 points if not, the text doesn’t not read as a cohesive text at all

• Q2c Check the richness in details. Does it contain enough details?
  – Give 4 points if yes, the text contains a lot of details.
  – Give 3 points if yes, the text contains details but has room for improvement.
  – Give 2 points if not really, the text contains limited details.
  – Give 1 point if not, the text contains very few details
  – Give 0 points if not, the text does not provide any additional details at all.

• Q2d Check the creativity. Does the added details novel and original while being relevant to the core main ideas?
  – Give 4 points if yes, all of the added details are novel and original
  – Give 3 points if yes, most of the added details are novel and original.
  – Give 2 points if not really, only some of the added details are novel and original.
  – Give 1 point if no, very few added details are repetitive.
  – Give 0 points if no, no added details are novel and original .

• Q2e Check the non-repetitiveness. Does it repeat a lot?
  – Give 4 points if no, all of the details are unique and different from each other
  – Give 3 points if no, most of the details are unique, only one is repeated
  – Give 2 points if not really, some of the details are repetitive
  – Give 1 point if yes, most the added details are repetitive
  – Give 0 points if not, all the added details are repetitive
• Q2f Rate the story plot. How interesting is it to you? Is it engaging and compelling?
  – Give 4 points if the text is very interesting.
  – Give 3 points if the text is quite interesting.
  – Give 2 points if the text is somewhat interesting.
  – Give 1 point if the text is only slightly interesting.
  – Give 0 points if the text is dull and not interesting at all.

After evaluating all the aspects of the expanded text, please give an overall score of 0-10 on the quality of the expanded text.

• Q3 Do you think expanded text is well written? Do you think it is a good read? On a scale of 0-10, how would you rate the overall quality?
  – 10: The text is perfect in every aspects
  – 8-9: The text is considered good. It contains minor issues in certain aspects but it meets all requirements with room for improvement.
  – 6-7: The text is moderate, it contains non-critical errors but to help the reader understand the source documents
  – 4-5: The text is below acceptable level. It contains critical errors that cause trouble to read
  – 2-3: The text contains very limited information that is relevant to the source summary
  – 0-1: The text is barely readable and comprehensible or it barely contains relevant information to the source summary.

Make sure you have answered all the questions for every expanded summary and long form text.

Task 2 You will be provided with around 15 questions depending on the length of the documents. You need to answer YES or NO to each QA set. Answer YES only when the answer is directly stated, heavily implied, or logically entailed in the text.

I Prompting Details

The prompts contain three parts: General guideline, domain-specific prompts, and input context.

The general guideline adapts the human guidelines (Appendix E) for the summarization and summary expansion, while the domain specific prompts give extra information about the domain as instructions of expected output. In the prompt template below, the general guideline is provided, {{domain-X}} denotes the domain-specific prompt. {{input}} is for the input document for the summarization task and human summaries for the summary expansion task.

Prompt for summarization

"""
You are a professional editor and reader.
You are reading a {{domain}}
{{domain-meta}}.
The {{domain}} starts with [START] and ends with [END].
After you have finished reading, please provide a summary of the {{domain}}.
{{domain-expect}}.
Make sure the summary has {{len(input) * ratio + 200}} words or less.
[START]
{{input}}
[END].
Write at least {{len(input) * ratio}} words.
"""

Prompt for summary expansion

"""
You are a professional editor and reader.
You are reading a summary of {{domain}}
{{domain-meta}}.
The {{domain}} starts with [START] and ends with [END].
After you have finished reading, write a well-structured, consistent {{domain}} that extends the summary.
{{domain-expect-expand}}.
[START]
{{input}}
[END].
Write at least {{len(source)}} words.
"""

The model-specific prompts for each domain are listed below. Note that not all domains have the prompt template for summary expansion.

• BookSum:

{{domain}}: "book chapter"
{{domain-meta}}: '"about the book [BOOK-TITLE], chapter [CHAP-NO], title [[CHAP-TITLE]]".'
{{domain-expect}}: '"'
{{domain-expect-expand}}: '"Please keep the main plot and characters if found in the summary.'
""

• LexGLUE:
{{domain}}: "legal document"
{{domain-meta}}: ""
{{domain-expect}}: ""Keep the main ideas and terms in the document.
"""

• SQuALITY:

{{domain}}: "short story"
{{domain-meta}}: ""
{{domain-expect}} : ""Keep the main character names and narratives of the story.
"""
{{domain-expect-expand}}: (same)

• Seahorse:

{{domain}}: "news article"
{{domain-meta}}: ""
{{domain-expect}}: ""

• FacetSum:

{{domain}}: "academic article"
{{domain-meta}}: "about [TITLE]"
{{domain-expect}}: ""Keep the structure of sections [SECTIONS]
"""
{{domain-expect-expand}}: (same)

• JRC-Acquis:

{{domain}}: "document"
{{domain-meta}}: "from European Commision"
{{domain-expect}}: ""

• MultiUN:

{{domain}}: "document"
{{domain-meta}}: "from United Nation"
{{domain-expect}}: ""

• GovReport:

{{domain}}: "government report"
{{domain-meta}}: ""
{{domain-expect}}: ""

• Wikipedia:

{{domain}}: "Wikipedia article"
{{domain-meta}}: "about [TITLE]"
{{domain-expect}}: ""

• Summscreen:

{{domain}}: "screenplay"
{{domain-meta}}: "about [TITLE]"
{{domain-expect}}: ""Keep the main plot and characters in the screenplay
"""
{{domain-expect-expand}}: ""Keep the main plot and characters in the summary.
Write in the dialogue form with multiple utterances
"""

J Detailed Results

Complementing Table 3 from Section 3, Tables 8 and 9 show results of different models in the summary expansion tasks on a different level of summary and expansion. Tables 10, 11 and 12 show the detailed results broken down by domain on the summary expansion task given the summaries (5%, 10% and 20%) and expanding by a respective factor of (20, 10, 5). It is shown that, despite having repeated instructions on the length, all models behave greatly differently in different domains. In particular, GPT-4o-mini can generate longer scientific/technical texts, but struggle to generate longer texts in conversational texts without sacrificing the qualities. On the other hand, medium-sized models such as LLAMA 3.1-8B generate more texts consistently across domains, but at a higher repetition.

The results of the summarization task of different levels are detailed in tables 13, 14 and 15. According to human evaluation, the best performing result in Table 13 is with GPT-4o-mini in the wikipedia domain and in Table 14 is with the same model in the legal domain (LexGlue). For table 15, the best results are with human output in the conversational domain (Summscreen).
<table>
  <tr>
    <th>Output</th>
    <th>%WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>4.789</b></td>
    <td>0.623</td>
    <td><b>0.910</b></td>
    <td>0.564</td>
    <td>0.450</td>
    <td>67.617</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>2.807</td>
    <td>1.830</td>
    <td>0.913</td>
    <td>0.625</td>
    <td>0.391</td>
    <td>54.344</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>1.788</td>
    <td>0.781</td>
    <td>0.906</td>
    <td>0.752</td>
    <td>0.501</td>
    <td>42.572</td>
  </tr>
</table>

Table 8: Performance on the summary expansion task by a factor of 10, given the 10% summary input.

<table>
  <tr>
    <th>Output</th>
    <th>%WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>7.960</b></td>
    <td>0.592</td>
    <td>0.909</td>
    <td>0.574</td>
    <td>0.455</td>
    <td>65.951</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>2.296</td>
    <td>0.631</td>
    <td>0.871</td>
    <td>0.720</td>
    <td>0.488</td>
    <td>51.539</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>3.805</td>
    <td>0.720</td>
    <td>0.967</td>
    <td>0.862</td>
    <td>0.562</td>
    <td>45.141</td>
  </tr>
</table>

Table 9: Performance on the summary expansion task by a factor of 20, given the 5% summary input.

<table>
  <tr>
    <th>DATASET</th>
    <th>Model</th>
    <th>% WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
  </tr>
  <tr>
    <td rowspan="3"><b>BookSum</b></td>
    <td>GPT-4o-mini</td>
    <td>7.116</td>
    <td>0.420</td>
    <td>0.953</td>
    <td>0.701</td>
    <td>0.523</td>
    <td>82.421</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>3.307</td>
    <td>0.631</td>
    <td>0.931</td>
    <td>0.863</td>
    <td>0.556</td>
    <td>66.252</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>5.070</td>
    <td>2.170</td>
    <td>0.975</td>
    <td>0.787</td>
    <td>0.442</td>
    <td>60.393</td>
  </tr>
  <tr>
    <td rowspan="3"><b>SQuALITY</b></td>
    <td>GPT-4o-mini</td>
    <td>7.335</td>
    <td>0.398</td>
    <td>0.954</td>
    <td>0.695</td>
    <td>0.523</td>
    <td>55.620</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>3.010</td>
    <td>0.530</td>
    <td>0.946</td>
    <td>0.769</td>
    <td>0.536</td>
    <td>47.703</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>3.166</td>
    <td>0.624</td>
    <td>0.947</td>
    <td>0.795</td>
    <td>0.539</td>
    <td>42.532</td>
  </tr>
  <tr>
    <td rowspan="3"><b>FacetSum</b></td>
    <td>GPT-4o-mini</td>
    <td>9.348</td>
    <td>0.635</td>
    <td>0.936</td>
    <td>0.593</td>
    <td>0.467</td>
    <td>59.238</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.253</td>
    <td>0.792</td>
    <td>0.769</td>
    <td>0.816</td>
    <td>0.475</td>
    <td>47.145</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.369</td>
    <td>0.998</td>
    <td>0.843</td>
    <td>0.841</td>
    <td>0.495</td>
    <td>27.660</td>
  </tr>
  <tr>
    <td rowspan="3"><b>Summscreen</b></td>
    <td>GPT-4o-mini</td>
    <td>8.042</td>
    <td>0.914</td>
    <td>0.794</td>
    <td>0.309</td>
    <td>0.307</td>
    <td>74.779</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>2.615</td>
    <td>0.572</td>
    <td>0.836</td>
    <td>0.432</td>
    <td>0.385</td>
    <td>50.914</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>3.370</td>
    <td>0.712</td>
    <td>0.840</td>
    <td>0.474</td>
    <td>0.391</td>
    <td>44.121</td>
  </tr>
</table>

Table 10: Performance on the summary expansion task by a factor of 20, given the 5% summary input, per dataset.

<table>
  <tr>
    <th>DATASET</th>
    <th>Model</th>
    <th>% WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
  </tr>
  <tr>
    <td rowspan="3"><b>BookSum</b></td>
    <td>GPT-4o-mini</td>
    <td>5.155</td>
    <td>0.443</td>
    <td>0.955</td>
    <td>0.689</td>
    <td>0.518</td>
    <td>82.421</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>3.596</td>
    <td>1.453</td>
    <td>0.981</td>
    <td>0.817</td>
    <td>0.503</td>
    <td>74.280</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>2.451</td>
    <td>0.778</td>
    <td>0.950</td>
    <td>0.859</td>
    <td>0.551</td>
    <td>60.163</td>
  </tr>
  <tr>
    <td rowspan="3"><b>SQuALITY</b></td>
    <td>GPT-4o-mini</td>
    <td>3.531</td>
    <td>0.418</td>
    <td>0.952</td>
    <td>0.665</td>
    <td>0.511</td>
    <td>56.867</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>3.104</td>
    <td>1.098</td>
    <td>0.957</td>
    <td>0.718</td>
    <td>0.485</td>
    <td>50.855</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>2.056</td>
    <td>0.659</td>
    <td>0.951</td>
    <td>0.813</td>
    <td>0.544</td>
    <td>38.008</td>
  </tr>
  <tr>
    <td rowspan="3"><b>FacetSum</b></td>
    <td>GPT-4o-mini</td>
    <td>5.220</td>
    <td>0.675</td>
    <td>0.939</td>
    <td>0.573</td>
    <td>0.459</td>
    <td>62.472</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.504</td>
    <td>3.005</td>
    <td>0.949</td>
    <td>0.707</td>
    <td>0.352</td>
    <td>33.248</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.356</td>
    <td>0.952</td>
    <td>0.882</td>
    <td>0.860</td>
    <td>0.517</td>
    <td>29.713</td>
  </tr>
  <tr>
    <td rowspan="3"><b>Summscreen</b></td>
    <td>GPT-4o-mini</td>
    <td>5.251</td>
    <td>0.954</td>
    <td>0.794</td>
    <td>0.331</td>
    <td>0.311</td>
    <td>68.707</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>4.023</td>
    <td>1.762</td>
    <td>0.766</td>
    <td>0.259</td>
    <td>0.224</td>
    <td>58.992</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>2.290</td>
    <td>0.736</td>
    <td>0.843</td>
    <td>0.475</td>
    <td>0.390</td>
    <td>42.404</td>
  </tr>
</table>

Table 11: Performance on the 10% summary expansion task per dataset.
<table>
  <tr>
    <th>DATASET</th>
    <th>Model</th>
    <th>% WC</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>HUM↑</th>
  </tr>
  <tr>
    <td rowspan="3">BookSum</td>
    <td>GPT-4o-mini</td>
    <td>0.641</td>
    <td>0.459</td>
    <td>0.960</td>
    <td>0.739</td>
    <td>0.536</td>
    <td>87.161</td>
    <td>6.691</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>1.539</td>
    <td>0.650</td>
    <td>0.867</td>
    <td>0.842</td>
    <td>0.526</td>
    <td>50.969</td>
    <td>5.123</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>1.695</td>
    <td>0.736</td>
    <td>0.936</td>
    <td>0.861</td>
    <td>0.550</td>
    <td>53.692</td>
    <td>5.160</td>
  </tr>
  <tr>
    <td rowspan="2">SQuALITY</td>
    <td>GPT-4o-mini</td>
    <td>3.014</td>
    <td>0.513</td>
    <td>0.961</td>
    <td>0.738</td>
    <td>0.532</td>
    <td>60.385</td>
    <td>6.320</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>1.107</td>
    <td>0.582</td>
    <td>0.952</td>
    <td>0.780</td>
    <td>0.539</td>
    <td>37.775</td>
    <td>5.434</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>1.351</td>
    <td>0.735</td>
    <td>0.955</td>
    <td>0.817</td>
    <td>0.542</td>
    <td>35.017</td>
    <td>5.360</td>
  </tr>
  <tr>
    <td rowspan="3">FacetSum</td>
    <td>GPT-4o-mini</td>
    <td>0.502</td>
    <td>0.642</td>
    <td>0.954</td>
    <td>0.640</td>
    <td>0.488</td>
    <td>70.061</td>
    <td>7.693</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.138</td>
    <td>0.705</td>
    <td>0.874</td>
    <td>0.877</td>
    <td>0.537</td>
    <td>26.587</td>
    <td>3.453</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.205</td>
    <td>0.935</td>
    <td>0.871</td>
    <td>0.871</td>
    <td>0.518</td>
    <td>28.462</td>
    <td>4.173</td>
  </tr>
  <tr>
    <td rowspan="2">Summscreen</td>
    <td>GPT-4o-mini</td>
    <td>3.569</td>
    <td>1.215</td>
    <td>0.776</td>
    <td>0.319</td>
    <td>0.284</td>
    <td>65.977</td>
    <td>5.000</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>1.449</td>
    <td>0.782</td>
    <td>0.814</td>
    <td>0.500</td>
    <td>0.386</td>
    <td>41.466</td>
    <td>3.800</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>1.495</td>
    <td>0.828</td>
    <td>0.851</td>
    <td>0.568</td>
    <td>0.418</td>
    <td>36.493</td>
    <td>4.480</td>
  </tr>
</table>

Table 12: Performance on the summary expansion task by a factor of 5, given the 20% summary input.

<table>
  <tr>
    <th>DATASET</th>
    <th>Model</th>
    <th>R-L(↑)</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>SH-4↑</th>
    <th>SH-5↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>HUM↑</th>
  </tr>
  <tr>
    <th colspan="11">LCFO5%</th>
  </tr>
  <tr>
    <td rowspan="3">LexGLUE</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.258</td>
    <td>0.930</td>
    <td>0.807</td>
    <td>0.617</td>
    <td>0.339</td>
    <td>0.528</td>
    <td>46.690</td>
    <td>6.360</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.342</td>
    <td>0.407</td>
    <td>0.956</td>
    <td>0.688</td>
    <td>0.657</td>
    <td>0.500</td>
    <td>0.479</td>
    <td>78.916</td>
    <td>7.747</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.386</td>
    <td>0.415</td>
    <td>0.954</td>
    <td>0.875</td>
    <td>0.625</td>
    <td>0.369</td>
    <td>0.482</td>
    <td>63.280</td>
    <td>6.987</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.378</td>
    <td>0.471</td>
    <td>0.972</td>
    <td>0.879</td>
    <td>0.617</td>
    <td>0.383</td>
    <td>0.476</td>
    <td>59.455</td>
    <td>6.907</td>
  </tr>
  <tr>
    <td rowspan="3">BookSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.226</td>
    <td>0.913</td>
    <td>0.762</td>
    <td>0.572</td>
    <td>0.315</td>
    <td>0.503</td>
    <td>71.006</td>
    <td>6.691</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.302</td>
    <td>0.257</td>
    <td>0.977</td>
    <td>0.857</td>
    <td>0.599</td>
    <td>0.485</td>
    <td>0.532</td>
    <td>93.168</td>
    <td>6.815</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.377</td>
    <td>0.362</td>
    <td>0.976</td>
    <td>0.846</td>
    <td>0.578</td>
    <td>0.374</td>
    <td>0.483</td>
    <td>76.871</td>
    <td>6.272</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.372</td>
    <td>0.400</td>
    <td>0.973</td>
    <td>0.846</td>
    <td>0.581</td>
    <td>0.347</td>
    <td>0.469</td>
    <td>72.999</td>
    <td>6.049</td>
  </tr>
  <tr>
    <td rowspan="3">SQuALITY</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.263</td>
    <td>0.922</td>
    <td>0.760</td>
    <td>0.520</td>
    <td>0.334</td>
    <td>0.497</td>
    <td>33.534</td>
    <td>5.173</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.285</td>
    <td>0.284</td>
    <td>0.980</td>
    <td>0.841</td>
    <td>0.548</td>
    <td>0.375</td>
    <td>0.492</td>
    <td>74.618</td>
    <td>6.600</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.340</td>
    <td>0.472</td>
    <td>0.961</td>
    <td>0.802</td>
    <td>0.463</td>
    <td>0.201</td>
    <td>0.391</td>
    <td>64.237</td>
    <td>5.227</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.339</td>
    <td>0.535</td>
    <td>0.968</td>
    <td>0.819</td>
    <td>0.488</td>
    <td>0.233</td>
    <td>0.395</td>
    <td>57.288</td>
    <td>5.827</td>
  </tr>
  <tr>
    <td rowspan="3">FacetSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.260</td>
    <td>0.945</td>
    <td>0.835</td>
    <td>0.691</td>
    <td>0.436</td>
    <td>0.571</td>
    <td>57.456</td>
    <td>7.053</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.404</td>
    <td>0.354</td>
    <td>0.921</td>
    <td>0.568</td>
    <td>0.682</td>
    <td>0.524</td>
    <td>0.468</td>
    <td>73.968</td>
    <td>7.434</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.412</td>
    <td>0.387</td>
    <td>0.962</td>
    <td>0.884</td>
    <td>0.696</td>
    <td>0.508</td>
    <td>0.533</td>
    <td>67.585</td>
    <td>6.213</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.419</td>
    <td>0.425</td>
    <td>0.967</td>
    <td>0.888</td>
    <td>0.704</td>
    <td>0.518</td>
    <td>0.530</td>
    <td>69.176</td>
    <td>6.733</td>
  </tr>
  <tr>
    <td rowspan="3">JRC-Acquis</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.247</td>
    <td>0.949</td>
    <td>0.849</td>
    <td>0.672</td>
    <td>0.464</td>
    <td>0.577</td>
    <td>52.092</td>
    <td>7.180</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.352</td>
    <td>0.383</td>
    <td>0.952</td>
    <td>0.539</td>
    <td>0.682</td>
    <td>0.593</td>
    <td>0.477</td>
    <td>82.239</td>
    <td>7.347</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.390</td>
    <td>0.424</td>
    <td>0.942</td>
    <td>0.883</td>
    <td>0.690</td>
    <td>0.470</td>
    <td>0.512</td>
    <td>60.948</td>
    <td>6.306</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.368</td>
    <td>0.427</td>
    <td>0.945</td>
    <td>0.882</td>
    <td>0.673</td>
    <td>0.449</td>
    <td>0.504</td>
    <td>59.209</td>
    <td>6.514</td>
  </tr>
  <tr>
    <td rowspan="3">MultiUN</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.255</td>
    <td>0.927</td>
    <td>0.862</td>
    <td>0.592</td>
    <td>0.276</td>
    <td>0.521</td>
    <td>44.466</td>
    <td>6.861</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.352</td>
    <td>0.364</td>
    <td>0.968</td>
    <td>0.549</td>
    <td>0.630</td>
    <td>0.528</td>
    <td>0.462</td>
    <td>76.639</td>
    <td>7.347</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.402</td>
    <td>0.400</td>
    <td>0.955</td>
    <td>0.903</td>
    <td>0.618</td>
    <td>0.303</td>
    <td>0.476</td>
    <td>76.121</td>
    <td>6.611</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.378</td>
    <td>0.443</td>
    <td>0.965</td>
    <td>0.907</td>
    <td>0.608</td>
    <td>0.320</td>
    <td>0.471</td>
    <td>59.683</td>
    <td>6.806</td>
  </tr>
  <tr>
    <td rowspan="3">Wikipedia</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.246</td>
    <td>0.961</td>
    <td>0.810</td>
    <td>0.664</td>
    <td>0.246</td>
    <td>0.527</td>
    <td>68.484</td>
    <td>6.893</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.341</td>
    <td>0.332</td>
    <td>0.974</td>
    <td>0.756</td>
    <td>0.693</td>
    <td>0.423</td>
    <td>0.503</td>
    <td>80.633</td>
    <td>7.754</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.382</td>
    <td>0.405</td>
    <td>0.968</td>
    <td>0.821</td>
    <td>0.660</td>
    <td>0.299</td>
    <td>0.469</td>
    <td>59.334</td>
    <td>6.551</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.379</td>
    <td>0.439</td>
    <td>0.963</td>
    <td>0.839</td>
    <td>0.672</td>
    <td>0.282</td>
    <td>0.463</td>
    <td>59.259</td>
    <td>5.841</td>
  </tr>
  <tr>
    <td rowspan="3">GovReport</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.226</td>
    <td>0.958</td>
    <td>0.803</td>
    <td>0.639</td>
    <td>0.336</td>
    <td>0.538</td>
    <td>35.157</td>
    <td>6.720</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.340</td>
    <td>0.333</td>
    <td>0.978</td>
    <td>0.722</td>
    <td>0.696</td>
    <td>0.538</td>
    <td>0.520</td>
    <td>81.420</td>
    <td>7.280</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.407</td>
    <td>0.363</td>
    <td>0.973</td>
    <td>0.870</td>
    <td>0.651</td>
    <td>0.430</td>
    <td>0.512</td>
    <td>54.626</td>
    <td>6.080</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.407</td>
    <td>0.353</td>
    <td>0.971</td>
    <td>0.855</td>
    <td>0.620</td>
    <td>0.354</td>
    <td>0.489</td>
    <td>52.231</td>
    <td>6.44</td>
  </tr>
  <tr>
    <td rowspan="3">Summscreen</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.243</td>
    <td>0.927</td>
    <td>0.739</td>
    <td>0.532</td>
    <td>0.384</td>
    <td>0.507</td>
    <td>62.003</td>
    <td>7.040</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.294</td>
    <td>0.289</td>
    <td>0.984</td>
    <td>0.832</td>
    <td>0.514</td>
    <td>0.346</td>
    <td>0.478</td>
    <td>60.347</td>
    <td>6.627</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.390</td>
    <td>0.328</td>
    <td>0.985</td>
    <td>0.849</td>
    <td>0.523</td>
    <td>0.259</td>
    <td>0.458</td>
    <td>70.638</td>
    <td>6.173</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.375</td>
    <td>0.373</td>
    <td>0.976</td>
    <td>0.854</td>
    <td>0.526</td>
    <td>0.266</td>
    <td>0.450</td>
    <td>69.116</td>
    <td>5.667</td>
  </tr>
  <tr>
    <td rowspan="3">Seahorse</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.213</td>
    <td>0.950</td>
    <td>0.819</td>
    <td>0.651</td>
    <td>0.440</td>
    <td>0.563</td>
    <td>51.057</td>
    <td>6.200</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.295</td>
    <td>0.279</td>
    <td>0.985</td>
    <td>0.832</td>
    <td>0.647</td>
    <td>0.556</td>
    <td>0.548</td>
    <td>67.220</td>
    <td>7.613</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.352</td>
    <td>0.382</td>
    <td>0.965</td>
    <td>0.842</td>
    <td>0.661</td>
    <td>0.434</td>
    <td>0.504</td>
    <td>65.295</td>
    <td>6.293</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.354</td>
    <td>0.369</td>
    <td>0.978</td>
    <td>0.846</td>
    <td>0.649</td>
    <td>0.472</td>
    <td>0.515</td>
    <td>65.893</td>
    <td>6.427</td>
  </tr>
</table>

Table 13: Performance on the 5% summarization task per dataset.
<table>
  <tr>
    <th rowspan="2">DATASET</th>
    <th rowspan="2">Model</th>
    <th colspan="10">LCFO.10%</th>
  </tr>
  <tr>
    <th>R-L(↑)</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>SH-4↑</th>
    <th>SH-5↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>HUM↑</th>
  </tr>
  <tr>
    <td rowspan="4">LexGLUE</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.351</td>
    <td>0.940</td>
    <td>0.829</td>
    <td>0.660</td>
    <td>0.362</td>
    <td>0.544</td>
    <td>62.141</td>
    <td>7.387</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.419</td>
    <td>0.494</td>
    <td>0.947</td>
    <td>0.599</td>
    <td>0.633</td>
    <td>0.500</td>
    <td>0.437</td>
    <td>80.094</td>
    <td>8.120</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.452</td>
    <td>0.566</td>
    <td>0.942</td>
    <td>0.882</td>
    <td>0.625</td>
    <td>0.397</td>
    <td>0.456</td>
    <td>59.666</td>
    <td>7.080</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.439</td>
    <td>0.641</td>
    <td>0.967</td>
    <td>0.876</td>
    <td>0.621</td>
    <td>0.376</td>
    <td>0.440</td>
    <td>59.349</td>
    <td>7.200</td>
  </tr>
  <tr>
    <td rowspan="4">BookSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.278</td>
    <td>0.907</td>
    <td>0.757</td>
    <td>0.610</td>
    <td>0.342</td>
    <td>0.512</td>
    <td>83.776</td>
    <td>7.601</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.327</td>
    <td>0.308</td>
    <td>0.978</td>
    <td>0.858</td>
    <td>0.578</td>
    <td>0.442</td>
    <td>0.510</td>
    <td>94.867</td>
    <td>7.062</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.427</td>
    <td>0.456</td>
    <td>0.967</td>
    <td>0.835</td>
    <td>0.573</td>
    <td>0.335</td>
    <td>0.451</td>
    <td>82.114</td>
    <td>6.469</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.415</td>
    <td>0.511</td>
    <td>0.966</td>
    <td>0.844</td>
    <td>0.551</td>
    <td>0.313</td>
    <td>0.432</td>
    <td>73.681</td>
    <td>6.420</td>
  </tr>
  <tr>
    <td rowspan="4">SQuALITY</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.313</td>
    <td>0.917</td>
    <td>0.773</td>
    <td>0.548</td>
    <td>0.312</td>
    <td>0.497</td>
    <td>51.501</td>
    <td>6.000</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.327</td>
    <td>0.329</td>
    <td>0.975</td>
    <td>0.819</td>
    <td>0.525</td>
    <td>0.341</td>
    <td>0.466</td>
    <td>76.441</td>
    <td>6.467</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.382</td>
    <td>0.367</td>
    <td>0.974</td>
    <td>0.836</td>
    <td>0.518</td>
    <td>0.320</td>
    <td>0.456</td>
    <td>46.731</td>
    <td>4.613</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.373</td>
    <td>0.406</td>
    <td>0.979</td>
    <td>0.856</td>
    <td>0.526</td>
    <td>0.324</td>
    <td>0.456</td>
    <td>50.129</td>
    <td>5.280</td>
  </tr>
  <tr>
    <td rowspan="4">FacetSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.328</td>
    <td>0.942</td>
    <td>0.840</td>
    <td>0.710</td>
    <td>0.425</td>
    <td>0.570</td>
    <td>69.570</td>
    <td>7.680</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.461</td>
    <td>0.409</td>
    <td>0.945</td>
    <td>0.658</td>
    <td>0.666</td>
    <td>0.506</td>
    <td>0.473</td>
    <td>78.381</td>
    <td>7.882</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.455</td>
    <td>0.538</td>
    <td>0.934</td>
    <td>0.890</td>
    <td>0.696</td>
    <td>0.527</td>
    <td>0.502</td>
    <td>63.663</td>
    <td>6.501</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.449</td>
    <td>0.547</td>
    <td>0.954</td>
    <td>0.892</td>
    <td>0.698</td>
    <td>0.496</td>
    <td>0.499</td>
    <td>63.768</td>
    <td>6.987</td>
  </tr>
  <tr>
    <td rowspan="4">JRC-Acquis</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.339</td>
    <td>0.959</td>
    <td>0.845</td>
    <td>0.702</td>
    <td>0.512</td>
    <td>0.590</td>
    <td>61.618</td>
    <td>7.680</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.433</td>
    <td>0.479</td>
    <td>0.947</td>
    <td>0.548</td>
    <td>0.668</td>
    <td>0.543</td>
    <td>0.445</td>
    <td>81.398</td>
    <td>7.819</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.440</td>
    <td>0.579</td>
    <td>0.922</td>
    <td>0.888</td>
    <td>0.662</td>
    <td>0.455</td>
    <td>0.470</td>
    <td>52.570</td>
    <td>6.542</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.443</td>
    <td>0.586</td>
    <td>0.938</td>
    <td>0.859</td>
    <td>0.673</td>
    <td>0.441</td>
    <td>0.465</td>
    <td>49.986</td>
    <td>7.02</td>
  </tr>
  <tr>
    <td rowspan="4">MultiUN</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.312</td>
    <td>0.942</td>
    <td>0.871</td>
    <td>0.612</td>
    <td>0.334</td>
    <td>0.539</td>
    <td>57.357</td>
    <td>7.902</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.422</td>
    <td>0.455</td>
    <td>0.961</td>
    <td>0.629</td>
    <td>0.621</td>
    <td>0.518</td>
    <td>0.455</td>
    <td>74.459</td>
    <td>7.875</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.447</td>
    <td>0.546</td>
    <td>0.912</td>
    <td>0.914</td>
    <td>0.622</td>
    <td>0.329</td>
    <td>0.446</td>
    <td>52.920</td>
    <td>6.903</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.446</td>
    <td>0.557</td>
    <td>0.950</td>
    <td>0.900</td>
    <td>0.606</td>
    <td>0.295</td>
    <td>0.439</td>
    <td>55.186</td>
    <td>7.014</td>
  </tr>
  <tr>
    <td rowspan="4">Wikipedia</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.286</td>
    <td>0.969</td>
    <td>0.812</td>
    <td>0.723</td>
    <td>0.286</td>
    <td>0.547</td>
    <td>78.316</td>
    <td>7.747</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.388</td>
    <td>0.428</td>
    <td>0.963</td>
    <td>0.640</td>
    <td>0.690</td>
    <td>0.446</td>
    <td>0.462</td>
    <td>78.149</td>
    <td>7.696</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.445</td>
    <td>0.557</td>
    <td>0.931</td>
    <td>0.830</td>
    <td>0.670</td>
    <td>0.278</td>
    <td>0.430</td>
    <td>57.310</td>
    <td>6.681</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.444</td>
    <td>0.489</td>
    <td>0.963</td>
    <td>0.822</td>
    <td>0.691</td>
    <td>0.322</td>
    <td>0.462</td>
    <td>58.038</td>
    <td>6.246</td>
  </tr>
  <tr>
    <td rowspan="4">GovReport</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.296</td>
    <td>0.956</td>
    <td>0.815</td>
    <td>0.670</td>
    <td>0.361</td>
    <td>0.548</td>
    <td>52.627</td>
    <td>6.720</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.402</td>
    <td>0.420</td>
    <td>0.972</td>
    <td>0.639</td>
    <td>0.683</td>
    <td>0.529</td>
    <td>0.480</td>
    <td>81.374</td>
    <td>7.72</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.454</td>
    <td>0.511</td>
    <td>0.923</td>
    <td>0.874</td>
    <td>0.667</td>
    <td>0.406</td>
    <td>0.472</td>
    <td>49.294</td>
    <td>6.57</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.459</td>
    <td>0.498</td>
    <td>0.972</td>
    <td>0.871</td>
    <td>0.650</td>
    <td>0.415</td>
    <td>0.482</td>
    <td>52.712</td>
    <td>6.987</td>
  </tr>
  <tr>
    <td rowspan="4">Summscreen</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.308</td>
    <td>0.930</td>
    <td>0.732</td>
    <td>0.557</td>
    <td>0.414</td>
    <td>0.514</td>
    <td>68.971</td>
    <td>7.733</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.314</td>
    <td>0.364</td>
    <td>0.974</td>
    <td>0.778</td>
    <td>0.511</td>
    <td>0.345</td>
    <td>0.449</td>
    <td>61.149</td>
    <td>6.667</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.417</td>
    <td>0.436</td>
    <td>0.984</td>
    <td>0.849</td>
    <td>0.505</td>
    <td>0.281</td>
    <td>0.437</td>
    <td>49.294</td>
    <td>6.413</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.402</td>
    <td>0.501</td>
    <td>0.987</td>
    <td>0.855</td>
    <td>0.506</td>
    <td>0.295</td>
    <td>0.428</td>
    <td>62.788</td>
    <td>6.067</td>
  </tr>
  <tr>
    <td rowspan="4">Seahorse</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.271</td>
    <td>0.952</td>
    <td>0.811</td>
    <td>0.651</td>
    <td>0.518</td>
    <td>0.576</td>
    <td>60.999</td>
    <td>7.067</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.353</td>
    <td>0.353</td>
    <td>0.977</td>
    <td>0.786</td>
    <td>0.632</td>
    <td>0.542</td>
    <td>0.517</td>
    <td>72.321</td>
    <td>7.720</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.417</td>
    <td>0.491</td>
    <td>0.963</td>
    <td>0.831</td>
    <td>0.660</td>
    <td>0.480</td>
    <td>0.489</td>
    <td>60.241</td>
    <td>6.440</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.402</td>
    <td>0.474</td>
    <td>0.968</td>
    <td>0.839</td>
    <td>0.642</td>
    <td>0.476</td>
    <td>0.490</td>
    <td>60.415</td>
    <td>7.080</td>
  </tr>
</table>

Table 14: Performance on the 10% summarization task per dataset
<table>
  <tr>
    <th rowspan="2">DATASET</th>
    <th rowspan="2">Model</th>
    <th colspan="10">LCFO.20%</th>
  </tr>
  <tr>
    <th>R-L(↑)</th>
    <th>REP-3(↓)</th>
    <th>CoLA↑</th>
    <th>COH-2↑</th>
    <th>SH-4↑</th>
    <th>SH-5↑</th>
    <th>AVG↑</th>
    <th>HE↑</th>
    <th>HUM↑</th>
  </tr>
  <tr>
    <td rowspan="3">LexGLUE</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.455</td>
    <td>0.940</td>
    <td>0.842</td>
    <td>0.688</td>
    <td>0.417</td>
    <td>0.559</td>
    <td>65.036</td>
    <td>7.800</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.516</td>
    <td>0.5822</td>
    <td>0.9446</td>
    <td>0.5874</td>
    <td>0.6286</td>
    <td>0.4820</td>
    <td>0.4121</td>
    <td>8.093</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.507</td>
    <td>0.702</td>
    <td>0.927</td>
    <td>0.860</td>
    <td>0.632</td>
    <td>0.369</td>
    <td>0.417</td>
    <td>54.710</td>
    <td>7.027</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.501</td>
    <td>0.824</td>
    <td>0.943</td>
    <td>0.882</td>
    <td>0.637</td>
    <td>0.410</td>
    <td>0.409</td>
    <td>49.870</td>
    <td>6.973</td>
  </tr>
  <tr>
    <td rowspan="3">BookSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.344</td>
    <td>0.918</td>
    <td>0.766</td>
    <td>0.621</td>
    <td>0.349</td>
    <td>0.517</td>
    <td>89.376</td>
    <td>7.605</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.355</td>
    <td>0.385</td>
    <td>0.975</td>
    <td>0.831</td>
    <td>0.573</td>
    <td>0.441</td>
    <td>0.487</td>
    <td>95.169</td>
    <td>8.452</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.455</td>
    <td>0.544</td>
    <td>0.936</td>
    <td>0.842</td>
    <td>0.511</td>
    <td>0.314</td>
    <td>0.416</td>
    <td>69.068</td>
    <td>6.296</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.453</td>
    <td>0.634</td>
    <td>0.971</td>
    <td>0.842</td>
    <td>0.550</td>
    <td>0.394</td>
    <td>0.425</td>
    <td>76.472</td>
    <td>6.457</td>
  </tr>
  <tr>
    <td rowspan="3">SQuALITY</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.395</td>
    <td>0.919</td>
    <td>0.782</td>
    <td>0.565</td>
    <td>0.339</td>
    <td>0.505</td>
    <td>61.257</td>
    <td>5.800</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.382</td>
    <td>0.425</td>
    <td>0.969</td>
    <td>0.774</td>
    <td>0.518</td>
    <td>0.328</td>
    <td>0.433</td>
    <td>79.698</td>
    <td>6.720</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.412</td>
    <td>0.498</td>
    <td>0.963</td>
    <td>0.797</td>
    <td>0.454</td>
    <td>0.205</td>
    <td>0.384</td>
    <td>41.584</td>
    <td>4.027</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.426</td>
    <td>0.601</td>
    <td>0.979</td>
    <td>0.835</td>
    <td>0.469</td>
    <td>0.233</td>
    <td>0.383</td>
    <td>47.011</td>
    <td>5.587</td>
  </tr>
  <tr>
    <td rowspan="3">FacetSum</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.415</td>
    <td>0.940</td>
    <td>0.829</td>
    <td>0.745</td>
    <td>0.490</td>
    <td>0.584</td>
    <td>72.317</td>
    <td>8.147</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.477</td>
    <td>0.483</td>
    <td>0.953</td>
    <td>0.685</td>
    <td>0.659</td>
    <td>0.526</td>
    <td>0.468</td>
    <td>80.937</td>
    <td>7.711</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.474</td>
    <td>0.622</td>
    <td>0.944</td>
    <td>0.899</td>
    <td>0.705</td>
    <td>0.507</td>
    <td>0.487</td>
    <td>55.197</td>
    <td>6.501</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.456</td>
    <td>0.565</td>
    <td>0.952</td>
    <td>0.894</td>
    <td>0.698</td>
    <td>0.471</td>
    <td>0.490</td>
    <td>46.081</td>
    <td>6.813</td>
  </tr>
  <tr>
    <td rowspan="3">JRC-Acquis</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.435</td>
    <td>0.971</td>
    <td>0.860</td>
    <td>0.713</td>
    <td>0.566</td>
    <td>0.605</td>
    <td>72.317</td>
    <td>7.902</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.513</td>
    <td>0.566</td>
    <td>0.952</td>
    <td>0.551</td>
    <td>0.685</td>
    <td>0.578</td>
    <td>0.440</td>
    <td>75.351</td>
    <td>7.681</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.493</td>
    <td>0.792</td>
    <td>0.854</td>
    <td>0.884</td>
    <td>0.648</td>
    <td>0.422</td>
    <td>0.403</td>
    <td>38.876</td>
    <td>6.653</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.490</td>
    <td>0.788</td>
    <td>0.929</td>
    <td>0.882</td>
    <td>0.633</td>
    <td>0.446</td>
    <td>0.420</td>
    <td>49.870</td>
    <td>6.833</td>
  </tr>
  <tr>
    <td rowspan="3">MultiUN</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.422</td>
    <td>0.942</td>
    <td>0.875</td>
    <td>0.605</td>
    <td>0.317</td>
    <td>0.531</td>
    <td>64.540</td>
    <td>8.139</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.484</td>
    <td>0.604</td>
    <td>0.954</td>
    <td>0.623</td>
    <td>0.615</td>
    <td>0.482</td>
    <td>0.414</td>
    <td>72.007</td>
    <td>7.917</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.483</td>
    <td>0.654</td>
    <td>0.907</td>
    <td>0.918</td>
    <td>0.625</td>
    <td>0.289</td>
    <td>0.417</td>
    <td>37.311</td>
    <td>6.694</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.512</td>
    <td>0.665</td>
    <td>0.925</td>
    <td>0.908</td>
    <td>0.603</td>
    <td>0.326</td>
    <td>0.419</td>
    <td>45.155</td>
    <td>7.028</td>
  </tr>
  <tr>
    <td rowspan="3">Wikipedia</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.355</td>
    <td>0.967</td>
    <td>0.817</td>
    <td>0.738</td>
    <td>0.333</td>
    <td>0.557</td>
    <td>81.923</td>
    <td>7.653</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.471</td>
    <td>0.516</td>
    <td>0.960</td>
    <td>0.596</td>
    <td>0.687</td>
    <td>0.432</td>
    <td>0.432</td>
    <td>76.772</td>
    <td>7.609</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.467</td>
    <td>0.779</td>
    <td>0.877</td>
    <td>0.849</td>
    <td>0.638</td>
    <td>0.300</td>
    <td>0.377</td>
    <td>55.349</td>
    <td>6.493</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.476</td>
    <td>0.701</td>
    <td>0.940</td>
    <td>0.786</td>
    <td>0.606</td>
    <td>0.257</td>
    <td>0.378</td>
    <td>51.110</td>
    <td>6.014</td>
  </tr>
  <tr>
    <td rowspan="3">GovReport</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.397</td>
    <td>0.954</td>
    <td>0.823</td>
    <td>0.712</td>
    <td>0.405</td>
    <td>0.563</td>
    <td>60.368</td>
    <td>8.027</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.488</td>
    <td>0.521</td>
    <td>0.968</td>
    <td>0.605</td>
    <td>0.684</td>
    <td>0.521</td>
    <td>0.451</td>
    <td>76.887</td>
    <td>7.680</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.489</td>
    <td>0.638</td>
    <td>0.916</td>
    <td>0.872</td>
    <td>0.634</td>
    <td>0.425</td>
    <td>0.442</td>
    <td>43.421</td>
    <td>6.347</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.479</td>
    <td>0.531</td>
    <td>0.971</td>
    <td>0.881</td>
    <td>0.623</td>
    <td>0.384</td>
    <td>0.466</td>
    <td>40.544</td>
    <td>7.093</td>
  </tr>
  <tr>
    <td rowspan="3">Summscreen</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.395</td>
    <td>0.939</td>
    <td>0.739</td>
    <td>0.552</td>
    <td>0.414</td>
    <td>0.513</td>
    <td>63.691</td>
    <td>8.373</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.347</td>
    <td>0.443</td>
    <td>0.969</td>
    <td>0.756</td>
    <td>0.503</td>
    <td>0.346</td>
    <td>0.426</td>
    <td>60.306</td>
    <td>6.200</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.432</td>
    <td>0.487</td>
    <td>0.979</td>
    <td>0.845</td>
    <td>0.502</td>
    <td>0.294</td>
    <td>0.426</td>
    <td>60.564</td>
    <td>6.627</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.432</td>
    <td>0.519</td>
    <td>0.987</td>
    <td>0.851</td>
    <td>0.522</td>
    <td>0.322</td>
    <td>0.433</td>
    <td>49.467</td>
    <td>6.413</td>
  </tr>
  <tr>
    <td rowspan="3">Seahorse</td>
    <td>Human</td>
    <td>n/a</td>
    <td>0.336</td>
    <td>0.964</td>
    <td>0.828</td>
    <td>0.673</td>
    <td>0.533</td>
    <td>0.586</td>
    <td>66.028</td>
    <td>7.907</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>0.415</td>
    <td>0.439</td>
    <td>0.970</td>
    <td>0.722</td>
    <td>0.607</td>
    <td>0.507</td>
    <td>0.474</td>
    <td>71.568</td>
    <td>8.173</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-70B</td>
    <td>0.454</td>
    <td>0.589</td>
    <td>0.957</td>
    <td>0.834</td>
    <td>0.615</td>
    <td>0.441</td>
    <td>0.452</td>
    <td>54.075</td>
    <td>6.600</td>
  </tr>
  <tr>
    <td>LLAMA 3.1-8B</td>
    <td>0.469</td>
    <td>0.642</td>
    <td>0.960</td>
    <td>0.847</td>
    <td>0.601</td>
    <td>0.456</td>
    <td>0.444</td>
    <td>54.142</td>
    <td>6.760</td>
  </tr>
</table>

Table 15: Performance on the 20% summarization task per dataset.

Reviewer paper summaries: This paper introduces the LCFO benchmark, a dataset and evaluation framework tailored for tasks involving long-context summarization and summary expansion. The dataset includes long documents (~5,000 words) from seven diverse domains, each paired with three levels of summaries (20%, 10%, 5%) and associated question-answer (QA) sets. The authors emphasize the importance of gradual summarization and structured evaluations, providing both human and automatic metric-based evaluations of state-of-the-art LLMs.
Summary of strengths: - This work addresses the lack of datasets designed specifically for long-context input and output tasks, filling an important gap in the field.
- LCFO provides high-quality references and evaluations comparing with other benchmark datasets.
- The inclusion of both summarization and summary expansion tasks is well-conceived, providing insights into model capabilities across related but distinct tasks.
- The results clearly demonstrate the strengths and weaknesses of current LLMs.
Summary of weaknesses: - While the paper focuses on summarization and QA, it does not explore other potential applications of the dataset, such as training supervised models for coherence or attribution evaluation.
- The performance of other models, models in other cases (e.g. summary expansions beyond the 5x factor), and analysis of model errors are not explored in depth.
- LCFO relies largely on human annotation at all stages, which may cause subjective bias.
Comments, suggestions and typos: See weaknesses.
- As the dataset is the cornerstone of the paper, its release should be prioritized to foster community engagement and further research.
- Incorporating more sophisticated coherence metrics or exploring novel automatic evaluation techniques could strengthen the evaluation framework.
- Consider extending the framework to include tasks like hierarchical summarization or multi-document summarization.
Overall assessment scores: 3.5