ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
Authors: Uri Shaham
Maor Ivgi
Avia Efrat
Jonathan Berant
Omer Levy
Venue: EMNLP 2023 Findings (2023)
Keywords: Long lexts
summarization
question answering
benchmark
zero-shot
TL;DR: We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, and conduct a comprehensive evaluation of both open-source and closed large language models on it.

Abstract: We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the
highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation
tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their
ideas on the live ZeroSCROLLS leaderboard.

Large language models (LLMs) have been improv- ing incredible pace, solving problems that seemed of reach, without any task-specific train- ing examples (Wei et al., 2022a; Ouyang et al., ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
Uri Shaham
Maor Ivgi"
Avia Efrat Jonathan Berant"
Omer Levy'M
The Blavatnik School of Computer Science, Tel Aviv University
A Meta Al
Abstract
Topp
14.3
We introduce ZeroSCROLLS, a zero-shot bench- mark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLSS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregat- ing the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation both open-source and closed large language models, finding that Claude oulper- forms ChatGPT, and that GPT-4 achieves the highest average score. However, there still room for improvement multiple open chal- lenges in ZeroSCROLLS. such as aggregation tasks, where models struggle pass the naive baseline. As the state of the art is moving larget, we invite researchers evaluate their ideas the live ZeroSCROLLS leaderboard."
Naive
19.6
Flan-TS (8K)
Flan-UL2 (9K)
30
DaVins0003 (4k)
33.7
ChatGPT (4k)
34.0
Claude (8k)
39.1
GPT-4 (8k)
41.7
10
20
30
40
50
Figure 1: ZeroSCROLLS
the average perfor- mance of state-of-the-art language models across 10 long text understanding tasks. The maximal amount of tokens each model can process is given in parentheses.
enables fine-tuning, adding four additional tasks: query-based summarization. multi-hop question an- swering, sentiment aggregation, and sorting book chapter summaries. We specifically design the latter two tasks examine model's ability aggregate and compare information across long sequences, while keeping evaluation simple and accurate. ZeroSCROLLS is designed to test zero- shot capabilities, and contains test sets with simple natural prompts and private gold references, small validation sets, and no train data. It has live leader- board to enable transparent and dynamic progress. Figure shows the state of the leaderboard based on our experiments, and Figure shows a per-task breakdown selected subset of models.
1 Introduction
2022: OpenAI, 2023). As commercial LLMs are adopted worldwide, becomes clear that they must also operate successfully over long sequences, such conversation histories or scientific documents. However, current LLM benchmarks that do evalu- ate models in zero-shot setting, such HELM (Liang al., 2022) and BigBench (Srivastava et al., 2022), mostly focus on short sequences; BigBench, for example, has an average of 77 words per input. To fill this gap, we introduce Z Zero- Shot CompaRison Over Long Language Sequences, benchmark for zero-shot long text reasoning over natural language, and conduct thorough investi- gation of state-of-the-an LLMs.
We use this new testbed to perform extensive evaluation and analysis across state of-te-of-theaar open and closed models. On question answering tasks, find that zero-shot LLMs bridge the gap with task-specific fine-tuned models; GPT-4 sets a new state of the art on the challenging QuALITY task (Pang et al., 2022), almost reaching human perfor- mance. contrast, LLMs generally struggle to
ZeroSCROLLS extends SCROLLS (Shaham et al., 2022), long text understanding benchmark that
1https: h/wp:://wwwwww.. zero. scrolls-bence www.bbnnhhhhccccom com/

---

R008 Naive
Flan-UL2 (8k)
Claude (8k)
GPT-4 (8k)
CoLTS (16k) finetuned
41.0
90
84
N922
61.85
60.6
80
75.4
90.0
47.4
70
450
28
22
24
21.0
22/8
62
00.7
47.0
51.3
365
60
17.3
41.1
16:
1338
11.5
10.5
30
26.6
2013
16.3
67
87
5.7
6
10
GovReport
SummScrrennFD
CMMSum
SQ
Queper
NarstiveaA
QUALITY
MusiQue
SpacaDigesl
BookSumSort
Figure 2: Per task scores of various LLMs and other baselines. In parentheses: the maximum number of tokens.
obtain such high scores for summarization tasks without a training set from which learn the nu- ances and artifacts of each dataset, even though GPT-4 does approach the fine-tuned state of the art on two of three datasets. We also observe that two of our new tasks, sentiment aggregation and sorting book chapter summaries, prove exception- ally challenging for all LLMs, with only GPT-4 surpassing the naive baseline in each task. Our code is available online.?
dedicated long sequence models pretrained from scratch (Guo al., 2022; Ainslie al., 2023).
All the aforementioned methods eventually fine- tune specialized model for every single task in SCROLLSS, setting that remains important for many applications. However, in the modern era of general purpose, zero-shot reasoning LLMs, a new evaluation setup is required, where this depen- dence on -ask-specific fine-tuning is alleviated.
When analyzing GPT-4 responses, we often find correct answers that do not match the requested format; e.g. producing a full sentence when asked to answer in single phrase. This problem is not unique to GPT-4, as different models may deviate from the specified format in different tasks. While ZEROSCROL.IS is primarily aimed facilitating re- search in understanding long texts, we encourage the community to use this benchmark to advance research in instruction understanding, prompt engi- neering, and evaluation of generated texts as well.
The ZeroSCROLLS Benchmark
ZeroSCROLLS is zero-shot benchmark containing test sets of ten natural language tasks, each one requiring reasoning
different type of long text. To ensure affordability, limit every task maximum of 500 examples.
3.1 Tasks
We describe the different ZeroSCROLLS datasets, six of which we adapt from Shaham et al. (2022), and four new tasks. Table provides overview.
2 Background: SCROLLS
ScRoLLs (Shaham et al., 2022) was introduced as a long text understanding benchmark. Its datasets were curated, cleaned, and reformatted to a single input-output format allowing for easy and fast us- age, with every example containing a single long document, such scientific paper or book. Since its launch, SCROLLS has facilitated signifi- cant progress, including new pretraining objectives (Tay et al., 2023), adaptations of short-text models to long sequences (Phang et al., 2022; Xiong et al., 2022: Ivgi et al., 2023; Bertsch et al., 2023), and
3.1.1 Summarization
We adopt the three summarization datasets from SCROLLS (GovReport, SummScreenFD, and QM- Sum), and add fourth (SQuALITY), GovReport and SummScreenFD full-document summa- rization tasks, while QMSum and SQuALITY are query-focused.
GovReport (Huang al., 2021) contains long reports by the Congressional Research Service and the U.S. Government Accountability Offices, with their expert written summaries.

---

Dataset
Task
Domain
Metric
Avg #Words
#Examples
GovReport (Huang al., 2021)
Summarization Summarization
Government
ROUGE
7,273
500
SummScreenFD (Chen al., 2022)
TV
ROUGE
5,663
337
QMSum (Zhong al., 2021)
QB-Summ
Meetings
ROUGE
10,839
281
SQuALITY (Wang
2022)
QB-Summ
Literature
ROUGE
4,971
260
Qasper (Dasigi et al., 2021)
QA
Science
F1
3,531
500
NarrativeQA (Kočiský et al., 2018)
QA
Literature, Film
F1
49,384
500
QuALITY (Pang et al., 2022)
MC-QA
Literature, Misc
Accuracy
4,248
500
MuSiQue (Trivedi al., 2022)
QA
Wikipedia
F1
1,749
500
SpaceDigest (New)
Aggregation
Reviews
5,481
500
BookSumSort (New)
Aggregation
Literature
Citx
6,840
Table I: An overview of the data statisties in ZeroSCROLLS. QB-Summ means query-based summarization, MC-QA abbreviates multiple-choiee question answering. ES refers to exponential similarity and Cia. refers concordance index. SpaceDigest data is from on the Space dataset (Angelidis et al., 2021) and BookSumSort data is from the BookSum dataset (Kryscinski et al., 2022).
SummScreenFD (Chen et al., 2022) contains episode scripts from TV shows with commu- nity contributed recaps that were collected from Wikipedia and TVMaze as their summaries.
from Wikipedia, and each question was answered by more annotators.
QuALITY (Pang al., 2022) contains stories and articles from Project Gutenberg, the Open American National Corpus, and more. Each in- stance contains a story and multiple choice ques- tion; question writers were guided to write ques- tions that require reading large portions of the story to answer correctly.
QMSum (Zhong et al., 2021) is a query-based summarization dataset over meetings transcripts. It contains academic meetings, industrial product meetings, and Welsh and Canadian parliament tran- scripts. Alongside the meeting transcript, each instance contains query, which aims to focus the summary on a particular topic.
MuSiQue (Trivedi et al., 2022) is multi-hop hop question answering dataset, where the inputs are 20 Wikipedia paragraphs and question that requires multiple hops between different paragraphs. In the original dataset, each question also has an uman- swerable twin question, where the correct answer is not present in the paragraphs. We randomly sample 100 unanswerable and 400 answerable questions for ZeroSCROLLS.
(Wang et al., 2022) is question- focused summarization dataset, where given story from Project Gutenberg, the task is produce summary of the story or aspects of it based on a guiding question. The questions and summaries are original and crowdsourced; experienced writers were told to design questions that require reading significant parts of the story to answer correctly.
SQuALITY
3.1.3 Aggregation
3.1.2 Question Answering
We create two new tasks that, by construction, quire contextualizing and aggregating information from different parts of the input. Despite the inher- ent complexity required to solve these tasks, we design their evaluation to simple and accurate.
We adopt the three question answering datasets from SCROLLS (Qasper, NarrativeQA, and QuAL- ITY), and add MuSiQue, which focuses on multi- hop question answering.
Qasper (Dasigi et al., 2021) contains NLP papers from the Semantic Scholar Open Research Corpus (S2ORC) (Lo et al., 2020). NLP practitioners pro- vided questions based on the abstracts, and another set of practitioners answered given the articles.
SpaceDigest is a new sentiment aggregation task. Given 50 hotel reviews (without their ratings) from the Space dataset (Angelidis et al., 2021), the task is to determine the percentage of positive reviews. We create example (50 reviews) per hotel from the 500 most rated hotels in the original dataset, keeping only strictly positive (rating ative (rating or 1) reviews, discarding ones with an ambivalent rating of 3. To verify that humans perform this task well, we gave 5 human annotators
NarrativeQA (Kočiský al., 2018) contains questions and answers over books from Project Gutenberg and movie scripts from various web- sites. To create questions and answers, annotators were provided summaries of the books and movies
4) neg-

---

shortened version of the examples (containing 10 reviews per example) and asked them write the percentage of positive reviews. Each annotator was assigned 10 examples (100 reviews per annotator, 500 overall). The annotators aggregated their in- dividual predictions perfectly, and had a total of 8 single-review classification errors out of the 500 reviews seen (~98.44 4%% accuracy).
You given meeting transcript and query containing question instruction. Answer the query one more sentences.
Transcript:
User Interface: That's the same uh the top of it uh with the the round uh button.
Industrial Designer: Like this onc.
User Interface: But uh we don't 't uh we don't 't uh disfmarker we do think it's um well disfmarker what if with ease of use. w which prefers the disfmarker which the the cus- tomer of the user prefers.
BookSumSort is task based the Book- Sum dataset (Kryscinski et al., 2022), which con- tains summaries of chapters (or parts) of novels, plays, and long poems from various sources. Given a shuffled list of chapter summaries, the task is to reorder them according to the original order of sum- maries in BookSum. We create the task by man- ually selecting the summaries of 125 books from BookSum, retaining only high-quality instances. We manually edit each summary by removing in- troductions, prefaces, overviews, and so forth, as well as any other information that may indicate the exact position of a summary; for example, "Chap- ter& 8 begins with Jane describing..." " is replaced with "This Chapter begins with Jane describing... and "As the play opens, Hippolytus announces..." becomes "Hippolyyus announces..." ". Each list of summaries contains between 3 and 86 chapter sum- maries, with median of 15 and an average of 18.8 chapters per instance. We select 4 random permutations of each list to create 500 instances.
Industrial Designer: It's important. Uh think th this is device which which has learning curve... [TTh rest of the transcript is omitted]
Query:
What did the group discuss ahout production costs of the product?
Answer:
Figure 3: An example input in ZeroSCROLLS, taken from the QMSum dataset. The meeting transcript and the question are in black, and the ZeroSCROLLS prompt is in blue. In copper is a string we append to the trimmed context when the model's context window too short to contain the entire input.
the query in one or more sentences." for QMSum). When the total input size too long for a model's s context window, we trim the context and append a string explicitly stating that the rest of the context is trimmed, to inform the model that it cannot see the entire context. We then concatenate the context with a header describing what kind of context it is, e.g. "Report:". "Reviews:". etc. For tasks that have queries, we append the question or query with appropriate header. The prompt ends with a header indicating the response type (e.g. "Answer:" or "Summary:").
3.2 Prompting
ZeroSCROLLS tests the ability to reason over long texts without any explicit training examples (zero- shot). We thus complement each data instance with an instruction that defines both the task and the desired output format (Efrat and Levy, 2020), without in-context demonstrations. While we in- vest effort in designing the canonical prompts for ZerOSCROLLS, the benchmark is open to further zero-shot prompt engineering (Radford et al., 2019; Schick and Schütze, 202 such as prompts that encourage tf reasoning (Wei ct al., 2022b). Table 5 contains the prompts for the sum- marization tasks and Table 6 contains prompts for question answering and agregation tasks.
A Accommodations for ChatBots Chat LLMs, such as ChatGPT and Claude, are designed in- teract with humans through a chat interface. We therefore adapt our canonical prompts to accom- modate these models. Specifically, omit the re- sponse header (e.g. "Summary:" or "Answer:") it is clear, in dialogue, that the input sequence has ended. In addition, any explanation." to the instructions of question answering and aggregation tasks. For Claude, we wrap each prompt with "Human:" and "Assistant:" dialogue indicators, and for the question answer- ing and aggregation tasks also add the instruction
Prompt Structure Figure 3 illustrates exam- ple from the benchmark. We manually craft prompt for each dataset, following generic tem- plate composed of instruction, context, query, and response. The instruction describes the task, and ends with the desired output format (e.g. "Answer
append "Do not provide

---

"please highlight your final answer with <(re- sponse__yppe typspos//resoosse http///rsoossstppppp>>>>>> tags"- rec- ommended by Anthropic's documentation."
Maximum Length
Model
Params
Open/Closed
TÖpp
11B
8.192
Open
Flan-T5
11B
8,192
Open
3.3 Automatic Evaluation
Flan-U1.2
20B
8,192
Open
DaVinci003
4,096
Closed
ZeroSCROLLSS evaluation is fully automatic. Given a model's response to every test instance, we apply per-task automatic evaluation metrics. These are then averaged across tasks to produce the model's ZeroSCROLLS score. For existing datasets, we follow Shaham et al. (2022) and use the metrics provided by each dataset's authors. For our newly proposed tasks (SpaceDigest and BookSumSort), we use two new automatic metries.
ChatGPT
4,096
Closed
Claude
8,192
Closed
GPT-4
8,192
Closed
Table 2: State the art LLMs we evaluate. Exact param-
eter counts of closed models not publicly available.
Concordance Index (BookSumSort) Assuming that the output is permutation of the given chapter summary IDs, we measure the amount of chapter summary pairs that are in the right order, divided by the total number of pairs (2) The average random permutation scores 50% on this metric.
ROUGE (GovReport, SunmSereenFD, QMSum, SQuALITY) ROUGE (Lin, 2004) measures ngram overlap between generated and reference sum- maries. For each instances, we combine ROUGE-I, ROUGE-2, and ROUGE-L into single score by computing their geometric mean. For SQuALITY, where there are multiple references, we take the maximal value of each ROUGE type before com- puting the geometric mean.
Evaluating State-of-the-Art LLMs
Using ZeroSCROLLS we conduct, to the best of our knowledge, the first systematic LLMs zero-shot performance comparison over tasks that require long text understanding.
F1 (Qasper, NarrativeQA, MuSiQue) Fl com- putes unigram overlap between generated and ref- erence answers, after normalizing white-spaces, lowercasing, omitting stopwords and punctuation (Rajpurkar et al., 2016), and transliterating any Uni- code to ASCII characters. For Qasper and Nar- rativeQA, where there are multiple reference an- swers, we take the maximal Fl score per instance.
4.1 Models
We evaluate both open-source models and closed products available via APIs. We apply greedy de- coding to all models, and leave further research into other decoding strategies to future work. Table 2 shows the selection of models we evaluate.
Open Source Models We experiment with Flan- T5-xxl (Wei al., 2022a) and Flan-UL,2 the instruction-tuned versions of T5 (Raffel et al., 2020) and UL. (Tay et al., 2023), as well as TOpp (Sanh et al.. 2022), an LM-adapted (Lester et al., 2021) version of T5 that was finetuned on vari- ous NLP tasks for zero shot generalization. For all open- source models length of 8,192 tokens (larger contexts were un- stable). We also experiment with shorter context lengths and smaller variants of Flan-T5.
Accuracy (QuALITY) For multiple choice ques- tions, we compare the predicted letter (A, B, C, or D) to the reference. We use the first valid option letter surrounded by word boundaries.
Exponential Similarity (SpaceDigest) Assum- ing that the output is percentage," we compute the exponential similarity between the gold reference percentage p and the predicted scalar p:
maximum input
ES(p. p) d clp A
We use = 2 and c = 10, which means that, intuitively, the score gets cut by half for every 10 point deviation from the correct answer.
Closed Models (Products) Using product APIs. we evaluate Claude v1.3 from Anthropic, 6 and Da Vinci0033 ChatGPT v0301. and GPT-4
"https://console anthropic. cww. com//docs/prooo propptttttpppppdd esign/clas httm//laasffccattin
'If the output is not permutation, we score 0%. We dis- card all characters but digits, commas, and white-spaces from the output string to eliminate prefixes such "Order:" htwp://wwwrrrooo..oooooeeennnnne hutps://paafomm.qeeai..oomo//// http:///iccoomooo///m//eeeel/eee researchers
*If the output is not percentage. we score 0%. We parse the first appearance of a percentage; e.g. for the output "Out of 50 reviews, 20 are positive and 30 are negative, so 40% of the neviews are positive 60% are negative." we automatically parse 40% as the answer.
"htts://chat.opemai.com

---

GvRp
SSFD
QMsm Rgco
SQAL Rgzo
Qspr F1
Nrty
QALT AC
MuSQ
SpDg
BKSS
Model
Tokens
Avg
Ryco
Rgco
Cax
Baselines
Naive
22.6
6.7
6.7
10.5
6.1
2.1
26.6
20.0
45.0
50.0
19.6
Human
23.6
67.7
58.2
93.5
74.8
93.3
Open Source Models
TOpp
8192
7.1
9.6
7.2
3.9
25.0
18.7
21.4
35.3
15.2
0.0
14.3
Flan-T5
8192
17.6
7.8
11.0
8.0
48.3
19.3
75.2
46.8
48.7
16.4
29.9
Flan-UL.2
8192
16.1
11.5
13.6
5.7
56.9
25.5
75.6
51.3
36.0
14.0
30.6
Closed Models
Da Vinci003
4096
21.7
16.1
16,9
22.0
52.7
24.6
69.0
33.5
31.3
49.5
33.7
ChatGT
4096
21.3
16.1
15.6
20.4
49.3
25.1
66.6
27.1
49.1
49.8
34.0
Claude
8000
24.2
16.1
14.6
21.0
52.3
32.6
84.8
36.1
61.6
47 4
39.1
GPT4
8192
26.3
17.3
18.5
22.6
50.7
27.6
89.2
41.1
62.8
60.5
41.7
Fine-tuned Models
CoLTS
16384
41.0
20.0
22.5
53.1
31.0
47.0
Table 3: The ZeToSCROLLS leaderboard, the time of writing. The dataset abbreviations stand for: GovReport, SummScreenFD, QMSum, SQuALITY, Qasper, NarrativeQA, QuALITY, MuSiQue, SpaceDigest, BookSumSort.
v0314 (OpenAI, 2023) from OpenAL. The max- imal context length of these models includes both input and output.
con-answerable sets from Trivedi et al. (2022). For SpaceDigest,
human annotations (Section 3.1.3) to estimate exponential similarity over 50 reviews.
Task-Specifi Models To compare general- purpose LLMs (zero-shot) to task-speeific mod- els (fine-tuned), use predictions by CoLTS-xl (Ainslie al., 2023), a transformer allocating more resources to important tokens, with a maximum input length of 16,384 tokens and is the current state of the art on SCROLLS.
4.2 Main Results
Table 3 shows the results for every model every ZeroSCROLLS task, along with the average. The overall best model is GPT- with average score of 41.7, and its closest competitor Claude with 39.1. both significantly higher than the other mod- cls. We discuss the results per task category.
Naive Baselines We implement simple baselines for all tasks. For GovReport, SummScreenFD, QM- Sum, SQuALITY and NarrativeQA, we select ran- dom spans from the input document of 500, 200, 50, 120 and words respectively. For Qasper, randomly decide whether choices ("Yes", "No", "Unanswerable") or choose a random span of 15 words. For MuSiQue, use "Unanswerable" for every instance. For QuAL- ITY, we randomly select an option from A, B, C, or D. For SigceDigest we always use 50%, and for BookSumSort we use the trivial permutation "1, n.....""
Summarization
There is a clear trend where the open-source models lag behind product-grade LLMs, and that GPT-4 reaches the highest ROUGE scores on four datasets. However, zero-shot LLMs struggle to compete with models fine-tuned per dataset (CoLT5) on those tasks, with some gap on SummScreenF and QMSum, and dramatic difference GovReport (41.0 compared to 26.3). In SQuALITY, GPT-4 is only one point away from the lower bound human performance.
of its fixed
Human Performance We provide human perfor- mance figures for 6 of the 10 tasks. For SQuALITY, Wang et al. (2022) estimate human performance by comparing reference against the other three. Similarly, for Qasper and NarrativeQA, we caleu- late inter-annotator Fl on the ZeroSCROLLS sub- sets. We use the human scores reported by Pang et al. (2022) on the full QuALITY test set, while for MuSiQue, we combine statistics on answerable and
Question Answering We see a different trend in question answering. GPT-4 achieves the best result on only one dataset, QuALITY, where it scores 89.2, close to human performance of 93.5. Flan- UL2 sets the high scores for Qasper and MuSiQue, while Claude has the best Fl on NarrativeQA. 5 points more than GPT-4. Our analysis Section 5 reveals that GPT-4 does not conform to the required answer format, resulting in a lower score.

---

GvRp Ryso
SSFD Rgco
QMsm Rgco
SQAL
Qspr F1
Nrtv
QALT
MuSQ
SpDg
BKSS
Model
Tokens
Avg
Rgeo
AC
Cat:
Flan-T5 Across
Model
Sizes
Flan-T5-s
8192
7.6
4.2
8.3
3.8
18.5
11.6
34.6
21.0
0.0
0.0
11.0
Flan-T5-b
8192
5.4
5.1
5.6
16.5
48.4
26.9
13.2
Flan-T5-1
8192
6.9
6.8
9.7
5.7
33.6
20.1
62.4
33.1
48.0
0.3
22.7
Flan-T5-xl
8192
15.2
10.2
6.6
46.6
21.6
69.6
42.8
32.8
25.5
Flan-T5-xxl
8192
17.6
8.0
48.3
75.2
46.8
29.9
Flan-75-xxd
Across Imput Lengths
Flan-T5-xxl
512
10.0
7.9
10.4
6.1
15.3
17.6
48.2
26.0
20.8
9.0
17.1
Flan-T5-xxl
1024
12.1
9.4
10.1
6.3
25.5
18.9
53.2
30.3
28.7
13.4
20.8
Flan-T5-xxl
2048
14.0
10.0
11.0
6.8
35.7
20.9
59.8
40.6
35.0
14.7
24.9
Flan-T5-xxl
4096
17.3
11.8
7.4
46.5
22.2
70.8
46.8
44.1
15.1
29.1
Flan-T5-xxl
8192
17.6
7.8
11.0
8.0
48.3
19.3
75.2
46.8
48.7
16.4
29.9
Claude Across Input Lengths
Claude
4096
23.0
15.0
14.3
20.2
47.7
31.7
76.8
35.8
61.1
37.6
36.3
Claude
8000
24.2
16.1
14.6
21.0
52.3
32.6
84.8
36,1
61.6
47.4
39.1
Table 4: Performance of Flan-T5 model sizes, and Flan-T5 and Claude across input lengths.
Aggregation Our new SpaceDigest and Book- SumSort datasets enrich ZeroSCROLLS with chal- lenges that explicitly require aggregating informa- tion across the sequence. Results indicate that both tasks are difficult for current LLMs. Performance figures for SpaceDigest show that even though sen- timent analysis, counting, and divisions are all "casy" tasks for contemporary models, their combi- nation can be quite challenging, only Claude and GPT-4 significantly outperform the naive baseline. The situation is even more dire in BookSumSort, where only GPT-4 outperforms the naive baseline.
69
39
Claude
GPT4
Flan-UL2
GPT4
Flan-UL2
GPT4
NamafveQA
Casper
Figure Human evaluation (accuracy) over 100 ques- tions from NarrativeQA, Qasper, and MuSiQue, compar- ing GPT-4 to the highest scoring model each dataset.
4.3 Impact of Model Size and Input Length
estingly, Flan-T5 also achieves higher scores on longer inputs in many cases, despite being trained much shorter sequences.
We now discuss the effects of increasing model size (parameters) and context length (tokens). As one may expect, both dimensions improve performance on ZeroSCROLLS, suggesting that the benchmark does indeed necessitate complex reasoning over long sequences.
Analysis
While GPT4 has the highest score on the ZeroSCROLLS leaderboard, we find it surprising that other models score higher on number of ques- tion answering tasks. We analyze model gener- ations and observe that GPT-4 responses do not match the desired output format (despite explicit instructions in the prompt), which results in penal- ization by the automatic metries. Further analysis reveals that format discrepancy is a phenomenon that occurs across different LLMs and tasks, and is not unique to GPT-4 and question answering.
Model Size The upper section of Table shows results of Flan-T5 of various sizes, ranging from (60M parameters) to XXL. (1IB parameters). As expected, increasing model size drives performance upwards across almost all tasks.
Input Length The middle and lower sections of Table 4 show the effect of increasing the maximum number of input tokens for T5 and Claude. In general, increasing the number of tokens helps the models preform the tasks better. Claude able to utilize the extra tokens more consistently, which results an almost point increment to its average score when going from 4k to 8k tokens. Inter-
Discrepancies in Question Answering We lyze the responses of GPT-4 and Claude for Narra- tiveQA (where Claude scores points higher), and the responses of GPT-4 and Flan-UL2 for Qasper and MuSiQue (where Flan-UL2 scores 6.2 and 10.2

---

points higher, respectively). Specifically, we sam- ple 100 instances from each dataset, and annotate whether the answer correct, ignoring formatting. fluency, or other factors. Figure shows that, in contrast to the FI scores, GPT-4 performs better than Claude and FLan-UL2 on NarrativeQA and Qasper, respectively, and that the gap between GPT- 4 and Flan-UL2 on MuSiQue is smaller in practice. From examining the generated texts, learn that GPT-4 consistently generates complete an- swers even though the prompt instructs otherwise (sce Section 3.2 and Appendix A). We further ana- lyze 200 random instances from NarrativeQA and check whether GPT-4 and Claude respond in the specified format, i.e. "using single phrase if pos- sible," regardless of whether the content is correct not. While Claude answers 191 questions in the correct format, GPT-4 does so for only 71 out of the 200 analyzed examples explaining why GPT-4 is penalized harder by the metric, despite being "correct" more often than Claude."
MuSiQue
10
SQUALITY
450
300
150
NarratieQA
Qasper
Format Discrepancy Figure 5 surveys the dis- tribution of output lengths across multiple tasks and models. In most cases, models generate out- puts that fall within the distribution of reference lengths, indicating that the format criteria provided in the prompts are sufficient. However, certain task-model combinations fall outside of the ref- erence distribution. While the NarrativeQA plot confirms our previous observation that GPT-4 gen- erates longer answers for this task, we find that format discrepancy is not unique to this dataset GPT-4, different models struggle to gener- ate texts in the correct format on different tasks; Claude generates long answers for QMSum, Flan- UL2 generates long summaries SummScreenFD. and all models generate short summaries for Gov- Report, which negatively impacts their scores.
0
QMSum
375
250
125
SummScreenFD
450
300
150
GovReport
1000
750
500
250
6 Conclusion
Flan-UL2 ChaiGPT Claude
GPT4
Refarance
We introduce ZeMOSCROLIS,, benchmark for zero- shot natural language understanding over long texts. ZeroSCroLLS enables systematic comparison of LLMs on tasks with naturally long input texts, and ones that require contextualizing and aggregating information from multiple documents. We evaluate
Figure 5: Distribution of the number of generated words.
open-source and production LLMs to find that GPT-4 and Claude are currently the best per- forming models, while open-source models such as Flan-UT.2 also prove powerful at long-context question answering tasks. ZeroSCROLLS remains an open challenge for LLM research, with our two
"Another interesting observation from analyzing Narra- tiveQA is that GPT-4 sometimes responds that it is unable to answer the question because the (trimmed) context does not contain the answer. It does so for 30 out of 200 cases, while Claude generates similar response for only 5, despite both models having similar context lengths (8k).

---

new aggregation tasks proving to be particularly difficult for contemporary LLMs.
8602-8615, Dublin, Ireland. Association for Compu- tational Linguistics.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers chored research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4599-4610, On- line. Association for Computational Linguisties.
7 Limitations
As language models improve, evaluating them presents a growing challenge given their ability to consistently generate coherent and reasonable text, which is harder to score, even with gold refer- ences at hand. Specifically in the zero-shot setting, where models must infer the output format from the prompt, ROUGE and F1 (ngram metrics) can assign low scores for semantically equivalent gen- erations, with different word choices or answer lengths. Additionally, to conduct fair evaluation, we use common prompt templates across models for every task, while model-specific prompts, as well as chain-of-thought prompting may improve model performance on this benchmark. Finally, the state of the art is moving target, and as write these lines new long-range models, alignment methods, decoding algorithms, and prompting tech- niques become available; we invite researchers to evaluate their ideas on the ZeroSCROLLS leader- board.
Avia Fraat and Omer Levy. 2020. The turking test: Can language models understand instructions?
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724- 736, Seattle, United States, Association for Compu- tational Linguistics.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics.
Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Efficient Lext Understanding with Sexrt-Text Models. Transactions of the Association for Compu- tational Linguistics, 11:284-299
Acknowledgements
This research is supported by the Yandex Initiative in Machine Learning. The benchmark is released by Tel Aviv University. All experiments were con- ducted by Tel Aviv University.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Asso- ciation,fr Computational Linguistics, 6:317-328.
References
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Associ- ation for Computational Linguistics: FMNLP 2022, pages 6536-65366 6558, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Joshua Ainslie, Tao Lei. Michiel de Jong, Santiago On- tañón, Siddharhaa Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Tee-Thorp, Yi Tay, Yun- Hsuan Sung, and Sumit Sanghai. 2023. Colt5: Faster long-range transformers with conditional computa- tion.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter- efficcieneei prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. Extractive opinion summarization in quantized trans- former spaces. Transactions of the Association for Computational Linguistics, 9:277-293.
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley, 2023. Unlimiformer: Long- range transformers with unlimited length input.
Percy Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan. Ce Zhang, Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for abstrac- tive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages

---

Wang, Keshav Santhanam, Laurel OIT, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie. Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Ieo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022. OpenReview.net.
Bhin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Kyle Lo. Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- national L Linguistics, pages 4969-4983 Online. Asso- ciation for Computational Linguisties.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan I Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Timo Schick and Hinrich Schüitze. 2021b. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 390- 402, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. SCROLLS: Standardized CompaRison over long lan- guage sequences. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 12007-1202) Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma. Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3356, Seattle, United States. Association for Computational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro. Aditya Gupta, and Adrià Garriga-Alonso et al. 2022. Beyond the imita- tion game: Quantifying and extrapolating the capa- bilitics of language models.
Jason Phang. Yao Zhao, and Peter J. Liu, 2022. Inves- tigating efficiently extending transformers for long input summarization.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):
Yi Tay, Mostafa Dehghami, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL22 Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 31:140:1-140:67
Harsh Trivedi. Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Ja- son Phang. and Samuel R. Bowman. 2022. SQuAL- ITY: Building a long-document summarization

---

dataset the hard way. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages I 1139-1159 156, Abu Dhabi, United Arab Emirates, Association for Computational Lin-
guistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Jason
Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. thain-of-thougee prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc.
Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen tau Yih. 2022. Adapt- ing pretrained text-to-text models for long text se- quences.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah. Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev, 2021. QMSum: A new benchmark for query- based multi multi-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association Computational Linguistics.
A Prompts
Table 5 shows ZeroSCROLLS prompts for summa- rization tasks, and Table shows our prompts for question answering and aggregation tasks. The prompts are designed to be simple, natural, and explicit. In braces are placeholders for the text of every example.

---

Task
Prompt
GovReport
You are given a report by govemment agency. Write a one-page summary of report.
Report:
(REPORT)
Summary:
SummSereen
You are given a script of episode. Summarize the episode paragraph.
Episode Script: (SCRIPT)
Summary:
QMSum
You given meeting transcript and query containing question instruction. Answer the query sentences.
Transcript:
(TRANSCRIPT)
Query: (QUERY)
Answer:
SQuALITY
You given a story and question. Answer the question in paragraph.
Story:
{STORY)
Question:
(QUESTION)
Answer:
Table 5: Summarization task prompts. For chat models (ChatGPT, Claude, and GPT-4), we and omit the response header, is less appropriate for dialogue.

---

Task
Prompt
Qasper
You given scientific article and question. the question concisely as you can, using single phrase or sentence if possible. question cannot answered based the information the article, write "unanswerable.. If the question yes/no question, answer "yes", "no", "unanswerable", Do provide any explanation.
Article:
(ARTICLE)
Question:
[QUESTION)
Answer:
NarrativeQa
You given a story, which can be either novel a movie script, and question. Answer question concisely as you can, using a single phrase if possible. not provide any explanation.
Story:
{STORY)
Question:
(QUESTION)
Answer:
QuALITY
You provided story and multiple-choice question with possible answers (marked by A, Choose the answer by writing corresponding letter (either C, or D). Do not provide any explanation.
Story: {STORY)
Question and Possible Answers: COUESTIONANNDOPTIONN
Answer:
MuSiQue
You are given several paragraphs from Wikipedia and question. Answer question as concisely as you can, using single phrase if possible. question cannot answered based the information the paragraphs, write "unanswerable" not provide any explanation.
Paragraphs:
[PARAGRAPHS]
Question:
(QUESTION)
Answer:
SpaceDigest
You are given a list reviews about a specific hotel. Each review is either positive or negative. What the percentage positive reviews (e.g. 60%, 34%, etc.)? Do not provide any explanation.
Reviews:
(REVIEWS)
Percentage Positive Reviews:
SookSumSort
You given (NUM_ SUMMARIES) summaries chapters or parts of novel, in shuffled order, where each summary denoted by numerical (e.g. Summary Summary etc.). Reorder summaries according to the original order chapters/parts in the novel by writing a list length (NUM SUMMARIES) of the summary IDs (e.g. if you were given summaries, one possible answer could "5, 1, 4, 2"). Do not provide any explanation.
Summaries:
(SUMMARIES)
Summary IDs in Corect Order:
Table 6: Question answering and aggregation task prompts. For chat models (ChatGPT, Claude, and GPT-4), we add an additional instruction (in grey), omit the response header, as it is less appropriate for dialogue.


Reviews:
[{"review_id": "V1ot6QPPiO", "reviewer": "Reviewer_Pt2m", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**Regarding your comment about the contribution being trivial:**\n\nThe rapid pace of advancement in LLMs requires that we update existing benchmarks frequently. ZeroSCROLLS is considerably more up to date than SCROLLS across three key dimensions:\n1. While SCROLLS assumes that the model can be fine-tuned per task, ZeroSCROLLS evaluates a single model in a zero-shot setting throughout all tasks. It therefore has no training data, smaller test sets, and instruction prompts.\n2. ZeroSCROLLS contains 4 new tasks upon SCROLLS, not 2. Of these, 2 are entirely new tasks, which are challenging for SotA LLMs, yet are easy to accurately evaluate. Furthermore, 3 of the 10 tasks in ZeroSCROLLS involve multi-document reasoning, which was absent in SCROLLS.\n3. ZeroSCROLLS rigorously evaluates the long-text understanding capabilities of SotA LLMs in the post-ChatGPT era. \n\nZeroSCROLLS therefore replaces SCROLLS as the most current benchmark for testing long text understanding in SotA models. This transition is essential especially considering the amount of new LLMs equipped with large context windows. \n\nWhile excitement about our work may be subjective, in practice, ZeroSCROLLS is being adopted by the research community, as evident by the increasing number of confidential submissions from prominent labs.\n\n**Regarding your comment about the analysis and insights:**\n\nWe present a comprehensive empirical analysis of the performance exhibited by various LLMs. We reveal that Flan-T5 is able to perform complex tasks on sequences significantly longer than it was trained on. Furthermore, in section 5, we investigate how well LLMs adhere to formatting instructions, employing both qualitative and quantitative assessments.\n\n**Regarding your comment about the dataset not being in the submission materials:**\n\nWe will add links to all of the data, code for running experiments and calculating metrics in an updated version, the paper currently contains an anonymized link to the live leaderboard that accepts submissions. We will be happy to share the code and samples of the data in the discussions upon request.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces ZeroSCROLLS, a benchmark for zero-shot natural language understanding over long texts and evaluate the performance of several LLMs.", "reasons_to_reject": "1. The contribution is trivial. The main contribution can be summarized that this paper extends the SCROLLS with two new tasks, sentiment aggregation and sorting book chapter summaries. More analysis and insights should be provided, such as the potential reasons of why other LLMs have poor performance among these tasks. \n2. The paper should be revised carefully. \n3. There is no dataset is the submission materials.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces ZeroSCROLLS, a benchmark for zero-shot natural language understanding over long texts. Meanwhile, it contains two new tasks, sentiment aggregation and sorting book chapter summaries. The evaluation results among several LLMs demonstrate that long text understanding is challenging. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H3eg8Ngcgv", "reviewer": "Reviewer_PkAz", "summary": "", "questions": "Question A: Are there plans to open source this benchmark in addition to submitting the model to the leader board for testing?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**Regarding your comment about the contribution being limited:**\n\nThe rapid pace of advancement in LLMs requires that we update existing benchmarks frequently. ZeroSCROLLS is considerably more up to date than SCROLLS across three key dimensions:\n1. While SCROLLS assumes that the model can be fine-tuned per task, ZeroSCROLLS evaluates a single model in a zero-shot setting throughout all tasks. It therefore has no training data, smaller test sets, and instruction prompts.\n2. ZeroSCROLLS is not a random subset of SCROLLS, as it contains 4 new tasks upon SCROLLS. Of these, 2 are entirely new, and are challenging for SotA LLMs, yet are easy to accurately evaluate. Furthermore, 3 of the 10 tasks in ZeroSCROLLS involve multi-document reasoning, which was absent in SCROLLS.\n3. ZeroSCROLLS rigorously evaluates the long-text understanding capabilities of SotA LLMs in the post-ChatGPT era.\n\nZeroSCROLLS therefore replaces SCROLLS as the most current benchmark for testing long text understanding in SotA models. This transition is essential especially considering the amount of new LLMs equipped with large context windows. \n\nWhile excitement about our work may be subjective, in practice, ZeroSCROLLS is being adopted by the research community, as evident by the increasing number of confidential submissions from prominent labs.\n\n**Regarding your comment about the a lack of explanation of why these datasets were selected:**\n\nSome of the tasks in ZeroSCROLLS are from the established SCROLLS benchmark, and test mainstream summarization and QA tasks over single documents across various language domains. The 4 new tasks are selected or created in this work especially to require aggregating information across the input, with 3 involving multiple documents. We will clarify this in an updated version.\n\n**Regarding your comment about the order the difficulty of the selected task/dataset:**\n\nWe include estimations for human performance for 6 tasks and naive baselines for all tasks to help grasp their difficulties. For example, we find that most LLMs demonstrate impressive QA capabilities, while still struggling on our two new aggregation tasks (see 4.2). \n\n**Regarding Flan-T5 sometimes unable to capitalize 8k input tokens:**\n\nWe find the result that Flan-T5 works well at all on long sequences actually quite surprising, as it was trained on sequences of only up to 512 tokens. To the best of our knowledge, we are the first to reveal such high generalization abilities of these models. In the same table you can see that a stronger model (Claude) is able to consistently benefit from the extra tokens.\n\n**Regarding your questions about publishing the data:**\n\nWe will add links to all of the data, code for running experiments and calculating metrics in an updated version, the paper currently contains an anonymized link to the live leaderboard that accepts submissions. We will be happy to share the code and samples of the data in the discussions upon request.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well-written paper. The paper is easy to follow.\n2. Extensive experimentation is conducted.", "reasons_to_reject": "1. The contribution of the whole paper is limited.  It feels like this benchmark is more like a randomly selected subset of SCROLLS, and all of the datasets are from published work. The paper lacks an explanation of why these datasets were selected and what specific aspects of long-text modeling capability were evaluated. This leaves the reader unclear as to whether the benchmark is a comprehensive measure of the model's ability in the direction of long text.\n2. The analysis of the experiment was insufficient. For example, order the difficulty of the selected task/dataset. At the same time, there is no more in-depth analysis of some experimental results, such as Table 4, the performance gain of FLAN-T5 tokens from 4096 to 8192 is very small or even reduced, which needs to be explained in more detail.", "questions_to_authors": "Question A: Are there plans to open source this benchmark in addition to submitting the model to the leader board for testing?", "paper_topic_and_main_contributions": "This paper proposes a new benchmark ZeroSCROLLS, for natural language understanding over long texts. ZeroSCROLLS contains 10 sub-sets within 500 samples in each. Based on ZeroSCROLLS, this paper evaluates a series of large language models and compares their differences in different tasks", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "vJiGXJmmdz", "reviewer": "Reviewer_vUFG", "summary": "", "questions": "See \"reasons to reject\" section. ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**Regarding your comments:**\n\nThe rapid pace of advancement in LLMs requires that we update existing benchmarks frequently. ZeroSCROLLS is considerably more up to date than SCROLLS across three key dimensions:\n1. While SCROLLS assumes that the model can be fine-tuned per task, ZeroSCROLLS evaluates a single model in a zero-shot setting throughout all tasks. It therefore has no training data, smaller test sets, and instruction prompts.\n2. ZeroSCROLLS contains 4 new tasks upon SCROLLS. Of these, 2 are entirely new, and are challenging for SotA LLMs, yet are easy to accurately evaluate. Furthermore, 3 of the 10 tasks in ZeroSCROLLS involve multi-document reasoning, which was absent in SCROLLS.\n3. ZeroSCROLLS rigorously evaluates the long-text understanding capabilities of SotA LLMs in the post-ChatGPT area.\n\nZeroSCROLLS therefore replaces SCROLLS as the most current benchmark for testing long text understanding in state of the art models. This transition is essential especially considering the amount of new LLMs equipped with large context windows. \n\nWhile excitement about our work may be subjective, in practice, ZeroSCROLLS is being adopted by the research community, as evident by the increasing number of confidential submissions from prominent labs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a comprehensive benchmark for long text understanding. The author also perform rigorous validation of their dataset construction methods. Facilitating further research in this area.\n\n2. The overall presentation of the paper is clear and easy to follow. The claims are well supported. ", "reasons_to_reject": "1. Motivation Unclear: What is the contribution of this paper compared to the existing SCROLLS benchmark other than making it for zero-shot evaluation. What is the reason why we cannot use the test set in SCROLLS for zero-shot evaluation? If there isn't a reason, why we still want to construct a new dataset for zero-shot evaluation other than using the existing test set.\n\n2. The paper offers limited insights other than simply benchmarking LLMs performance on long text understanding. ", "questions_to_authors": "See \"reasons to reject\" section. ", "paper_topic_and_main_contributions": "This paper introduces a new benchmark for evaluating the zero-shot performance of large language models on long text understanding tasks. The benchmark includes 10 datasets spanning summarization, question answering, and information aggregation over long texts like scientific articles, books, meeting transcripts. The authors evaluate several state-of-the-art LLMs, including Flan-T5, Claude and GPT-4. They find that GPT-4 achieves the highest overall score, but there is still significant room for improvement on certain challenging tasks like sentiment aggregation. The paper also analyzes issues like output formatting discrepancies between models and references.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new zero-shot benchmark, built upon the SCROLLS benchmark and called ZeroSCROLLS. 6 tasks are from the SCROLLS dataset (and adapted) and 4 are completely new. The objective of this new dataset is to evaluate both the zeroshot capability and the long-text understanding capabilities of SOTA LLMs.
Analysis and results are provided for 8 LLMs (3 open source models and 5 closed models) and 2 baselines for most of the tasks.
Data and code will be made available (anonymized link in the submission).
The motivation, difference with the SCROLLS benchmark and the need for such a benchmark could be better presented.
Decision: Accept-Findings