NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
Authors: Cunxiang Wang
Ruoxi Ning
Boqi Pan
Tonghui Wu
Qipeng Guo
Cheng Deng
Guangsheng Bao
Xiangkun Hu
Zheng Zhang
Qian Wang
Yue Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Long-context
Large Language Models
Question Answering
Primary area: datasets and benchmarks
TL;DR: We introduce NovelQA, the first question answering dataset on documents with an average length exceeding 200K tokens.

Abstract: Recent advancements in Large Language Models (LLMs) have pushed the boundaries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. NovelQA, constructed from English novels, offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construction of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension.  Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, averaging over 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis.

Published as a conference paper at ICLR 2025
NOVELQA: BENCHMARKING QUESTION ANSWERING ON DOCUMENTS EXCEEDING 200K TOKENS
Cunxiang Wang'", Ruoxi Ning 12 Boqi Pan Tonghui Wur, Qipeng Guo Cheng Deng":
Guangsheng Bao', Xiangkun Hu Zheng Zhang Qian Wang and Yue Zhang
Westlake University; UUniversity of Waterloo; 'Hangzhou Normal University;
SShangai AI Lab; SSTTU "NYU Shanghai
waagguuxxanng, zhangyue| @esttakee.du..cn ruxxining outlook.com: 22200122@hzn.edu.co
ABSTRACT
Recent advancements in Large Language Models (LL.Ms) have pushed the bound- aries of natural language processing, especially in long-context understanding. However, the evaluation of these models' long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, introduce NovelQA, a benchmark tailored for evaluating LLMs with complex, extended narratives. Constructed from English novels, NovelQA offers unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper details the design and construe- tion of NovelQA, focusing on its comprehensive manual annotation process and the variety of question types aimed at evaluating nuanced comprehension. Our evaluation of long-context LLMs on NovelQA reveals significant insights into their strengths and weaknesses. Notably, the models struggle with multi-hop reasoning, detail-oriented questions, and handling extremely long inputs, with average lengths exceeding 200,000 tokens. Results highlight the need for substantial advancements in LLMs to enhance their long-context comprehension and contribute effectively to computational literary analysis.
INTRODUCTION
Recent years have remarkable surge in the development of Large Language Models (LLMs) (OpenAl 2023b Touvron et al.) 2003). Among these developments, long-context LLMs stand out for their ability to process and interpret extended pieces of text (Tworkowski et al. 2023 Team
2023 Anthropic 2023). This capability is essential for complex tasks that require deep and nuanced understanding of lengthy documents, such as legal cases (Xiao et al.] [02T] or academic papers (Groeneveld et al. 2024)) where the key is to understand extended narratives (Xu et al. (023).
In addition, the ability to analyze extremely long documents and multiple documents simultaneously is increasingly crucial, supporting more informed decision-making in various fields (Deng et al.
2023
Lin et al. 2023; Boiko et al. 2023).
The evaluation of extremely long-context capabilities presents challenges, existing benchmarks (Yang et al. (2018) Tay et al. 202T) no longer align with the advanced processing abilities of current [LMs (Anthropic 2023 Team 2023 OpenAl| 2023a). The increasing context window size of LLMs now outpaces the average token lengths found in long-range datasets. This gap is evident, as the most advanced long-context LLMs capable of processing over 250,000 tokens, stark contrast the longest average token length in current benchmarks, which is around 60,000 tokens. This mismatch underscores the need for updated evaluation methods that can accurately reflect the capabilities of current and future LLMs, as illustrated in Figure [
To fill this gap, we introduce NovelQA, a benchmark crafted specifically evaluate LLMs' per- formance on texts with averaged context windows exceeding 200,000 tokens. Unlike existing
Equal contribution, paper finished at Westlake University Correspondence Yue Zhang @haaggue@werslkke.ddhu .ed.cc and Qian Wang (202001426) trzu.edu.cm

---

Published as a conference paper at ICLR 2025
250K
XVERSE-Long-2SOK7
NuvelQA
Size (Datasel) Inpunt Lenghh (Model)
Yi-6B-200K
200K
Claude-2. 2. 1200K
Bochuan2-192K
150K
CharGPT-4TTrto-1228
ChaGGPT-128K
C3
100K
50K
NarrativeQA
VLSP
FINDSum
Jama- JoK
HotposQA
QMSum
OMSumQasper Quiper
A03 Style
CROILS RAN Long CongBBerch AMB0O
DuReader
GPT-2-IK
GovReport SumnSqreeanMuSiQuee
2018
2019
2021
2022
2023
2024
Timeline
Figure 1: Trend of context window size of LLMs (Orange) and average token length of long-range benchmarks (Green). NovelQA is highlighted with star.
Jane Eyre
Complexit: Multihop Aspect Clanetee
Les Misérables
Aspeet: Selting
Complexiny Denii
Question: Someone mamried clergyman in Chapter X, who did he'she company to walk the garden Chapter
How caodles Madame Magloire lighted when Rishop had his last dinner with Jean Valjean?
Question:
Choices:
A. Burns
B. Brocklehurst
C. Lleyd
D. Scatcherd
Choices
period married, removed with husband (a clergyman, an excellent man, almost worthy of such wife) distant county, and consequently was lost me.
A. Two candles
B. One candle Four candles
C. T.rre candles
When he had any dinner, Madome Magloire lighted the candles and set the candlesticks on the table.
confirmed in this idea by the fact of her once or twice coming downstaizs on very warn aflernoons, and being laken by Miss Temple into the garden;
Gold:
Gold: A
Illustrative examples from NovelQA: This figure showcases two sample questions. For each question, models evaluated under two distinct settings multichoice, where the task is to select the correct answer from four options, and Generative, where the model generates an answer.
Figure 2:
benchmarks (Shaham et al., 2023 al. 2023 Adams et al.) 2024). NovelQA addresses the need for assessing extremely long-context understanding. offering refined and comprehensive tool for ad- vancing natural language processing capabilities. We construct NovelQA based on novels in English, which also ideal for testing cong-context modeling because they long and complex. with plots that are closely linked from start to end. We select novels from various eras, genres, and formats to enhance diversity. The annotation process is performed by group of expert annotators, all of whom are holding or pursuing a degree in English Literature and have a strong interest in and familiarity with the novels they annotate. Each question paired with a 'golden answer' and corresponding textual evidences from the novels. And we categorize them by complexity and aspect for detailed analysis. Figure 2 presents two examples, while Table T details the distribution of question types. The dataset includes multi-hop, single-hop, and detail questions, which test the model's 's abilities retrieve and integrate scattered information, retrieve information and summarize, and precisely identify specific and subtle details, respectively. The above capabilities are more challenging in particularly long context. For example, finding relevant information a particularly big challenge.
We assess various long- Long-context LLMs using NovelQA, including commercial models such GPT-4- 128K (OpenAl (023a) and Claude-2. 2-1-2. 3-00kk Claude-3.5-2004 (Anthroppi 2023). alongside open-source models like InternLM2-Chal (Team 2023) and Llama-3.1 (Touvron et al. 2023). Results show that even the most advanced long-conter context LLMs face challenges in consistently extracting and processing accurate information from extended texts. For example, Claude-3.5, the top performer, achieves a 62.30% accuracy rate, whereas the open-soure Llama-3.1 achieves 51.50% in generative setting. In addition, some models unable answer properly when the context
2

---

Published as a conference paper at ICLR 2025
exceeds 100K tokens, even if their context windows are much longer (xverse 2023 01 -ai 2023
Technically, operating LLMs on inputs exceeding 200,000 tokens presents challenges, particularly regarding memory requirements and associated costs. The difficulty of valid models is particularly apparent in answering multi-hop questions and queries that probe meanings, relationships, spans, and timelines, highlighting significant gap in the models' long-range comprehension. Moreover, our data shows a decline in performance for evidence situated beyond the 100,000-token mark, including information the novel's end, diverging from the anticipated lost-in-middle phenomenon (Liu et al. 2023). This shift suggests distinct challenge faced by LLMs when processing texts exceeding 100K tokens in length. These results highlight challenges not only in memory optimization but also in the nuanced comprehension and integration of lengthy texts, indicating substantial obstacle on the path to truly effective long-context LLMs.
To the best of our knowledge, NovelQA is the first long-context QA benchmark featuring manually crafted questions, golden answers, and evidences, with contexts extending beyond 200,000 tokens.
2 RELATED WORK
Long-Range Benchmarks Evaluating the ability of llog-coonte Large Language Models has been a hot topic (Kočiský et al. [018]] (aa et al. 2021 Shaham et al., [023 Pang et al. 2022 Wang et al., 2024b; Yu et al. 2024). When entering the era of Large Language Models, the context window length has been much longer than ever, and the decoder-only LLMs have been the mainstream of language models and they have faced specific problems, such as the fost-in-middle issue et et al. 2023). Thus, increasing benchmarks
created for evaluating long-context LLMs. Among those benchmarks, several are representatives. The Needle in A Haystack test (gkamradt et al. 2023) involves inserting an unrelated sentence into long context and asking the model to retrieve it. However, this test does not represent natural language question-answering answering task that requires composite and complex capabilities for processing long contexts. Narrative (Kočiský al. 2018) asks annotators from MTurr to read novel summaries to write questions. Narrative (Moskvichev & Mail 2023) utilizes GPT-3.5 to summarize approximately 150 scenes per book from a collection of 1,500 books, and automatically annotates around one million questions, ZeroSCROLLS (Shaham et al.] (2023). LooGLE (Li et al.) 2023a), Counting-Stann Saariing-StrrSSonnnn (Song et al.) 2024), and LongICI.Bench (Li et al. (024) emphasize the importance of understanding and aggregating information from long texts, presenting challenges that highlight areas for improvement in current models. L-Eval (An et al. 2023) introduces a suite of tasks with human-labeled query-response pairs to assess LLMs performance in processing long inputs effectively, using advanced metrics for a more accurate evaluation. Furthermore, benchmarks like LongBench (Bai et al.] 2023), BAMBOO (Dong et al., 2023a), and LongBench-Chai (Bai ett al., (024) offer a diverse set of tasks across languages and domains, from reasoning and coding summarization and multilingual translation. These benchmarks designed rigorously test. the ability of LLMS to manage extensive contexts, with LongBench-Chat specifically focusing on instruction-follloing capabilities in long-contex: interactions. Additionally, LongHealth (Adams) et al. 2024) addresses the need for LLMs to interpret long clinical documents accurately, providing a specialized benchmark for evaluating models on medical texts. This focus on domain-specift challenges underscores the broader necessity for LLMs to not only handle long texts but to do so in manner that is accurate and contextually relevant across various fields.
NovelQA distinguishes from existing benchmarks in three points. Firstly, NovelQA features an average token length exceeding 200,000, far surpassing the tens of thousands typically found in other benchmarks. Secondly, while other datasets often rely on Al-generation or existing datasets for content creation, the questions, golden answers, and evidences of NovelQA are entirely crafted through human effort of experts. We present detailed comparison between NovelQA and other benchmarks in Appendix Fig
'We have released the demonstrations and input of NovelQA, and created a leaderboard. More details can be found in https:::p... / wwvelleveeeeeoooooooooeo qa. atttbiiooo And NovelQA released under the Apache-2.0 License. For the public access, we have released all constructed data on Huggingface dattsets htt///oveel//aaeeeeeee Car.veeooA/Noveeeo Nove 1QA and an evaluation system on Codabench httpp: /www. codabench However, we only release public domain novels. Therefore, we offer two types of metrics in the evaluation system and leaderboard: one for evaluating QAs within public domain novels and another for evaluating QAs across all novels.
/o/
org/ wwtpe/ttiiios/////////oo
3

---

Published as a conference paper at ICLR 2025
Min Count
Median
Mean
2008
300K
400%
S00K
600K
84721
200K
400K
S00K
600K
Figure 3: Token Count Distribution in NoveIQA, including Copyrighted (left) and Public Domain Only (right). The token counts of both the novel and the questions are counted. The tokenization procedure is accomplished through gpt-3.5-turbo 16k tokenizer.
Table I:
Distribution of Question Types in NovelQA: This table provides a breakdown of questions across different complexity categories (Multi-hop, Single-hop, Detail) and aspect categories (Times, Meaning, Span, Setting, Relation, Character, Plot).
Multi-hop Single-hop
Detail
Sum
Times
463
463
Meaning
34
126
206
366
Span
34
34
Setting
264
Relation
119
14
32
165
Character
69
255
98
422
Plot
64
414
113
591
Sum
807
986
512
2305
We also discuss related Long-Context Languaag Modeling methods in Appendix Sec C.I
3
DATA
3.1 DATASET OVERVIEW
Data Formulation Every novel (N) the dataset corresponds to multiple pieces annotated data (d;). Each piece of data consists of the following domains, question (Q:), answer (A;), multichoices (a,,0- 01,1, 0g,2, and a,,3), gold label (Os.gota), evidences ($t.0, 8,,1- ...), and rype (Complæ; and serve for either generative multichoice task. In the generative setting, a novel text N and a question Qi are combined to send into the model each time, and the generated answer is compared with the answer A; In the multichoice setting, the novel N, question Qi and the four choices 0g to ag, are sent into the model, and the output is evaluated according to the gold label di.gold- The evidences domain consists of either the original excerpts from the novel or the reasoning steps written by the annotator.
Aspectz). NovelQA
Book Selection The books in NovelQA contain 65 free public-domain books from the Project Gutenberg and 24 copyright- books follows criterion to ensure the diversity the eras, styles, and themes, thus we decided to include portion of copyrighted novels. All selected books are in English and exceed 50K words (approximately 67k tokens). The distribution the token count is illustrated in Figure 3
books purchased from the Internet. The selection of the
Question Distribution The annotated questions can be classified by the complexity of solving the question and the aspect that the question focuses on. By complexity, the data are categorized into three complexity levels, multi-hop (35.0%). single-hop (42.8%). and detail (22.2%). The order of complexity is as follows: multi-hop > detail > single-hop. By the aspect that each question focuses on, the data entails seven types. A detailed specification of each type listed in Appendix C.2.2 We have a total of 2305 questions, of which 1640 are from 65 public domain novels, while the remaning 665 are from 24 copyrighted novels. According the classification above, the distribution of the
hhttps//www.gueenhrrg.rrg
4

---

Published as a conference paper at ICLR 2025
questions in our dataset is displayed in Table | We also list the ability tested by each kind of question in Appendix Table [0]
Advantages Our NovelQA dataset serves new benchmark for evaluating long long-context under- standing, distinguished by several key advantages. Firstly, surpasses existing benchmarks in length, offering rigorous test of model's 's ability to navigate and comprehend significantly longer texts. Secondly, the inclusion of clear evidences alongside questions ensures that evaluations are grounded in concrete textual support, enhancing the reliability of assessments. Furthermore, the dataset empha- sizes questions that require attention to detailed information, challenging models to move beyond superficial impressions to extract specific, nuanced answers. Questions, golden answers, and evi- dences of the dataset are entirely manually annotated and carefully checked, ensuring high-quality, nuanced questions and answers that reflect complex human thought processes. To prevent against data leakage, we will not release golden answers for the test set, minimizing the risk of overfitting. These features, combined with the dataset's comprehensive coverage of diverse narratives and meticulous construction, make NovelQA valuable resource for advancing long-context understanding.
3.2 DATA ANNOTATION
Procedure Overview The annotation process is performed by group of expert annotators. The annotation procedure consists of two phases: (1) Template-based phase: The annotators can fill entities into 10+ templates (see Appendix Sec C.2.1] that we design to be related to multi-hop or detailed information. This phase entails half of the data, mainly contributing to the multi-hop ones. (2) Free-form phrase: To ensure the diversity of question expression and align the questions to the natural distribution, the second half of our data is annotated without a template, namely, the annotators contribute any difficult questions that they come up with freely.
Annotator Recruitment Our annotators are predominantly English Language and Literature uni- versity students those with keen interest in English novels, recruited from local universities. These students are accustomed to closely reading English novels for coursework or as a hobby, and writing reports or papers on them. Before annotation, each annotator was instructed read the annotation instructions to understand the requirements and sign to agree to participate. Annotators are allowed to select novels for annotation based on their familiarity, ensuring they have previously read and comprehensively understood the texts. Meanwhile, we make sure the selected books meet our standards of enough word count and well-developed narratives. We also ensure that each selected novel is either annotated by only one individual, or consistent in version across annotators, despite minor variations among different editions. Each annotator contributes to a typically small number of 20-30 questions per novel. This approach avoids forcing annotators to annotate questions unfamiliar content.
Annotation Guideline While creating the QA tuples, our annotators are instructed to follow several principles below: (1) The annotators are either senior English language and literature college students, or students with high English test scores. (2) The annotators required to read through the GPT-4 responses on example questions. (3) The annotators should choose novels above 50K tokens and read through the books they choose before annotation.
(4) Evidence of each answer should be provided for validation purposes. Evidence should be as sufficient as possible.
Template Design The first annotation phase relies on a question template, which requires the annotator to fill in the entities from the novel to form valid questions. To design templates, we carried out sufficient pre-tests on several LLMs to analyze their possible weaknesses in long-input QA and novel knowledge memorization. Our pre-test shows that they usually fail to tackle information spanning over multiple chapters, as well as lack attention to details that have no contribution to the main theme. We also refer to around fifteen books on novel and narration theories (e.g. Forster 1927 Tobias 2012] Schmidi 2012, McKee 2005) to ensure our template covers more aspects that a novel can discuss (c.g., character, setting, theme). Templates ensured to test on facts (c.g., events, entities, numbers) that can be traced back to specific evidences from the books, instead of on any subjective feelings or analysis of the readers.
Time Consumption and Rewards Given the annotator's familiarity with their chosen novels and their experience with similar questions in their academic assignments, creating questions based on their knowledge becomes a manageable task within a reasonable time cost. The annotation reward is of $1.11 $1.39 per tuple. As an average annotator can write 5 to 6 pieces of data at full speed
5

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation of Long-Context LLMs on NovelQA. This table presents the performance of four long-context LLMs, including both commercial models (GPT-4, GPTT-4o-inni Claude-2.1, Claude-3- Sonnet and Prtuue-3.5-Someeee and open-source, locally deployed models LnneenLM2-Chhal Chat-7b/20b and 1.-mn--..--b/00b) Accuracy percentages reported under two testing scenarios: multichoice and generative. The Max Length column denotes maximum token length of each model.
Max Length
Multichoice
Generative
GPT-4
128K
71.80
46.88
GPT-4o-mini
128K
71.85
53.32
Claude-2.1
200K
66.84
46.04
Claude-3
200K
71.11
53.66
Claude-3.5-Sonee
200K
77.92
62.30
Internl M2rmnlMM2-Cn M2-Chat-7h
200K
43.51
30.90
MMermllM--Cha---e
200K
49.18
32.37
Llama-3.1-88
128K
62.31
42.65
Llama-3,1-12 1-70B
128K
69.39
51.50
Human bascline
00
97.00
90.00
according to our observation, the $5.56 to $8.34 hourly wage is above the local legal minimum wage of $2.78/hour. The annotation process costs around $3,500.
Quality Control The created data is manually double-checked by three authors of this work. The review procedure follows the criteria of minimizing factual errors, enhancing expression clarity, and maintaining challenges to LLMs. Besides, we ensured that all questions are based on factual descriptions and eliminated any subjective ones. Consequently, only 79.4% of the collected data preserved, resulting in final dataset of 2305 QA tuples. Meanwhile, we have also conducted inter-annotator agreement (IAA) test, focusing on evaluating the quality of annotated question- answer pairs. Annotators are required to choose books they are familiar with but have not annotated themselves to answer questions on. As annotators are mostly from the same local university and share similar courses and projects, many can find books they have read in common. We select books read by at least two annotators and have the other reader answer multichoice questions. As the respondents quite familiar with the target novels and have strong academic background in English Literature, it takes them around 2-3 hours or less to complete each novel. The IAA test shows a score of 94.6% in Cohen's Kappa, indicating high agreement among annotators.
Distractions for Multichoice Setting We use GPT-4 to generate three distracting options for each question and its golden answer, and randomly permute the four answers We manually checked those distractions and rewrote those with similar meaning with the golden answers when we double check the data.
4 EXPERIMENTS
We focus long long-context models meeting three criteria: context window of at least 128,000 tokens, accessibility via a full API or public release, and chat functionality. For commercial models, our selection includes GPT-4-128K (OpenAl 2023a) and Claude 2. 2.1-00KK (Anthropic 2023). Among open-source options, we evaluated models like InternLM2-chat (Team 2023).
4.1 IMPLEMENTATIONS
Settings We employ two evaluation settings: generative setting where models directly generate short answers, and a multichoice setting with four provided options.
Prompts We uniform prompts for all LLMs, with start part, novel content, questions, choices in the multichoice setting, and an end part. The prompt structure is shown Appendix Table
Our pilot study compared distractors generated by both GPT-4 and Claude 2.1. Interestingly, GPT-4 generated slightly more challenging distractors- hoth models scored approximately 0.5% lower on GPT-4's distractors compared Claude 2.1's. This indicates advantageous bias for GPT-4.

---

Published as a conference paper at ICLR 2025
Table 3: Model Performance by Question Type in Generative Setting: This table details the accuracy scores of four models across different question types. Question types include details (dtl), multi- (mh), single-hop (sh), character (chara), meaning (mean), plot, relation (relat), setting (settg), others, and an average score (avg) for each category, with '-' indicating the absence of data for a category.
(a) GPT-4
(b) Claude 2.1
chara mean plot
relat
settg
span times avg
chara mean
plot
avg relat settg span times
mh 57.81 61.76 52.46 45.30 56.52 18.18 21.23 32.83 mh 48.94 64.29 58.18 40.00 82.61 18.52 17.10 30.34
sh
62.12 56.56 69.33 21.43 57.23
63.93
sh 72.41
55.24 67.96 23.08 59.60
65.47
dtl
52.63 12.87 61.68 37.50 58.06
37.58
dil 55.22
12.43
66.30 27.59 55.77
37.65
avg 62,04 32.40 65.93 41.72 57.38 18.18 21.23 46.88
avg 65.90
31.61
66.60 35.61 61.06 18.52 17.10
46.04
(c) InternI.M2-Chat11
(d) InternL.M2-Chht-222
chara mean plot relat settg span times avg
chara mean plot relat settg span times
avg
mh 23.81
38.24 42.62 24.35 39.13 15.15 21.32 24.62 mh 32.84
44.12 29.03 37.61 25,00 15.15 26.81 29.29
sh 35.80
28.10 42.18 14.29 32.10
36.42
42.97
34.17 43.22 21.42 36,90
40.57
dtl 26.67
9.18
55.14
29.03
34,43
27.02
dil 30.93
7.00
48.21 25.00 29.03
24.65
avg 32.02
18.52
44.77
24.38
33.33
15.15 21.32 30.90
avg 38.50
19.77
42.66
33.74 33.86
15.15 26.81 33.07
Truncation Due to input length limitations, we truncated the novel content from the end the front following Bai et al. (2023); Li et al.] (2023a); (n et al. (2023). to meet with the max input length, while keeping questions and other prompts complete.
Evaluating Generative Results Previous researches have proved the ability of LLMs in evaluating the machine-generated answers align with human judgements (Wang et al. 2023 An et al. 2023 Li et al. 2023a) After a pilot study showing that the models show no several preference towards the answers generated by its own kinds (See Appendix [C4.2 we choose GPT-4 to evaluate our generated results. We further conducted human evaluation 800 pieces of generative outputs and carried out an inter-evaluator agreement (IEA) test between two human evaluators and the GPT-4 evaluator. In the IEA test, annotators also serve human evaluators of the novels they were familiar with. Their evaluations were compared to those of GPT-4 evaluators, with Cohen's 's kappa score calculated to measure agreement. As shown in Table 7 the result of 89.25% in Cohen's 's Kappa indicates a high agreement towards the GPT-4 evaluating results. NovelQA primarily consists of objective questions, which have clear, verifiable answers with factual basis in the text. This objectivity significantly reduces the impact of model bias in LLM-as-Judge evaluation
Commercial LLMs The APIs of commercial LLMs utilized are gp1-4-0125-previee, gpt-4o-mini, Claude-2.1, Claude-3-sonnet and 3.5-onet.
Open-source LLMs Running long-context L.MMs on extremely long inputs, such 200K tokens, is a challenge due to the immense GPU memory required, for example, it takes roughly 2.5T memory to calculate one attention matrix for 7B model with 200K-token input, while our local device is 4 X 80G A100. To address this, we utilize the LMDeploy (Contributors) 2023) (based on Dynamic NTK (emozilla 2023) and vLLM (Kwon et al.) 2023) for memory and time reduction, which is only compatible with several LLMs. Therefore, we choose MeemLLM2-Chat2200-2000
Llama-3.1-80 and Llama-3. 1-b for our experiments.
Human Performance We selected books that have been read by multiple annotators, and then have had the readers engage in a two-round answering process on novels they had not previously annotated. The first round was in generative setting, and the second round was in multichoice setting. This process was conducted on 5 novels with a total of 100 questions. The result shows that human performance scored 90 in the generative setting and 97 in the multiple-choice setting.
4.2 MAIN RESULTS
We present the main results in Table Even the highest scores (71.80% and 46.88% for GPT-4 in generative and multichoice settings, respectively) suggest there is considerable room for improvement in long-context understanding compared with human readers. This is especially true in the generative setting where understanding and recall over long contexts are more challenging. Additionally, commercial models (Claude-3.5-Somme and GPT-40-mini) outperform open-source models in this benchmark. Among closed-sone soosre-source models, LLaMA family models outperform the InternLM models. All models show drop in performance in the generative setting compared to the multichoice setting.
7

---

Published as a conference paper at ICLR 2025
100
100
gpt4
gpt4 claude
claude
80-
internim7b
80-
intemnm77
Internim20b
interni220b
Accuracy (%)
80
60
40
30-
20-
10
10-
OK 26K 40K 60K BŌK10OK 130K 160K 190K Token Position
10% 20%
30%
40%
50%
60%
70%
B0%
90%
Percentile
Figure 4: Analysis of Accuracy in Generative Setting by Absolute and Relative Token Positions: The two figures above illustrate the accuracy, plotted against the absolute token indexes (left) and the percentage position (right) of each question's 's relevant evidences. The x-axis of the absolute token position figure (left), reflecting token indexes, is folded on the right due to the long-tails.
This indicates the increased challenge generating a correct answer from scratch, as opposed to selecting from provided options.
We appended a typical error analysis in Appendix Sec C.4.6
4.3
RESULTS BY THE QUESTION TYPE
An in-depth analysis model performance across question types reveals nuanced insights into their comprehension abilities in both generative and multichoice settings, detailed in Table BableBaand Appendix C.4.3 respectively. This analysis not only highlights the models' weaknesses across different formats but also illuminates the challenges in narrative comprehension, contributing to both NLP and computational literary research.
The examination of accuracy relation, and setting highlights distinet patterns in performance, pointing to the models' differential capabilities and limitations. Notably, models exhibit partieular difficulty with questions centered around meaning, relation, span, and times. This difficulty suggests several underlying challenges:
question categories such character, meaning, plot,
The struggle with meaning questions indicates challenge in grasping abstract concepts and locating entities or sentences through interpretations within the text, which requires an advanced level of semantic understanding and inference.
(1) Meaning Questions:
(2) Relation Questions: Difficulty with relation questions points to gap in the models' ability to identify and interpret the dynamic and often nuanced relationships between characters, events, or concepts, which are crucial for holistic understanding of narratives.
(3) Span and Times Questions: The lower performance on span and times questions suggests limitation in tracking temporal sequences and spatial extents within the narrative, reflecting challenges maintaining and applying contextual information over long stretches of text.
The above findings underscore critical aspect in both computational literary studies and long- context comprehension
models are adept handling certain types of narrative questions, they encounter significant hurdles when required to synthesize abstract concepts, interpret complex relationships, or maintain a coherent understanding temporal and spatial narratives long context. These be the domains requiring further improvement to enhance narrative comprehension and reasoning capabilities.
4.4 RESULTS BY POSITION
Our analysis delves into how the positioning of evidence within novels affects the accuracy of long- context LLMs. Specifically, we explore the impact of both absolute and relative positions of evidence, where the absolute position refers to the specific token index within the text, and the relative position normalized against the total length of the novel, scaled 0%-1100 range.
8

---

Published as a conference paper at ICLR 2025
Table 4: Model Performance Analysis Pre and Post 100K Tokens: The accuracies of questions on evidences pre-100K and post-100K are calculated separately and compared, where the '100K' indicates the aboslute starting token position of evidences relating to answer. The result shows an obvious contrast as the accuracy drops sharply after the 100K, in both generative and multichoice settings on each model.
Multichoice pre- 100K
Generative pre-
post-100K
post-100K 100K
GPT-4
83.17
59.22
53.64
32.01
Claude2.1 1
77.65
54.46
61.96
30.57
InternI.M-7b
51.78
38.17
37.71
18.23
InternI.M-20 55.59
40.87
37.74
21.10
Weighted Avg.
66.24
48.08
46.84
25.36
Table 5: Evidence recalling results of four LLMs: During the evidence recall test, each model is required output the relevant evidence from the original novel contents. Each recalled piece of evidence is further compared by GPT-4, which has been proven to generate reliable evaluations, with the gold standard evidence. The scores for each model, assessed three dimensions, Correctness, Relevance, and Sufficiency, are listed below.
Correctness
Relevance
Sufficiency
Avg.
GPT-4 Claude 2.1
29.47
38.05
27.67
31.73
23.51
29.08
22.27
24.95
InternL.M2-Chatt Cb
2.05
3.88
1.90
2.61
InternLM2-Chhate
6.36
12.51
7.50
8.79
Absolute Position In the generative setting, depicted in Figure (left), all evaluated models show improved performance on questions where the necessary evidence located before the 100K token mark. This trend highlights challenge for L.LMs in accessing and processing information beyond this threshold, suggesting diminished capacity to handle very long inputs. The multichoice setting, detailed in Appendix [C.44] follows a similar pattern, reinforcing the importance of evidence position in model performance. We also present the relationship between the accuracy and the absolute position of evidence, grouping by pre- 100K and post- 100K, Table
Relative Position By normalizing the evidence positions within the entire novel, we aim to understand the proportional location of evidence influences model accuracy. This analysis, shown Figure (right) for the generative setting and in the Appendix C.4.4 for the multichoice setting, indicates that models maintain relatively consistent performance across various relative positions, suggesting that long-context LLMs' effectiveness is not significantly affected by the evidence's relative position within the standardized length of novels.
To delve deeper into how long-context LLMs navigate extremely long inputs, segment novels into two categories based on length: 65k- 100K and over 100K tokens. We then examine model accuracy in relation to relative evidence positions within these ranges, with results presented in Appendix Figure 7 For novels within the 65k- 100K token range, we observe lost- in in-middee phenomenon akin findings by Liu al. (023). This pattern
in GPT-4, InternL_M2-bb, and InternI.22-25 indicates stronger performance at the beginning and end of texts but weaker in the middle. Conversely, novels exceeding 100K tokens, model performance generally declines towards the end, potentially due to the scarcity of training data for contexts of this length. This behavior underscores a unique challenge faced by LLMs when processing exceptionally long texts over 100K tokens.
This analysis highlights the critical role of absolute evidence positioning in determining the accuracy of LLMs in processing long texts. Challenges arise when context beyond a specific token threshold. Conversely, the relative position within normalized length has a minimal effect.
4.5 EVIDENCES RECALL
We evaluate the ability to recall evidence, prompting the four models above to answer the questions in NovelQA again, with printing the supporting evidence simultaneously. We then prompt GPT-4

---

Published as a conference paper at ICLR 2025
Table 6: Close-book Performance across four LLMs on NoveIQA. Unlike the standard scenario, models rely solely on internal knowledge without access to the novels. The parentheses indicate the performance drop from the standard to the Close-book scenario, highlighting the models' dependency on external text for answering.
Close-book QA
Multichoice
Generative
GPT-4
60.94 (-9.06)
34.30 (-13.58)
Claude 2.1
51.77 (-15.01)
22.36 (-20.68)
33.58 (-10.71)
14.12 (-16.81)
IntemL.M2-Chatt
33.05 (-16.13)
15.51 (-16.86)
with the generated evidence alongside the annotated evidence to obtain its evaluation the quality of retrieved evidence pieces. The evaluating matrix consists of the following three dimensions: correctness refers to whether the retrieved evidence is the same as the annotated evidence with a similar correct meaning; relevance indicates whether the evidence is consistent with the answer; sufficiency, whether the retrieved pieces of evidence are enough support the answer. Fach dimension is scored between 0 and 100 and an average score is further obtained through calculating the algorithmic mean on these three dimensions. Prompts involved this evaluation procedure are presented in Appendix C.2.3
The results, detailed Table show higher performances of GPT-4 and Claude 2.1. Moreover, though the scoring range is from O to 100, the four models all perform with low scores in evidence recall, possibly because the models do not always follow the instructions after inputs and outputs of such an above-average length, resulting in high percentage of O scores. This phenomenon is and MitemLM-Chht-200 models, whose outputs consist of large proportion of invalid placeholders. Still, current long-context LLMs generally demonstrate inadequate abilities recalling the correct and supportive evidences from the context.
particularly severe for InterL.M-Chat
4.6 CLOSE-BOOK QUESTION ANSWERING
We employ a Close-book QA scenario to assess the extent to which models rely on the content of novels versus using their internal knowledge to answer questions. In this approach, the models are not given access to the text of the novels and must rely solely on their pre-existing knowledge to provide answers. Given that our selected novels are well-known and representative of their genres, it's inevitable that LLMs have encountered their texts during training and retained some of their content. The results of this evaluation are presented in Table Models like GPT-4 and Claude 2. achieve notable scores in the Close-book setting (60.94% and 51.77% in multichoice, 34.30% and 22.36% in generative, respectively) indicating that they have internalized significant portions of the novels' content during training. This internal knowledge allows them to perform reasonably well without direct access to the text. The difference in performance between the Close-book and standard settings underscores the challenges in long-contex: understanding. While models can retain and recall information from well-known texts, their ability comprehend and use such information to answer questions accurately diminishes in the absence of the text. This suggests that long-context understanding, as measured by the main results. might still be more challenging than appears, as models benefit from having the text directly available.
5 CONCLUSION
We introduced NovelQA, a long-range benchmark designed to assess the long-context comprehension abilities of LLMs. Utilizing representative English novels, NovelQA presents LLMs with the challenge of navigating complex, real-world texts. Our evaluations reveal that both commercial and open-source models face challenges with detailed understanding, multi-hop reasoning, and accurately retrieving specific information from lengthy contexts, especially for lengths are 100,000, Moreover, operating LLMs on inputs exceeding 200,000 tokens faces technical challenges, notably terms of memory requirements and associated costs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We appreciate the contributions of our annotators: Jingyi Wu, Ziyue Li, Qiangyi Zhu, Xinyao Feng. Zhangrui Xu, Xiao Wu, Yu Qiu, Hanxin Ye, Mengting Wei, Hanyu Cai, and Yaokai Tu from Hangzhou Normal University; Ziyang Zhu, and Xinyu Zhu from Ningbo University; and Rizwan Abbas, Khadija Azhar, and Snawar Hussainfrom Zhejiang University. This publication has emanated from research conducted with the financial support of National Natural Science Foundation of China (NSFC 6o.62161160339
REFERENCES
Ol-ai. Yi-6b-200k, 2023. URL UKL/rrpss//ugggiggee..////.---
Lisa Adams, Felix Busch, Tianyu Han, lean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, Hugo JWI.. Aerts, Jakob Nikolas Kather, Daniel Truhn, and Keno Bressem. LongHealth: A question answering benchmark with long clinical documents, 2024.
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-Eval: Instituting standardized evaluation for long context language models, 2023.
Anthropic. Cluade-2.1, 2023. URL httpt::t////m // wwwwwwwwww//mm..oo aww.ooiicccoiccccc wwtprr/ppp....c............ wwt.
/ с aude- 2-1
Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang. Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. ArXiv, abs/2308. 14508, 2023. https/ 7/api stmant cccchlaaaasc wwtp:oa..rrgoooouuostoooa 6122445564
Yushi Bai, Xin Lv, Jiajie Zhang. Yuze He, Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models, 2024.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 150, 2020.
Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical re- search with large language models. Nature, 624:570 - 578, 2023. URL https://api
semanti cscho ar www.olla.rrrrrrggooooo wtt.arrr/ooooosst 26:
Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An ar- chacology of books known ChatGPT/GPT-4, In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7312-7327, Singapore, December 2023. Association for Computational Linguis- URL http::ss//eclaatoollo.ooo///////
tics. doi: 10. htt.///3//1/222emmmleee
emmlpp main. 453
Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arX0v:2310.05002, 2023a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of
large language models via positional interpolation. arXiv preprint 2tt.:2306.115995 2023b.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint a305v:2305. /4788, 2023.
LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm. https
//github. com htt//nnteennn./ wttt.//.....ttett///////catc///t http:/eteeeii//medep..o.o... Imdeploy 2023.
Cheng Deng. Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. Learning a foundation language model for geoscience knowledge understanding and utilization. arXiv preprint arXiv:2306.05066 2023.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models, 2023a.
11

---

Published as a conference paper at ICLR 2025
Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey on long text modeling with transformers. arXiv preprint 200X:::2002.1500,,, 2023b.
emozilla. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning, 2023. URL
Mat.LLLMMA//oenttsssssss comments/
14mrgpr/dynami ca 1y_ sca
Guhao Feng, Yuntian Gu, Bohang Zhang, Hao-Tong Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. ArXiv, abs/2305. 15408, 2023. URL https://api wwwanticssciiine
Edward Morgan Forster. Aspects of the Novel. Harcourt, Brace, 1927.
gkamradt et al. Needle in a haystack pressure testing Ilms, 2023. URL BRLLttts://github...
gkamradt deteratt/LLLMM /LLMTest Lee.eeeestteeeeerrllaaaastte
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. OLMo: Accelerat- ing the science of language models. arXiv preprint arXivv2402.00838, 2024.
Chi Han, Qifan Wang. Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:330..66137, 2023.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards million context length Ilm inference with kv cache quantization. arXiv preprint arXiv:2401. 18079, 2024.
Yunpeng Huang, Jingwei Xu, Zixu Jiang, Junyu Lai, Zenan Li, Yuan Yao, Taolue Chen, Lijuan Yang. Zhou Xin, and Xiaoxing Ma. Advancing transformer architecture in long-context large language models: A comprehensive survey, ArXiv, abs/2311. 2023. URL https://api
wwwpcchhlla...oooooouuususeee.ee semanticscholar.co org. oee.aaa....ccooo.....oooooooooot
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing lims in long context scenarios via prompt compression. arXiv preprint arXiv:2310.0663 2023.
Tomá's Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
doi: 10.1162/tacl a a_0023. URL
1023
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE; Can long-context language models understand long contexts?, 2023a.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-context lims struggle with long in-context learning, 2024.
Xianming Li, Zongxi Li, Xiaotian Luo, Haoran Xie, Xing Lee, Yingbin Zhao, Fu Lee Wang, and Qing Li. Recurrent attention networks for long-text modeling. arXiv preprint arXiv:2306. 06843, 2023b.
Yucheng Li. Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06200 2023c.
Bin Lin. Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao. Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, et al. Infinite-LLM Efficient 11m service for long context with distattention and distributed kvcache, arXiv preprint 2xXiv:241.006660 2024.
12

---

Published as a conference paper at ICLR 2025
Zhouhan Lin, Cheng Deng, Le Zhou, Tianhang Zhang, Yi Xu, Yutong Xu, Zhongmou He, Yuanyuan Shi, Beiya Dai, Yunchong Song, et al. GeoGalactica: A scientific large language model in geoscience. arXiv preprint arXiv:2401.00434, 2023.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilaequa, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023.
Robert McKee. Story. Dixit, 2005.
Arseny Moskvichev and Ky-Vinil Mai. Narrativexl: A large-scale dataset for long-term memory models, 2023. URL httttss:::: http://://wwwww..r//....oo wrtp://////bbb/3......./ 13877 13877
OpenAl. GPT-4 technical report. arXiv, 2023a.
OpenAl. GPT-4 technical report, 2023b.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long imput texts, yes! In Marine Carpuat, Marie-Catherine de Marn- effe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technolo- gies, pp. 5336-5358, Seattle, United States, July 2022. Association for Computational Linggis- URL httpsttss///aaaaaaoooooooooooe
tics. doi: 10.18653/v 10.18653/w1/0222mmaal maacl-maze 391
Ofrr Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
Victoria Lynn Schmidt. 45 Master Characters, Revised Edition: Mythic Models for Creating Original Characters. Penguin, 2012.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: zero-shot benchmark for long text understanding, 2023.
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars' A multi-evidencee, position-aware, and scalable benchmark for evaluating long-context large language models, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568: 127063, 2024.
Yi Tay, Mostafa Dehghami, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https
7/openreview net/ www.enrrreee..nt/Eomrniiieeee..e Eor.mmidd--vv----ge---
InterLM Team. InterLM: A multilingual language model with progressively enhanced capabilities. https://gi http:///thhuu..oooo//eetttteee www.//ooo/nnniiieeeeeee httph//...../eetetttttttettttatn InnterIneeiiim 2023.
Ronald B Tobias. 20 master plots: And how to build them. Penguin, 2012.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288, 2023.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mitoš. Focused transformer: Contrastive training for context scaling, 2023.
Cunxiang Wang, Sirui Cheng. Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang,
Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-QA evaluation. Thirty-seventh Conference on Neural Information P Processing Systems Datasets and Benchmarks Track, 2023.
URL htttps://ppereeiiiiee wwt.eeeiiwwweeeeeeeeeeee http://ww......rommmmddd-eeeeeee
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 0alo:0006.0766.. 2020.
13

---

Published as a conference paper at ICLR 2025
Y Wang, D Ma, and D Cai. With greater text comes greater necessity: Inference-time time training helps long text generation. arXiv preprint arXiv:2401.11004 2024a.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. 2024b.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: memory--ugmenned transformer for sequence modeling. In Findings of the Association for Computational Linguistics; AACL-IJCNLP 2022, November 2022a.
Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers.
arXiv preprint wxXv:22033008133 2022b.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. Lawformer: A pre-trained language model for chinese legal long documents. Al Open, 2:79-84, 2021.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023.
xverse. XVERSE 136-2
2023.
URL https://bcgginnfaccccc//eee
XVERSE- 138-566-3B
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering, 2018.
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-grounded interactive evaluation framework for large language models. arXiv preprint wwXi:22402.10043 2024.
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou, Soaring from 4k to 400k: Extending Ilm's context with activation beacon. arXiv preprint arX1v:2401.03462, 2024.
Wanjun Zhong. Lianghong Guo, Qiqi Gao, and Yanlin Wang. MemoryBan Bank: Enhancing large language models with long-term memory, arXiv preprint 102550 2023.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
Genre Limitation: NovelQA focus the novel genre. The contents of several popular novels might be mentioned in existing datasets, resulting in the risk of containing the correct answers of our copyright protected books, although this dataset has managed contain non-trivial details and reasoning questions to avoid this risk.
Language Limitation: NovelQA, and all associated data are exclusively in English. The following researches may consider to extend the language covered.
Truncation: Truncation standard practice for adapting longer texts fit within the constrained context windows of LLMs, especially when evaluating their performance on long-context tasks. While it's possible that some evidence may fall within the truncated parts.
B ETHICS STATEMENTS
We are dedicated to ensuring that NovelQA serves exclusively for academic and scientific endeavors. Our plan includes the launch of an evaluation website, leaderboard website, and the provision of an API for data access. As certain novels used in our project are protected by copyright, we affirm that we will not release these novels.
NovelQA does not contain any personally identifiable information or offensive content.
C APPENDIX
C.I RELATED WORKS
Long-Conters Language Modeling can be divided into several parts including efficient attention, preserving long-term memory like using KV cache, extrapolative positional embedding module, context pre/post-processingg
(Huang et al.] [023] Dong et al.) 2023b). Using efficient attention
reduce computational complexity and memory usage (Beltagy al. 2020 Wang et al.) [020
Tworkowski et al.] 2023] Li et al. 2023b). Preserving KV cache or context-level cache allows models to recall and leverage past information without reprocessing, enhancing coherence over long
texts (Chevalier et al.
2023 Wu et al.] [022albb Lin et al.) 2024 Zhong et al.] 2023] Chen et al.
2023a Hooper et al. 2024 Deploying
extrapolative positional embedding can extend beyond the sequence lengths seen during training (Chen et al. 2023b) Su et al. 2024).
the context can make models focus key information (Li al. 2023c liang et al. 2023) [Zhng
iet al. 2024: Han et al.) 2023). Moreover, some training methods are put forward to support further long-context language modeling development (Wang et al. 2024a| Press al. 202T). Since these models are more about exploring Long-context language modeling without scaling in data and model
size, the focus is on language modeling indicators, such as BLEU and Perplexity, NovelQA can serve as benchmark for evaluating those long-context language modeling methods.
C.2 DATA
Table 7: The Cohen's Kappa score in Inter Inter-Evaluator Agreement test different model outputs between human-evaluator and @PP4-as-evaluator Higher Cohen's 's Kappa scores indicate higher agreement between human-evaluators and the GPT-4-evaluator.
To Human Evaluator A
To Human Evaluator
Model
Avg.
GPT-4 Claude 2.1
91.97
95.84
93.91
88.04
90.00
89.02
InternL.M-Chat-1
91.88
86.53
84.88
InterL
85.68
84.08
87.25
Avg.
89.39
89.11
89.25
15

---

Published as a conference paper at ICLR 2025
Avg. Length Over 200K
Long Dependency
Require Reasoning
Details
Multiple Tasks
BAMBOO
X
X
LongBench
X
X
X
ZeroSCROLLS
X
X
X
V
LongICI.Bench
X
X
Counting Starts
X
L-Eval
X
X
LooGLE X
NovelQA
Figure 5: Comparison between NovelQA and other long-context benchmarks highlights several key dimensions. Long dependency refers to questions that require multi-hop evidence spanning extensive text resolve. Details pertains to questions that need answers with minimal contextual support. Reasoning involves questions without direct answers in the text, necessitating inference. NovelQA excels in these dimensions and features the longest context length among the benchmarks, catering to the current situation where number of LLMs have context length equal to over 200K.
Table 8: Selected question templates adopted data annotation. Question types include character (chara), meaning (mean), plot, relation (relat), setting (settg), times and span. '<>' label indicates the entity for annotators to fill in. Note that the annotators are also encouraged annotate questions beyond the given templates.
Aspect Template
Answer Format
Has the plot plot> happened in the novel? If so, how many times docs happen in the text?
Yes/No number
How many times has <a character> done <a doing-verb phrase> in the novel?
Yes/No + number
How many times have <a character> and <a character> <do somethings together novel?
Yes/No + number
times
How many times have characters and character> each other appear together)> the novel?
Yes/No number
How many times have character> and <a character> <communicate have verbal confficts with cach other> in the novel?
Yes/No number
Explain the meaning implication of the symbol metaphor <a symbol metaphor>
An explunation
mean
sentence, which appears the novel. In which chapter does there exist sentence in the novel <the novel> with the same or similar meaning as "ca sentence not from the original text>"? Please output the chapter name or index.
A chapter index title
<A character> is used to be <positive negative> and finally becomes <negative or positive> one in the novel. Tell in one sentence which episnde marks this character's change.
plot in one sentence
Who are mentioned with names in the <an organization, a family, or clubs in the novel?
A list of names
chara
Please list aliases designations of characters the novel.
aliases designations
Who is minor character, character without name, character that appears only
description character
the novel?
story take place in the novel? which year does carliest event happen, and which year does latest happen in the novel?
which ccities, countries>
of cities or countries range years
settg
What relationship between character> and alias a nickname of this character> the novel?
relationship
C.2.1 QUESTION TEMPLATES
Question templates adopted in the annotation procedure are presented in Table 8 The templates designed using a combination of literary theories and widely-used long-context tests on LLMs. During annotation, annotators required to fill in the missing entities (e.g., character, place, sentence) in the templates and are allowed to modify the expressions to suit specific questions. This approach balances the workload of annotation and ensures diverse set of questions.
16

---

Published as a conference paper at ICLR 2025
C.2.2 DATA CLASSIFICATION
Table 9: Data Distribution of NovelQA. Each question is labeled with Complexity and an Aspect label. The Complexity dimension comprises three categories, multi-hop, single-hop, and detail, while the Aspect dimension comprises seven categories including times, meaning, and so forth.
Dimension
Type
Percentage
Description
Example
Multi-hop
34.98%
Questions requiring knowledge across multiple paragraphs, even multiple chupters, to be solved.
Has Crusoe been on a voyuge? If so, how many times has this plot happened? in The Life Adventures of Robinson Crusse
Single-hop
42.74%
Questions roquiring knowledge from or sev- eral adjacent single sentences to be solved.
According to the Colonel, what did he smoke in McQuuenn's com- partment?? Munder ent Express
Complexity
Detail
22.19% Questions requiring knowledge from adjacent sentences to he solved. Detail questions are distinguished from the single-hop class by involving information that is too minor impuct other plots, making the details difficult to recall.
How many candles Madame Ma- glaire lighted when the Bishop had last dianer with Jean Valjean? ? Les Misérables
Times
20.07%
About the number of times that character, lo- cation, or plot uppears in the novel.
How many simes has Kitty kissed Walter? in The Painted Veil
Meaning
15.86%
The understanding of certain sentences or metaphors, c.g. interpret relationship certain metuphor and the novel itself. or find specific sentence according to a paraphrase pro- vided by annotator.
which chapter does there ex- a sentence with the or similar meaning as 'he Mar- quis responded, "You do me too much honor. In any case, I lean toward that assumption." "? in Tale of Two Cities
Span
1.47%
About the range of the novel setting. To be spe- cific. they either ask about the starting and end- ing yeur of the story require listing all cities countries that involved in the story. About the time place settings, besides those in the span type, are classified in this type.
In which year does the earliest event happen, and in which year does the latest even happen? in Tess of the d' d'Urbervilles
By Aspect
Setting
11.44%
Where did Diana's 's cousins leave for the Debating Club con- cert? in Anne of Green Gables
Relation
7.15%
About the relationship multiple character en- tities. To he specitic, they ask either about the relationship of a character and their alias or des- ignation, or about the relationship between dif- ferent characters.
What is the relationship be- Jean Voljean and Ul- time Fauchelevent? (designa- tion) Who are members of ABC frienuss? ? (membership) Les Misérables
Character
18.29%
About the information of characters. besides those in the relation type, ure classified into this type.
Who is Miss Beirne? Dublin- e75
Plot
25.62%
define a plot "some character does some- thing for once". Questions that ask about the information of any plots are classified into this type.
What does Clarissa repair in preparation the night's parry? in Mrs. Dallory
The data in NovelQA are classified into complexity levels and focus on 7 aspects. Table 9 presents detailed description of each class's criteria and examples.
C.2.3 PROMPTS
We present all the prompts involved in both test and evaluation phases in Table 11 The prompts generally follow the formula of zero-shot prompting. In each prompt, the model is firstly assigned the identity of a literature professor reviewing student answers. Then the prompt provides the model with detailed task description and clear specifications for the input and desired output.
C.2.4 THE LENGTH DISTRIBUTION OF EVIDENCES
Of the 3,666 evidence instances, 134 are summaries or comments without exact matches in the document. The distribution of the remaining 3,532 instances is as follows: Absolute distribution (by
17

---

Published as a conference paper at ICLR 2025
Table 10: Question types and their corresponding LLM abilities. Each type of question in NovelQA is designed to test one more abilities of LLMs, including the abilities to retrieve information, to identify details, and so forth.
Question Type
Abilities Required
Multi-hop
Ability to retrieve and integrate scattered information.
Single-hop
Ability to retrieve information and summarize.
Detail
Ability to precisely identify specifie, subtle details.
Times
Ability to retrieve facts, integrate facts, and reason.
Meaning
Ability to retrieve ambiguous information and reason.
Span, Setting, Relation
Ability to retrieve facts, integrate facts, and reason.
Character, Plot
Ability to subtle ambiguous information and reason.
absolute token index) can be found in Table 12 Approximately 18.4% of the evidence is located beyond the 130K token mark (outside GPT-4's context window), and 6.4% is beyond 210K tokens (outside Claude and InternLM's context windows).
Given the varying lengths of novels, we also analyzed the relative position distribution (by relative token position: token position / total tokens in the corresponding book) in Table
Our analysis reveals that the highest proportion of evidence instances 25.17%) occurs within the first 10% of the novels, while the distribution across the middle sections relatively uniform. This concentration in the early parts of the novels can be attributed to the initial introduction of characters and plot elements. The first occurrence of evidence related to these introductions naturally falls the earlier sections of the novels. It's important to note that during the question formulation process, we did not deliberately adjust the distribution of questions. The observed pattern in answer locations emerges naturally from the narrative structure of the novels.
C.3 MULTICHOICE DISTRACTORS GENERATION
Our pilot study compared distractors generated by both GPT-4 and Claude 2.1. Interestingly, GPT-4 generated slightly more challenging distractors both models scored approximately only 0.5% lower on GPT-4's distractors compared to Claude 2.1ss. Thus, we consider the bias caused by the model choice not significant in the distractors generation, and safely chose the choices generated by GPT-4.
C.4 EXPERIMENTS AND ANALYSIS
C.4.1 IMPLEMENTATIONS
We also tried to test Gemini-1.5 but the generation of 816 questions from 29 novels was blocked for unknown reasons (with only 14 of these novels under copyright protection). Consequently, we have decided not present Gemini's result.
Experiment Configs Given the cost of running long-context APIs, we request that each model respond to all questions for a given book in single session. To ensure fair comparisons, local- deployed LLMs also answer all questions for a book at once. We set 'temperature = 0' to eliminate randomness and keep other hyper-parameters default.
C.4.2 EVALUATOR MODEL
To prove that the model bias has little or no effect final results, conducted thorough analysis comparing different evaluator models to check their potential, demonstrated Table 15] The results show minimal variance between evaluators, with Claude-3.5 being slightly stricter but showing no significant model preference. This consistency is largely due to NovelQA's objective nature, with well-defined questions and answers that leave little room for evaluator bias.
/faah/
18

---

Published as a conference paper at ICLR 2025
Table 11: Prompts used in each setting of question answering. In each prompt, the model is assigned the identity of a literature professor reviewing student answers, with a detailed task description and clear specifications for the input and desired output. The angle brackets '<>' indicate the contents varying among each input.
QA Setting
Prompt
You are a literature professor. will provide you with full text novel along with a series of questions.
Please thoroughly analyze the novel's content accurately respond each of the following questions.
Book tide: <title;; Book Content: <content>; Book ends. Questions start here:
x x(uuestin:: <question>); Questions end here.
Generative
Try your to answer the questions based on the given novel full text.
The answer should be in short with only one several words.
Your outuu format should be 'Answertk Aonswer>AAsswerl <answer>... Answem: <answer>", each answer in one line without outputing the questions and other info.
You literature professor. I will provide you with the full text novel along with a series of questions and corresponding choices pertaining to it.
Pleuse thoroughly unulyze the novel content to accurately respond to each of the following questions.
Book title: <titl>; Book Content: <content>; Book ends. Questions start here:
MultiChaice
N xQuestion; <question>> Choices: O: <choicell> <choice >> <choice2s> 3: <choice3>); Questions end here,
Try your best to select the correct choice to each question based on the given full text the novel.
Your should output the choice to cach question with the format
AAnwwrO:; <choice> Answerl: <choice>... Answern: <choice>'
(only the choice index required), each answer in one line without outputing the questions and other info.
You are a literature professor. I will provide you a series of questions.
Please accurately respond to each of the following questions.
Book title: <title;; Book Content: <contents; Book ends.
Closebook- Generative
Questions start here: N (Question: <question>); Questions end here.
Try your best to answer the questions based on your own knowledge.
answer should in short with only one several words.
Your output formut should be 'AnswerO: Answer:-Answerl:: <answer>... Answem; <answer>', each answer in one line without outputing the questions and other info.
You are a literature professor. I will provide you a series of questions along with four choices for each question. Please accurately select the correct choice each the following questions.
Book title <title; Book Content: <content>; Book ends. Questions start here:
Closebook- MultiChoice
N x x(uesttin:: <question>> Choices: O: <choice0> I: <choicel>> 1> 2: <choice2> 3: <choice3>); Questions end here.
Try your best to answer the questions based on your own knowledge.
Your should output the choice each question with the format
AnswerO: <choice> Answerl: <choice>... Answern: <choiee>'
(only the choice index is required), each answer in one line w without outputing the questions and other info.
You literature professor reviewing student's quiz paper.
The question is about the novel <novel titlex: <question>. The related evidences from the novel are: <evidences>,
Correct ans is: <ea>. Student ans is: <su>.
Evaluating Generative
Piz check whether the student's 's ans is correct wit. the correct ans, and return "C" for correct and "N" for not correct. esp.. the student grabs the correct ans's meuning, return "C".
However, if there are factuality errors in student ans.
or the question requires a specific number but the student answers rough number, you should retum 'N".
Please only return the char or N w/o any other output.
You are a literature professor. I will provide you with the full text of novel along with a series of questions.
Please thoroughly analyze the novel's ss's content to accurately respond to each of the following questions.
Book titl: <title:: Book Content: <contents: Book ends. Questions start here:
x(Questinn: <question>); Questions end here.
Evidence Rocall
Try your best to answer the questions based on the given full text of the novel.
The answer should be in short with only one or several words. Your output format should be ÁnswerD: <answer>$ <evidences> Answerl: <answer>S <evidences>... Answern <answer>$ cevidencesoy each answer in one line with the supporting evidences.
Each evidence should be sentence exactly from the original text without any paraplurase.
You are a literature professor reviewing student's is evidence for their answer about novel <novel title>.
Question: <ques>. Correct answer: <ca>. Student answer: <sa>- Correct evidence: <oe>, Student evidence: <se>. You should evaluate the student evidence aspects:
Evaluating Evidence Recall
C) conrectness: whether the student evidence is the same with the correct evidence with similar correct meaning.
R) relevance: whether the evidence is relevant to the ans.
S) sufficiency: whether sufficient evidences are retrieved to support the ans.
And give score of 1-100 to only evidence (not the ans).
You should **only** return score numbers, e.g.in format C50R66S33, without any other outputs.
C.4.3 MULTICHOICE PERFORMANCE RELATED TO QUESTION TYPE
Besides the evaluation in generative settings, also prompted the models to collect their responses for the multichoice versioned questions. Table 14 presents the accuracies of four models in multi- choice settings in each question type.
19

---

Published as a conference paper at ICLR 2025
Table 12:
The distribution of evidences by absolute token index.
Range 0-20K
20-40K
40-60K
60-80K
80-100K
100-130K
130-170K
170-210K
>210K
Count 959
631
446
359
249
239
214
209
226
Percentage 27.15
17.87
12.63
10.16
7.05
6.77
6.06
5.91
6.40
Table 13: The distribution of evidences by relative token index.
Range
0-10
20
30
40
50
60
70
80
90
100
Count 889
431
360
326
288
271
274
250
205
238
Percentage
25.17
12.20
10.19
9.23
8.15
7.67
7.76
7.08
5.80
6.74
Table 14: Model Performance by Question Type in Multichoice Setting: This table details the accuracy scores of four models across different question types within the Multichoice setting of NoveIQA. Question types include character (chara), meaning (mean), plot, relation (relat), setting (settg), and others, with '-' indicating the absence of data for category. The table also provides an average score (avg) for each question category and model. Abbreviations used are dtl (details), mh (multi-hop), sh (single-hop).
GPT-4
(b) Claude 2.1
chara mean plot
relat
settg
span times
avg
chara mean
plot relat settg span times avg
mh 55.88 44.12 53.12 56.78 66.67 44.12 38.53 45.15 mh 71.88
76.47 85.00 70.69 78.26 51.52 47.71 58.17
sh 53.94 53.97 58.94 35.71 62.15
57.26
82.28
76.72 83.29 50.00 80.89
81.19
dtl
47.96 25.98 55.75 18.75 55.56
30.00
dtl
63.51
38.12 76.92 68.75 79.03
58.02
avg 52.86 37.36 57.70 47.56 60.98 44.12 38.53 49.18 avg
76.80
54.55 82,21 68.52 80.17 51.52 47.71 66.78
(c) InterLLM2-Chat-71
(d) 1nterLLM2-Chann
chara mean plot
relat settg span times
avg
chara mean
plot relat settg span times avg
42.19 38.24 45.90
65.23 39.39 42.92 43.87 mh 76.81 88 24 87.50 79.83 91.67 52.94 45.79 60.22
sh
44.44
39.34 44.56 28.57 48.15
44.23
86.27
88.10 92.03 57.14 87.01
88.64
dtl 52.63
26.24
55.14 31.25 59.68
41.54
dtl 69.39
30.10 85.84 53.12 80.95
57.62
avg 45.69
31.84
46.79
41.72 52.63 39.39 42,92 43.51
avg 80.81
55.46
90.36
72.73
85.98
52.94
45.79 71.80
C.4.4 MULTICHOICE PERFORMANCE RELATED TO EVIDENCE POSITION
Figure 6 which presents the relationships between the accuracy and the absolute or relative positions accordingly shows similar trends those observed in the generative setting. To be specific, the accuracy by absolute token position remains high when related evidences are before 100K's text length, while it drops after 100K. Meanwhile, the accuracy by relative position remains relatively even. A comparison is made between the accuracies within two ranges, (thethe lowest token count) 100K (namely pre-100K) and 100K to the end (namely post-100K), Figure and Table 4 present a clearer contrast between these two ranges, where the precision drops dramatically after the 100K token.
C.4.5 MULTI-HOP ACCURACY RELATED TO EVIDENCE DISTANCE
We also measured the relationship between the evidence distance within each multi-hop question and the accuracy under the generative task. For each multi-hop question, we obtain the distances among all evidences, and consider the max distance among them as the evidence distance. This can be interpreted as the model must memorize at least one of its evidences for the max distance to meet the final evidence in order to obtain the answer. The correlation between the evidence distance and the accuracy for multi-hop questions is demonstrated in Table Among the indices, Pearson correlation assumes that the two input distributions have linear correlation. Spearman correlation assumes that the two input distributions have monotonic correlation. Kendall correlation assumes that the two input distributions have ordinal correlation (in ranks). The results suggest that the max distance among evidence has negative correlation between the accuracy and the max distance among evidences, which can be interpreted as a lower accuracy is expected on the questions with more distantly distributed evidences.
20

---

Published as a conference paper at ICLR 2025
Table 15: Two evaluator models' (GPT-4o-mini and Claude-3.5-somme evaluations on the answers of these two models. The results show minimal variance between evaluators, with Claude-3.5 5 being slightly stricter but showing no significant model preference.
Evaluated By GPT-4o-mini
Evaluated By Claude-3.5-some
Claude-3.5-somi GPT-4o-mini
62.30
61.39
52.32
51.56
100
100
90-
90
80-
80
Accuracy (9)
70-
60-
60
Accuracy
50
40
gpt4
30
gpt4
claude
claude
internim 7b
20:
internim 7b
internlm20b
10
internim20b
OK 20K 40K 60K 80K100K 130K 160K 190K Token Position
0
20%
30%
40%.
50%
60%
70%
80%
90%
Percentile
Figure 6: Analysis of Accuracy in multichoice Setting by Absolute and Relative Token Positions: This figure illustrates the accuracy the multichoice setting NovelQA, plotted against the absolute position (left) and percentage position (right) of each question's 's relevant evidence within the novel. Each subplot represents different model. The x-axis of the absolute position figure (left), reflecting token indexes, is folded on the right due to the long tail distribution in the lengths of the selected novels.
C.4.6 REPRESENTATIVE ERRORS
Through reviewing the generation above, concluded three common error types: hallucination, miscounting, and overlooking. Examples for each type are presented in Table [7
Hallucination refers to the information generated by the model with factual errors. In our generative
QA setting, typical hallucination mistakes encompass two types: (I) factual errors about the fictional settings (e.g., mixing entities within the setting between the settings of different books) and (2) factual errors about the narrations (e.g., whether a fact is narrated). The first category usually appears in questions asking minor characters, plots, and settings, where the model might output non-existing one. Meanwhile, the second category is often associated with sentence-location questions, which ask the model to locate sentence. In this case, the model may fake sentence that does exist in the original text.
Overlooking refers the model's neglect of details. As mentioned above, the questions in detailed category involve minor characters, plots, or settings. Diving further, the reasons why these details are difficult be recalled lie in two aspects: (1) They do not contribute the character development, other plots, or the main themes, and thus reading the rest of the novel does not help to remind this detail; (2) Since most novels have derivative works (e.g., films, fan works, and book reviews), where the detailed information is eliminated to form a condensed narration. As the derivative works spread further and appear more frequently in the model's training data, they have higher probability of becoming the models' inner knowledge, which is similar to (Chang et al. (023)² 's observation, and vice versa for those omitted details. These two factors contribute to the difficulty in the model's recalling details and thus result in overlooking errors.
Miscounting Researchers (Li al. 2023a Feng et al. 2023) have revealed shortcomings in the counting ability of LLMs, especially nutoregressivv-decooe based models, and methods unfolding the outputs such as th-hougt prompting can enhance their counting ability. Our test does show that models make mistakes with numbers. Though the emror in the of generative responses
21

---

Published as a conference paper at ICLR 2025
135
18C
(a) GPT-4 65-100K
(b) GPT-4
(c) Claude 2.1 65-100K
(d) Claude 2.1 100+K
100+K
180
Ale
(e) InternLMM-hh 65-100K
( InternI
(g) InternI-M-20h
(h) InternL-M220h
100+K
150K
100+K
Figure 7: Performance Analysis by Relative Positions within Different Token Ranges: This figure illustrates the accuracy of various long-context LLMs in the multichoice setting, segmented by novels' length categories: 65h(the lowest token count) to 100K tokens and over 100K tokens. It highlights the models' performance trends relative to the evidence position within these ranges, showcasing the "lost-in-middle" phenomenon in shorter texts and a performance decline towards the end in longer texts, thus revealing distinct comprehension challenges faced by LLMs when processing texts of varying lengths.
Table 16: Correlation between the distance among multiple evidences in multi-hop questions and the accuracy in the generative scenario. The max distance among all evidence distances are considered the evidence distance of the question. The overall negative correlations show that the evidence distances are negatively correlated with the accuracies. The interpretation of matrices can be found in Appendix C.4.5
Model
Pearson
Spearson
Kendall
GPT-4
-0.0734 (0.0879)
-0,2511 (3.0696xe*) 099
-0.2031 (4.6129 xe -09)
Claude-3
-0.0120 (0.7899)
-0.1826 (4.6228)
-0.1385 (9.0407 xe -05)
InternLM-7bb
-0.1140 (0.0083)
-0.2287 (8.8721 xe -
-0.1841 (1.2663 xe -07)
InternLM-20b
-0.0634 14.14 -0.2017 (2.1954xx -06)
-0.1626 (2.722 008
may be due to not following instructions and simply outputting 'multiple times' instead of the desired specific times, the accuracy in the multichoice setting has still only reached 38.53% to 49.56% for the chosen four models, shown in Table [7 Even in the simplest question, which asks for the appearing frequency of certain phrases, the models still make mistakes.
22

---

Published as a conference paper at ICLR 2025
Table 17: Categories of representative errors observed in evaluating LLMs NovelQA.
Fault Type
Subtype
Setting
Example QA
Book: Mansfield Park
Q: What is relationship between Miss Maria Ward
Fictional Setting
GPT-4, generative
Lady Bertram?
Hallucination
Correct A: Miss Maria Ward and Lady Bertram are
same person.
Model A: Sisters.
Book: The Night Land
GPT-4,
Q: How many times has Aesworpth shouted?
On Narration
generative
Correct A: 1
Model A: Not mentioned.
Book: Light in August
Q: Who Percy Grimm?
GPT-4,
Overlooking
Correct A: Percy Grimm the captain of the State National
generative
Guard who kills Christmas and castrates him.
Model Percy Grimm does not appear in novel.
Book: Can You Forgive Her?
Q: Has the word phrase 'take away another man' appeared?
GPT-4, generative
Miscounting
If so, how many times does it appear in the text? Correct A: Yes, 2. Model Yes,
23


Reviews:
[{"review_id": "wSqc7ocvlh", "reviewer": "Reviewer_aWGC", "summary": "This paper introduces a new benchmark called NovelQA, which is designed to evaluate the performance of LLMs on extremely long and complex texts. NovelQA uses English novels as contexts and annotates questions across three levels of complexity and seven aspects by literature experts. The proposed benchmark surpasses existing ones in length, includes evidence alongside questions, and emphasizes detailed comprehension. This paper also presents experimental results from a suite of LLMs on this benchmark and analyses their performance across different aspects.", "questions": "1. How was the evidence annotated?\n2. It was mentioned in the paper that LLMs \"usually fail to tackle information spanning over multiple chapters.\" How does the proposed benchmark assess this aspect?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well-written and clear.\n2. The proposed NovelQA benchmark features the longest context to date\n3. The experiments are generally well-designed", "weaknesses": "1. The benchmark only focuses on novels, which is somewhat limited. It is essential to include other forms of long-context in order to provide a more comprehensive evaluation of LLMs' long-context understanding capabilities.\n2. The templates mainly focus on information extraction types of tasks. It would be beneficial to design more complex questions, e.g. ones \nthat require reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHLt4VlRUF", "reviewer": "Reviewer_E41W", "summary": "This paper presents a new dataset on long-context understanding. It uses novel as the long context and evaluate the model with questions from different complexity and aspect categories. Those questions and answers are annotated by human annotators, with a high inter-annotator agreement on answers. The answers are also with distractions for multi-choice setting generated from GPT-4. They evaluate open-source and commercial LLMs on the constructed dataset, revealing some findings such as performance degradation after 100K tokens, difficulty with questions on meaning, relation, span and times and low performance on evidences recall.", "questions": "1. How are the question types and evidence annotated?\n\n2. What does the distribution of the evidence positions look like?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a new dataset on long-context understanding. It has some unique features that are not covered by previous work, such as the long averaged context length, and human annotation efforts. It has the potential to be used in later work.\n\n2. The benchmarking results include some interesting findings, such as performance degradation after 100K tokens and low performance on evidence recall. Those findings may enlighten future research in this area,\n\n3. This paper is well-organized and easy to follow, with comprehensive appendix on the details of the data.", "weaknesses": "1. One major concern of this work is that this paper only focuses on using novels as the genre for long-context understanding. It would be better if the author could cover other long-context genres, such as other nonfiction books, to evaluate the long-context understanding comprehensively.\n\n2. Another concern I have is about the analysis of the performance based on the position of corresponding evidence. Because the annotation on the annotation does not include whether the evidence is unique in the long context, there may be some cases where the annotated evidence can also be found in another snippet of the context. And it may ultimately impact the analysis or the conclusion of the analysis based on the evidence position.\n\n3. As we can see in Table 1, most of the questions are either single-hop or details, which is generally based on a short text snippet of the long context. It would be also beneficial to include more discussion on the multi-hop questions, for example, the performance with regard to the relative distances of the pieces of evidence for a multi-hop question, as these questions are more difficult to be produced by LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrjSlrYfCz", "reviewer": "Reviewer_EUJu", "summary": "NovelQA is a new benchmark designed for evaluating large language models (LLMs) on complex, extended narratives with average context windows exceeding 200,000 tokens. This is relevant as the most advanced long-context LLMs can process over 250,000 tokens. NovelQA provides a combination of complexity, length, and narrative coherence, using a diverse selection of English novels from various eras, genres, and formats. Professional expert annotators carry out the annotation process, and all of them hold or are pursuing degrees in English Literature. The dataset includes multi-hop, single-hop, and detail questions, which assess the model's abilities to retrieve and integrate scattered information, summarize information, and accurately identify specific and subtle details. Both closed APIs and open models have been evaluated using this benchmark, and a comprehensive analysis is proposed based on the results.", "questions": "— Somehow the Table 7 and Figure 5 are above the Appendix section, and needs to be formatted.\n\n— No Table number in Line 1120\n\n— Consider using VLLM models to reduce running costs and time.\n\n— Better to write `multichoice` or `multi-choice` consistently during the paper\n\n— Add Human Baseline in Table 2\n\n— Write in bold the best scores in Tables, better readability", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "— Annotators performed a huge amount of work. The questions, golden answers, and evidence of NovelQA are crafted through the efforts of experts. The context is extensive, making it challenging to create such a set.\n\n— An informative, descriptive Appendix with details, annotation agreements, and error analysis.\n\n— The contribution is clear and high", "weaknesses": "— No ethical consideration and Limitations sections. The Limitation section is highly recommended, as mentioning the restrictions is important.\n\n— There are efficiency problems with the benchmark running for the long-context models. \n\n— To add in the limitations: data leakage. `To prevent against data leakage, we will not release golden answers for the test set, minimizing the risk of overfitting.`\nThe novels are not created from scratch. Many of them, particularly public domain works, are already included in the training datasets of models like GPT-4. This represents an indirect issue of data leakage. It is also crucial to discuss the problems related to data contamination and leakage.\n\n— The biases of the annotators also need to be mentioned in the Limitation section.\n\n— Truncation is also a limitation as nobody checked whether the truncated part influences the result.\n\n— The creation of templates based on the most difficult cases for GPT-4 and Claude2.1 is understandable but may cause biases.\n\n— Syntactic choices for the multiple choices:\n`We use GPT-4 to generate three distracting options for each question and its golden answer and randomly permute the four answers`\nAuthors further evaluate API models on the same sets: it will be with a high probability of a strong bias for GPT4 to answer the questions.\nThus, the authors can not claim that the GPT performs better than others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oEYmgelgr7", "reviewer": "Reviewer_Y5jR", "summary": "This paper presents a new benchmark to assess LLMs on reading and understanding very long inputs (200k in average). Concretely, the task is Question Answering (QA) in two forms, multiple-choice and generative, over novels. The dataset is constructed based on public and copyrighted novels from the Project Gutenberg (purchasing ones when necessary) and manually crafted questions, gold answers (distractors generated automatically but manually inspected), and evidence. Human annotators are Language and Literature students. The benchmark contains questions of different complexity (multi-hop, single-hop, and details) and about different aspects (e.g., relationships or narrative settings -- 8 categories in total). Five large-context LLMs are evaluated on the benchmark (3 commercial and 2 open source). Results reveal that LLMs struggle to read, find and process the content in the long context necessary to formulate the answer; this happens even with commercial LLMs (GPT-4) with open-source LLMs exhibiting worse performance. \n\nThe paper is easy to follow and with illustrative examples. It includes performance analysis per question type and by content position.", "questions": "The authors could incorporate experiments with the new Llama 3.1 family.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed benchmark is a valuable resource for the evaluation of LLMs performance on reading and reasoning to answer questions over long texts.", "weaknesses": "While the evaluation of LLMs shows that these struggle to understand, attend, and recall all the content from the long context necessary to answer questions (main focus of the paper), I wonder whether an initial extractive step would be a strong baseline in this setup. For instance, a common practice in summarisation of long inputs, is an initial extractive step (e.g., by tf-idf) that is applied before carrying out the actual task with the selected text. Maybe adding such a baseline would contribute to highlight the robustness of the proposed QA benchmark (i.e., how difficult is it to solve the task in this way). Also, some of the questions (e.g., counting) could be a LLM reasoning weakness rather than a long context processing issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4obehNVto8", "reviewer": "Reviewer_RTuY", "summary": "This paper introduces NovelQA, a benchmark designed to evaluate LLMs on long-context question-answering tasks. NovelQA addresses limitations in existing benchmarks by focusing on texts exceeding 200,000 tokens, derived from a diverse selection of English novels. NovelQA incorporates human-annotated questions that require detailed comprehension of lengthy narratives. The dataset evaluates multiple aspects of comprehension, including multi-hop reasoning, detail retrieval, and narrative coherence, revealing significant gaps in LLMs' ability to manage large, complex contexts.", "questions": "1. I’m curious about the performance of other open-source models, such as mistral, llama, and so on, or perhaps smaller closed-source models like GPT-4o-mini. Could you provide additional insights into how these models perform compared to the primary models evaluated in the paper?\n\n2. Regarding the use of gpt-4 as an evaluator, as noted above, this may introduce potential bias. Could you clarify how gpt-4 , or an alternative model, acted as an evaluator in the paper?\n\n3. Are the questions and sample sentences in the dataset accessible online for user verification in the future, or are they restricted and not publicly available for sourcing and comparison?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "NovelQA benchmark addresses an important challenge in long-context LLMs by introducing a dataset that emphasises extended context comprehension within complex narratives, representing a significant advancement over standard benchmarks. The dataset is carefully constructed with expert annotations, which are detailed in the paper, ensuring that question complexity aligns appropriately with model capabilities. The paper provides comprehensive documentation on question types and evaluation metrics, establishing a strong foundation for interpretability and contributing to a reliable benchmark. Last but not least, NovelQA evaluates narrative comprehension, multi-step reasoning, and retrieval-based tasks in LLMs, marking a substantial contribution to long-context evaluation.", "weaknesses": "Though the transparency of the annotation process is addressed, it could be strengthened. Although the authors state that questions are created by expert annotators with backgrounds in English Literature, the paper lacks clarity regarding annotator instructions, experience levels, and quality control measures above basic inter-annotator agreement scores — this could be demonstrated more clearly. Secondly, evaluation methods are insufficiently justified and may introduce bias due to the use of automated scoring. For instance, gpt-4 is used as an evaluator, with limited details on its evaluation process, which is particularly concerning given that the benchmark aims to capture complex reasoning across extended narratives.  eg. gpt-4 as an evaluator may score higher for gpt-4's answer. This could lead to bias in the results. While two human evaluators are mentioned, the paper does not provide information on their backgrounds or expertise. Other than these issues, the paper is well-written!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)