NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

## Abstract

Abstract The rapid advancement of Large Language Models (LLMs) has introduced a new frontier in natural language processing, particularly in understanding and processing long-context information. However, the evaluation of these models’ long-context abilities remains a challenge due to the limitations of current benchmarks. To address this gap, we introduce NovelQA, a benchmark specifically designed to test the capabilities of LLMs with extended texts. Constructed from English novels, NovelQA offers a unique blend of complexity, length, and narrative coherence, making it an ideal tool for assessing deep textual understanding in LLMs. This paper presents the design and construction of NovelQA, highlighting its manual annotation, and diverse question types. Our evaluation of Long-context LLMs on NovelQA reveals significant insights into the models’ performance, particularly emphasizing the challenges they face with multi-hop reasoning, detail-oriented questions, and extremely long input with an average length more than 200,000 tokens. The results underscore the necessity for further advancements in LLMs to improve their long-context comprehension and computational literary studies.

## 1 Introduction

Recent years have seen a remarkable surge in the development of Large Language Models (LLMs) (OpenAI, ; Touvron et al., ). Among these developments, long-context LLMs stand out for their ability to process and interpret extended pieces of text (Tworkowski et al., ; Team, ; Anthropic, ). This capability is essential for complex tasks that require a deep and nuanced understanding of lengthy documents, such as legal cases Xiao et al. () or academic papers Groeneveld et al. (), where the key is to understand extended narratives Xu et al. (). Meanwhile, the ability to analyze extremely long documents and multiple documents simultaneously is increasingly crucial, supporting more informed decision-making in various fields Deng et al. (); Lin et al. (); Boiko et al. ().

Figure: Figure 1: Trend of context window size of LLMs (Orange) and average token length of long-range benchmarks (Green). NovelQA is highlighted with a star.
Refer to caption: x1.png

However, evaluating the long-context capabilities of models presents challenges, as existing benchmarks (Yang et al., ; Tay et al., ) no longer align with the advanced processing abilities of current LLMs (Anthropic, ; Team, ; OpenAI, ). The disparity is further highlighted by the increasing context window size of LLMs, which now outpaces the average token lengths found in long-range datasets. This gap is evident, as the most advanced long-context LLMs are capable of processing over 250,000 tokens, a stark contrast to the longest average token length in current benchmarks, which is around 60,000 tokens. This mismatch underscores the need for updated evaluation methods that can accurately reflect the capabilities of current and future LLMs, as illustrated in Figure 1.

To fill this gap, we introduce NovelQA, a benchmark crafted to specifically evaluate LLMs’ performance on texts with context windows exceeding 200,000 tokens. Unlike existing benchmarks (Shaham et al., ; An et al., ; Adams et al., ), NovelQA addresses the need for assessing extremely long-context understanding, offering a refined and comprehensive tool for advancing natural language processing capabilities.
We construct NovelQA based on novels in English, which are also ideal for testing long-context modeling because they are long and complex, with plots that are closely linked from start to end. We select novels from various eras, genres, and formats to enhance diversity. The annotation process is performed by a group of expert annotators, all of whom are holding or pursuing a degree in English Literature and have a strong interest in and familiarity with the novels they annotate.
Each question is paired with a ‘golden answer’ and corresponding textual evidences from the novels. And we categorize them by complexity and aspect for detailed analysis.
Figure 2 presents two examples, while Table 6 details the distribution of question types. The dataset includes multi-hop, single-hop, and detail questions, which test the model’s abilities to retrieve and integrate scattered information, retrieve information and summarize, and precisely identify specific and subtle details, respectively. The above capabilities are more challenging in a particularly long context. For example, finding relevant information is a particularly big challenge.

Figure: Figure 2: Illustrative Examples from NovelQA: This figure showcases two sample questions. For each question, models are evaluated under two distinct settings – multi-choice, where the task is to select the correct answer from four options, and Generative, where the model generates an answer.
Refer to caption: x2.png

Using the dataset, we assess various long-context LLMs, including commercial models such as GPT-4-128K (OpenAI, ) and Claude 2.1-200K (Anthropic, ), alongside open-source models like InternLM2-Chat (Team, ).
The primary findings offer a significant insight: even the most advanced long-context LLMs face challenges in consistently extracting and processing accurate information from extended texts.
For example, GPT-4, the top performer, achieves a 46.88% accuracy rate, whereas the open-source InternLM2-20b achieves 32.37% in a generative setting.
In addition, some models are unable to answer properly when the context exceeds 100K tokens, even if their context windows are much longer (xverse, ; 01-ai, ). Moreover, operating LLMs on inputs exceeding 200,000 tokens presents technical challenges, particularly regarding memory requirements and associated costs.
The difficulty of valid models is particularly apparent in answering multi-hop questions and queries that probe meanings, relationships, spans, and timelines, highlighting a significant gap in the models’ long-range comprehension.
Moreover, our data shows a decline in performance for evidence situated beyond the 100,000-token mark, including information at the novel’s end, diverging from the anticipated lost-in-middle phenomenon (Liu et al., ).
This shift suggests a distinct challenge faced by LLMs when processing texts exceeding 100K tokens in length.
These results highlight challenges not only in memory optimization but also in the nuanced comprehension and integration of lengthy texts, indicating a substantial obstacle on the path to truly effective long-context LLMs.

To the best of our knowledge, NovelQA is the first long-context QA benchmark featuring manually crafted questions, golden answers, and evidences, with contexts extending beyond 200,000 tokens. (^1^11We have released the demonstrations and input of the testset, and created a leaderboard. More details can be found in [https://github.com/NovelQA/novelqa.github.io](https://github.com/NovelQA/novelqa.github.io). And NovelQA is released under the Apache-2.0 License.)

## 2 Related Work

Long-Range Benchmarks
Evaluating the ability of Long-Context Pretrained Language Models has been a hot topic (Tay et al., ; Shaham et al., ; Pang et al., ; Wang et al., ; Yu et al., ). When entering the era of Large Language Models, the context window length has been much longer than ever, and the decoder-only LLMs have been the mainstream of language models and they have faced specific problems, such as the Lost-in-middle issue (Liu et al., ). Thus, increasing benchmarks are created for evaluating long-context LLMs. Among those benchmarks, several are representatives.
The *Needle in A Haystack* test (gkamradt et al., ) involves inserting an unrelated sentence into a long context and asking the model to retrieve it. However, this test does not represent a natural language question answering task that requires composite and complex capabilities for processing long contexts.
ZeroSCROLLS (Shaham et al., ), LooGLE (Li et al., ), Counting-Stars(Song et al., ), and LongICLBench(Li et al., ) emphasize the importance of understanding and aggregating information from long texts, presenting challenges that highlight areas for improvement in current models. L-Eval (An et al., ) introduces a suite of tasks with human-labeled query-response pairs to assess LLMs’ performance in processing long inputs effectively, using advanced metrics for a more accurate evaluation.
Furthermore, benchmarks like LongBench (Bai et al., ), BAMBOO (Dong et al., ), LongBench-Chat (Bai et al., ) offer a diverse set of tasks across languages and domains, from reasoning and coding to summarization and multilingual translation. These benchmarks are designed to rigorously test the ability of LLMs to manage extensive contexts, with LongBench-Chat specifically focusing on instruction-following capabilities in long-context interactions.
Additionally, LongHealth (Adams et al., ) addresses the need for LLMs to interpret long clinical documents accurately, providing a specialized benchmark for evaluating models on medical texts. This focus on domain-specific challenges underscores the broader necessity for LLMs to not only handle long texts but to do so in a manner that is accurate and contextually relevant across various fields.

NovelQA distinguishes from existing benchmarks in three points. Firstly, NovelQA features an average token length exceeding 200,000, far surpassing the tens of thousands typically found in other benchmarks. Secondly, while other datasets often rely on LLMs or existing datasets for content creation, the questions, golden answers, and evidences of NovelQA are entirely crafted through human effort. We present a detailed comparison between NovelQA and other benchmarks in Appendix Fig 4.

We also discuss related Long-Context Language Modeling methods in Appendix Sec 4.

## 3 Data

### 3.1 Dataset Description

Data Formulation Every novel ($N$) in the dataset corresponds to multiple pieces of annotated data ($d_{i}$). Each piece of data consists of the following domains, *question* ($Q_{i}$), *answer* ($A_{i}$), *multi-choices* ($a_{i,0}$, $a_{i,1}$, $a_{i,2}$, and $a_{i,3}$), *gold label* ($a_{i,gold}$), *evidences* ($s_{i,0}$, $s_{i,1}$, …), and *type* ($Complx_{i}$ and $Aspect_{i}$), among which the *answer* domain is a short answer to the question, while the *multi-choices* and *gold label* domain forms multi-choices, indicating that our dataset can serve to the evaluation of both the generative task and the multi-choice task. In the generative setting, a novel text $N$ and a question $Q_{i}$ are combined to send into the model each time, and the generated answer is compared with the answer $A_{i}$. In the multi-choice setting, the novel $N$, a question $Q_{i}$, and the four choices $a_{i,0}$ to $a_{i,3}$ are sent into the model, and the output is evaluated according to the gold label $a_{i,gold}$. Meanwhile, the *evidences* domain consists of either the original excerpts from the novel, or the reasoning steps written by the annotator.

Book Selection
We source public domain novels from Project Gutenberg, and purchase e-books from internet if it is necessary.
We aim to enhance diversity by selecting novels from various eras, genres, and formats. Recognizing that some newer, popular novels are still under copyright protection, NovelQA inevitably includes some copyrighted novels, comprising a mixture of 65 novels in public domain and 24 ones in copyright-protection. (^2^22We evaluate all annotated questions in this paper and conduct analysis. For the public access, we have released all constructed data. However, we only release public domain novels. Therefore, we offer two types of metrics in the evaluation system and leaderboard: one for evaluating QAs within public domain novels and another for evaluating QAs across all novels.) All selected books exceed 50K words (approximately 67k tokens) and are in English. A visualization of the distribution of book token lengths is provided in Appendix Sec D.2.

Question Distribution
The annotated questions can be classified by the complexity of solving the question and the aspect that the question focuses on. By complexity, The data are categorized into three complexity levels, multi-hop (35.0%), single-hop (42.8%), and detail (22.2%). The order of complexity is as follows: multi-hop > detail > single-hop. By the aspect that each question focuses on, the data entails seven types. A detailed specification of each type is listed in Appendix D.3. We have a total of 2305 questions, of which 1640 are from 65 public domain novels, while the remaining 665 are from 24 copyrighted novels. According to the classification above, the distribution of the questions in our dataset is displayed in Appendix Table 6.
We also list the ability tested by each kind of question in Appendix Table 8.

### 3.2 Data Collection and Annotation

Procedure Overview The annotation process is performed by a group of expert annotators.The annotation procedure consists of two phases: (1) Template-based phase: The annotators can fill entities into 10+ templates (see Appendix Sec D.1) that we design to be related to multi-hop or detailed information. This phase entails half of the data, mainly contributing to the multi-hop ones. (2) Free-formed phrase: To ensure the diversity of question expression and align the questions to the natural distribution, our second half of the data is annotated without a template, namely, the annotators contribute to any difficult questions that they come up with freely.

Annotator Recruitment
Our annotators are predominantly English Language and Literature university students or those with a keen interest in English novels, recruited from local universities. These students are accustomed to closely reading English novels for coursework or as a hobby, and writing reports or papers on them. Annotators are allowed to select novels for annotation based on their familiarity, ensuring they had previously read and comprehensively understood the texts. Meanwhile, we make sure the selected books meet our standards of enough word count and well-developed narratives. We also ensure that the each selected novel is either annotated by only one individual, or consistent in version across annotators, despite minor variations among different editions. Each annotator contribute to a typically small number of 20-30 questions per novel. This approach avoids forcing annotators annotating questions on unfamiliar content.

Time Consumption and Rewards
Given the annotator’s familiarity with their chosen novels and their experience with similar questions in their academic assignments, creating questions based on their knowledge becomes a manageable task within a reasonable time cost.
The annotation reward is of $1.11 to $1.39 per tuple. As an average annotator can write 5 to 6 pieces of data at full speed according to our observation, the $5.56 to $8.34 hourly wage is above the local legal minimum wage of $2.78/hour. The annotation process costs around $3,500.

Template Design The first annotation phase relies on a question template, which requires the annotator to fill in the entities from the novel to form valid questions. To design templates, we carried out sufficient pre-tests on GPT-4 and Claude2.1 to analyze their possible weaknesses in long-input QA and novel knowledge memorization. Our pre-test shows that they usually fail to tackle information spanning over multiple chapters, as well as lack attention to details that have no contribution to the main theme. Meanwhile, we also refer to around fifteen books on novel and narration theories (e.g. Forster, ; Tobias, ; Schmidt, ; McKee, ) to ensure our template covers more aspects that a novel can discuss (e.g., character, setting, theme).

Quality Control The created data is manually double-checked by three authors of this work. The review procedure follows the criteria of minimizing factual errors, enhancing expression clarity, and maintaining challenges to LLMs. Consequently, only 79.4% of the collected data are preserved, resulting a final dataset of 2305 QA tuples. Meanwhile, we have also conducted the inter-annotator agreement (IAA) test on the multi-choice versioned QA annotation on 100 cases, shows a score of 94.6% in Cohen’s Kappa, indicating a high agreement among annotators. We present the details of the IAA test in the Appendix Sec D.4.

Distractions for Multi-choice Setting
We use GPT-4 to generate three distracting options for each question and its golden answer and randomly permute the four answers. And we check those distractions and rewrite those with similar meaning with the golden answers manually when we double check the data.

### 3.3 Advantages

Our NovelQA dataset serves as a new benchmark for evaluating long-context understanding, distinguished by several key advantages.
Firstly, it surpasses existing benchmarks in length, offering a rigorous test of a model’s ability to navigate and comprehend significantly longer texts.
Secondly, the inclusion of clear evidences alongside questions ensures that evaluations are grounded in concrete textual support, enhancing the reliability of assessments.
Furthermore, the dataset emphasizes questions that require attention to detailed information, challenging models to move beyond superficial impressions to extract specific, nuanced answers.
Questions, golden answers, and evidences of the dataset are entirely manually annotated and carefully checked, ensuring high-quality, nuanced questions and answers that reflect complex human thought processes.
To prevent against data leakage, we will not release golden answers for the test set, minimizing the risk of overfitting.
These features, combined with the dataset’s comprehensive coverage of diverse narratives and meticulous construction, make NovelQA a valuable resource for advancing long-context understanding.

## 4 Experiments

**Table 1: Evaluation of Long-Context LLMs on NovelQA. This table presents the performance of four long-context LLMs, including both commercial models (GPT-4, Claude 2.1 and Claude 3 Sonnet) and open-source, locally deployed models (InternLM2-Chat-7b/20b). Accuracy percentages are reported under two testing scenarios: multichoice and generative. The Max Length column denotes the maximum token length of each model.**
|  | Max Length | Multi-choice | Generative |
| --- | --- | --- | --- |
| GPT-4 | 128K | 71.80 | 46.88 |
| Claude 3 | 200K | 71.11 | 53.66 |
| Claude 2.1 | 200K | 66.84 | 46.04 |
| InternLM2-Chat-7b | 200K | 43.51 | 30.90 |
| InternLM2-Chat-20b | 200K | 49.18 | 32.37 |

**Table 2: Model Performance by Question Type in Generative Setting: This table details the accuracy scores of four models across different question types. Question types include details (dtl), multi-hop (mh), single-hop (sh), character (chara), meaning (mean), plot, relation (relat), setting (settg), others, and an average score (avg) for each category, with ’-’ indicating the absence of data for a category.**
| (a) GPT-4 | (b) Claude 2.1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | chara | mean | plot | relat | settg | span | times | avg |  | chara | mean | plot | relat | settg | span | times | avg |
| mh | 57.81 | 61.76 | 52.46 | 45.30 | 56.52 | 18.18 | 21.23 | 32.83 | mh | 48.94 | 64.29 | 58.18 | 40.00 | 82.61 | 18.52 | 17.10 | 30.34 |
| sh | 66.12 | 56.56 | 69.33 | 21.43 | 57.23 | - | - | 63.93 | sh | 72.41 | 55.24 | 67.96 | 23.08 | 59.60 | - | - | 65.47 |
| dtl | 52.63 | 12.87 | 61.68 | 37.50 | 58.06 | - | - | 37.58 | dtl | 55.22 | 12.43 | 66.30 | 27.59 | 55.77 | - | - | 37.65 |
| avg | 62.04 | 32.40 | 65.93 | 41.72 | 57.38 | 18.18 | 21.23 | 46.88 | avg | 65.90 | 31.61 | 66.60 | 35.61 | 61.06 | 18.52 | 17.10 | 46.04 |
| (c) InternLM2-Chat-7b | (d) InternLM2-Chat-20b |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | chara | mean | plot | relat | settg | span | times | avg |  | chara | mean | plot | relat | settg | span | times | avg |
| mh | 23.81 | 38.24 | 42.62 | 24.35 | 39.13 | 15.15 | 21.32 | 24.62 | mh | 32.84 | 44.12 | 29.03 | 37.61 | 25.00 | 15.15 | 26.81 | 29.29 |
| sh | 35.80 | 28.10 | 42.18 | 14.29 | 32.10 | - | - | 36.42 | sh | 42.97 | 34.17 | 43.22 | 21.42 | 36.90 | - | - | 40.57 |
| dtl | 26.67 | 9.18 | 55.14 | 29.03 | 34.43 | - | - | 27.02 | dtl | 30.93 | 7.00 | 48.21 | 25.00 | 29.03 | - | - | 24.65 |
| avg | 32.02 | 18.52 | 44.77 | 24.38 | 33.33 | 15.15 | 21.32 | 30.90 | avg | 38.50 | 19.77 | 42.66 | 33.74 | 33.86 | 15.15 | 26.81 | 33.07 |

We focus on long-context models meeting three criteria: a context window of at least 128,000 tokens, accessibility via a full API or public release, and chat functionality. For commercial models, our selection includes GPT-4-128K (OpenAI, ) and Claude 2.1-200K (Anthropic, ). Among open-source options, we evaluated models like InternLM2-chat (Team, ).

### 4.1 Implementations

Settings To thoroughly test the abilities of these LLMs, we employ two evaluation settings: a generative setting where models directly generate short answers, and a multi-choice setting with four provided options.

Prompts
We use uniform prompts for all LLMs, with a start part, novel content, questions, choices in the multi-choice setting, and end part. The prompt structure is shown in Appendix Table 9.

Truncation Due to input length limitations, we truncated the novel content from the end to the front following Bai et al. (); Li et al. (); An et al. (), to meet with the max input length, while keeping questions and other prompts complete.

Evaluating Generative Results: Following the findings in Wang et al. () which highlight GPT-4’s proficiency in assessing the accuracy of short machine-generated answers, we employ GPT-4 (gpt-4-0125-preview) for the evaluation of generative responses in our study, which is also applied in other long-range benchmark studies (An et al., ; Li et al., ). We further conducted a human evaluation on 800 pieces of generative outputs and carried an inter-evaluator agreement test between two human evaluators and the GPT-4 evaluator. The result of 89.25% in Cohen’s Kappa indicates a high agreement towards the GPT-4 evaluating results (details in Appendix D.4).

Commercial LLMs:
The APIs of commercial LLMs utilized are gpt-4-0125-preview(^3^33[https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo)), Claude 2.1 and Claude 3-sonnet(^4^44[https://aws.amazon.com/bedrock/claude/](https://aws.amazon.com/bedrock/claude/)).

Open-source LLMs:
Running long-context LLMs on extremely long inputs, such as 200K tokens, is a challenge due to the immense GPU memory required, for example, it takes roughly 2.5T memory to calculate one attention matrix for a 7B model with a 200K-token input, while our local device is a 4 $\times$ 80G A100.
To address this, we utilize the LMDeploy (Contributors, ) (based on Dynamic NTK emozilla ()) for memory reduction, which is only compatible with several LLMs. Therefore, we choose InternLM2-Chat-7b-200K and InternLM2-Chat-20b-200K for our experiments.
Attempts to run models like Yarn-Mistral-7b-128K (Peng et al., ), chatglm3-6b-128K (Zeng et al., ), XVERSE-13B-256K (xverse, ) and Yi-6B-200K (01-ai, ) on inputs exceeding 100K tokens are unsuccessful. The challenges range from GPU memory overflow due to incompatibility with LMDeploy, to models imitating the input content, or continuing to generate following the inputs without answering the questions. However, reducing input lengths to under 50K tokens can normalize most outputs, following the given instructions. This situation underscores the current limitations in technical support for processing and generating content based on extremely long contexts.

Additional Details Given the cost of running long-context APIs, we request that each model respond to all questions for a given book in a single session. To ensure fair comparisons, local-deployed LLMs also answer all questions for a book at once. We set ‘temperature = 0’ to eliminate randomness and keep other hyper-parameters default.

Human Performance: As most of the annotators are from the same university and share common courses and projects, several books have been read by more than one student. We selected such books and have had the readers engaged in a two-round answering process on novels they had not previously annotated. The first round was in a generative setting, and the second round was in a multichoice setting. This process was conducted on 5 novels with a total of 100 questions. The result shows that human performance scored 90 in the generative setting and 97 in the multiple-choice setting.

### 4.2 Main Results

We present the main results in Table 12.
Even the highest scores (71.80% and 46.88% for GPT-4 in generative and multi-choice settings, respectively) suggest there is considerable room for improvement in long-context understanding compared with human readers. This is especially true in the generative setting where understanding and recall over long contexts are more challenging.
Additionally, commercial models (GPT-4 and Claude 3/2.1) outperform open-source models (InternLM2-Chat-7b and 20b) in this benchmark.
All models show a drop in performance in the generative setting compared to the multi-choice setting. This indicates the increased challenge in generating a correct answer from scratch, as opposed to selecting from provided options.

We have also observed three typical errors, hallucination, overlooking, and miscounting, a detailed analysis is conducted in Appendix Sec E.4. In addition, we have also test the extent to which models rely on the content of novels versus using their internal knowledge to answer questions, and the details can be found in Appendix Sec E.5.

### 4.3 Results by the Question Type

An in-depth analysis of model performance across question types reveals nuanced insights into their comprehension abilities in both generative and multi-choice settings, detailed in Table 2 and Appendix E.2, respectively.
This analysis not only highlights the models’ weaknesses across different formats but also illuminates the challenges in narrative comprehension, contributing to both NLP and computational literary research.

The examination of accuracy scores across question categories such as character, meaning, plot, relation, and setting highlights distinct patterns in performance, pointing to the models’ differential capabilities and limitations. Notably, models exhibit particular difficulty with questions centered around meaning, relation, span, and times. This difficulty suggests several underlying challenges:

(1) Meaning Questions: The struggle with meaning questions indicates a challenge in grasping abstract concepts and locating entities or sentences through interpretations within the text, which requires an advanced level of semantic understanding and inference.

(2) Relation Questions: Difficulty with relation questions points to a gap in the models’ ability to identify and interpret the dynamic and often nuanced relationships between characters, events, or concepts, which are crucial for a holistic understanding of narratives.

(3) Span and Times Questions: The lower performance on span and times questions suggests a limitation in tracking temporal sequences and spatial extents within the narrative, reflecting challenges in maintaining and applying contextual information over long stretches of text.

The above findings underscore a critical aspect in both computational literary studies and long-context comprehension of LLMs—while models are adept at handling certain types of narrative questions, they encounter significant hurdles when required to synthesize abstract concepts, interpret complex relationships, or maintain a coherent understanding of temporal and spatial narratives over long context. These can be the domains requiring further improvement to enhance narrative comprehension and reasoning capabilities.

Figure: Figure 3: Analysis of Accuracy in Generative Setting by Absolute and Relative Token Positions: The two figures above illustrates the accuracy, plotted against the absolute token indexes (left) and the percentage position (right) of each question’s relevant evidences in the novels. The x-axis of the absolute token position figure (left), reflecting token indexes, is folded on the right due to the long-tails.
Refer to caption: x3.png

### 4.4 Results by the Position

Our analysis delves into how the positioning of evidence within novels affects the accuracy of long-context LLMs. Specifically, we explore the impact of both absolute and relative positions of evidence, where the absolute position refers to the specific token index within the text, and the relative position is normalized against the total length of the novel, scaled to a 0%-100% range.

Absolute Position Analysis: In the generative setting, as depicted in Figure 3 (left), all evaluated models show improved performance on questions where the necessary evidence is located before the 100K token mark. This trend highlights a challenge for LLMs in accessing and processing information beyond this threshold, suggesting a diminished capacity to handle very long inputs. The multi-choice setting, detailed in Appendix E.3, follows a similar pattern, reinforcing the importance of evidence position in model performance. We also present the relationship between the accuracy and the absolute position of evidence, grouping by pre-100K and post-100K, in Appendix Table 10.

Relative Position Analysis: By normalizing the evidence positions within the entire novel, we aim to understand if the proportional location of evidence influences model accuracy. This analysis, shown in Figure 3 (right) for the generative setting and in the Appendix E.3 for the multi-choice setting, indicates that models maintain relatively consistent performance across various relative positions, suggesting that long-context LLMs’ effectiveness is not significantly affected by the evidence’s relative position within the standardized length of novels.

To delve deeper into how long-context LLMs navigate extremely long inputs, we segment novels into two categories based on length: 65k-100K and over 100K tokens. We then examine model accuracy in relation to relative evidence positions within these ranges, with results presented in Appendix Figure 7. For novels within the 65k-100K token range, we observe a lost-in-middle phenomenon in GPT-4, InternLM2-7b, and InternLM2-20b, akin to findings by Liu et al. (). This pattern indicates stronger performance at the beginning and end of texts but weaker in the middle. Conversely, in novels exceeding 100K tokens, model performance generally declines towards the end, potentially due to the scarcity of training data for contexts of this length. This behavior underscores a unique challenge faced by LLMs when processing exceptionally long texts over 100K tokens.

**Table 3: Evidence recalling results of four LLMs: During the evidence recall test, each model is required to output the relevant evidence from the original novel contents. Each recalled piece of evidence is further compared by GPT-4, which has been proven to generate reliable evaluations, with the gold standard evidence. The scores for each model, assessed in three dimensions, Correctness, Relevance, and Sufficiency, are listed below.**
|  | Correctness | Relevance | Sufficiency | Avg. |
| --- | --- | --- | --- | --- |
| GPT-4 | 29.47 | 38.05 | 27.67 | 31.73 |
| Cluade 2.1 | 23.51 | 29.08 | 22.27 | 24.95 |
| InternLM2-Chat-7b | 2.05 | 3.88 | 1.90 | 2.61 |
| InternLM2-Chat-20b | 6.36 | 12.51 | 7.50 | 8.79 |

### 4.5 Evidences Recall

We evaluate the ability to recall evidence, prompting the four models above to answer the questions in NovelQA again, with printing the supporting evidence simultaneously. We then prompt GPT-4 with the generated evidence alongside the annotated evidence to obtain its evaluation on the quality of retrieved evidence pieces. The evaluating matrix consists of the following three dimensions: *correctness* refers to whether the retrieved evidence is the same as the annotated evidence or with a similar correct meaning; *relevance* indicates whether the evidence is consistent with the answer; *sufficiency*, whether the retrieved pieces of evidence are enough to support the answer. Each dimension is scored between 0 and 100 and an average score is further obtained through calculating the algorithmic mean on these three dimensions. Prompts involved in this evaluation procedure are presented in Appendix D.5.

The results, detailed in Table 3, show higher performances of GPT-4 and Claude 2.1. Moreover, though the scoring range is from 0 to 100, the four models all perform with low scores in evidence recall, possibly because the models do not always follow the instructions after inputs and outputs of such an above-average length, resulting in a high percentage of 0 scores. This phenomenon is particularly severe for InternLM-Chat-7b and InternLM-Chat-20b models, whose outputs consist of a large proportion of invalid placeholders. Still, current long-context LLMs generally demonstrate inadequate abilities recalling the correct and supportive evidences from the context.

## 5 Conclusion

We introduced NovelQA, a long-range benchmark designed to assess the long-context comprehension abilities of LLMs. Utilizing representative English novels, NovelQA presents LLMs with the challenge of navigating complex, real-world texts. Our evaluations reveal that both commercial and open-source models face difficulties with detailed understanding, multi-hop reasoning, and accurately retrieving specific information from lengthy contexts, especially for lengths are 100,000. Moreover, operating LLMs on inputs exceeding 200,000 tokens faces technical challenges, notably in terms of memory requirements and associated costs.

## Appendix

## Appendix A Limitations

Access to Close-source LLMs: One significant limitation is our inability to obtain APIs for certain close-source long-context LLMs, such as Baichuan-192K, GLM4-200K, and Moonshot-192K.

Technical Constraints for Open-source LLMs: The majority of open-source LLMs are not optimized for processing inputs exceeding 128K tokens with high GPU memory cost. This technical challenge renders direct inference on NovelQA infeasible for these models. Consequently, our experiments were limited to LLMs that have been adapted using LMDeploy.

Language Limitation: NovelQA, and all associated data are exclusively in English.

## Appendix B Ethics Statements

We are dedicated to ensuring that NovelQA serves exclusively for academic and scientific endeavors. Our plan includes the launch of an evaluation website, a leaderboard website, and the provision of an API for data access. As certain novels used in our project are protected by copyright, we affirm that we will not release these novels.

NovelQA does not contains personally identifiable information or offensive content.

## Appendix C Related Work

Figure: Figure 4: Comparison between NovelQA and other long-context benchmarks highlights several key dimensions. *Long dependency* refers to questions that require multi-hop evidence spanning extensive text to resolve. *Details* pertains to questions that need answers with minimal contextual support. *Reasoning* involves questions without direct answers in the text, necessitating inference. NovelQA excels in these dimensions and features the longest context length among the benchmarks, catering to the current situation where a number of LLMs have a context-length equal to or over 200K.
Refer to caption: x4.png

Long-Context Language Modeling
can be divided into several parts including efficient attention, preserving long-term memory like using KV cache, extrapolative positional embedding module, context pre/post-processing Huang et al. [], Dong et al. [].
Using efficient attention can reduce computational complexity and memory usage Beltagy et al. [], Wang et al. [], Tworkowski et al. [], Li et al. [].
Preserving KV cache or context-level cache allows models to recall and leverage past information without reprocessing, enhancing coherence over long texts Chevalier et al. [], Wu et al. [, ], Lin et al. [], Zhong et al. [], Chen et al. [], Hooper et al. [].
Deploying an extrapolative positional embedding to extend beyond the sequence lengths seen during training Chen et al. [], Su et al. [].
Pre/post-processing the context can make models focus on key information Li et al. [], Jiang et al. [], Zhang et al. [], Han et al. [].
Moreover, some training methods are put forward to support further long-context language modeling development Wang et al. [], Press et al. [].
Since these models are more about exploring long-context language modeling without scaling in data and model size, the focus is on language modeling indicators, such as BLEU and Perplexity, NovelQA can serve as a benchmark for evaluating those long-context language modeling methods.

## Appendix D Data

**Table 4: The Cohen’s Kappa score in Inter-evaluator Agreement test on different model outputs between human-evaluator and GPT4-as-evaluator. Higher Cohen’s Kappa scores indicate higher agreement between human-evaluators and the GPT-4-evaluator.**
| Model | Human<br>Evaluator A | Human<br>Evaluator B | Avg. |
| --- | --- | --- | --- |
| GPT-4 | 91.97 | 95.84 | 93.91 |
| Claude 2.1 | 88.04 | 90.00 | 89.02 |
| InternLM-Chat-7b | 91.88 | 86.53 | 84.88 |
| InternLM-Chat-20b | 85.68 | 84.08 | 87.25 |
| Avg. | 89.39 | 89.11 | 89.25 |

### D.1 Question Templates

**Table 5: Selected question templates adopted in data annotation. Question types include character (chara), meaning (mean), plot, relation (relat), setting (settg), times and span. ‘<>’ label indicates the entity for annotators to fill in. Note that the annotators are also encouraged annotate questions beyond the given templates.**
| Aspect | Template | Answer Format |
| --- | --- | --- |
|  | Has the plot <a plot> happened in the novel? If so, how many times does it happen in the text? | Yes/No + number |
| times | How many times has <a character> done <a doing-verb phrase> in the novel? | Yes/No + number |
| How many times have <a character> and <a character> <do something> together in the novel? | Yes/No + number |  |
| How many times have <a character> and <a character> <met each other (or appear together)> in the novel? | Yes/No + number |  |
| How many times have <a character> and <a character> <communicate or have verbal conflicts with each other> in the novel? | Yes/No + number |  |
| mean | Explain the meaning or implication of the symbol or metaphor <a symbol or a metaphor> in one sentence, which appears in the novel. | An explanation |
| In which chapter does there exist a sentence in the novel <the novel> with the same or similar meaning as "<a sentence not from the original text>"? Please output the chapter name or index. | A chapter index or title |  |
|  | <A character> is used to be <positive or negative> and finally becomes a <negative or positive> one in the novel. Tell in one sentence which episode marks this character’s change. | A plot in one sentence |
| chara | Who are mentioned with names in the <an organization, a family, or a club> in the novel? | A list of names |
| Please list 3 aliases or designations of <a character> in the novel. | 3 aliases or designations |  |
| Who is <a minor character, or a character without name, or a character that appears only once> in the novel? | A description of character |  |
| settg | In which <cities, or countries> does this story take place in the novel? | A list of cities or countries |
| In which year does the earliest event happen, and in which year does the latest even happen in the novel? | A range in years |  |
| relat | What is the relationship between <a character> and <an alias or a nickname of this character> in the novel? | A relationship |

Question templates adopted in the annotation procedure are presented in Table 5. The templates are designed using a combination of literary theories and widely-used long-context tests on LLMs. During annotation, annotators are required to fill in the missing entities (e.g., character, place, sentence) in the templates and allowed to modify the expressions to suit specific questions. This approach balances the workload of annotation and ensures a diverse set of questions.

### D.2 Distribution of Input Token Count

Figure: Figure 5: Token Count Distribution in NovelQA, including Copyrighted (left) and Public Domain Only (right). The token count of both the novel and the questions are counted. The tokenization procedure is accomplished through gpt-3.5-turbo-16k tokenizer.
Refer to caption: x5.png

The input token count of each novel in NovelQA is calculated by adding the book-length to the lengths of its related questions. The distribution of the token count is illustrated in Figure 5.

### D.3 Data Classification

**Table 6: Distribution of Question Types in NovelQA: This table provides a breakdown of questions across different complexity categories (Multi-hop, Single-hop, Detail) and aspect categories (Time, Meaning, Span, Setting, Relation, Character, Plot).**
|  | Multi-hop | Single-hop | Detail | Sum |
| --- | --- | --- | --- | --- |
| Times | 463 | 0 | 0 | 463 |
| Meaning | 34 | 126 | 206 | 366 |
| Span | 34 | 0 | 0 | 34 |
| Setting | 24 | 177 | 63 | 264 |
| Relation | 119 | 14 | 32 | 165 |
| Character | 69 | 255 | 98 | 422 |
| Plot | 64 | 414 | 113 | 591 |
| Sum | 807 | 986 | 512 | 2305 |

**Table 7: Data Distribution of NovelQA. Each question is labeled with a Complexity and an Aspect label. The Complexity dimension comprises three categories, multi-hop, single-hop, and detail, while the Aspect dimension comprises seven categories including times, meaning, and so forth.**
| Dimension | Type | Percentage | Description | Example |
| --- | --- | --- | --- | --- |
|  | Multi-hop | 34.98% | Questions requiring knowledge across multiple paragraphs, or even multiple chapters, to be solved. | *Has Crusoe been on a voyage? If so, how many times has this plot happened?* in *The Life and Adventures of Robinson Crusoe* |
| By<br>Complexity | Single-hop | 42.74% | Questions requiring knowledge from one or several adjacent single sentences to be solved. | *According to the Colonel, what did he smoke in McQueen’s compartment?* in *Murder on the Orient Express* |
| Detail | 22.19% | Questions requiring knowledge from one or two adjacent sentences to be solved. Detail questions are distinguished from the single-hop class by involving information that is too minor to impact other plots, making the details difficult to recall. | *How many candles Madame Magloire lighted when the Bishop had his last dinner with Jean Valjean?* in *Les Misérables* |  |
| Times | 20.07% | About the number of times that a character, location, or plot appears in the novel. | *How many times has Kitty kissed Walter?* in *The Painted Veil* |  |
|  | Meaning | 15.86% | The understanding of certain sentences or metaphors, e.g., to interpret the relationship of a certain metaphor and the novel itself, or find a specific sentence according to a paraphrase provided by the annotator. | *In which chapter does there exist a sentence with the same or similar meaning as ‘The Marquis responded, ‘You do me too much honor. In any case, I lean toward that assumption.”?* in *A Tale of Two Cities* |
| By<br>Aspect | Span | 1.47% | About the range of the novel setting. To be specific, they either ask about the starting and ending year of the story or require listing all the cities or countries that are involved in the story. | *In which year does the earliest event happen, and in which year does the latest even happen?* in *Tess of the d’Urbervilles* |
| Setting | 11.44% | About the time or place settings, besides those in the span type, are classified in this type. | *Where did Diana’s cousins leave for the Debating Club concert?* in *Anne of Green Gables* |  |
| Relation | 7.15% | About the relationship of multiple character entities. To be specific, they ask either about the relationship of a character and their alias or designation, or about the relationship between different characters. | *What is the relationship between Jean Valjean and Ultime Fauchelevent?* (designation) *Who are members of ABC friends?* (membership) in *Les Misérables* |  |
| Character | 18.29% | About the information of characters, besides those in the relation type, are classified into this type. | *Who is Miss Beirne?* in *Dubliners* |  |
| Plot | 25.62% | We define a plot as "some character does something for once". Questions that ask about the information of any plots are classified into this type. | *What does Clarissa repair in preparation for the night’s party?* in *Mrs.Dallory* |  |

**Table 8: Question types and their corresponding LLM abilities. Each type of question in NovelQA is designed to test one or more abilities of LLMs, including the abilities to retrieve information, to identify details, and so forth.**
| Question Type | Abilities Required |
| --- | --- |
| Multi-hop | Ability to retrieve and integrate scattered information. |
| Single-hop | Ability to retrieve information and summarize. |
| Detail | Ability to precisely identify specific, subtle details. |
| Times | Ability to retrieve facts, integrate facts, and reason. |
| Meaning | Ability to retrieve ambiguous information and reason. |
| Span, Setting, Relation | Ability to retrieve facts, integrate facts, and reason. |
| Character, Plot | Ability to subtle or ambiguous information and reason. |

The data in NovelQA are classified into 3 complexity levels and focus on 7 aspects. Table 7 presents a detailed description of each class’s criteria and examples.

### D.4 Inter-Annotator Agreement Tests

We have conducted two separate Inter-Annotator Agreement (IAA) tests in this work.

The first IAA experiment focuses on evaluating the quality of annotated question-answer pairs. Annotators are required to choose books they are familiar with but have not annotated themselves to answer questions on. As annotators are mostly from the same local university and share similar courses and projects, many can find books they have read in common. We select books read by at least two annotators and have the other reader answer multichoice questions.
As the respondents are quite familiar with the target novels and have a strong academic background in English Literature, it takes them around 2-3 hours or less to complete each novel.

The second IAA test assesses model answer evaluation by calculating the correlation between human evaluators and GPT-4 evaluators. Annotators also serve as human evaluators for model-generated answers to the novels they were familiar with. Their evaluations were compared to those of GPT-4 evaluators, with Cohen’s kappa score calculated to measure agreement.
Human evaluators, familiar with the content, typically completed their reviews in under half an hour per novel.
We also perform an inter-evaluator agreement test between human-evaluators and the GPT-4-evaluator. The results of Cohen’s Kappa score are presented in Table 4.

### D.5 Prompts

**Table 9: Prompts used in each setting of question answering. In each prompt, the model is assigned the identity of a literature professor reviewing student answers, with a detailed task description and clear specifications for the input and desired output. The angle brackets ‘<>’ indicate the contents varying among each input.**
| QA Setting | Prompt |
| --- | --- |
| Generative | You are a literature professor. I will provide you with the full text of a novel along with a series of questions.<br>Please thoroughly analyze the novel’s content to accurately respond to each of the following questions.<br>Book title: <title>; Book Content: <content>; Book ends.<br>Questions start here:<br>N $\times$(Question: <question>); Questions end here.<br>Try your best to answer the questions based on the given novel full text.<br>The answer should be in short with only one or several words.<br>Your output format should be ’Answer0: <answer>Answer1: <answer>… Answern: <answer>’,<br>each answer in one line without outputing the questions and other info. |
| MultiChoice | You are a literature professor. I will provide you with the full text of a novel<br>along with a series of questions and corresponding choices pertaining to it.<br>Please thoroughly analyze the novel ’s content to accurately respond to each of the following questions.<br>Book title: <title>; Book Content: <content>; Book ends.<br>Questions start here:<br>N $\times$(Question: <question> Choices: 0: <choice0> 1: <choice1> 2: <choice2> 3: <choice3>); Questions end here.<br>Try your best to select the correct choice to each question based on the given full text the novel.<br>Your should output the choice to each question with the format<br>’Answer0: <choice> Answer1: <choice>… Answern: <choice>’<br>(only the choice index is required), each answer in one line without outputing the questions and other info. |
| Closebook-<br>Generative | You are a literature professor. I will provide you a series of questions.<br>Please accurately respond to each of the following questions.<br>Book title: <title>; Book Content: <content>; Book ends.<br>Questions start here: N $\times$(Question: <question>); Questions end here.<br>Try your best to answer the questions based on your own knowledge.<br>The answer should be in short with only one or several words.<br>Your output format should be ’Answer0: <answer>Answer1: <answer>… Answern: <answer>’,<br>each answer in one line without outputing the questions and other info. |
| Closebook-<br>MultiChoice | You are a literature professor. I will provide you a series of questions along with four choices<br>for each question. Please accurately select the correct choice to each of the following questions.<br>Book title: <title>; Book Content: <content>; Book ends.<br>Questions start here:<br>N $\times$(Question: <question> Choices: 0: <choice0> 1: <choice1> 2: <choice2> 3: <choice3>); Questions end here.<br>Try your best to answer the questions based on your own knowledge.<br>Your should output the choice to each question with the format<br>’Answer0: <choice> Answer1: <choice>… Answern: <choice>’<br>(only the choice index is required), each answer in one line without outputing the questions and other info. |
| Evaluating<br>Generative | You are a literature professor reviewing a student’s quiz paper.<br>The question is about the novel <novel title>: <question>. The related evidences from the novel are: <evidences>.<br>Correct ans is: <ca>. Student ans is: <sa>.<br>Plz check whether the student’s ans is correct wrt. the correct ans, and return "C" for correct and "N" for not correct.<br>esp., if the student grabs the correct ans’s meaning, return "C".<br>However, if there are factuality errors in student ans,<br>or the question requires a specific number but the student answers a rough number, you should return "N".<br>Please only return the char C or N w/o any other output. |
| Evidence<br>Recall | You are a literature professor. I will provide you with the full text of a novel along with a series of questions.<br>Please thoroughly analyze the novel’s content to accurately respond to each of the following questions.<br>Book title: <title>; Book Content: <content>; Book ends.<br>Questions start here:<br>N $\times$(Question: <question>); Questions end here.<br>Try your best to answer the questions based on the given full text of the novel.<br>The answer should be in short with only one or several words. Your output format should be<br>Ánswer0: <answer>$ <evidences> Answer1: <answer>$ <evidences>… Answern: <answer>$ <evidences>,́<br>each answer in one line with all the supporting evidences.<br>Each evidence should be a sentence exactly from the original text without any paraphrase. |
| Evaluating<br>Evidence<br>Recall | You are a literature professor reviewing student’s evidence for their answer about novel <novel title>.<br>Question: <ques>.<br>Correct answer: <ca>. Student answer: <sa>.<br>Correct evidence: <ce>. Student evidence: <se>.<br>You should evaluate the student evidence in 3 aspects:<br>C) correctness: whether the student evidence is the same with the correct evidence or with a similar correct meaning.<br>R) relevance: whether the evidence is relevant to the ans.<br>S) sufficiency: whether sufficient evidences are retrieved to support the ans.<br>And give a score of 1-100 to only the evidence (not the ans).<br>You should **only** return 3 score numbers, e.g.in format C50R66S33, without any other outputs. |

We present all the prompts involved in both test and evaluation phases in Table 9. The prompts generally follow the formula of zero-shot prompting. In each prompt, the model is firstly assigned the identity of a literature professor reviewing student answers. Then the prompt provides the model with a detailed task description and clear specifications for the input and desired output.

## Appendix E Experiments and Analysis

**Table 10: Model Performance Analysis Pre and Post 100K Tokens: The accuracies of questions on evidences pre-100K and post-100K are calculated separately and compared, where the ‘100K’ indicates the aboslute starting token position of evidences relating to answer. The result shows an obvious contrast as the accuracy drops sharply after the 100K, in both generative and multichoice settings on each model.**
|  | Multichoice | Generative |  |  |
| --- | --- | --- | --- | --- |
|  | pre-100K | post-100K | pre-100K | post-100K |
| GPT-4 | 83.17 | 59.22 | 53.64 | 32.01 |
| Claude2.1 | 77.65 | 54.46 | 61.96 | 30.57 |
| InternLM-7b | 51.78 | 38.17 | 37.71 | 18.23 |
| InternLM-20b | 55.59 | 40.87 | 37.74 | 21.10 |
| Weighted Avg. | 66.24 | 48.08 | 46.84 | 25.36 |

**Table 11: Model Performance by Question Type in Multichoice Setting: This table details the accuracy scores of four models across different question types within the Multichoice setting of NovelQA. Question types include character (chara), meaning (mean), plot, relation (relat), setting (settg), and others, with ’-’ indicating the absence of data for a category. The table also provides an average score (avg) for each question category and model. Abbreviations used are dtl (details), mh (multi-hop), sh (single-hop).**
| (a) GPT-4 | (b) Claude 2.1 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | chara | mean | plot | relat | settg | span | times | avg |  | chara | mean | plot | relat | settg | span | times | avg |
| mh | 55.88 | 44.12 | 53.12 | 56.78 | 66.67 | 44.12 | 38.53 | 45.15 | mh | 71.88 | 76.47 | 85.00 | 70.69 | 78.26 | 51.52 | 47.71 | 58.17 |
| sh | 53.94 | 53.97 | 58.94 | 35.71 | 62.15 | - | - | 57.26 | sh | 82.28 | 76.72 | 83.29 | 50.00 | 80.89 | - | - | 81.19 |
| dtl | 47.96 | 25.98 | 55.75 | 18.75 | 55.56 | - | - | 30.00 | dtl | 63.51 | 38.12 | 76.92 | 68.75 | 79.03 | - | - | 58.02 |
| avg | 52.86 | 37.36 | 57.70 | 47.56 | 60.98 | 44.12 | 38.53 | 49.18 | avg | 76.80 | 54.55 | 82.21 | 68.52 | 80.17 | 51.52 | 47.71 | 66.78 |
| (c) InternLM2-Chat-7b | (d) InternLM2-Chat-20b |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | chara | mean | plot | relat | settg | span | times | avg |  | chara | mean | plot | relat | settg | span | times | avg |
| mh | 42.19 | 38.24 | 45.90 | 46.15 | 65.22 | 39.39 | 42.92 | 43.87 | mh | 76.81 | 88.24 | 87.50 | 79.83 | 91.67 | 52.94 | 45.79 | 60.22 |
| sh | 44.44 | 39.34 | 44.56 | 28.57 | 48.15 | - | - | 44.23 | sh | 86.27 | 88.10 | 92.03 | 57.14 | 87.01 | - | - | 88.64 |
| dtl | 52.63 | 26.24 | 55.14 | 31.25 | 59.68 | - | - | 41.54 | dtl | 69.39 | 30.10 | 85.84 | 53.12 | 80.95 | - | - | 57.62 |
| avg | 45.69 | 31.84 | 46.79 | 41.72 | 52.63 | 39.39 | 42.92 | 43.51 | avg | 80.81 | 55.46 | 90.36 | 72.73 | 85.98 | 52.94 | 45.79 | 71.80 |

### E.1 Implementations

We also tested Gemini-1.5(^5^55[https://deepmind.google/technologies/gemini/flash/](https://deepmind.google/technologies/gemini/flash/)), but generation for 816 questions from 29 novels was blocked for unknown reasons (with only 14 of these novels under copyright protection). Consequently, we have decided not to present Gemini’s result.

### E.2 Multi-choice Performance on Each Type of question

Besides the evaluation in generative settings, we also prompted the models to collect their responses for the multi-choice versioned questions. Table 11 presents the accuracies of four models in multi-choice settings in each question type.

### E.3 Relation between Multi-choice Accuracy and Evidence Positions

Figure: Figure 6: Analysis of Accuracy in multi-choice Setting by Absolute and Relative Token Positions: This figure illustrates the accuracy in the multi-choice setting of NovelQA, plotted against the absolute position (left) and percentage position (right) of each question’s relevant evidence within the novel. Each subplot represents a different model. The x-axis of the absolute position figure (left), reflecting token indexes, is folded on the right due to the long-tail distribution in the lengths of the selected novels.
Refer to caption: x6.png

Figure 6 which presents the relationships between the accuracy and the absolute or relative positions accordingly shows similar trends to which are observed in the generative setting. To be specific, the accuracy by absolute token position remains high when related evidences are before 100K’s text length, while drops after 100K. Meanwhile, the accuracy by relative position remains relatively even. A comparison is made between the accuracies within two ranges, 65K(the lowest token count) to 100K (namely pre-100K) and 100K to the end (namely post-100K). Figure 7 and Table 10 present a clearer contrast between these two ranges, where the accuracies drop sharply after the 100K token.

Figure: (a)
Refer to caption: x7.png

### E.4 Representative Errors

**Table 12: Categories of representative errors observed in evaluating LLMs on NovelQA.**
| Fault Type | Subtype | Setting | Example QA |
| --- | --- | --- | --- |
| Hallucination | On Fictional<br>Setting | GPT-4,<br>generative | Book: *Mansfield Park*<br>Q: What is the relationship between Miss Maria Ward and<br>Lady Bertram?<br>Correct A: Miss Maria Ward and Lady Bertram are the<br>same person.<br>Model A: Sisters. |
| On Narration | GPT-4,<br>generative | Book: *The Night Land*<br>Q: How many times has Aesworpth shouted?<br>Correct A: 1<br>Model A: Not mentioned. |  |
| Overlooking | GPT-4,<br>generative | Book: *Light in August*<br>Q: Who is Percy Grimm?<br>Correct A: Percy Grimm - the captain of the State National<br>Guard who kills Joe Christmas and castrates him.<br>Model A: Percy Grimm does not appear in the novel. |  |
| Miscounting | GPT-4,<br>generative | Book: *Can You Forgive Her?*<br>Q: Has the word or phrase ’take away another man’ appeared?<br>If so, how many times does it appear in the text?<br>Correct A: Yes, 2.<br>Model A: Yes, 1. |  |

Through reviewing the generation above, we concluded three common error types: hallucination, miscounting and overlooking. Examples for each type are presented in Table 12.

Hallucination refers to the information generated by the model with factual errors. In our generative QA setting, typical hallucination mistakes encompass two types: (1) factual errors about the fictional settings (e.g., mixing entities within the setting or between the settings of different books) and (2) factual errors about the narrations (e.g., whether a fact is narrated). The first category usually appears in questions asking minor characters, plots, and settings, where the model might output a non-existing one. Meanwhile, the second category is often associated with sentence-locating questions, which ask the model to locate a sentence. In this case, the model may fake a sentence that does not exist in the original text.

Overlooking refers to the model’s neglect of details. As mentioned above, the questions in *detailed* category involve minor characters, plots, or settings. Diving further, the reasons why these details are difficult to be recalled lie in two aspects: (1) They do not contribute to the character development, other plots, or the main themes, and thus reading the rest of the novel does not help to remind this detail; (2) Since most novels have derivative works (e.g., films, fan works, and book reviews), where the detailed information is eliminated to form a condensed narration. As the derivative works spread further and appear more frequently in the model’s training data, they have a higher probability of becoming the models’ inner knowledge, which is similar to [Chang et al., ]’s observation, and vice versa for those omitted details. These two factors contribute to the difficulty in the model’s recalling details and thus result in overlooking errors.

Miscounting Researches [Li et al., , Feng et al., ] has revealed shortcomings in the counting ability of LLMs, especially autoregressive-decoder-based models, and methods unfolding the outputs such as chain-of-thought prompting can enhance their counting ability. Our test does show that models make mistakes with numbers. Though the errors in the case of generative responses may be due to not following instructions and simply outputting ‘multiple times’ instead of the desired specific times, the accuracy in the multi-choice setting has still only reached 38.53% to 49.56% for the chosen four models, as shown in Table . Even in the simplest question which asks for the appearing frequency of certain phrases, the models still make mistakes.

**Table 13: Close-book Performance across four LLMs on NovelQA. Unlike the standard scenario, models rely solely on internal knowledge without access to the novels. The parentheses indicate the performance drop from the standard to the Close-book scenario, highlighting the models’ dependency on external text for answering.**
|  | Close-book QA |  |
| --- | --- | --- |
|  | Multichoice | Generative |
| GPT-4 | 60.94 (-9.06) | 34.30 (-13.58) |
| Claude 2.1 | 51.77 (-15.01) | 22.36 (-20.68) |
| InternLM2-Chat-7b | 33.58 (-10.71) | 14.12 (-16.81) |
| InternLM2-Chat-20b | 33.05 (-16.13) | 15.51 (-16.86) |

### E.5 Close-book Question Answering

We employ a Close-book QA scenario to assess the extent to which models rely on the content of novels versus using their internal knowledge to answer questions. In this approach, the models are not given access to the text of the novels and must rely solely on their pre-existing knowledge to provide answers. Given that our selected novels are well-known and representative of their genres, it’s inevitable that LLMs have encountered their texts during training and retained some of their content. The results of this evaluation are presented in Table 13.

Models like GPT-4 and Claude 2.1 achieve notable scores in the Close-book setting (60.94% and 51.77% in multi-choice, 34.30% and 22.36% in generative, respectively) indicating that they have internalized significant portions of the novels’ content during training. This internal knowledge allows them to perform reasonably well even without direct access to the text.
The difference in performance between the Close-book and standard settings underscores the challenges in long-context understanding. While models can retain and recall information from well-known texts, their ability to comprehend and use such information to answer questions accurately diminishes in the absence of the text. This suggests that long-context understanding, as measured by the main results, might still be more challenging than it appears, as models benefit from having the text directly available.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How was the evidence annotated?
2. It was mentioned in the paper that LLMs "usually fail to tackle information spanning over multiple chapters." How does the proposed benchmark assess this aspect?
soundness: 3
strengths: 1. The paper is generally well-written and clear.
2. The proposed NovelQA benchmark features the longest context to date
3. The experiments are generally well-designed
summary: This paper introduces a new benchmark called NovelQA, which is designed to evaluate the performance of LLMs on extremely long and complex texts. NovelQA uses English novels as contexts and annotates questions across three levels of complexity and seven aspects by literature experts. The proposed benchmark surpasses existing ones in length, includes evidence alongside questions, and emphasizes detailed comprehension. This paper also presents experimental results from a suite of LLMs on this benchmark and analyses their performance across different aspects.
weaknesses: 1. The benchmark only focuses on novels, which is somewhat limited. It is essential to include other forms of long-context in order to provide a more comprehensive evaluation of LLMs' long-context understanding capabilities.
2. The templates mainly focus on information extraction types of tasks. It would be beneficial to design more complex questions, e.g. ones 
that require reasoning

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How are the question types and evidence annotated?

2. What does the distribution of the evidence positions look like?
soundness: 3
strengths: 1. This paper presents a new dataset on long-context understanding. It has some unique features that are not covered by previous work, such as the long averaged context length, and human annotation efforts. It has the potential to be used in later work.

2. The benchmarking results include some interesting findings, such as performance degradation after 100K tokens and low performance on evidence recall. Those findings may enlighten future research in this area,

3. This paper is well-organized and easy to follow, with comprehensive appendix on the details of the data.
summary: This paper presents a new dataset on long-context understanding. It uses novel as the long context and evaluate the model with questions from different complexity and aspect categories. Those questions and answers are annotated by human annotators, with a high inter-annotator agreement on answers. The answers are also with distractions for multi-choice setting generated from GPT-4. They evaluate open-source and commercial LLMs on the constructed dataset, revealing some findings such as performance degradation after 100K tokens, difficulty with questions on meaning, relation, span and times and low performance on evidences recall.
weaknesses: 1. One major concern of this work is that this paper only focuses on using novels as the genre for long-context understanding. It would be better if the author could cover other long-context genres, such as other nonfiction books, to evaluate the long-context understanding comprehensively.

2. Another concern I have is about the analysis of the performance based on the position of corresponding evidence. Because the annotation on the annotation does not include whether the evidence is unique in the long context, there may be some cases where the annotated evidence can also be found in another snippet of the context. And it may ultimately impact the analysis or the conclusion of the analysis based on the evidence position.

3. As we can see in Table 1, most of the questions are either single-hop or details, which is generally based on a short text snippet of the long context. It would be also beneficial to include more discussion on the multi-hop questions, for example, the performance with regard to the relative distances of the pieces of evidence for a multi-hop question, as these questions are more difficult to be produced by LLMs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: — Somehow the Table 7 and Figure 5 are above the Appendix section, and needs to be formatted.

— No Table number in Line 1120

— Consider using VLLM models to reduce running costs and time.

— Better to write `multichoice` or `multi-choice` consistently during the paper

— Add Human Baseline in Table 2

— Write in bold the best scores in Tables, better readability
soundness: 3
strengths: — Annotators performed a huge amount of work. The questions, golden answers, and evidence of NovelQA are crafted through the efforts of experts. The context is extensive, making it challenging to create such a set.

— An informative, descriptive Appendix with details, annotation agreements, and error analysis.

— The contribution is clear and high
summary: NovelQA is a new benchmark designed for evaluating large language models (LLMs) on complex, extended narratives with average context windows exceeding 200,000 tokens. This is relevant as the most advanced long-context LLMs can process over 250,000 tokens. NovelQA provides a combination of complexity, length, and narrative coherence, using a diverse selection of English novels from various eras, genres, and formats. Professional expert annotators carry out the annotation process, and all of them hold or are pursuing degrees in English Literature. The dataset includes multi-hop, single-hop, and detail questions, which assess the model's abilities to retrieve and integrate scattered information, summarize information, and accurately identify specific and subtle details. Both closed APIs and open models have been evaluated using this benchmark, and a comprehensive analysis is proposed based on the results.
weaknesses: — No ethical consideration and Limitations sections. The Limitation section is highly recommended, as mentioning the restrictions is important.

— There are efficiency problems with the benchmark running for the long-context models. 

— To add in the limitations: data leakage. `To prevent against data leakage, we will not release golden answers for the test set, minimizing the risk of overfitting.`
The novels are not created from scratch. Many of them, particularly public domain works, are already included in the training datasets of models like GPT-4. This represents an indirect issue of data leakage. It is also crucial to discuss the problems related to data contamination and leakage.

— The biases of the annotators also need to be mentioned in the Limitation section.

— Truncation is also a limitation as nobody checked whether the truncated part influences the result.

— The creation of templates based on the most difficult cases for GPT-4 and Claude2.1 is understandable but may cause biases.

— Syntactic choices for the multiple choices:
`We use GPT-4 to generate three distracting options for each question and its golden answer and randomly permute the four answers`
Authors further evaluate API models on the same sets: it will be with a high probability of a strong bias for GPT4 to answer the questions.
Thus, the authors can not claim that the GPT performs better than others.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors could incorporate experiments with the new Llama 3.1 family.
soundness: 3
strengths: The proposed benchmark is a valuable resource for the evaluation of LLMs performance on reading and reasoning to answer questions over long texts.
summary: This paper presents a new benchmark to assess LLMs on reading and understanding very long inputs (200k in average). Concretely, the task is Question Answering (QA) in two forms, multiple-choice and generative, over novels. The dataset is constructed based on public and copyrighted novels from the Project Gutenberg (purchasing ones when necessary) and manually crafted questions, gold answers (distractors generated automatically but manually inspected), and evidence. Human annotators are Language and Literature students. The benchmark contains questions of different complexity (multi-hop, single-hop, and details) and about different aspects (e.g., relationships or narrative settings -- 8 categories in total). Five large-context LLMs are evaluated on the benchmark (3 commercial and 2 open source). Results reveal that LLMs struggle to read, find and process the content in the long context necessary to formulate the answer; this happens even with commercial LLMs (GPT-4) with open-source LLMs exhibiting worse performance. 

The paper is easy to follow and with illustrative examples. It includes performance analysis per question type and by content position.
weaknesses: While the evaluation of LLMs shows that these struggle to understand, attend, and recall all the content from the long context necessary to answer questions (main focus of the paper), I wonder whether an initial extractive step would be a strong baseline in this setup. For instance, a common practice in summarisation of long inputs, is an initial extractive step (e.g., by tf-idf) that is applied before carrying out the actual task with the selected text. Maybe adding such a baseline would contribute to highlight the robustness of the proposed QA benchmark (i.e., how difficult is it to solve the task in this way). Also, some of the questions (e.g., counting) could be a LLM reasoning weakness rather than a long context processing issue.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I’m curious about the performance of other open-source models, such as mistral, llama, and so on, or perhaps smaller closed-source models like GPT-4o-mini. Could you provide additional insights into how these models perform compared to the primary models evaluated in the paper?

2. Regarding the use of gpt-4 as an evaluator, as noted above, this may introduce potential bias. Could you clarify how gpt-4 , or an alternative model, acted as an evaluator in the paper?

3. Are the questions and sample sentences in the dataset accessible online for user verification in the future, or are they restricted and not publicly available for sourcing and comparison?
soundness: 4
strengths: NovelQA benchmark addresses an important challenge in long-context LLMs by introducing a dataset that emphasises extended context comprehension within complex narratives, representing a significant advancement over standard benchmarks. The dataset is carefully constructed with expert annotations, which are detailed in the paper, ensuring that question complexity aligns appropriately with model capabilities. The paper provides comprehensive documentation on question types and evaluation metrics, establishing a strong foundation for interpretability and contributing to a reliable benchmark. Last but not least, NovelQA evaluates narrative comprehension, multi-step reasoning, and retrieval-based tasks in LLMs, marking a substantial contribution to long-context evaluation.
summary: This paper introduces NovelQA, a benchmark designed to evaluate LLMs on long-context question-answering tasks. NovelQA addresses limitations in existing benchmarks by focusing on texts exceeding 200,000 tokens, derived from a diverse selection of English novels. NovelQA incorporates human-annotated questions that require detailed comprehension of lengthy narratives. The dataset evaluates multiple aspects of comprehension, including multi-hop reasoning, detail retrieval, and narrative coherence, revealing significant gaps in LLMs' ability to manage large, complex contexts.
weaknesses: Though the transparency of the annotation process is addressed, it could be strengthened. Although the authors state that questions are created by expert annotators with backgrounds in English Literature, the paper lacks clarity regarding annotator instructions, experience levels, and quality control measures above basic inter-annotator agreement scores — this could be demonstrated more clearly. Secondly, evaluation methods are insufficiently justified and may introduce bias due to the use of automated scoring. For instance, gpt-4 is used as an evaluator, with limited details on its evaluation process, which is particularly concerning given that the benchmark aims to capture complex reasoning across extended narratives.  eg. gpt-4 as an evaluator may score higher for gpt-4's answer. This could lead to bias in the results. While two human evaluators are mentioned, the paper does not provide information on their backgrounds or expertise. Other than these issues, the paper is well-written!

Meta-review: The paper presents NovelQA, a benchmark of question answering where the context can be of length up to than 200k tokens.  The benchmark is quite thorough, the paper well written and the reviewers are all positive wrt the paper.  The paper also benchmarks many LLMs on this task which is a valuable set of datapoints.  Given this, I suggest acceptance.

Decision: Accept (Poster)