Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions (2024)

Abstract: This work introduces an original and practical paradigm for narrative comprehension, stemming from the characteristics that individual passages within narratives tend to be more cohesively related than isolated. Complementary to the common end-to-end paradigm, we propose a fine-grained modeling of narrative context, by formulating a graph dubbed NARCO, which explicitly depicts task-agnostic coherence dependencies that are ready to be consumed by various downstream tasks. In particular, edges in NARCO encompass free-form retrospective questions between context snippets, inspired by human cognitive perception that constantly reinstates relevant events from prior context. Importantly, our graph formalism is practically instantiated by LLMs without human annotations, through our designed two-stage prompting scheme. To examine the graph properties and its utility, we conduct three studies in narratives, each from a unique angle: edge relation efficacy, local context enrichment, and broader application in QA. All tasks could benefit from the explicit coherence captured by NARCO.

heading: 1 Introduction
text: Since the advent of Large Language Models (LLMs), document comprehension has been improved significantly by simply employing the endto-end generative paradigm. Especially, with long context window enabled via techniques such as position interpolation (Xiong et al., 2023; Peng et al., 2024), cached or efficient attention (Wang et al., 2023; Ge et al., 2024a; Munkhdalai et al., 2024), context compression or pruning (Chevalier et al., 2023; Anagnostidis et al., 2023; Ge et al., 2024b), the end-to-end paradigm is deemed undoubtedly simple and effective for comprehension tasks (e.g. question answering) on various documents.
However, while the typical benchmarks are continually enhanced by more advanced LLMs (Tou*Corresponding author.
vron et al., 2023; Jiang et al., 2024; OpenAI et al., 2024), the end-to-end paradigm may not suffice for all comprehension scenarios. In this work, we focus around the narrative context, i.e. stories or novels, and propose a conceptually original framework of fine-grained context modeling: a graph is formulated that depicts the relations between context snippets, abstracting over the context to reflect a high-level understanding of the narrative. The graph itself is practically realized by LLMs to harness their rapidly evolving strengths, and the resulting graph could serve to facilitate various downstream narrative comprehension tasks.
Our motivation arises from the distinctive nature of narratives: multiple development of characters or events in a story could be entangled over long context ranges, where each local passage usually serves specific purposes for others. Thus, individual passages tend to be cohesively interconnected than being isolated. As the end-to-end paradigm implicitly grasps these context connections through sequence modeling, our approach explicitly models these dependency relations to capture coherence, offering a directly-applicable alternative path orthogonal to the end-to-end paradigm.
Concretely, drawing inspiration from the cognitive process on narrative perception, whereas humans can constantly reinstate relevant or causal events from past context during reading (Trabasso and Sperry, 1985; Graesser et al., 1994), our formalism, termed NARrative COgnition graph (NARCO), splits the entire context into chunks that act as graph nodes, with edges representing the relations between node pairs. In particular, edge relations are constituted by free-form questions. As humans could relate to past context in retrospect, accordingly, each question in NARCO edges arises from the succeeding node (latter context), asking necessary background or causes that can be clarified by the preceding node (prior context). Hence, graph edges consist of inquisitive questions that naturally
5822
reflect retrospection. Overall, the resulting graph explicitly depicts task-agnostic understanding of fine-grained coherence flow that could be flexibly utilized by downstream tasks.
Though our graph formulation partially shares motivations with discourse parsing that characterizes how each proposition relates to others within a close context (Grosz and Sidner, 1986), our method targets on practical utility for narrative comprehension, where edges in NARCO are designed to be easily obtained and effectively consumed by downstream tasks. Consequently, NARCO is formulated in a different scope from discourse parsing by two main perspectives. First, as most discourse frameworks, such as Rhetorical Structure Theory (Mann and Thompson, 1988), Penn Discourse Treebank (Prasad et al., 2008), or the recent Questions Under Discussion (QUD) (Ko et al., 2022, 2023) are rooted upon linguistic principles, their relation types are oriented for formal discourse analysis, requiring trained experts to annotate edges according to a defined linguistic taxonomy. Whilst for NARCO, the relation space is larger without taxonomy constraints, offering diverse high-level semantic signals for narrative tasks. Second, NARCO practically leverages LLMs to derive edge relations, without reliance on human annotations. Thus, the edge quality is not restricted by annotation resources, and shall be continuously enhanced along with the ongoing LLM advancement.
The key difficulty of NARCO lies in the edge realization between two nodes, which itself demands strong context understanding to determine which aspects to inquire upon the context, and to assess their saliency for comprehension. Such process is especially strenuous due to the large hypothesis space compared to conventional discourse formalisms. To this end, we pose soft semantic constraints on relations, and employ LLM’s capabilities to construct edges automatically through our proposed prompting scheme, of which consists a question generation stage and a self verification stage (Section 3). The obtained edges could then be utilized by downstream tasks in two primary ways. First, edges themselves directly provide information flow to guide various comprehension tasks. Second, they offer global coherence view for each node, thereby augmenting the local context to deepen the digest of independent passages.
To empirically demonstrate the practical utility of NARCO, we present three studies on narrative comprehension tasks, each from a unique angle:
• Our first study examines the edge efficacy on whether the relation questions capture capable retrospective coherence (Section 4). We conduct experiments on the recap identification task (Li et al., 2024), where NARCO is shown to recognize coherence dependencies between context, boosting up to 4.7 F1 over the GPT-4 baseline. • Our second study concerns the exploitation of enriched local embeddings, by injecting edges of relation dependencies into node representation (Section 5). Evaluated on the plot retrieval task (Xu et al., 2023b), our proposed approach with NARCO outperforms the zero-shot baseline by 3% and the supervised baseline by 2.2%. • Lastly, we utilize NARCO in a long document question answering task (Section 6), as a broader application of Retrieval-Augmented Generation (RAG) (Lewis et al., 2020). Experiments on QuALITY that requires global context evidence (Pang et al., 2022) suggest that, NARCO consistently raises zero-shot accuracy by 2-5% upon retrievalbased baselines with various LLMs, able to identify more relevant context through edge relations.
Overall, our key contributions in this work are: • We propose a new paradigm of fine-grained con-
text modeling to facilitate narrative comprehension, orthogonal to the end-to-end paradigm. • Our introduced NARCO framework describes flexible relations of context dependencies by retrospective questions, which are realized by LLMs through our designed prompting scheme, without reliance on human annotations. • We present three studies effectively utilizing NARCO on narrative tasks, empirically examining its edge properties and broader utilization.

heading: 2 Related Work
text: Questions Under Discussion QUD is a linguistic framework with rich history that approaches discourse and pragmatics analysis by repeatedly resolving queries triggered by prior context (Kuppevelt, 1995; Roberts, 1996; Benz and Jasinskaja, 2017). QUD has been adapted by recent works for discourse analysis (De Kuthy et al., 2018, 2020; Ko et al., 2020, 2022, 2023) or other applications (Wu et al., 2023b; Newman et al., 2023). Our proposed NARCO also adopts question-form relations; though, the scope and motivation is different from discourse analysis. Consequently, NARCO differs from QUD works considerably on the following design choices.
• Coarse Granularity While QUD tends to employ sentences as the basic discourse unit, NARCO opts for a coarser granularity, adopting passages (or chunks) as the graph nodes. It is driven by the fact that in narratives, complex events or interactions may often be conveyed beyond sentence-level, thus relations in NARCO could target higher-level understanding between context.
• Retrospection-Oriented Unlike conventional QUD that inquires from prior context to be addressed by subsequent context (forward direction), which could yield unanswerable questions (Westera et al., 2020; Ko et al., 2020), NARCO takes the backward direction, by asking retrospective questions from latter context, such that all generated questions in NARCO are naturally grounded by the corresponding prior context.
• Precision-Focused Unlike previous QUD works that require dedicated human annotations, NARCO is formulated attainable by LLMs. Accordingly, we prioritize precision over recall for practical instantiation of graph edges, and do not necessitate strict linguistic criteria, as long as edges contribute positively for narrative understanding.
Narrative Comprehension Assessments A major task direction on narratives is question answering (QA), where past works have proposed several datasets with human annotations, such as NarrativeQA (Kočiský et al., 2018), TellMeWhy (Lal et al., 2021), FAIRYTALEQA (Xu et al., 2022b), QuALITY (Pang et al., 2022). We adopt QuALITY as the broader application in this work, due to its challenging long context, requirement of global evidences, and simple evaluation by multi-choices.
Recently, several tasks have emerged focusing on modeling the reading process of long narratives, including TVShowGuess (Sang et al., 2022), PERSONET (Yu et al., 2023), TOM-IN-AMC (Yu et al., 2024), and retrieval tasks such as RELiC (Thai et al., 2022), PLOTRETRIEVAL (Xu et al., 2023b). These tasks require a holistic understanding of the long narratives to enhance contextual comprehen-
sion of specific segments. We reckon the significance of explicitly modeling context dependencies as a crucial aspect of narrative comprehension, motivating the inception of .
LLM Understanding and Reasoning LLMs have demonstrated remarkable capabilities on a wide spectrum of comprehension and reasoning tasks (Chen et al., 2024; Sun et al., 2024). The simple end-to-end solution is especially appealing with long context window enabled, using techniques such as scaling positional embeddings (Chen et al., 2023b; Xiong et al., 2023; Peng et al., 2024), efficient attention (Munkhdalai et al., 2024), cached attention (Wang et al., 2023; Ge et al., 2024a), recurrent attention (Dai et al., 2019), context compression (Chevalier et al., 2023; Ge et al., 2024b), context pruning (Anagnostidis et al., 2023), etc. Though being effective, certain narrative tasks demand beyond the end-to-end solution. Recently, new methods have been proposed for fine-grained task processing, e.g. reading agents such as MEMWALKER (Chen et al., 2023a). Nevertheless, our proposed approach depicts explicit context dependencies as an alternative paradigm, which is orthogonal to the existing directions and could be even further combined.
Structured Representation Various relational structures in text documents has attracted much attention by previous works, such as syntactic relations (Strubell et al., 2018; Xu et al., 2022a), discourse relations (Ji and Smith, 2017; Nair et al., 2023; Hu and Wan, 2023), entity or event relations (Ding et al., 2019; Li et al., 2020, 2021; Xu and Choi, 2020, 2022; Nguyen et al., 2022). As all these structures encompass pre-defined taxonomies on edge types, our propose graph representation is motivated to comprise open-world edge types that have been practiced in other tasks (Wu et al., 2019; Xu et al., 2023a; Su et al., 2024), while being practical and attainable by LLMs without requiring efforts of human annotations.

heading: 3 NARCO: Narrative Cognition Graph
text: In this section, we start by delineating our graph formulation, which is itself not tied to any particular implementation. Subsequently, we elaborate our specific graph realization using LLMs, without dependence on human annotations.

heading: 3.1 Graph Formulation
text: Nodes For a narrative, the entire context is split into short consecutive chunks (or passages), such that each is within a maximum word limit and constituted by sentences or paragraphs. Graph nodes are then all the chunks adhering the left-to-right sequential order, denoted by V = {v1, v2, .., vN}, with N being the total number of chunks.
Edges An edge connecting two nodes indicates the relations between the context. These relations are articulated as free-form questions that are not constrained by fixed taxonomies. All edges follow the backward direction, such that for an edge eij (i < j), the expressed questions always arise from the succeeding node vj , asking clarification regarding specific events or situations appeared in vj , which could be addressed by the preceding context vi. Since the hypothesis space is huge without any regularization, we pose soft semantic constraints on questions, such that questions should primarily reflect on causal and temporal relations, which are significant to the narrative coherence.
Functionally speaking, these backward edges resemble the human cognitive process for narrative perception: when reading a certain passage, humans are able to reinstate previous relevant parts in retrospect that lay out the build-up or causes, so to achieve a coherent comprehension of the global context (Trabasso and Sperry, 1985; Graesser et al., 1994; Song et al., 2020). Unlike conventional QUD that features curiosity-driven questions in a forward direction, which could yield unanswerable questions, all edge in NARCO are fully grounded by the prior context, since all retrospective questions are addressable by prior nodes.
Derived upon the above formulation, an edge eij in NARCO has the following features: • It may have zero or many questions. An empty
edge (zero questions) signifies vj is vaguely independent from vi in terms of coherence. • Each question should be salient towards the comprehension of narrative development, rather than inquiring trivial details. Hence, the number of
questions in an edge should reflect how cohesively related between two nodes. • As we adopt coarse granularity for nodes, questions could probe higher-level relations based on the extrapolation over multiple sentences, which may be useful towards broader understanding.

heading: 3.2 Graph Realization
text: To obtain graph nodes, the full context is split by paragraph and sentence boundaries. We impose each node within 240 words in this work, though the exact limit can be task-specific. For a graph characterized by N total nodes, there are O(N2) full edges available, which can become cumbersome and excessive. It is also task-dependent to determine which pairs of nodes should be gathered edges upon, e.g. for enriching local representation, it is sufficient to obtain relation dependencies from neighboring nodes within a context window.
Despite the daunting formulation on edge relations, the emergence of LLMs presents an opportunity: through designed LLM prompting, it becomes conceivable to actualize the entire graph without any human annotations involved. To this end, we introduce a two-stage prompting scheme to tackle the challenging edge construction.
Question Generation For an edge eij to be instantiated, LLMs need to determine important aspects to ask upon vj that reflect the retrospective coherence towards the prior context in vi. Similar utilization of LLMs for question generation (QG) has been explored in other applications, such as performing QG for QUD (Wu et al., 2023a) and passage decontextualization (Newman et al., 2023), where a LLM is prompted to generate questions directly based on task-specific criteria. For our case, such direct generation can be briefly outlined as:
Given a current context vj and its prior context vi, generate questions upon vj, such that each question asks about the cause or background of specific events or situations in vj, which can be clarified by vi, so to reflect their causal or temporal relations between the two context.
However, our preliminary experiments suggest that although LLMs can generate plausible questions by following the instructions, their quality is often unsatisfactory for NARCO, with common errors as follows (examples in Appx A.2): • Self-answerable: LLMs often ask questions upon vj but also answerable by vj as well. Such selfanswerable pattern aligns with the more conven-
tional QG setting (Du et al., 2017) that may exist plentifully during the supervised finetuning of LLMs, causing a bias towards this type of question. However, they are not desirable for NARCO, since they do not express dependencies between nodes to reflect their relations. • Hallucination: LLMs could hallucinate the relations of two nodes by guessing and inferring extra underlying connections not grounded by the provided context, resulting in questions not directly answerable by vi. In essence, QG for NARCO requires LLM simultaneously aware of questions being: 1) arising from vj ; 2) not answerable by vj ; 3) answerable by vi. As this is empirically challenging even for strong LLMs (e.g. GPT-4), we perform QG with two heuristic turns that can be viewed as human-guided Chain-of-Thoughts (Wei et al., 2022):

heading: 1. List concrete parts in vi that contribute as the preceding background or cause for specific events or situations mentioned in
text: vj, along with brief explanations.
2. Convert each above listed connection to a question, such that it asks about the cause or background upon vj and can be clarified by the corresponding concrete part in vi, helpful to comprehend their causal or temporal relations.
The designed two-turn QG scheme yields higherquality questions than the rudimentary generation, mainly alleviating the self-answerable problem. However, noisy questions of the two identified error types still occur due to imperfect instruction following by LLMs. In light of these noises, we apply an optional second stage to filter out noisy questions through self verification.
Self Verification The second stage takes the generated questions from QG and in turn, performs question answering on the context:
Given a context Cij and a related question, determine whether it is answerable. If yes, reason the answer and provide original sentences of key supporting evidences.
In Particular, Cij is the concatenated context from vi and vj without disclosing their boundary. If the question is answerable, we then parse the response and identify whether the supporting sentences are from the prior context vi. If not, the question is attested noisy and gets discarded, as it does not bridge the two context effectively.
With the second stage, only questions that could be answered by prior nodes are eventually retained
in NARCO, being a precision-focused approach. In this work, we adopt GPT-4 for the challenging QG stage, and ChatGPT for the easier verification stage. NARCO may also be derived with strong open-source LLMs as well. Our full prompts and more details are provided in Appx A.1.
As NARCO targets the practical utility to facilitate narrative comprehension, the obtained edges shall be directly consumed by downstream tasks. Sections 4-6 present three empirical studies, each from a distinctive perspective, to examine the edge properties and their utilization.

heading: Candidate A

heading: Query

heading: 4 Study I: Edge Efficacy
text: Our first study examines the graph edges on whether they express useful relations, such that the generated retrospective questions should bridge the coherence between two context. For appropriate assessment, we conduct recap identification on RECIDENT dataset (Li et al., 2024), a task on narratives that identifies whether certain preceding snippets can function as a recap or prelude to the audience in regards to a current context.
Concretely, the input takes a short snippet from a novel or show script, along with a provided list of its preceding snippets. The task resolves which preceding snippets are directly related with the current one in terms of plot progression, requiring contextual understanding of narrative development. As NARCO is designed to capture the coherence relations between context, edges of retrospective questions could be leveraged to link the current snippet to related preceding ones. Therefore, RECIDENT serves as a natural testbed for comprehensive evaluation of edge efficacy.

heading: 4.1 Approach
text: For this study, our proposed approach targets upon the zero-shot baseline with LLMs in (Li et al., 2024), where ChatGPT is originally asked to select the related recap snippets from the list of preceding candidates based on their text content.
With NARCO, we regard each current snippet as a target graph node vt, and the list of its N pre-
ceding snippets {vc|c = 1, .., N} as the candidates. For vt and each of its candidate vc, the edge is realized as ect. As questions in ect should reflect causal or temporal coherence, we directly utilize these questions in two following ways.
Edge Relations Normally, candidate snippets are processed by their text content as in the baseline. To evaluate the coherence depicted by edges, we instead propose to identify recap snippets solely based on the edge relations: for a candidate node vc, we concatenate all its questions in edge ect, denoted by qc, to identify recap, and completely neglect original text content, so to ensure an entirely isolated assessment of edge relations.
Specifically, given the context of a target snippet vt, and N candidates {qc|c = 1, .., N} represented by questions, we now ask a LLM to score which qc addresses important questions that are significant to comprehend the current context, with higher scores indicating better overall questions that provide recap information. Candidates with empty edges are directly assigned 0 score.
Edge Degrees Alternatively, as mentioned in Section 3.1, the number of questions between two nodes could suggest how cohesively related they are. We take this number as the edge degree, and propose to simply deem it as the score to rank candidates, without any inference on the node context or edge relations at all. Though being rather unconventional, ranking candidates by edge degrees further reflects the edge quality.
With either the relation score or degree score, it can be used standalone or interpolated with the baseline selection. More formally, we obtain the rank ∈ [1, N ] of each candidate i by relation scores, denoted as rreli , and the rank by degree scores r deg i , along with binary selection bi from the original baseline. The final score s of each candidate is:
si = α · rreli + β · rdegi − λ · I(bi) (1) I is the indicator function that boosts the baseline decision bi by λ rank; relation and degree ranks are interpolated by α and β. The final score is then ranked to select top candidates with recap information (lower is better). Setting α/β/λ to 0 can thereby evaluate each method standalone.

heading: 4.2 Experiments
text: Data As RECIDENT includes multiple novels and show scripts, we pick one classic novel NotreDame de Paris (NDP) in English and one TV show
Game of Thrones (GOT) to reduce the evaluation API cost from OpenAI. The test set of each source consists of 169 / 204 target snippets respectively. Each target is provided 60 candidate snippets, with 5.6 / 4.9 candidates being positive on average.
Evaluation Metric We follow Li et al. (2024) and adopt F1@5 (F1 on top-5 selected candidates) as the main evaluation metric.
Methods We conduct zero-shot LLM experiments with both ChatGPT (gpt-3.5-turbo-1106) and GPT-4 (gpt-4-1106-preview) from OpenAI. • BL: the original ChatGPT baseline (Listwise +
Char-Filter from Li et al. (2024).) We additionally run GPT-4 for comprehensive evaluation. • Rel: standalone ranking by edge relations, without using any candidate context itself. • Full: full interpolation by Eq (1) with both edge relations and degrees. Coefficients are set through a holdout set from another novel.

heading: 4.3 Results
text: Table 1 shows the zero-shot evaluation results on the test set of RECIDENT. Notably, the interpolation with NARCO edges (Full) consistently brings improvement upon the baseline (BL), by 4.9 / 2.4 F1 on NDP / GOT respectively with ChatGPT, up to a 21.7% relative improvement. The stronger GPT-4 boosts performance for all methods as expected, and NARCO still advances 3.5 / 4.7 F1 upon BL on NDP / GOT as well.
Moreover, selection solely based on edge relations (Rel) obtains comparable or better performance than the baseline, with the only exception on GOT with ChatGPT. Overall, Table 1 effectively demonstrates the edge efficacy of NARCO that expresses coherence through retrospective questions.
For in-depth analysis, we further perform two additional evaluation with ChatGPT: • Deg: standalone ranking by edge degrees; for tied
degrees, closer candidates are prioritized. • Full-F: the Full setting with all generated ques-
tions, without Filtering by self verification.
Table 2 shows the additional evaluation results, where ranking by edge degrees of NARCO exhibits decent performance. It even surpasses the baseline on NDP by 1+%, which is impressive for the fact that it does not undergo any task-specific inference. Understandably, it indeed lags behind the baseline on GOT by a noticeable margin.
For Full-F, the degradation is trivial from Full. It is also expected, as the LLM scoring on relations is based on the presence of “good” questions that reflect recap information, which should be retained by the verification stage. Thus, our approach with NARCO is shown robust against noisy questions.

heading: 4.4 Graph Insights
text: The majority of generated questions in NARCO are what/why/how-type of questions. Their ratios are provided in Table 3, along with the averaged number of questions per edge before / after the self verification stage (Section 3.2).

heading: 5 Study II: Node Augmentation
text: Our second study underscores the NARCO utility of local context augmentation, examining whether the graph typology could enrich node representation with global contextual information.
Specifically, for a node vj , a preceding node vi and succeeding node vk such that i < j < k, eij depicts outgoing questions arising from vj to vi, and ejk specifies incoming questions from vk that can be clarified by ej . These questions either highlight important aspects of events or situations in the current context, or provide implication of subsequent development. Such auxiliary information from neighboring nodes is especially useful for retrieval on narratives, as each passage tends to be more interconnected with others than isolated.
We hence investigate if an embedding function on top of NARCO could lead to enriched local representation. Towards this objective, we consider the plot retrieval task defined in (Xu et al., 2023b), which aims to find the most relevant story snippets given a query of short plot description. It is challenging as queries are often abstract based on readers’ overall understanding of the stories, requiring essential background information clarified on candidates, analogous to the concept of decontexualization (Choi et al., 2021). Retrieval on narratives thereby fits our evaluation purpose well.

heading: 5.1 Approach
text: For this task, candidate snippets from stories are retrieved upon a given query. We regard all candidate snippets as graph nodes to be retrieved from, and derive NARCO edges of neighboring nodes. Our proposed method focuses on fusing edge questions into node representation for enhanced retrieval.
Xu et al. (2023b) follows the classic paradigm of contrastive learning that learns a BERT-based encoder (Devlin et al., 2019) on queries and candidates. As its trained model is not released as of this writing, our approach adopts the public BGE encoder (Xiao et al., 2023) in this work that ranks top on the MTEB leaderboard1. For comprehensive evaluation, we propose methods with NARCO for both zero-shot and supervised settings.

heading: 5.1.1 Zero-Shot Retrieval
text: Since edge questions are available to provide auxiliary information, edges can be directly integrated in the zero-shot retrieval process. Our motivation is straightforward: if there can be improvement with zero-shot retrieval, it ensures that these questions bring positive information gain, thus confirming the efficacy for augmenting local context.
Concretely, the hidden states (embeddings) for the query, nodes and edges are obtained by the 1https://huggingface.co/spaces/mteb/leaderboard
encoder. Let hvi be the L2-normalized hidden state for the ith node, heij for its jth outgoing questions, hq for the query. The interpolated similarity Si between the query and ith candidate is defined as:
S = hq · hvi + λ ·max(hq · heij)|Mj=1 (2)
The final similarity S is the typical query-node similarity interpolated with the query-edge similarity by λ, which is then the max query-question similarity out of total M questions. S among all nodes are then sorted for retrieval ranking, being a zero-shot approach without task-specific training.

heading: 5.1.2 Supervised Learning
text: We then introduce our proposed supervised approach that reranks candidates with augmented node embeddings. Specifically, the enrichment is formulated as an attention, with the user query as query, edge questions as both key and value, such that a new node embedding is obtained attending its edge questions conditioned on the query. Let Ai be the attention scores of the ith candidate node, the augmented node embedding hai is denoted as:
Ai = softmax ((hqWQ)(heijWK)T )√
d
) |Mj=1 (3)
hai = h v i +Ai (heijWV )|Mj=1 (4)
WQ/K/V is the parameter for query/key/value in attention, and d is the query dimension size. For a node vi, we provide both outgoing and incoming questions to/from its direct neighbor node for bidirectional contextual information.
With the augmented embedding for the ith node hai , the model simply reranks top retrieved candidates from a baseline system. It is trained with the supervised contrastive loss (Khosla et al., 2020) to maximize the similarity between each query q and its positive targets P (q) among N in-batch candidates (details in Appx A.3):
L = −1|P (q)| ∑
x∈P (q) log exp(hq · hax)∑N y=1 exp(h q · hay) (5)

heading: 5.2 Experiments
text: Data For experiments situating our purpose, we adapt the data from (Xu et al., 2023b) with slight modification. First, we use the available data of Notre-Dame de Paris in Chinese for training and evaluation, instead of using all available novels to avoid large-scale graph realization. Second, the original task operates retrieval on sentence-level.
Similar to Section 4, we take short snippets as graph nodes, and label positive snippets converted from the original positive sentences. The resulting dataset has 1288 candidate snippets in total, with 29484/1000/510 queries for the train/dev/test split.
Evaluation Metric A query may have one or many positive snippets (up to 7). We take the typical information retrieval metric normalized Discounted Cumulative Gain (nDCG), assigning the same relevance for each positive snippet equally.
Methods Four methods are evaluated as follows; all methods adopt BGE-Large encoder2. • Zero Shot (ZS): the zero-shot method that ranks
candidates based on the query-node similarity. • ZS+NARCO: our proposed interpolation with
query-edge similarity; λ is tuned on the dev set. • Supervised (SU): the baseline supervised model
without leveraging NARCO. • SU+NARCO: our proposed rerank model that
utilizes the global-contextualized embeddings; the inference reranks top 50 candidates by SU.

heading: 5.3 Results
text: Table 4 shows the evaluation results of the four settings. Notably, our proposed zero-shot interpolation with query-edge similarity improves upon its baseline on all nDCG metrics, leading 3.4% on nDCG@10 (λ = 0.1), confirming the positive information gain from edges that contribute useful contextual information. The same trend still holds up for the supervised model that learns enriched embeddings leveraging edge relations, especially by the 2.4% improvement on nDCG@1.
Overall, NARCO is shown helpful towards the acquisition of better local representation, through the explicit relational dependencies beyond local context. The empirical results advocate the direc-
2https://huggingface.co/BAAI/bge-large-zh-v1.5
tion of fine-grained context modeling, which could foster a more nuanced comprehension.

heading: 6 Study III: Broader Application
text: Our last study sheds light on the potentials of graph utility in broader applications. As a first step towards this new direction, in this work, we evaluate with Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) in the task of long document question answering. Experiments are conducted on QuALITY (Pang et al., 2022), a multi-choice QA dataset on narrative documents, mostly being fiction stories from Project Gutenberg. With an averaged length of 5k+ tokens per document, we adopt the retrieval-based approaches, where relevant snippets conditioned on the question are retrieved first, then fed to a LLM to generate answers, following a standard RAG paradigm.
Especially, QuALITY was constructed with global evidences in mind: questions may require multiple parts in the document to reason upon. Therefore, NARCO may assist to recognize more relevant snippets through the extracted relations across the narrative context, leading to improved QA performance benefited from enhanced retrieval.
Methods Retrieval-based approaches are commonly adopted for tackling long context, which have been evaluated on QuALITY by previous works (Pang et al., 2022; Xu et al., 2024; Sarthi et al., 2024). Following these setup, we split the full document by short snippets, and retrieve relevant snippets with regard to the question, which are then concatenated as the shortened context for subsequent zero-shot QA inference by LLMs. To leverage NARCO, we apply the same retrieval process described in Section 5.1.1 to identify relevant snippets, where the query-edge similarity is interpolated as in Eq (2) using BGE-Large encoder.
Experiments We employ Llama2 (Touvron et al., 2023) and ChatGPT for the zero-shot QA inference. As evaluation on the test set requires submission to the ZeroSCROLLS leaderboard (Shaham et al., 2023), we first perform fine-grained performance analysis on the dev set with short retrieved context (<1k tokens), then submit the final test set results using ChatGPT with 1.5k context limit, aligned with Xu et al. (2024) for direct comparison. The baseline retrieval method and our Enhanced retrieval are denoted by R and ER respectively.
Table 5 & 6 present the evaluation results on the
dev set and test set respectively. Results on the dev set suggest that ER can boost QA performance with all LLMs, especially with the smaller 7B model by 5% accuracy, fulfilling our initiative to effectively utilize NARCO in broader applications. The improvement from enhanced context retrieval is consistent, further confirmed by the 2% leading margin with ChatGPT on both the dev and test set.
Having demonstrated that NARCO can improve RAG in narratives through enhanced retrieval, its utility beyond the retrieval process may be further exploited, e.g. potential facilitation on LLM pretraining or inference directly. We leave future research to explore additional integration of finegrained context modeling.

heading: 7 Conclusion
text: We address the distinctive characteristics of narratives, and propose a novel paradigm of fine-grained context modeling, which explicitly captures the inter-connective coherence within narrative context. A graph is thereby formulated, dubbed NARCO, with edges encompassing free-form retrospective questions to depict the relational dependencies. NARCO is practically realized by LLMs via our designed two-stage prompting scheme, leveraging the promising development of LLMs without reliance on human annotations. To examine the graph properties and its utility, three unique studies are conducted, where NARCO is shown to bring empirical improvement on various narrative applications.

heading: Limitations
text: While we have demonstrated the usefulness of our proposed NARCO, upon manually verifying the generated edge questions, deficiencies do exist in the current graph generation approach: • The generated questions are not free from noises,
as mentioned in Section 3. One common scenario occurs when pairs of context chunks are irrelevant to each other. GPT-4 struggles to accurately identify irrelevancy, leading it to ask questions that lack informativeness. • Our approach does not handle the scenario where there is joint dependency among three or more chunks. As we generate questions upon pairs, sometimes the key connecting information exists in the third chunk and is missing, preventing the recognition and formulation of useful questions.
Despite the aforementioned issues, our graph still proves beneficial in various applications. This is partly due to the fact that Large Language Models (LLMs) and our learned models possess the capability to automatically discern which information to utilize. Still, enhancing the quality of questions could further augment the benefits derived from our graph, highlighting the potentials of our proposed representation of narrative context.
An additional limitation lies in our filtering algorithm. For LLMs that struggle with following instructions accurately, the current filtering strategy may prove inadequate. For instance, if an LLM repeatedly poses questions that could be understood and answered solely by referring to prior texts, our filtering process is inefficiency to rule out these questions. One potential solution to mitigate this issue could involve implementing a matching model between the questions and the target texts. However, since our work employs GPT-4 alongside Chain-of-Thought, which effectively reduces such instances of shortcut-taking, we have opted to retain the current strategy. We acknowledge the possibility of exploring alternative LLMs with more sophisticated filtering strategies in future work.

heading: A Graph Realization

heading: A.1 Full Prompts and Details
text: Full prompts of our designed two-stage LLM prompting scheme (Section 3) are provided in Figure 3-5. We specify the maximum number of generated questions for a node pair as 4 in the prompt.
For the task of plot retrieval (Section 5) and long document QA (Section 6), we construct edges within a neighboring window of 4 preceding nodes, such that the graph realization is proportional to the input instead of being quadratic. For recap identification (Li et al., 2024), edges are obtained on the provided preceding snippets.
For a context with Tk tokens, it takes approximately 6Tk tokens to obtain all edge questions of NARCO using GPT-4, which costs $0.03T as of this writing.

heading: A.2 Qualitative Examples
text: Examples of generated questions on Game of Thrones from RECIDENT (Li et al., 2024).
A.2.1 Case1

heading: Current Context:
text: At Craster’s Keep, Locke scouts the keep for the party of the Night’s Watch sent to eliminate the traitors holed up there; in his reconnaissance, Locke finds the hut where Bran Stark, Jojen, Meera and Hodor are being held captive. Reporting back to Jon Snow and the others, Locke tells them that only eleven traitors are present and most of them are drunk and won’t prove much of a threat. He also lies about the hut where Jon’s brother Bran and his group are being imprisoned, claiming there are only hounds kept inside and that they should keep away from it to prevent the dogs alerting their enemy. Believing Locke, Jon agrees and tells the party they attack at nightfall. Prior Context: Having received word of the wildlings’ raids down south, the Lord Commander states that they do not have the manpower to afford venturing away from the Wall. They are interrupted when Edd and Grenn return to Castle Black after escaping Craster’s Keep. Jon reveals he told Mance Rayder that a thousand men armed Castle Black and therefore points out that when Mance reaches Craster’s Keep, Rast and Karl Tanner will not hesitate in revealing the truth. Jon then insists the Night’s Watch send a party to Craster’s Keep to kill their traitor brothers before Mance gets to them first. Generated Question (Valid): What prompted the Night’s Watch to act with urgency in sending a party to Craster’s Keep to eliminate the traitors? Generated Question (Invalid): What was the reason behind Jon Snow’s insistence on a strategic assault to silence the traitors before a specific event could occur? (Note: it is a question asked upon the prior context and can be answered by it directly, as addressed in the Limitations Section, not bridging two context.)
A.2.2 Case2

heading: Current Context:
text: In what becomes known as the infamous Red Wedding, Lothar draws a knife and repeatedly stabs the pregnant Talisa in the stomach, killing her unborn child. Talisa collapses to the ground as chaos surrounds. Before he can react, Robb is shot by the musicians with crossbows several times and falls to the floor. Numerous other Stark men are killed by the crossbow bolts or set upon by Frey soldiers. Catelyn is shot by one of the musicians in the back and falls to the floor. Prior Context: In Gendry’s quarters, Melisandre seduces Gendry long enough to distract him, then promptly ties him to the bed and places leeches on his body. She explains as Stannis and Davos enter the room that Davos wanted a demonstration of the power in king’s blood, then removes the leeches and lights a fire in a nearby brazier. As part of the magical ritual that follows, Stannis throws the leeches into the flames at Melisandre’s direction, and recites the names of three people he wants dead as they burn: "The usurper Robb Stark, the usurper Balon Greyjoy, the usurper Joffrey Baratheon." Generated Question (Valid): What ritual was performed prior to the Red Wedding that sought the death of Robb Stark and might have influenced his fate? (Note: it is an open question whether Melisandre’s ritual really worked and is widely discussed among fans. The question uses might which adds its accuracy.)
A.2.3 Case3

heading: Current Context:
text: In King’s Landing, Eddard is summoned to the throne room by "King Joffrey"; Robert has died. He arrives to find Littlefinger and Varys waiting for him, along with Commander Janos Slynt and a detachment of the City Watch. Varys tells him that Renly has fled the city, along with Ser Loras Tyrell and a number of retainers. They were last seen heading south. The party enters the throne room, where Joffrey sits on the Iron Throne. He demands oaths of fealty from his councilors and subjects. Instead, Eddard gives Ser Barristan Selmy the proclamation naming him as Lord Protector of the Realm. To Barristan’s shock, Cersei takes the "paper shield" and tears it up. Instead, she suggests that Eddard bend the knee and swear allegiance. Prior Context: Lord Eddard Stark meets with Cersei Lannister. He tells her that he knows the secret that Jon Arryn died for: that Cersei’s three children are not Robert’s, but the product of incest between her and Jaime. Cersei does not deny the charge and in fact is proud of it, comparing their love to the old Targaryen practice of marrying brother to sister; she also admits to having despised Robert ever since their wedding night, when Robert drunkenly stumbled into Cersei’s bed and called her "Lyanna". Eddard angrily tells her to take her children and leave the city immediately. When Robert returns from his hunt, he will tell him the truth of the matter and Cersei should run as far as she can before that happens, lest Robert’s wrath find her. Generated Question (Valid): What is the reason behind Eddard Stark’s refusal to swear fealty to Joffrey and his decision to present a proclamation in the throne room? Generated Question (Invalid):
What prevented Eddard Stark from informing King Robert about the illegitimacy of Cersei’s children, which could have significantly altered the succession to the Iron Throne?
A.2.4 Case4

heading: Current Context:
text: In what becomes known as the infamous Red Wedding, Lothar draws a knife and repeatedly stabs the pregnant Talisa in the stomach, killing her unborn child. Talisa collapses to the ground as chaos surrounds. Before he can react, Robb is shot by the musicians with crossbows several times and falls to the floor. Numerous other Stark men are killed by the crossbow bolts or set upon by Frey soldiers. Catelyn is shot by one of the musicians in the back and falls to the floor. Prior Context: At Harrenhal, Jaime speaks one last time to Brienne before he leaves. Jaime remarks that he owes Brienne a debt for both keeping him alive on their journey and for giving him a reason to live to rouse him from his suicidal depression after losing his hand. Brienne tells Jaime to repay his debt by keeping his pledge. Jaime promises that he will keep his word to return Catelyn Stark’s daughters to her. Generated Question (Invalid): What prior commitment made by Jaime Lannister could influence the fate of the Stark family following the Red Wedding, where Catelyn Stark is among those attacked? (Note: the question is rather irrelevant in regards to the two context snippets.)
A.2.5 Case5

heading: Current Context:
text: Tormund and Beric Dondarrion review the defenses atop the Wall at Eastwatch-by-the-Sea. Tormund remarks that the crows say he’ll get used to the height, but he admits it’ll probably be a while. Suddenly, the pair sees movement at the edge of the Haunted Forest. A White Walker emerges atop an undead horse, followed shortly by a horde of wights. More and more White Walkers emerge as the Night Watch’s horns sound three times. However, the army of the dead stops some distance from the foot of the Wall and Tormund looks relieved; despite their numbers, the dead don’t have anything that could possibly get them past the barrier. But then all on the Wall stop in horror as they hear a very familiar sound; a screeching roar mixed with the heavy thumping of huge wings beating the air. Prior Context: At Eastwatch, Sandor carries the struggling Wight into a boat. Tormund and Beric tell him they will meet again but Sandor retorts he hopes not. Daenerys sends Drogon and Rhaegal to scour the surrounding mountains for Jon. Jorah tells Daenerys that it is time to leave but she insists on waiting a bit longer. Before she can leave, they hear a horn blowing signaling a rider approaching. Looking down from the battlements, Dany sees a wounded Jon Snow approaching on horseback. Aboard their ship, Davos and Gendry remove the frozen-stiff garments and tend to Jon Snow, who has suffered severe hypothermia and several minor injuries. Daenerys also notes the massive scars on his chest from his previous fatal wounds. Geneated Question (Invalid): What was Daenerys waiting for at Eastwatch before Jon Snow’s wounded arrival on horseback?
(Note: this is another example of asking upon the prior context, which could happen more often than irrelevant questions.)

heading: A.3 Experiments
text: LLM The usage of ChatGPT (gpt-3.5-turbo) and GPT-4 (gpt-4-1106-preview) is through OpenAI’s paid API service. For the open-source Llama-2 (Touvron et al., 2023), we perform inference on Nvidia A100 GPUs.
Training For training a rerank model in Section 5, we initialize a BERT model with weights from BGE-Large (Xiao et al., 2023), and use the mean-pooled token embeddings as the sequence representation, following the standard S-BERT setup (Reimers and Gurevych, 2019). The training is conducted on one Nvidia A100 GPU, taking around 6 hours to finish, with 20 epochs, 20 queries within each batch, learning rate 2× 10−5, cosine learning rate schedule, and a warmup ratio of 5× 10−2.

heading: Future Work
text: We acknowledge the possibility of exploring alternative LLMs with more sophisticated filtering strategies in future work