Venue: NAACL 2025

# Context-Efficient Retrieval with Factual Decomposition

Yanhong Li

University of Chicago / TTIC

yanhongli@uchicago.edu

&David Yunis

Toyota Technological Institute at Chicago

dyunis@ttic.edu

&David McAllester

Toyota Technological Institute at Chicago

mcallester@title.edu

&Jiawei Zhou

Stony Brook University

jiawei.zhou.1@stonybrook.edu

###### Abstract

There has recently been considerable interest in incorporating information retrieval into large language models (LLMs). Retrieval from a dynamically expanding external corpus of text allows a model to incorporate current events and can be viewed as a form of episodic memory. Here we demonstrate that pre-processing the external corpus into semi-structured "atomic facts" makes retrieval more efficient. More specifically, we demonstrate that our particular form of atomic facts improves performance on various question answering tasks when the amount of retrieved text is limited. Limiting the amount of retrieval reduces the size of the context and improves inference efficiency.

## 1 Introduction

Although large language models (LLMs) demonstrate remarkable capabilities across various tasks, their inability to continuously adapt to dynamic or domain-specific knowledge without parameter updates remains a substantial limitation. To address this limitation retrieval-augmented generation (RAG) supplements models with some external knowledge source during inference Lewis et al. (2020); Ram et al. (2023); Borgeaud et al. (2022a). Typically these models treat the external source as a set of arbitrarily segmented blocks of raw text. However, there has also been interest in using more structured external knowledge sources such as knowledge graphs Edge et al. (2024); Peng et al. (2024), compressed documents Xu et al. (2024) or document trees Sarthi et al. (2024). In each case one can identify a "unit of retrieval" where one retrieves some set of such units, such as a set of documents or a set of knowledge graph triples. Various candidates for units of retrieval, or "atomic facts", have been formulated Chen et al. (2023); Jiang et al. (2024c); Min et al. (2023); Gunjal and Durrett (2024).

In designing units of retrieval there is a tension between concise but brittle logical representations, such as knowledge graph triples, and highly expressive and nuanced, but verbose and unstructured, chunks of raw text. We propose an intermediate retrieval unit that we call an entity-description pair (EDP). This is a pair of an "entity"1 and some form of description of that entity. For example the entity might be "Alan Turing's contributions" and the

Figure 1: Example of datastore used for knowledge retrieval in our approach compared with typical fixed-size text chunks in RAG. We retrieve much shorter contexts.



factual description could be "Pioneered modern computing, laid the foundation of computer science and engineering". Each EDP is a structured piece of information, like in structured databases, but also enjoys the flexibility of natural language. See Figure 1. We use a three-step language model prompting protocol to decompose a chunk of free text into a collection of EDPs and use the EDPs as the unit of retrieval in the resulting EDP knowledge base (KB). See Figure 2.

Our main result is a demonstration that on various challenging question answering benchmarks EDP KB retrieval achieves better accuracy when the amount of retrieval (the number of retrieved tokens) is limited. This can be phrased as improving the "context-efficiency" of RAG. We are also optimistic that our formulation of EDP KBs is a significant step toward more structured yet expressive internal representations of knowledge.

## 2 Related Work

Context-Efficient RetrievalAs we will see in experiments, our approach achieves superior performance in context-efficient retrieval, which we define as RAG methods aiming to reduce retrieved contexts for cost-effective LLM generations. Previous related work involves various compression methods. Some focus on vector-based compression, where models learn to compress long contexts into compact memory slots through end-to-end training Ge et al. (2024); Cheng et al. (2024). Others are text-based compression, which includes training rerankers Pradeep et al. (2023), applying extractive summarization Xu et al. (2024), or training abstractive summarizers to compress the retrieved context Xu et al. (2024); Jiang et al. (2024b). We also reduce retrieved contexts, but rather than compressing them _post-retrieval_, we achieve context efficiency from the outset through improved knowledge representation _pre-retrieval_. Post-retrieval context compression methods still rely on token chunks as coarse units of knowledge for retrieval, whereas we structure the knowledge more efficiently with clear, well-defined representations that maintain high expressivity.

Knowledge Representation for RetrievalMost previous works directly segment source documents into equal-length text chunks, each containing hundreds of tokens Lewis et al. (2020); Ram et al. (2023); Borgeaud et al. (2022a). Recent research has explored alternative formats for knowledge representation, such as indexing source documents using knowledge graphs Edge et al. (2024), hierarchical tree structures Sarthi et al. (2024), or more relaxed versions of knowledge graphs Liang et al. (2024).

Unlike these works, we don't rely on any explicit relational structures; our knowledge datastore consists of flat, semi-structured entity-description pairs The work most similar to ours is Chen et al. (2023), which decomposes each sentence in Wikipedia into individual propositions for retrieval. However, we propose a novel method where speculated queries generated by the LM guide the fact extraction, and repeated samples of factual decompositions enhance our database. These techniques yield significant performance improvements by enabling more targeted information extraction and increased coverage of constructed facts, resulting in more relevant and concise data being incorporated during inference.2

Footnote 2: More general background of RAG is in Appendix A.

## 3 Methodology

As shown in Figure 2, our method consists of three main steps: (1) question speculation, (2) atomic fact extraction, and (3) knowledge base (KB) augmentation. This approach allows to decompose long documents into concise, factual entity-description pairs (EDPs), building up a semi-structured KB for RAG, reducing retrieval overhead on the lengths of contexts to improve inference efficiency.

### Question Speculation

Let \(D\) represent a long document, which is split into \(N\) equal-length chunks \(D=\{D_{1},D_{2},\ldots,D_{N}\}\), where each chunk \(D_{i}\) contains approximately the same number of tokens. For each chunk \(D_{i}\), we prompt a LM to speculate a set of possible questions \(Q_{i}=\{q_{i1},q_{i2},\ldots,q_{iJ}\}\)

Figure 2: Overview of our method.

 

where \(q_{ij}\) represents a potential question one might ask about the chunk \(D_{i}\). This question speculation process helps direct the extraction of relevant knowledge and facilitates targeted information retrieval later.

Formally, given a document chunk \(D_{i}\), we define the question speculation function as: \(Q_{i}=\text{LM}_{\text{speculate}}(D_{i})\), where \(\text{LM}_{\text{speculate}}\) denotes the question-speculation language model. The result is a set of speculative questions \(Q_{i}\) for each document chunk \(D_{i}\).

Prompts for \(\text{LM}_{\text{speculate}}\) are shown in Table 5 and Table 6 in Appendix G.

### Query-Guided Factual Decomposition

Once we have the speculative questions \(Q_{i}\), we feed both the set of questions \(Q_{i}\) and the corresponding document chunk \(D_{i}\) into the language model to extract relevant information that can be used to answer the questions. The goal is to retrieve concise, atomic facts that are highly specific and contextually relevant.

We prompt the language model to produce a set of EDPs for each chunk, where EDP is defined as a pair \(k_{im}\) consisting of an entity \(e_{im}\) and a fact \(f_{im}\). The entity \(e_{im}\) represents a key concept, while the fact \(f_{im}\) encapsulates the essential information regarding \(e_{im}\). Notably, the entity needs not be limited to a noun or an entry from a traditional knowledge graph; it can be a short noun phrase, sentence, or even a question.

The extraction process for a chunk \(D_{i}\) is as: \(K_{i}=\text{LM}_{\text{extract}}(D_{i},Q_{i})\), where \(K_{i}=\{k_{im}=(e_{im},f_{im})\}^{M}_{m=1}\) is the set of EDPs for chunk \(D_{i}\). This method ensures that the extracted knowledge is both flexible and informative. Note that each EDP \(k_{im}\) does not have to correspond to a particular query \(q_{ij}\) as \(K_{i}\) are generated collectively with guidance from all \(Q_{i}\), and the total number of of EDPs \(M\) could vary, regardless of the size of \(Q_{i}\). Prompts for \(\text{LM}_{\text{extract}}\) are shown in Table 7 and Table 8 in Appendix G.

### Sample Augmentation

To further enrich the knowledge base, we apply a sampling-based approach that augments the fact extraction across multiple runs. By repeating the extraction process multiple times using the same prompt and leveraging the inherent randomness of the LM's outputs, we capture diverse sets of EDPs and prevent information gaps. We aggregate the knowledge extracted from these different runs to build a more comprehensive and robust KB.

Let \(S\) denote the number of sampling runs. For each document chunk \(D_{i}\), we repeat the extraction process, including both question speculation and EDP extraction, \(S\) times, yielding multiple KBs: \(K_{i}^{(1)},K_{i}^{(2)},\ldots,K_{i}^{(S)}\). We then merge these KBs to form a final, augmented knowledge base \(K_{i}^{\text{final}}\) for chunk \(D_{i}\): \(K_{i}^{\text{final}}=\bigcup_{s=1}^{S}K_{i}^{(s)}\). The final knowledge base for the entire document \(D\) is then constructed by merging the augmented knowledge from all chunks: \(K^{\text{final}}=\bigcup_{i=1}^{N}K_{i}^{\text{final}}\). \(K^{\text{final}}\) provides a rich semi-structured knowledge repository for retrieval, where units are each EDP.

## 4 Experiments

### Setup

Following prior work [30], we evaluate our method on three long-context QA datasets:

**NarrativeQA**[9] consists of questions based on books and movie transcripts, requiring comprehension of entire stories. We report BLEU-4 [20], ROUGE-L [15], and METEOR [19]

Figure 3: Results on NarrativeQA (top three plots), Qasper (bottom right), and quality (bottom left). The x-axis represents the number of tokens fixed in the retrieval context, and y-axis are different QA metrics used for each dataset.

 

scores on the test set to measure the quality of generated answers, following previous work (Sarthi et al., 2024).

Qasper Dasigi et al. (2021)includes questions from NLP research papers, focusing on detailed information extraction from full texts. Answers are categorized as Answerable/Unanswerable, Yes/No, Abstractive, and Extractive. We evaluate using the F1 metric on the test set, reflecting the overlap between predicted and reference answers, following previous work (Sarthi et al., 2024).

QuALITY Pang et al. (2022)contains multiple-choice questions paired with context passages averaging around 5,000 tokens from various English articles (e.g., sci-fi, magazine articles, nonfiction). Since the test set is not public, we report accuracy on the validation set, measuring the proportion of correctly answered questions, following previous work (Sarthi et al., 2024). We use BM25 (Robertson and Zaragoza, 2009) as the retriever for both standard retrieval and our method, due to its effectiveness in prior studies. For our EDP-based knowledge base construction, we employ ChatGPT (gpt-4-2024-08-06) (OpenAI et al., 2024), which generates entity decomposition propositions efficiently. For question answering, we use Mixral-8x7B-Instruct-v0.1 (Jiang et al., 2024a), a state-of-the-art instruction-tuned language model suitable for downstream QA tasks. Following RECOMP (Xu et al., 2024), we compare our approach to the following baselines to ensure a fair evaluation:

* **Decomposition into Propositions**(Chen et al., 2023): Uses ChatGPT to decompose documents into propositions, aiming to enhance retrieval by indexing finer-grained units.
* **Retrieve-then-Summarize**(Xu et al., 2024): Utilizes off-the-shelf summarizers like T5-large (Raffel et al., 2023) and GPT-3.5 (Brown et al., 2020) to condense retrieved documents before answering.
* **Standard Retrieval**: Applies BM25 on raw document chunks without any decomposition or summarization.

### Results

Figure 3 shows results on NarrativeQA, Qasper, and QuALITY. We see that our method consistently outperforms all baselines when the number of tokens in the context is kept at _all_ different levels, shown with our curve above all others. Our method performs especially well in the short-context regime when the retrieved tokens are very limited. This can effectively reduce LLM inference cost with more efficient usage of context in RAG. We also find that decomposing sentences into propositions (Chen et al., 2023) does not generalize well to domains with lower fact density, such as novels or scientific papers. Some qualitative examples of retrieved documents for each method are provided in Appendix C.

### Analysis

Here we ablate each component of our method (Section 3) on their contribution to the overall performance.

Why Question Speculation?Despite extensive prompt tuning, providing speculated queries to LM when generating the KB consistently yields better performance compared to letting the LM extract facts without guidance (see Figure 4). To investigate this, we randomly select 20 stories (617 associated queries) from NarrativeQA and compute the similarities between the speculated questions and the real queries. Surprisingly, using similarity thresholding heuristics and manual inspection, we find that 11.18% of the speculated questions

Figure 4: Comparison of performance (y-axis) vs. number of retrieved tokens (x-axis) between Fact-Only KB construction and Question-speculated KB construction on a subset of NarrativeQA’s validation set.

Figure 5: Performance (y-axis) vs. number of retrieved tokens (x-axis) on NarrativeQA for the number of re-sampled KBs equal to 1, 3, and 5.

 

closely align with or rephrase the real queries, and 53.97% focus on the same topic (for more details, see Appendix D). This significant overlap aligns with previous research showing that LLMs are effective at generating synthetic queries Wu and Cao (2024). In fact, these speculated questions function like a chain-of-thought process Wei et al. (2024), allowing the LM to gather relevant information before answering the query.

Why KB Augmentation?We observe that the questions speculated and facts extracted vary between different runs due to the LM's inherent stochasticity. Figure 5 shows that augmenting KBs improves performance, indicating that the sampling process effectively captures a more diverse range of meaningful knowledge pairs. Full results for NarrativeQA, Qasper, and QuALITY are in Appendix E.

### Quality Checks on Speculative Questions and EDPs

While our method demonstrates strong performance, we carefully evaluated the quality of the speculative questions that guide fact extraction and the generated EDPs.

Automatic Evaluation of Speculative QuestionsAs a proxy for assessing the quality of the generated questions, we measure their similarity to real queries in the validation set of the corresponding dataset (using 20% of that set for this evaluation). Following standard practice, we employ an embedding-based similarity approach using the all-MiniLM-L6-v2 model from Hugging Face's sentence-transformers.3 The higher the average similarity scores, the more closely the speculative questions resemble real queries, which is desirable. Examples of some of the highest-similarity pairs can be found in Appendix F (Table 4).

Footnote 3: Available at [https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)

Manual Evaluation of EDPs.To ensure consistency and accuracy of EDPs, we also conducted a thorough manual evaluation. Specifically, we randomly selected 200 examples from our generated datastore for each dataset (NarrativeQA, Qasper, QuALITYY). A team of three reviewers independently assessed the quality, coherence, and correctness of the EDPs. We did not identify any contradictions or significant issues in these sampled EDPs.

## 5 Conclusion

We have demonstrated that on various challenging question answering benchmarks EDP KB retrieval achieves better accuracy when the amount of retrieval (the number of retrieved tokens) is limited. This improves the "context-efficiency" of RAG, which has real cost-effective implications for LLM inference. We are also optimistic that our formulation of EDP KBs is a significant step toward more structured yet expressive internal representations of knowledge, and we encourage future research to build upon and expand this approach.

LimitationsWhile our approach demonstrates improved context-efficiency in retrieval-augmented generation for question answering tasks, several limitations warrant discussion. First, our method relies heavily on the performance of large language models for both question speculation and factual decomposition. Any biases or errors inherent in these models could propagate through the process, potentially affecting the quality and reliability of the extracted entity-description pairs.

Second, the stochastic nature of our sampling-based augmentation introduces variability in the generated knowledge bases. Although multiple samples help capture a broader range of information, this approach may lead to inconsistencies across different runs. Further research is needed to assess the stability and reproducibility of the results when applying our method in diverse settings.

In summary, while our method enhances context-efficiency, it remains vulnerable to inherent LLM biases and sampling-induced variability. Addressing these issues is crucial for improving the reliability and consistency of our approach in various applications.

## References

* Asai et al. (2024) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In _The Twelfth International Conference on Learning Representations_.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Arbor, 

Michigan. Association for Computational Linguistics.
* Bertsch et al. (2023) Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. 2023. Unlimformer: Long-range transformers with unlimited length input. _Advances in Neural Information Processing Systems_, 36.
* Borgeaud et al. (2022a) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022a. Improving language models by retrieving from trillions of tokens. In _Proceedings of the 39th International Conference on Machine Learning_, volume 162 of _Proceedings of Machine Learning Research_, pages 2206-2240. PMLR.
* Borgeaud et al. (2022b) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022b. Improving language models by retrieving from trilions of tokens. In _International conference on machine learning_, pages 2206-2240. PMLR.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prrafulla Dhariwal, Arvind Neelakantan, Panav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _Preprint_, arXiv:2005.14165.
* Chen et al. (2023) Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. 2023. Dense x retrieval: What retrieval granularity should we use? _arXiv preprint arXiv:2312.06648_.
* Cheng et al. (2024) Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Siqing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. 2024. xrag: Extreme context compression for retrieval-augmented generation with one token. _Preprint_, arXiv:2405.13792.
* Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4599-4610, Online. Association for Computational Linguistics.
* Edge et al. (2024) Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. _Preprint_, arXiv:2404.16130.
* Ge et al. (2024) Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. 2024. In-context autoencoder for context compression in a large language model. In _The Twelfth International Conference on Learning Representations_.
* Gunjal and Durrett (2024) Anisha Gunjal and Greg Durrett. 2024. Molecular facts: Desiderata for decontextualization in llm fact verification. _Preprint_, arXiv:2406.20079.
* Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In _International conference on machine learning_, pages 3929-3938. PMLR.
* Jiang et al. (2024a) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lilio Renard Lavaud, Lucile Saulnier, Marie-Anne Lacnaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniakis, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2024a. Mixtral of experts. _Preprint_, arXiv:2401.04088.
* Jiang et al. (2024b) Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024b. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1658-1677, Bangkok, Thailand. Association for Computational Linguistics.
* Jiang et al. (2024c) Ziyan Jiang, Xueguang Ma, and Wenhu Chen. 2024c. Longrag: Enhancing retrieval-augmented generation with long-context llms. _arXiv preprint arXiv:2406.15319_.
* Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In _International Conference on Learning Representations_.
* Kocisky et al. (2018) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. _Transactions of the Association for Computational Linguistics_, 6:317-328.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.

 

Xun Liang, Simin Niu, Zhiyu li, Sensen Zhang, Shichao Song, Hanyu Wang, Jiawei Yang, Feiyiu Xiong, Bo Tang, and Chenyang Xi. 2024. Empowering large language models to set up a knowledge retrieval indexer via self-learning. _Preprint_, arXiv:2405.16933.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2024) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Parajape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. _Transactions of the Association for Computational Linguistics_, 12:157-173.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannanach Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12076-12100, Singapore. Association for Computational Linguistics.
* OpenAI et al. (2021) OpenAI, Josh Achiam, Steven Adler, Sandhnii Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Atlenschmidt, Sam Altman, Shymal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jakse Berndiern, Gabriel Bernatef-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brahman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dolhan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Nike Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Eile Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Gou, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschelle, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhu Jin, Denny Jin, Shino Jonto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanine Lin, Mateusz Litvin, Theresa Lopez, Ryan Lowe, Patricia Luc, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menickx, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vininc Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mely, Ashvin Nair, Reitichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascalondo, Joel Parish, Empy Paraparita, Alexa Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henririque Pond e Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rolsted, Henri Roussee, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shock, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simons, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeeline B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Ceron Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Wong, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Vojeci Laremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Pang et al. (2022) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phanga, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5336-5358, Seattle, United States. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 

40th Annual Meeting on Association for Computational Linguistics_, ACL '02, pp. 311-318. External Links: Link Cited by: SS5.
* B. Peng, Y. Zhu, Y. Liu, X. Bo, H. Shi, C. Hong, Y. Zhang, and S. Tang (2024)Graph retrieval-augmented generation: a survey. arXiv preprint arXiv:2408.08921. Cited by: SS6.
* R. Pradeep, S. Sharifymoghaddam, and J. Lin (2023)RankVicuna: zero-shot listwise document reranking with open-source large language models. arXiv:2309.15088. Cited by: SS7.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2023)Exploring the limits of transfer learning with a unified text-to-text transformer. PreprintarXiv:1910.10683. Cited by: SS 



## Appendix A Background on RAG

Retrieval-Augmented Generation Retrieval-augmented generation Lewis et al. (2020) (RAG) is the process of dynamically adding additional information at inference time through a similarity search process in order to improve generation quality. It is typically used in domains where it may be difficult for the language model to rely on parametric knowledge alone, for example long-tail question answering, or for current events past the training data cutoff date. The simplest form of RAG is to add text related to the query directly to the input Ram et al. (2023). There are also vector-based variants, for example injecting information at deeper layers of the network Borgeaud et al. (2022); Wu et al. (2022); Bertsch et al. (2023), or interpolating with a nearest neighbor generation Khandelwal et al. (2019). Some works tune with retrieval-augmentation Guu et al. (2020), or to induce retrieval behavior Asai et al. (2024). Though retrieval-augmentation is generally quite beneficial, language models can be distracted depending on the order Liu et al. (2024) or content Yoran et al. (2023) of the data retrieved.

## Appendix B Experiment Setup

### Datasets

* **NarrativeQA Kocisky et al. (2018) is a dataset containing 1,572 documents, including books and movie transcripts. It requires answering questions based on the full text of these narratives. The task tests the model's ability to comprehend entire stories, with performance measured using BLEU (B-1, B-4), ROUGE (R-L), and METEOR metrics. We report BLEU-4, ROUGE-L and METEOR on the entire test set.**
* **QASPER Dasigi et al. (2021) consists of 5,049 questions drawn from 1,585 NLP papers, with answers categorized as Answerable/Unanswerable, Yes/No, Abstractive, and Extractive. The questions focus on extracting detailed information embedded within the full text of the papers. Accuracy is evaluated using the F1 metric, reported on the entire test set.**
* **QuALITY Pang et al. (2022) contains multiple-choice questions, each paired with context passages averaging around 5,000 tokens. Since the QuALITY test set is not public, accuracy is reported on the validation set.**

### Details on Setup

For the standard retrieval baseline, we experiment with different token counts within a chunk (see Appendix B.3) and select the best-performing one as the final baseline. In all experiments, we follow Sarthi et al. (2024), using cl100k_base from Tiktoken as the tokenizer to split source documents into chunks and compute final token usage. We use BM25 as the retriever, ChatGPT (gpt-4o-2024-08-06) for our EDP-based KB construction, and Mixtral-8x7B-Instruct-v0.1 for question answering.

### Best Chunk Length for Standard Retrieval Baseline

We perform comprehensive ablation studies to find the optimal chunk length for each retrieved document (see Figure 6, Figure 8, Figure 7). We test chunk lengths of 50, 100, 150, 200, 250, 300, and 350 tokens, ensuring sentence boundaries are respected when chunking the book into fixed-size documents. For each chunk length, we select 5-10 different numbers of documents. We find that a chunk length of 250 tokens achieves the best performance on NarrativeQA, Qasper, and QuALITY, and we use this as the naive retrieval baseline reported in the main text.

## Appendix C Qualitative Examples of Retrieved Documents

We provide datastore examples that are retrieved when answering a question from NarrativeQA. Table 1 shows our retrieval compared to the standard retrieval, and Table 2 shows the retrieval following Chen et al. (2023)'s proposition method. We find that our retrieval leads to the best final answer, while the other two approaches struggle to retrieve the correct information from their datasores. The standard baseline fails to find the relevant chunk from the book, and the proposition baseline decomposes all human dialogue into even smaller units, which makes the information more scattered and harms retrieval

- "how does happy harry hardon describe his listeners?: happy harry hardon refers to his listeners as 'horny'.",
- "what actions does happy harry hardon encourage his listeners to take when contacting him?: happy harry hardon encourages his listeners to send him their most pathetic moments, as long as they are real, and wants details like size, shape, feel, and smell.",
- "who is happy harry hardon and what role does he play?: happy harry hardon is a radio host, engaging with his listeners over the air, and helping chris discuss his experiences and feelings.",
- "what contest does happy harry hardon mention during his show and how does he engage his listeners?: happy harry hardon runs a contest on the best way to put parents out of their misery and challenges listeners to amaze and discuss the sensational content of his show.",
- "what concern does mark have about the effects of his message?: mark feels ill about the situation and perceives it as chaotic, tangled with his life",
- "what does mark express about his feelings in the conversation with his parents?: he finds it difficult to talk to girls in his new place and feels different.",
- "what reaction does mark have to the police car pulling up?: mark is not worried and thinks the police are just dropping in on his neighbor.",
- "what internal conflict does mark express after his interaction with nora?: mark expresses that the whole thing is making him ill and that it's his life that mora is messing with.",
- "what critical message does mark want to convey to his audience about taking control of their lives?: mark conveys that it's their life, they should take charge of it, and they should fill the air and speak out.",
- "english class question: jan emerson asks mark to share his feelings about what he wrote." Our answer: take charge of their lives. Standard retrieval baseline: -happy harry hardon
- you interview a student and then you rat on her, you betray her trust, isn't that right sir <deaver hangups upvells well as you can see, these guys are played out society is mutating so rapidly that anyone over the age of twenty has really no idea err alright, back down to business i share a room with my older brother and nearly every night after he turns off his light the come over to my bed and gives me a few arm nookies and stuff and then makes me scratch his back and other refinements!!'s about time we had some refinements on this show then sooner or later he gets worked up and further a do he rub shis thing and makes me watch signed i'm just screwed upwell first of all you're not screwed up, your an unscrewed up reaction to a screwed up situation feeling screwed up at a screwed up time, in a screwed up place does not make you necessarily screwed up, if you catch dry drift well as you know fear listers if you enclose your number a reply is guaranteed <rings miss screwed up>",
- it's the trouble makers, you can't run a top school with trouble makers in the mix brian
- okay, so what exactly as a trouble marker creswood
- someone who has no interest in education brain
- oh c'mon that includes every teenager i know crewsood
- can't yout understand that nothing is more important than a good education brain
- except for the basic right to it crewsoud
- the point is i have the highest s at scores in the state brain
- yeah but how crewood
- i stand by my record <the school field> sheep sheppard
- mr watts, shepberand channel six news here watts
- good evening shepberand
- how does washington intend to deal with this situation watts
- we at the f c c feel that democracy is about protecting the rights of the ordinary citizen un-regulated radio would result in programming of the lowest common denominator, the rule of the mob <watts looks round to see one of his f c c vans has picked up some garffitiotis this is vandalism, not free expression <everyone gathers outside the school to listen to the happy harry hardon show>" Standard retrieval's answer: mark does not urge his listeners to do anything

\begin{table}
\begin{tabular}{l} \hline
**Question**: What does Mark urge his listeners to do? Ground Truth: to do something about their problems instead of committing suicide \\ \hline
* "how does happy Harry hardon describe his listeners?: happy harry hardon refers to his listeners as ’horny’.”,
- "what actions does happy harry hardon encourage his listeners to take when contacting him?: happy harry hardon encourages his listeners to send him their most pathetic moments, as long as they are real, and wants details like size, shape, feel, and smell.",
- "who is happy harry hardon and what role does he play?: happy harry hardon is a radio host, engaging with his listeners over the air, and helping chris discuss his experiences and feelings.",
- "what contest does happy harry hardon mention during his show and how does he engage his listeners?: happy harry hardon runs a contest on the best way to put parents out of their misery and challenges listeners to amaze and discuss the sensational content of his show.",
- "what concern does mark have about the effects of his message?: mark feels ill about the situation and perceives it as chaotic, tangled with his life",
- "what does mark express about his feelings in the conversation with his parents?: he finds it difficult to talk to girls in his new place and feels different.",
- "what reaction does mark have to the police car pulling up?: mark is not worried and thinks the police are just dropping in on his neighbor.",
- "what internal conflict does mark express after his interaction with nora?: mark expresses that the whole thing is making him ill and that it's his life that mora is messing with.",
- "what critical message does mark want to convey to his audience about taking control of their lives?: mark conveys that it’s their life, they should take charge of it, and they should fill the air and speak out.",
- "english class question: jan emerson asks mark to share his feelings about what he wrote." Our answer: take charge of their lives. Standard retrieval baseline: -happy harry hardon
- you interview a student and then you rat on her, you betray her trust, isn’t that right sir <deaver hangups upvells well as you can see, these guys are played out society is mutating so rapidly that anyone over the age of twenty has really no idee arr alright, back down to business i share a room with my older brother and nearly every night after he turns off his light the come over to my bed and gives me a few arm nookies and stuff and then makes me scratch his back and other refinements!’s about time we had some refinements on this show then sooner or later he gets worked up and further a do he rub shis thing and makes me watch signed i’m just screwed upwell first of all you're not screwed up, your an unscreved up reaction to a screwed up station feeling screwed up at a screwed up time, in a screwed up place does not make you necessarily screwed up, if you catch my drift well as you know dear liesters if you enclose your number a reply is guaranteed <rings miss screwed up>”,
- it’s the trouble makers, you can’t run a top school with trouble makers in the mix brian
- okay, so what exactly as a trouble maker creswood
- someone whose ha nos interest in education brain
- oh c’mon that includes every teenager i know crewsood
- can’t you understand that nothing is more important than a good education brain
- except for the basic right to it crewsoud
- the point is i have the highest s at x cores in the state brain
- yeah but how creswood
- i stand by my record ¡the school field¿ sheep sheppard
- mr watts, shepberand channel six news here watts
- good evening shepberand
- how does washington intent to deal with this situation watts
- we at the f c c feel that democracy is about protecting the rights of the ordinary citizen un-regulated radio would result in programming of the lowest common denominator, the rule of the mob <watts looks round to see one of his f c c vanas has picked up some garffitotis this is vandalism, not free expression <everyone gathers outside the school to listen to the happy harry hardon show>’ Standard retrieval’s answer: mark does not urge his listeners to do anything \\ \end{tabular}
\end{table}
Table 1: Example of answering a question from NarrativeQA: In our method, the blue highlight represents the evidence most likely to contribute to the final answer.

Figure 6: Metrics vs Number of Tokens for Different Chunk Lengths on NarrativeQA





## Appendix D Ablations on Question Speculation

Table 3 shows the similarity between real queries and speculative queries in a subset of NarrativeQA. The similarity is measured by computing the similarity between embeddings encoded with the all-MiniLM-L6-v2 model from Hugging Face's sentence-transformers. We examined 617 questions and found that 11.18% of the speculated questions closely align with or rephrase the real queries, while 53.97% focus on the same topic.

## Appendix E Ablations on KB Augmentation

Figure 9, Figure 10 and Figure 11 show the effect of different numbers of KBs in NarrativeQA, Qasper, and QuALITY.

\begin{table}
\begin{tabular}{p{42.7pt} p{227.6pt}}  & **Question**: What does Mark urge his listeners to do? \\  & **Ground Truth**: to do something about their problems instead of committing suicide \\  & **Proposition Baseline (Chen et al., 2023)**: “creswood states, so what does this prove, not everyone goes to college.", “mark goes to collect his post.”, “The speaker asserts, happy harr hardon will go to any language to keep his three listeners glued with huwy bluuwy to their radios.’,”. “happy harry hardon asks, are you willing to tell my listeners what you told me here in this letter?”, “happy harry hadron invites listeners to share their most real moments..”, “nora finds mark burning his happy harron letters.”, “mark adds, i know exactly what it means.”, “market protests, i swear, what are you doing?”, “mrs. kaiser invites malcolm to join his parents downstairs.”, “happy harry hardon states that listeners are interested in the decision to expel cheryl bates.”, “mark explains that it is his mom’s jeep and that she kind of loaned it to him.”, “chris expresses, j didn’t know what to do.”, “mark says, io it’s outside,and shows nora his converted radio jeep.”, “happy harry hardon asks david deaver to explain his work.”, “mark collects his post from the postal center, exits, and starts to read the eat me beat me lady’s letter.”, “donald shakes he head in disgust.”, “chris asks, so what are we going to do about this?”, “happy harry hardon suspects a lie if miss screwed up does not remember or tell the truth. ”, “happy harry hardon concludes, but you know what you have to do.”, “happy harry hardon lists, you have parents, teachers telling you what to do.“, “nora pulls mark into the clayroom and reassures, it’s cool, it’s safe guess what i heard?”, “back outside the lockers, dough asks dnonald, so what did they do to you?”, “nonquestions, frank what is with you?”, “malcolm’s mother, ms. kaiser, asks malcolm about his homework.”, “happy harry hardon continues, you have movies, magazines, and tv telling you what to do.”, “happy harry hardon questions what david deaver says to young people about the world’s trustworthiness.”, “detective denny, holding ub his badge, implies that the postal clerk can give the information to him.”, “mark asks, ¡close to what?”, “malcolm tells mrs. kaiser that he has finished his homework.”, “happy harry hardon notes, now they’ve all run home to tune in and listen to what they’ve all been talking about.”, “mark comments, yeah, back to you.”, “happy harry hardon addresses his audience as ill my horby listeners.”, “marta hurre salis brain hunter, have you noticed bis behaviour lately?”, “brian questions, okay, so what exactly is a troublemaker?”, “nora points out, f.c.co. you know what that means.”, “happy harry hardon asks, so what did you do?”, “happy harry hardon prompts, so tell us what happened.”, “mark adds, ¿i can’talk to them!”, “mark mentions having something to show nora.”, “mark comments to nora, you’ve so different.”, “mark clarifies, i can’t talk to you.”, “rora greets, hit what are you doing? you having fun?”, “brain asks, Iorteta what the hell is going on there?”, “cheryl asks, can you tell me what this is about?”, “crewood assets, nonsense, she doesn’t know what she’s talking about.”, “happy harry hardon claims, happy party just happens to have in his very hands a copy of a memo written by mr.”, “mark asserts, i can’t talk to you people.”, “mark declares, steat il, it belongs to you.”, “happy harrdon acknowledges all of my horny listeners would love it if i would call up the eat me hear me lady.”, “jan reveals, last night one of our students, malcolm kaiser, took his own life.” \\  & **Proposition Baseline’s answer**: Mark does not urge his listeners to do anything. No specific action is mentioned. \\ \end{tabular}
\end{table}
Table 2: Example for answering one question from NarrativeQA.





## Appendix F Additional Examples of Generated Questions

Here, we provide additional examples of speculative questions and their real-query counterparts from NarrativeQA. Table 4 lists some of the highest-similarity pairs according to the all-MiniLM-L6-v2 model. These examples show that speculative questions are semantically aligned with real queries, which helps guide the LM to extract relevant facts without exact repetition.

## Appendix G Prompts

We detail all the prompts used in our method and baselines. For our method, prompts for question speculation are shown in Table 5 and Table 6. Prompts for EDP KB construction are shown in Table 7 and Table 8. Prompts for question answering are shown in Table 9, Table 10, Table 11, and Table 12. Note that for NarrativeQA, we use a two-step prompting approach to obtain the final answer: first, perform regular question answering based on

Figure 8: Avg Max F1 vs number of retrieved tokens for different chunk lengths as retrieval units in the standard RAG approach on Qasper dataset.

\begin{table}
\begin{tabular}{l l c} \hline \hline
* 0.85)**} \\ \hline  & What does Klaatu want to discuss with representatives from Earth? & 0.7529 \\ What did Klaatu say would happen if his message was ignored by & What is Klaatu’s demeanor when he discusses the stakes for Earth’s future if his message is not needed? & 0.7783 \\ Earth’s people? & How does Klaatu react to the replies from world leaders regarding the meeting? & 0.7284 \\  & What alternative does Klaatu say Earth would face if his proposals are rejected? & 0.7517 \\  & What message does Klaatu ask to be delivered and to whom? & 0.7136 \\  & What ultimatum is being given to the audience in Klaatu’s message? & 0.7272 \\ \hline Who did Bobby suggest was the greatest living person? & How does Bobby respond to Klaatu’s question about the greatest man in America? & 0.7343 \\  & Who does Bobby identify as the greatest scientist in the world? & 0.7316 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Examples of Speculated Questions and Their Similarity to Real Questions

Figure 7: Accuracy vs number of retrieved tokens for different chunk lengths as retrieval units in the standard RAG approach on QuALITY dataset.



the query and retrieved documents; second, compress the answer to make it more concise. This is because answers in NarrativeQA are typically just a few words, but Mixtral tends to generate lengthy responses regardless of prompt adjustments, prompting us to adopt a two-step process.

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Real Question** & **Speculative Question** & **Similarity** \\ \hline How does Liza get a black eye? & What causes Liza’s black eye? & 0.9264 \\ What does Dr. Varava reveal about Esther? & What does Dr. Varava reveal to Kate about Esther? & 0.9189 \\ What is Mr. Roundhay’s profession? & What is Mr. Roundhay’s occupation and hobby? & 0.9327 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Highest-similarity speculative questions vs. real questions from the NarrativeQA validation set.

Figure 11: Results on different number of KBs on QuALITY.

Figure 10: Results on different number of KBs on Qasper.

Figure 9: Results on different number of KBs on NarrativeQA.

 



**System: You are a highly attentive assistant focused on generating specific and concise questions about the narrative elements of a text. Your goal is to produce clear and direct questions that help a reader deeply understand the concrete aspects of the story. User: Task: Generate Specific, Concrete, and Contextual Narrative Questions **Objective**: Given a section of text from the book, generate a set of specific, concise, and detailed questions that are directly related to the narrative elements--such as characters, actions, events, settings, and their historical or cultural significance. If the text contains irrelevant information like publisher details, web content, or other non-narrative elements, do not generate questions and instead return 'no questions extracted.

**Instructions**:
1. **Read the Text Carefully**: Pay close attention to the provided section of the text to fully understand the narrative context, including any historical or cultural references.
2. **Check for Irrelevant Information**: Identify whether the text contains non-narrative elements such as publisher details, web content, disclaimers, or any information not directly related to the narrative. If such content is found, return 'no questions extracted.'
3. **Identifify Key Narrative and Contextual Elements**: If the text is free from irrelevant information, focus on identifying the key events, actions, characters, settings, and any historical or cultural references. Consider what is happening, who is involved, where and when these events are taking place, and the historical or symbolic significance of these elements.
4. **Formulate Questions**: Create questions that are specific to the identified narrative and contextual elements. Ensure each question is concise, detailed, and directly connected to the content of the narrative, including its historical, cultural, or symbolic context.
5. **Question Variety and Depth**: Aim for a diverse set of questions that cover various aspects of the narrative, including specific locations, character roles, relationships, and cultural or historical context. Avoid redundancy by ensuring each question explores a different element or angle of the narrative.
6. **Avoid Abstract and Meta-Content*: Refrain from generating questions about abstract themes, philosophical ideas, or meta-information such as publication details or background information unrelated to the narrative itself. **Example**:

Here is an excerpt from the book: The Great Peace towards which people of good will throughout the centuries have inclined their hearts, of which seers and poets for countless generations have expressed their vision, and for which from age to age the sacred scriptures of mankind have constantly held the promise, is now at long last within the reach of the nations. For the first time in history it is possible for everyone to view the entire planet, with all its myriad diversified peoples, in one perspective. World peace is not only possible but inevitable. It is the next stage in the evolution of this planet--in the words of one great thinker, 'the planefication of mankind'. Whether peace is to be reached only after unimaginable horrors precipitated by humanity's stubborn clinging to old patterns of behaviour, or is to be embraced now by an act of consultative will, is the choice before all who inhabit the earth. At this critical juncture when the intractable problems confronting nations have been fused into one common concern for the whole world, failure to stem the tide of conflict and disorder would be unconscionably irresponsible." --

**Example Questions**:

- Where is the Great Peace expected?
- Who has expressed the vision of the Great Peace?
- What does 'planetization of mankind' mean?
- How does the text describe the current world state?
- What critical choice is presented?

**Your Turn**: Now, using the provided section of text, check for any irrelevant information. If you find any, return 'no questions extracted.' If not, generate a list of specific, concise questions covering various narrative elements such as characters, actions, settings, historical or cultural references, and symbolic meanings. --

*Section of the book*[INSERT EXCERPT HERE]"

\begin{table} \end{table}
Table 5: Prompts for generating speculative questions on NarrativeQA and QuALITY



**Qasper (Question Speculation)****System: You are an AI language model that generates insightful and analytical questions about a given passage. Your goal is to create questions that encourage deeper understanding and critical thinking about the content, themes, and details within the passage. The questions should resemble the style of the example questions provided. User: ++Instructions:**

1. Carefully read the passage provided, paying special attention to any mention of the experimental design, dataset details, evaluation methods, and results.

2. Generate a list of questions focusing on the following aspects: - Experimental setup - Dataset characteristics (e.g., size, composition) - Evaluation methods and metrics - Results and conclusions

3. The questions should be clear, specific, and thought-provoking, encouraging a deep understanding of the methodology and results presented.

4. **Each question must contain only one question.**

5. **Extract as many questions as possible.**

**Example:**

_Passage:. "Minimally Supervised Learning of Affective Events Using Discourse Relations Recognizing affective events that trigger positive or negative sentiment has a wide range of natural language processing applications but remains a challenging problem mainly because the polarity of an event is not necessarily predictable from its constituent words. In this paper, we propose to propagate affective polarity using discourse relations. Our method is simple and only requires a very small seed lexicon and a large raw corpus. Our experiments using Japanese data show that our method learns affective events effectively without manually labeled data. It also improves supervised learning results when labeled data are small.

Introduction

Affective events are events that typically affect people in positive or negative ways. For example, getting money and playing sports are usually positive to the experiecers; catching cold and losing one's wallet are negative. Understanding affective events is important to various natural language processing (NLP) applications such as dialogue systems, question-answering systems, and humor recognition. In this paper, we work on recognizing the polarity of an affective event that is represented by a score ranging from \(-1\) (negative) to \(1\) (positive).

Learning affective events is challenging because, as the examples above suggest, the polarity of an event is not necessarily predictable from its constituent words. Combined with the unbounded combinatorial nature of language, the non-compositionality of affective polarity entails the need for large amounts of world knowledge, which can hardly be learned from small annotated data.

In this paper, we propose a simple and effective method for learning affective events that only requires a very small seed lexicon and a large raw corpus. As illustrated in Figure 1, our key idea is that we can exploit discourse relations to efficiently propagate polarity from seed predicates that directly report one's emotions (e.g., "to be glad" is positive). Suppose that events \(x_{1}\) are \(x_{2}\) are in the discourse relation of Cause (i.e., \(x_{1}\) causes \(x_{2}\)). If the seed lexicon suggests \(x_{2}\) is positive, \(x_{1}\) is also likely to be positive because it triggers the positive emotion. The fact that \(x_{2}\) is known to be negative indicates the negative polarity of \(x_{1}\). Similarly, if \(x_{1}\) and \(x_{2}\) are in the discourse relation of Concession (i.e., \(x_{2}\) in spite of \(x_{1}\)), the reverse of \(x_{2}\)'s polarity can be propagated to \(x_{1}\). Even if \(x_{2}\)'s polarity is not known in advance, we can exploit the tendency of \(x_{1}\) and \(x_{2}\) to be of the same polarity (for Cause) or of the reverse polarity (for Congestion) although the heuristic is not exempt from counterexamples. We transform this idea into objective functions and train neural network models that predict the polarity of a given event.

 chatWe trained the models using a Japanese web corpus. Given the minimum amount of supervision, they performed well. In addition, the combination of annotated and unannotated data yielded a gain over a purely supervised baseline when labeled data were small."

_Example Questions:-

1. What is the seed lexicon?

2. How are relations used to propagate polarity?

3. How does their model learn using mostly raw data?

4. How big is the Japanese data?

5. How large is the raw corpus used for training?

6. How big is the seed lexicon used for training?

7. What are the results?

8. What are the labels available in the dataset for supervision?

9. How significant are the improvements of supervised learning results trained on smaller labeled data enhanced with the proposed approach compared to the basic approach?

***Task:**

Now, read the following passage and generate a list of questions that resemble the style of the example questions. Passage:.

[INSERT EXCERPT HERE]

## Appendix A

\begin{table}
\begin{tabular}{l} \hline
**Task:**

Now, read the following passage and generate a list of questions that resemble the style of the example questions. Passage:. \\ \hline INSERT EXCERPT HERE] \\ \hline \end{tabular}
\end{table}
Table 6: Prompt for generating speculative questions on Qasper.





**NarrativeQA & QuALITY (KB Construction)**

**System: You are a helpful assistant. User: Please extract all relevant entities and facts from the provided passage that are useful for answering specific questions. Only return entity and facts for information that is explicitly mentioned in the passage. If a question does not have a corresponding fact in the passage, omit that entity and fact entirely. For example, if the question is "Who visits the philosopher at the beginning of the story?" and the passage mentions that a friend visits the philosopher, the response should be (Visitor, A friend visits the philosopher). However, if the passage does not provide specific information on a question and there is no mention of the location, do not include anything in your response for that question. Your returned output should be a series of tuples, like (Visitor, A friend visits the philosopher), (Philosopher's stance on law, Breaking the law is equivalent to betraying a contract with the state). Passage: [INSERT EXCERPT HERE] Questions: [INSERT SPECULATED QUESTIONS HERE]**

\begin{table}
\begin{tabular}{l} \hline \hline
**NarrativeQA \& QuALITY (KB Construction)** \\ \hline
**System**: You are a helpful assistant. \\
**User**: Please extract all relevant entities and facts from the provided passage that are useful for answering specific questions. Only return entity and facts for information that is explicitly mentioned in the passage. If a question does not have a corresponding fact in the passage, omit that entity and fact entirely. For example, if the question is “Who visits the philosopher at the beginning of the story?” and the passage mentions that a friend visits the philosopher, the response should be (Visitor, A friend visits the philosopher). However, if the passage does not provide specific information on a question and there is no mention of the location, do not include anything in your response for that question. Your returned output should be a series of tuples, like (Visitor, A friend visits the philosopher), (Philosopher’s stance on law, Breaking the law is equivalent to betraying a contract with the state). \\ Passage: [INSERT EXCERPT HERE] Questions: [INSERT SPECULATED QUESTIONS HERE] \\ \hline \hline \end{tabular}
\end{table}
Table 7: Prompt for constructing knowledge bases using speculative questions from NarrativeQA or QuALITY.

\begin{table}
\begin{tabular}{p{284.5pt}} \hline \hline
**Qasper (KB Construction)** \\ \hline
**System**: You are a helpful assistant. \\
**User**: Please provide answers to the following questions based on the passage. Whenever possible, prioritize using ***+**direct quotes***+ from the passage instead of summarizing. Only summarize when a direct quote does not provide a clear answer. Format each answer as a pair of: \\
**(Question, Answer)** \\ \hline
**If a direct quote is used, place it within quotation marks. \\ Example format: \\ (What is the seed lexicon?, A vocabulary of positive and negative predicates that helps determine the polarity score of an event.) \\ (How big is the Japanese data?, 7,000,000 pairs of events were extracted from the Japanese Web corpus, and 529,850 pairs of events were extracted from the ACP corpus.) \\ (How does the proposed method compare to previous techniques?, ”Compared to existing methods, the proposed approach ’achieves a 15% increase in classification accuracy while reducing computational complexity by approximately 30%.” This substantial improvement highlights the efficiency and effectiveness of the new algorithm in large-scale data settings.” \\ Passage: [INSERT EXCEPT HERE] Questions: [INSERT SPECULATED QUESTIONS HERE] \\ \hline \hline \end{tabular}
\end{table}
Table 8: Prompt for constructing knowledge bases using speculative questions from Qasper.





**System: You are a helpful assistant. User: ***Instructions:**

1. If you find direct evidence from the context, extract the relevant span as your answer. Ensure it is concise and faithful to the text.

2. If the answer requires a rephrasing or cannot be directly extracted, use your own words to provide a clear, concise response.

3. For yes/no questions, simply respond with 'Yes' or 'No' based on the context.

4. If no answer is found within the context, output 'Unanswerable.'

**Context:** [INSERT RETRIEYED DOCUMENTS HERE]

**Question:** [ INSERT QUESTION HERE]

**Options:** 1. options[0] 2. options[1] 3. options[2] 4. options[3]

Choose the option that seems most appropriate based on the context, even if you're unsure. Respond with only the number of the selected option and do not provide any additional text or explanation.

\begin{table}
\begin{tabular}{p{56.9pt}} \hline \hline
**System: You are a helpful assistant. User: **Transductions:**

1. If you find direct evidence from the context, extract the relevant span as your answer. Ensure it is concise and faithful to the text. \\
2. If the answer requires a rephrasing or cannot be directly extracted, use your own words to provide a clear, concise response. \\
3. For yes/no questions, simply respond with ’Yes’ or ‘No’ based on the context. \\
4. If no answer is found within the context, output ’Unanswerable.’ \\
****Context:** [INSERT RETRIEYED DOCUMENTS HERE] \\
**Question:** [INSERT QUESTION HERE] \\ \hline \hline \end{tabular}
\end{table}
Table 10: Prompt for answering questions from Qasper.