CSTree-SRI: Introspection-Driven Cognitive Semantic Tree for Multi-Turn Question Answering over Extra-Long Contexts
Authors: Zhaowen Wang, Xiang Wei, Kangshao Du, Yiting Zhang, Libo Qin, Yingjie Xia, Li Kuang
Accepted at: acl25

CSTree-SRI: Introspection-Driven Cognitive Semantic Tree for Multi-Turn Question Answering over Extra-Long Contexts

Zhaowen Wang1 Xiang Wei1 Kangshao Du1 Yiting Zhang1 Libo Qin1 Yingjie Xia2,3 Li Kuang1*
1School of Computer Science and Engineering, Central South University
2Micro-Electronics Research Institute, Hangzhou Dianzi University
3College of Computer Science and Technology, Zhejiang University
{wizxech, 8205220302, 8210220928, 8209220418}@csu.edu.cn
m15840540712@163.com, xiayingjie@zju.edu.cn, kuangli@csu.edu.cn

Abstract
Large Language Models (LLMs) have achieved remarkable success in natural language processing (NLP), particularly in single-turn question answering (QA) on short-text. However, their performance significantly declines when applied to multi-turn QA over extra-long context (ELC), as they struggle to capture the logical correlations across multiple chunks of ELC and maintain the coherence of multi-turn Questions. To address the challenges, we propose the CSTree-SRI framework (Cognitive Semantic Tree through Summarization, Retrieval, and Introspection). CSTree-SRI dynamically constructs the CSTree to preserve logical coherence within ELC through hierarchical synthesis and introspective validation. Then a logic-driven traversal strategy on CSTree is designed to provide efficient information retrieval for question answering. Additionally, we construct a suite of multi-turn QA datasets and an evaluation benchmark tailored for ELC tasks, and comprehensive experiments demonstrate the framework’s superiority in addressing the challenges of multi-turn QA over ELC.

1 Introduction
The rapid proliferation of digital information has intensified the demand for understanding extra-long context (ELC) in multi-turn question answering (MTQA) with LLM. ELC involves both single documents (e.g., legal contracts) and cross-document synthesis (e.g., academic literature reviews) that exceed the context window of LLM (Bai et al., 2024b). MTQA over ELC scenarios further complicates the problem. Users often engage in iterative questioning, such as consulting legal clauses or exploring academic topics. As shown in Fig. 1, such kind of tasks require capturing the logical correlation among multiple chunks of ELC, the coherence among multi-turn QA, as well as the alignment between questions and partially overlapping retrieved segments (Zhu et al., 2023), thus placing higher demands on LLMs’ ability to precisely and efficiently extract key information in ELC.

There are mainly two kinds of approaches in processing ELC (Huang et al., 2023): (1) modifying LLM’s architecture to extend the context window, e.g., optimizing attention mechanisms (Chen et al., 2023b), introducing recurrence (Borgeaud et al., 2022), or modifying positional encoding (Su et al., 2024); (2) employing external tools (e.g., RAG and Agents) to assist LLM in efficient retrieval and information processing (Topsakal and Akinci, 2023). These approaches primarily focus on single-turn tasks, lacking effective mechanisms for maintaining coherence across multi-turn interactions.

Research on multi-turn conversation abilities has largely been confined to short-text domains, where evaluation benchmarks have been well-established. Traditional methods on MTQA simply concatenate historical turns, where context is utilized inefficiently (Zhang et al., 2018a), and noise may be introduced. Moreover, when the context window is

![An Example of MTQA over ELC. The 1st question of summarizing multiple papers involves the correlation among multiple chunks of ELC; the 2nd question of recommending the latest one involves the coherence among multiple questions, and the following two questions of the paper details involve partially overlapping retrieved segments. We propose a cognitive semantic tree to capture logical relationships and coherence across MTQA over ELC.](page_1012_370_420_320.png)

Figure 1: An Example of MTQA over ELC. The 1st question of summarizing multiple papers involves the correlation among multiple chunks of ELC; the 2nd question of recommending the latest one involves the coherence among multiple questions, and the following two questions of the paper details involve partially overlapping retrieved segments. We propose a cognitive semantic tree to capture logical relationships and coherence across MTQA over ELC.
exceeded, truncation mechanisms may discard critical information, adversely affecting the model’s reasoning and comprehension.

In summary, current research on MTQA over ELC exhibits three key limitations: (1) logical fragmentation: Existing context window extension methods address length constraints but fail to preserve inter-document, across-MTQA relationships (Liao et al., 2024). (2) noise accumulation: Concatenating multi-turn inputs causes noise accumulation from redundant information, and increases computational costs (Zhang et al., 2018b). (3) evaluation gaps: Existing benchmarks focus on short-text MTQA (Kwan et al., 2024; Wang et al., 2023; Bai et al., 2024a), lacking datasets and metrics for evaluating MTQA reasoning performance over ELC.

To address these challenges, we propose CSTree-SRI (Cognitive Semantic Tree through Summarization, Retrieval, and Introspection), a framework for multi-turn QA over extra-long context (ELC), which dynamically constructs a hierarchical Cognitive Semantic Tree (CSTree) to organize ELC into document/paragraph/sentence nodes, preserving logical coherence for efficient retrieval. CSTree-SRI integrates four expert modules: (1) Retrieval Expert (RE), for relevant segments filtering; (2) Summary Expert (SE), which generates hierarchical summaries; (3) Introspection Expert (IE), which dynamically makes decisions on retrieval and response optimization; and (4) Answer Expert (AE), produces final responses. To address challenge 1, CSTree-SRI first dynamically builds the CSTree through hierarchical synthesis and introspective validation by a collaboration of RE, SE, and IE. To address challenge 2, it then introduces a logic-driven hierarchical traversal strategy on CSTree to retrieve relevant information for the next question by RE and IE. Subsequently, the framework iteratively optimizes responses through collaboration between AE and IE, ensuring both relevance and grounding in the retrieved information. To address challenge 3, we construct an MTQA-ELC benchmark and assess LLM performance in extra-long context QA tasks. Our contributions transcend prior work in three dimensions:

(1) Framework Innovation: CSTree-SRI is the first attempt to construct and utilize the introspection-driven CSTree through the collaboration of multiple expert modules for understanding ELC in MTQA precisely and efficiently.

(2) Benchmark Rigor: We introduce the first MTQA-ELC benchmark, containing over 500 articles spanning 391k words, nearly 4k groups of correlated questions, and new metrics for reasoning time, accuracy, and LLM-human gaps.

(3) Empirical Superiority: On tasks with 256k+ tokens, CSTree-SRI improves multi-turn QA performance by an average of 21.48%, reduces inference time by 41.11% (ETScore) compared to RAG/Agent solutions while improving answer accuracy by 44.17%.

2 Related Work

2.1 Long-Text Processing in LLMs

Current challenges in enhancing the long-text processing capabilities of LLMs include (Huang et al., 2023): quadratic complexity in attention computation, the lack of context memory mechanisms, and limitations on the maximum length of training samples. Existing approaches can be broadly categorized into two classes:

Architectural Optimization. Existing approaches to enhance Transformer-based LLMs’ long-text processing capabilities focus on the following architectural optimizations: (1) Attention mechanism refinement improves computational efficiency through blockwise processing or hierarchical attention (Qiu et al., 2019; Chen et al., 2023b; Yang et al., 2016), yet often sacrifices global contextual awareness; (2) Recurrent memory augmentation integrates external memory databases to preserve long-term dependencies (Borgeaud et al., 2022; Tworkowski et al., 2024), but struggles with precise memory retrieval; (3) Positional encoding extension employs rotary operations or NTK-aware scaling to expand context windows (Su et al., 2024; Chen et al., 2023a; Peng and Quesnelle, 2023), but they require additional adjustments and optimizations, potentially increasing training difficulty.

Unlike existing work focused on specific Transformer optimizations, we propose CSTree-SRI that enhances LLMs’ multi-turn QA over ELC beyond architectural-level improvements.

External Tool Augmentation. These approaches employ LLMs as black-box processors combined with external mechanisms: (1) Multi-agent collaboration frameworks delegate long-text processing through role specialization and interaction protocols (Zhao et al., 2024), though coordination overhead increases complexity; (2) Attention modification techniques like LongHeads adapt attention patterns for extended contexts without archi-
tectural changes (Lu et al., 2024), but lack dynamic reasoning adaptation; (3) Retrieval-Augmented Generation (RAG) enhances inputs through external knowledge bases (Gao et al., 2023), with recent improvements incorporating LLM-guided retrieval evaluation (Li et al., 2023) and reflection mechanisms (Asai et al., 2023). While effectively circumventing context window limitations, such methods often underutilize LLMs’ native reasoning capacities for complex textual analysis.

Our work combines chunked retrieval and reflective analysis, leveraging multi-module experts and the Cognitive Semantic Tree to extract and maintain logical information in ELC. This enables efficient information filtering and organization, offering a new pathway for MTQA over ELC.

2.2 Benchmarks for MTQA on Long-Context
Evaluating long-context models is challenging due to the inherent difficulty of collecting and analyzing long texts (Li et al., 2024). Bai et al. (2024b) introduced the LongBench benchmark, comprising six major task categories and 21 tasks, covering key long-text application scenarios. An et al. (2024) proposed the L-Eval benchmark, which includes long documents from domains such as law, finance, academic papers, novels, and conferences, along with various tasks. However, these benchmarks primarily evaluate single-turn QA tasks and lack assessments for MTQA tasks. Zheng et al. (2023) developed MT-Bench, a dataset of 80 multi-turn questions, but each dialogue consists of only two turns. Kwan et al. (2024) increased the number of turns, proposing MT-Eval, which includes multiple task types within a single dialogue to evaluate LLMs’ comprehensive multi-turn dialogue capabilities. However, these benchmarks involve relatively short texts and do not address ELC.

In summary, these works lack evaluations of LLMs’ reliability and efficiency in MTQA over ELC, and the distinction between evaluating models and augmenting long-text processing with external tools remains underexplored. In contrast, our work evaluates mainstream long-text LLMs and external tools (e.g., RAG, Agents) in MTQA over ELC, addressing gaps in existing research.

3 CSTree-SRI
The input to CSTree-SRI consists of a text sequence \( X = \{x_1, x_2, \ldots, x_t\} \), which can be a single long document or a collection of documents, and a sequence of logically dependent queries \( Q = \{q_1, q_2, \ldots, q_m\} \) across multiple rounds. The framework aims to generate answers for these queries based on the input \( X \). To handle this, CSTree-SRI initially segments the input text \( X \) into chunks of a predefined size \( sz = 512 \), resulting in \( X = \{C_1, C_2, \ldots, C_t\} \) where \( t = \lceil l/sz \rceil \). These chunks act as the fundamental processing units, enabling effective multi-turn QA (MTQA) over extra-long context (ELC) by maintaining and leveraging historical information throughout the queries.

3.1 Framework Components
The CSTree-SRI framework comprises a Cognitive Semantic Tree (CSTree) and four expert modules.

The CSTree is a three-layer tree structure where the nodes are classified into document-level nodes, paragraph-level nodes, and sentence-level nodes. Each node contains a summary or raw text, with edges between the nodes of various layers formed due to their common logical relationships.

The four Expert modules include: (1) A Retrieval Expert (RE) that filters out relevant text segments to reduce noise. (2) A Summary Expert (SE) generates concise summaries after each QA turn to maintain logical consistency. (3) An Answer Expert (AE) that produces final responses. (4) An Introspection Expert (IE) that dynamically refines retrieval precision. The IE module will conduct introspection from two aspects: retrieval precision and response accuracy, with specific introspection questions detailed in Table 1.

Specifically, for each query \( q_i \), CSTree-SRI performs two core operations: (1) **Dynamic Structure Construction** of the tree through collabora-

<table>
  <tr>
    <th>Type</th>
    <th>Specific Question</th>
  </tr>
  <tr>
    <td>Relative</td>
    <td>Are the retrieved text chunks relevant to the current query \( q_i \)?</td>
  </tr>
  <tr>
    <td>NodeRetr</td>
    <td>For the summarized information of a node, is further retrieval necessary?</td>
  </tr>
  <tr>
    <td>ExtraRetr</td>
    <td>Is the retrieved information from the CSTree sufficient, or is further retrieval from the original text needed?</td>
  </tr>
  <tr>
    <td>Support</td>
    <td>Can the retrieved information support the AE’s answer?</td>
  </tr>
  <tr>
    <td>Useful</td>
    <td>Does the AE’s answer effectively address \( q_i \)?</td>
  </tr>
</table>

Table 1: Introspection Questions for the IE Module (See Appendix C.1 for Detailed Prompts)
tion among the RE, IE, and SE modules, and (2) Hierarchical Information Selection on the tree via collaboration between the RE and IE modules. After retrieving relevant information blocks, responses are refined through iterative optimization between the AE and IE modules to ensure enhanced answer precision. Appendix C.2 contains specific prompts for each module. The following sections describe how these modules interact collaboratively with the CSTree during each QA round \( q_i \).

3.2 Dynamic Structure Construction

Inspired by the hierarchical structure of human reading notes (paragraph-chapter-book), we propose a dynamic CSTree construction that mimics cognitive processes through introspection-driven hierarchical synthesis. The RE, IE, and SE modules collaborate to implement the "structured note-taking" approach. They retrieve context segments, validate logical coherence through introspection, and synthesize summaries at paragraph and document levels. This process transforms ELC into navigable information structures. Below, we detail the technical implementation of constructing paragraph-level and document-level nodes.

Para-Level Node Construction. As shown in Fig. 2(a), our framework combines flat retrieval with introspective validation for paragraph-level construction, known as the flat information retrieval strategy. The RE first retrieves candidate text chunks \( X \) using the BM25 algorithm. Meanwhile, the IE assesses whether the retrieved chunks truly represent the key information \( C_{key} \) relevant to the query \( q_i \), effectively addressing the "keyword bias" commonly found in traditional sparse retrieval methods. This two-stage filtering process—merging statistical relevance with semantic introspection—ensures that only logically coherent fragments proceed to the synthesis phase. The SE then dynamically creates a hierarchical parent node \( F_{para} \) by abstracting the relationships among the \( C_{key} \) nodes, thereby establishing explicit edges to maintain content associations and provenance.

Doc-Level Node Construction. The framework constructs \( C_{doc} \) through a ratio-controlled triggering mechanism. When \( C_{para} \) with logical relationships are identified during CSTree traversal, a predefined 1:3 doc-to-para ratio threshold governs the construction process. This ensures that the number of \( C_{doc} \) never exceeds one-third of the \( C_{para} \), preventing structural redundancy. The proportional constraint activates the SE module only when sufficient \( C_{para} \) nodes exist. This activation asks the SE module to generate the doc-level parent node \( F_{doc} \) by summarizing relationships across paragraphs.

This hierarchical summarization structure enables dynamic CSTree evolution through progressive QA interactions. Our "structured note-taking" approach preserves critical relationships within the ELC, enhancing QA accuracy. Additionally, the CSTree improves reasoning efficiency by maintaining the logical coherence of the text, which accelerates information retrieval. These advantages are validated in our ablation studies.

3.3 Hierarchical Information Selection

Unlike conventional tree traversal methods with fixed depth-first or breadth-first strategies, our approach introduces a logic-driven hierarchical traversal strategy where the IE module evaluates node relevance at each hierarchy level. The RE and IE modules collaborate to strategically navigate the
CSTree, balancing retrieval depth with computational efficiency to address ELC challenges. This logic-driven approach mirrors human top-down comprehension, starting with high-level summaries and drilling down to details as needed. After retrieval, CSTree-SRI uses a sufficiency validation mechanism to ensure the retrieved information meets query requirements. The hierarchical information selection process is detailed below.

Logic-Driven Hierarchical Traversal Strategy As shown in Fig. 2(b), we have implemented a dynamic hierarchical traversal strategy that adjusts exploration depth through semantic introspection. The process begins with the IE module analyzing the summary information of each non-leaf node, and a dynamic continuation probability \( \phi(C) = IE(q_i, C_l, NodeRetr) \) is calculated for each node chunk C. The hierarchical retrieval automatically terminates at level l when \( \phi(C_l) < \tau \), implementing principled depth control that prevents over-retrieval while maintaining query relevance.

For the retrieval results across the entire CSTree, paragraph-level nodes \( C_{para} \) will have their corresponding document-level parent nodes constructed as outlined in Section 3.2. For sentence-level nodes \( C_{sen} \), to prevent excessive information retrieval, the BM25 algorithm is employed to efficiently filter the top-K most relevant nodes, which are then used as the retrieved text chunks from the CSTree. The entire hierarchical information screening process can be formalized as follows:

\[
\phi(C) \geq \tau \Rightarrow Select(Child)
\]

\[
Select(C_{doc}) \Rightarrow Select(C_{para}) \Rightarrow Select(C_{sen})
\]

\[
Chunk_{tree} = BM25(q_i, C_{sen}, topk)
\]

Here, \( A \Rightarrow B \) indicates that operation B is performed based on the result of operation A; \( Select(\cdot) \) represents the selection operation, where the child nodes Child of the selected node become the target of the next layer of retrieval; \( Chunk_{tree} \) refers to the text chunks retrieved from the CSTree; and \( BM25(\cdot) \) denotes the retrieval operation using the BM25 algorithm.

Sufficiency Validation Mechanism After completing the CSTree retrieval, the IE module introspects the ExtraRetr to evaluate whether the retrieved text chunks are sufficient to answer the query \( q_i \). If necessary, additional relevant text chunks are retrieved from the extra-long context using the flat information retrieval strategy described in Section 3.2. Finally, all retrieved text chunks are consolidated and provided to the AE module to generate the final response.

3.4 Iterative Response Optimization

This step represents the final stage of the framework, synthesizing text chunks retrieved through the processes detailed in Sections 3.2 and 3.3. Through iterative collaboration between the IE and AE modules, the response to query \( q_i \) is refined.

The IE module evaluates the AE’s output across two critical dimensions: <Support>, which ensures that the response is grounded in the retrieved text chunks, and <Useful>, which assesses the response’s relevance to \( q_i \). This dual-focused evaluation facilitates iterative optimization, ultimately leading to the final answer, as formalized below.

\[
Resp = AE(q_i, Chunk_{tree} + Chunk_{flat})
\]

\[
IE(q_i, Resp, Support&Useful) \Rightarrow Iter(Resp)
\]

\[
Resp^* = Iter(Resp)
\]

Here, \( AE(\cdot) \) denotes the generation of a response by the AE, and \( Chunk_{flat} \) represents the text chunks retrieved using the flat information retrieval method. \( IE(Support&Useful) \) indicates the IE module performing the <Support> and <Useful> introspection. \( Iter(\cdot) \) represents the iterative process of generating and refining responses through the AE and IE. \( Resp^* \) refers to the updated response generated in a new iteration.

4 MTQA-ELC Benchmark

Current benchmarks for evaluating LLMs primarily focus on language modeling and generation tasks. However, these benchmarks may not fully capture the models’ abilities to handle complex, multi-turn question-answering tasks, particularly with extra-long contexts. To address this gap, we have developed a benchmark specifically designed to assess LLM performance in information retrieval, key information extraction, and logical reasoning—skills

<table>
  <tr>
    <th>Benchmark</th>
    <th>#words in Text<br>Max.</th>
    <th>#Words in Text<br>Avg.</th>
    <th>#Turns<br>Max.</th>
    <th>#Turns<br>Avg.</th>
  </tr>
  <tr>
    <td>LongBench(QA task)</td>
    <td>18409</td>
    <td>8640</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>L-Eval(QA task)</td>
    <td>26918</td>
    <td>9133</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>MT-Bench</td>
    <td>330</td>
    <td>68</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>MT-Eval</td>
    <td>2574</td>
    <td>760</td>
    <td>12</td>
    <td>7</td>
  </tr>
  <tr>
    <td>MT-Bench-101</td>
    <td>817</td>
    <td>202</td>
    <td>323</td>
    <td>67</td>
  </tr>
  <tr>
    <th>MTQA-ELC (Ours)</th>
    <th>217273</th>
    <th>217264</th>
    <th>100</th>
    <th>100</th>
  </tr>
</table>

Table 2: Data Statistics. Detailed data sources are provided in Appendix A.1.
that are crucial for real-world applications involving long-text processing.

4.1 Data Construction

Table 2 shows the key statistics of MTQA-ELC. Our dataset consists of reading comprehension passages from major exams such as the NMET, CET, PGEE, and TPO. Each passage is carefully divided into paragraphs, with unique identifiers added at the beginning and end of each segment to indicate the article and paragraph. This structure enables explicit tracking of relationships between paragraphs when multiple segments from different articles are concatenated into an ELC. These identifiers allow benchmarks to evaluate LLMs’ ability to process and integrate information across paragraphs.

To further assess the integration and reasoning capabilities of models, we generated multi-turn question sets based on texts of varying lengths (32k, 64k, 128k, 256k). For fair evaluation, we randomize paragraph order, shuffle options, and compare model performance with human test-taker scores, as detailed in Appendix A.2.

4.2 Task Set

Reading comprehension tasks assess various cognitive skills. To evaluate LLMs’ capabilities in multi-turn QA over ELC, we categorize tasks based on required abilities, including paragraph retrieval, information integration, detail/main idea comprehension, and logical reasoning.

Tasks are divided into four types: Detail Understanding (DU), Semantic & Reference (SR), Main Idea (MI), and Inference & Judgment (IJ). The former two tasks focus on single-paragraph retrieval and understanding, while the latter two require integrating information across multiple paragraphs to grasp the main idea or perform complex reasoning. For all tasks, the input consists of an ELC and a set of questions with multiple choices, and the output is the correct choice. Appendix D contains examples of various evaluation tasks.

4.3 Metrics

Our Benchmark evaluates LLMs using three key metrics: Accuracy (ACC), Effective Time Score (ETScore), and Human-Adjusted Overall Score. Accuracy is a commonly used metric, while ETScore and Human-Adjusted Overall Score are newly proposed metrics in our Benchmark.

ETScore measures LLMs’ reasoning time and their ability to answer correctly within a specific time frame, addressing the limitation of traditional accuracy metrics in capturing time efficiency. The Human-Adjusted Overall Score compares LLM performance to human test-takers, highlighting strengths and weaknesses relative to people.

Accuracy is calculated as:

\[
ACC = \frac{\sum_{i=1}^{N} Check(O_i, A_i)}{N} \times 100\%
\]

Here, \( N \) is the total number of questions, and \( Check(O_i, A_i) \) verifies if the model’s output \( O_i \) matches the correct answer \( A_i \).

ETScore’s calculation formulas are as follows:

\[
AvgTime = \frac{\sum_{i=1}^{M}(EndTime_i - StartTime_i)}{M}
\]
(2)

\[
ETScore = Acc \times \frac{K}{1 + \beta \times AvgTime}
\]
(3)

Here, \( M \) is the number of test papers, \( AvgTime \) is the average reasoning time per question, \( \beta \) controls time sensitivity, and \( K \) scales the score. We set \( \beta = 0.002 \) and \( K = 100 \), with higher \( ETScore \) indicating better performance.

Human-Adjusted Overall Score accounts for task difficulty by incorporating test-taker accuracy:

\[
Overall = \frac{\sum_{i=1}^{N} W_i}{N}
\]
(4)

\[
W_i = \frac{\sum_{j=1}^{Q_i} f(p_{ij}, a_{ij}, k_i)}{Q_i} \times 100
\]
(5)

\[
f(p, a, k) = e^{0.5 \cdot k} + ae^{k(0.5-p)a}
\]
(6)

\[
a_{ij} = \begin{cases} 1, & Resp_{ij} = Answer_{ij} \\ -1, & Resp_{ij} \neq Answer_{ij} \end{cases}
\]
(7)

Here, \( Q_i \) is the number of questions in test paper \( i \), \( p_{ij} \) is the human accuracy for question \( j \), and \( a_{ij} \) indicates correctness (1 for correct, -1 for incorrect). Hyperparameter \( k \) adjusts sensitivity: higher difficulty (\( p_{ij} \) low) increases rewards for correct answers and softens penalties for mistakes, while low-difficulty errors incur heavier penalties.

5 Experiments

In this section, we evaluate the performance of the CTree-SRI framework on both single-turn and multi-turn QA tasks. For single-turn QA, we use the LongBench benchmarks. For multi-turn QA, we conduct experiments on the MTQA-ELC to assess the capabilities of various long-text LLMs over ELC. We also compare CTree-SRI with mainstream RAG and Agent methods, demonstrating its superior performance in MTQA. Additionally, ablation studies validate the contributions of individual modules within the CTree-SRI framework.
<table>
  <tr>
    <th rowspan="2">Model/Framework</th>
    <th colspan="4">Single-Doc QA</th>
    <th colspan="4">Multi-Doc QA</th>
  </tr>
  <tr>
    <th>NQA</th>
    <th>Qspr.</th>
    <th>MulFi</th>
    <th>Avg.</th>
    <th>HQA</th>
    <th>WMQA</th>
    <th>Musq.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Llama-2-7B-chat †</td>
    <td>18.7</td>
    <td>19.2</td>
    <td>36.8</td>
    <td>24.9</td>
    <td>25.4</td>
    <td>32.8</td>
    <td>9.4</td>
    <td>22.6</td>
  </tr>
  <tr>
    <td>-LongHeads w/NTK init †</td>
    <td>16.87</td>
    <td><b>30.32</b></td>
    <td>38.59</td>
    <td><b>28.59</b></td>
    <td><b>36.04</b></td>
    <td>26.72</td>
    <td>10.21</td>
    <td>24.32</td>
  </tr>
  <tr>
    <td>-LongLora</td>
    <td>17.36</td>
    <td>28.97</td>
    <td>38.37</td>
    <td>28.30</td>
    <td>34.81</td>
    <td>32.57</td>
    <td>12.72</td>
    <td>26.70</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td><b>19.42</b></td>
    <td>23.34</td>
    <td><b>41.25</b></td>
    <td>28.00</td>
    <td>35.73</td>
    <td><b>35.21</b></td>
    <td><b>21.23</b></td>
    <td><b>30.72</b></td>
  </tr>
</table>

Table 3: The results of different methods based on the Llama-2-7B-chat model on LongBench. † means the data are sourced from the LongBench and LongHeads papers

5.1 Experiments Setting

All evaluations were conducted with float16 precision on 4 Nvidia V100-32G GPUs. Configuration details for each benchmark are described below.

LongBench We evaluated six English datasets from LongBench: NarrativeQA, Qasper, MultiFieldQA, HotpotQA, 2WikiMQA, and Musique, spanning single- and multi-document QA tasks. The CSTree-SRI framework used Llama-2-7B-chat as the AE with gpt-3.5-turbo for SE/IE modules. Baseline included: 1) vanilla Llama-2-7B-chat, 2) LongLoRA (Chen et al., 2023b, attention-optimized fine-tuning), and 3) LongHeads (Lu et al., 2024, attention head selection).

MTQA-ELC We conducted 100-round multi-turn QA sessions. Vanilla LLMs processed texts by concatenating the first and last halves of their context windows due to inherent context window limitations. We evaluated three open-source LLMs with 128K context windows (GLM-4-9B-Chat, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct), all locally deployed. Additionally, we tested three API-accessed models: gpt-4o-mini (128K), DeepSeek-chat (64K), and gpt-3.5-turbo (16K). External tools compared included RAG (using jina-embeddings-v2-base-en with cosine similarity), LongAgent (with gpt-4o-mini), and CSTree-SRI (with gpt-4o-mini for SE/IE), all using Llama-3.1-8B-Instruct as the QA module. Appendix C.3 contains prompts for evaluation tasks.

5.2 Single-Turn QA Evaluation

Table 3 compares our method with LongHeads and LongLora on single-turn QA tasks within LongBench. Our method achieves performance that is comparable to the baseline in single-document QA. However, in multi-document QA, CSTree-SRI significantly outperforms the others in average scores, demonstrating its effectiveness in handling more complex long-text QA tasks. This improvement is due to our framework’s enhanced ability to capture the logical relationships within lengthy and intricate texts.

To further validate the generalizability of our method across different models, we conducted additional experiments on the L-Eval benchmark. The results demonstrate consistent performance improvements, as detailed in Appendix B.1.

5.3 Multi-Turn QA Evaluation

Table 4 presents the performance comparison of Long-Context LLMs and external tool-enhanced methods on MTQA tasks across different context lengths. For Long-Context LLM with a 128K native context window, both ACC and ETS decline significantly when handling texts beyond this limit (128K, 256K) compared to shorter contexts (32K, 64K), highlighting their constraints in extra-long context processing.

Comparing the overall performance of CSTree-SRI (with Qwen-2.5-7B-Instruct as AE) to gpt-4o-mini and deepseek-chat in Table 4, our method achieves the highest Overall score (213.78) while maintaining consistently high ACC and ETS across different context lengths. This highlights CSTree-SRI’s effectiveness in mitigating performance degradation in ELC scenarios.

Table 4 also compares models enhanced with external tools. Experimental results show that traditional RAG methods offer minimal gains within context limits and only slight improvements for ELC. LongAgent improves long-text QA capability but incurs high time costs due to excessive inter-agent interactions, especially at longer contexts (e.g., ETS of 37.13 at 256K length). In contrast, CSTree-SRI outperforms these methods across all context lengths, especially beyond 256K tokens, boosting MTQA performance by 21.48%, reducing inference time by 41.11% (ETScore), and increasing accuracy by 44.17% (calculated based on CSTree-SRI with Llama-3.1-8B-Instruct as AE). We attribute this improvement to the dynamic construction of CSTree, which preserves key information in multi-turn QA, and its logic-driven hierarchical traversal strategy, effectively reducing retrieval time in extra-long context scenarios and leading to superior overall performance.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">32k</th>
    <th colspan="2">64k</th>
    <th colspan="2">128k</th>
    <th colspan="2">256k</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>ACC(%)</th><th>ETS</th>
    <th>ACC(%)</th><th>ETS</th>
    <th>ACC(%)</th><th>ETS</th>
    <th>ACC(%)</th><th>ETS</th>
  </tr>
  <tr>
    <th colspan="9">Locally Deployed Models</th>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td><td>56.00</td><td>51.11</td><td>60.33</td><td>48.26</td><td>48.00</td><td>43.31</td><td>54.33</td><td>48.94</td><td>157.75</td>
  </tr>
  <tr>
    <td>GLM-4-9B-Chat</td><td>64.00</td><td>57.14</td><td>68.33</td><td>52.43</td><td>56.67</td><td>48.68</td><td>61.00</td><td>52.32</td><td>173.62</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td><td>75.00</td><td>69.44</td><td>65.33</td><td>53.95</td><td>64.67</td><td>58.98</td><td>71.00</td><td>62.77</td><td>187.17</td>
  </tr>
  <tr>
    <th colspan="9">API-Based Models</th>
  </tr>
  <tr>
    <td>gpt-3.5-turbo</td><td>25.67</td><td>25.50</td><td>27.33</td><td>27.11</td><td>26.00</td><td>25.77</td><td>23.33</td><td>23.20</td><td>97.66</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td><td>84.33</td><td>83.25</td><td>86.33</td><td>84.23</td><td>75.33</td><td>73.15</td><td>74.67</td><td>69.14</td><td>208.10</td>
  </tr>
  <tr>
    <td>deepseek-chat</td><td>93.67</td><td>90.17</td><td>87.67</td><td>84.56</td><td>77.33</td><td>72.72</td><td>75.33</td><td>70.71</td><td>210.38</td>
  </tr>
  <tr>
    <th colspan="9">Models Enhanced with External Tools</th>
  </tr>
  <tr>
    <td>RAG</td><td>57.33</td><td>56.59</td><td>56.00</td><td>55.02</td><td>58.33</td><td>56.81</td><td>57.00</td><td>54.55</td><td>162.74</td>
  </tr>
  <tr>
    <td>LongAgent</td><td>66.67</td><td>46.69</td><td>65.67</td><td>51.12</td><td>62.67</td><td>38.31</td><td>65.67</td><td>37.13</td><td>179.26</td>
  </tr>
  <tr>
    <th colspan="9">CSTree-SRI (The Open-Source Model Only as Answer Expert)</th>
  </tr>
  <tr>
    <td>Ours (Llama-3.1 as AE)</td><td>72.67</td><td>65.03</td><td>76.33</td><td>68.71</td><td>78.33</td><td>70.45</td><td>78.33</td><td>69.06</td><td>202.35</td>
  </tr>
  <tr>
    <td>Ours (GLM-4 as AE)</td><td>81.00</td><td>74.27</td><td>79.67</td><td>72.86</td><td>82.67</td><td>73.22</td><td>80.67</td><td>71.15</td><td>211.74</td>
  </tr>
  <tr>
    <td>Ours (Qwen-2.5 as AE)</td><td>83.67</td><td>76.21</td><td>80.33</td><td>72.56</td><td>83.00</td><td>73.46</td><td>81.00</td><td>75.79</td><td>213.78</td>
  </tr>
  <tr>
    <th colspan="9">CSTree-SRI (A Pure Open-Source Configuration)</th>
  </tr>
  <tr>
    <td>Ours (Purely Llama-3.1)</td><td>68.33</td><td>60.69</td><td>68.67</td><td>59.04</td><td>65.33</td><td>58.56</td><td>69.67</td><td>58.77</td><td>182.85</td>
  </tr>
  <tr>
    <td>Ours (Purely GLM-4)</td><td>70.33</td><td>64.86</td><td>72.33</td><td>66.21</td><td>68.00</td><td>61.96</td><td>70.67</td><td>64.72</td><td>193.23</td>
  </tr>
  <tr>
    <td>Ours (Purely Qwen-2.5)</td><td>79.33</td><td>73.36</td><td>71.00</td><td>65.82</td><td>73.33</td><td>65.96</td><td>75.67</td><td>70.79</td><td>198.16</td>
  </tr>
</table>

Table 4: Results of MTQA with LLMs and External Tool-Enhanced Methods under Different Context Lengths. The best performance is shown in bold, while the second best performance is represented with an underline.

The bottom three rows in the table 4 present that any LLM in our method can be effectively replaced with purely open-source models. Furthermore, this configuration of purely open-source models is still effective for MTQA over ELC compared to the original locally deployed models. To evaluate the resource efficiency, we also analyzed the number of LLM interactions and token consumption across different configurations. The detailed results are presented in Appendix B.3.

To further validate CSTree-SRI, we analyzed its MTQA performance across task types and difficulty levels. CSTree-SRI remains robust as question difficulty increases and excels in complex reasoning and multi-paragraph retrieval, demonstrating strong logical consistency and long-range dependency capture. Detailed results are in Appendix B.2.

5.4 Ablation Study

We conducted ablation experiments to assess the contributions of the CSTree, SE, and IE modules. Experiments were performed on the MTQA-ELC dataset with 256K-length contexts. Results are presented in Table 5, using Llama-3.1-8B-Instruct as the AE, gpt-4o-mini for SE/IE, and CSTree construction enabled as the default configuration.

Impact of Different LLMs for Expert Modules. Replacing gpt-4o-mini with the more powerful gpt-4o or deepseek-chat in the SE/IE improves both ACC and ETS. Conversely, substituting these modules with the weaker gpt-3.5-turbo leads to a decline in overall performance, highlighting the importance of strong LLMs in expert modules.

<table>
  <tr>
    <th>Model Setting</th>
    <th>ACC(%)</th>
    <th>ETS</th>
  </tr>
  <tr>
    <td>CSTree-SRI</td><td>78.33</td><td>69.06</td>
  </tr>
  <tr>
    <td>-w SE/IE use deepseek-chat</td><td>81.33(+3.8%)</td><td>73.35(+6.2%)</td>
  </tr>
  <tr>
    <td>-w SE/IE use gpt-4o</td><td>79.33(+1.3%)</td><td>72.46(+4.9%)</td>
  </tr>
  <tr>
    <td>-w SE/IE use gpt-3.5-turbo</td><td>73.00(-6.8%)</td><td>63.54(-8.0%)</td>
  </tr>
  <tr>
    <td>-w/o CSTree</td><td>73.67(-6.0%)</td><td>64.84(-6.1%)</td>
  </tr>
  <tr>
    <td>-w/o SE</td><td>70.01(-10.6%)</td><td>64.63(-6.4%)</td>
  </tr>
  <tr>
    <td>-w/o IE</td><td>60.00(-23.4%)</td><td>58.60(-15.2%)</td>
  </tr>
</table>

Table 5: Ablation Study on MTQA-ELC (256k-length)

Impact of CSTree-SRI Modules. The ablation experiment results in Figure 6 show that removing CSTree results in declines in both ACC and ETS, as the framework loses logical relationships from historical information, weakening its ability to process multi-turn questions. Similarly, excluding the SE module, where non-leaf nodes store concatenated child node information instead of summarized data, reduces accuracy by 10.6% due to redundancy. This redundancy overloads the IE module’s retrieval process and impairs its ability to determine further retrieval needs accurately. Notably, removing the IE module leads to the most significant performance drop, with ACC decreasing by 23.4% and ETS by 15.2%, as this module
is essential for guiding the reasoning process. The introspective questioning mechanism enables the LLM to process ELC, ensuring successful multi-turn QA efficiently.

Overall, these results validate the effectiveness of each CSTree-SRI module in maintaining logical consistency, reducing retrieval redundancy, and enhancing multi-turn QA performance.

6 Conclusion

In this paper, we propose CSTree-SRI, a framework to enhance LLM performance on multi-turn QA tasks over extra-long contexts. CSTree-SRI follows an introspection-driven way to construct and search on CSTree, where logical relationships and coherence within ELC are preserved, through the collaboration of the Summary, Retrieval, Introspection and Answer expert modules. We also design the MTQA-ELC benchmark and conduct comprehensive experiments. The results demonstrate the effectiveness of our proposed CSTree-SRI.

For future work, we will refine the design of each expert module and integrate mechanisms like position encoding modifications, pre-training, and fine-tuning techniques to further improve the accuracy and efficiency of relevant context retrieval.

7 Limitations

Dependency on External LLMs for Expert Modules. The CSTree-SRI framework’s reliance on third-party LLMs (e.g., gpt-4, gpt-3.5) for critical modules—Summary Expert (SE), Introspection Expert (IE), and Answer Expert (AE)—introduces systemic risks in terms of operational stability and cost efficiency. Performance bottlenecks may arise from API latency, model availability fluctuations, or unexpected service interruptions. To mitigate these risks, the framework’s modular design inherently supports alternative implementations, including open-source LLMs (e.g., Llama-3, Qwen) or locally deployed models. This flexibility allows users to reduce dependency on specific vendors and enhance robustness against service disruptions. However, the financial burden of deploying high-tier LLMs—whether proprietary or self-hosted—could still render the framework economically impractical for resource-constrained users or organizations, particularly in scenarios requiring frequent or large-scale ELC processing.

8 Acknowledgments

This work has been supported by the National Key R&D Program of China under grant No.2022YFF0902500, the National Natural Science Foundation of China under grant No.62472447, the Science and Technology Innovation Program of Hunan Province under grant No.2023RC1023, and Hunan Provincial Natural Science Foundation of China under grant No.2024JK2006, the Fundamental Research Funds for the Central Universities of Central South University. This work was carried out in part using computing resources at the High Performance Computing Center of Central South University.

References

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14388–14411, Bangkok, Thailand. Association for Computational Linguistics.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024a. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7421–7454, Bangkok, Thailand. Association for Computational Linguistics.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024b. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand. Association for Computational Linguistics.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milligan, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023a. Extending context window
of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Yunpeng Huang, Jingwei Xu, Junyu Lai, Zixu Jiang, Taolue Chen, Zenan Li, Yuan Yao, Xiaoxing Ma, Li-juan Yang, Hao Chen, et al. 2023. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.12351.

Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. arXiv preprint arXiv:2401.16745.

Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Llatrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07838.

Zhuowen Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 881–893.

Zihan Liao, Jun Wang, Hang Yu, Lingxiao Wei, Jianguo Li, and Wei Zhang. 2024. E2llm: Encoder elongated large language models for long-context understanding and reasoning. arXiv preprint arXiv:2409.06679.

Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Longheads: Multi-head attention is secretly a long context processor. arXiv preprint arXiv:2402.10685.

Bowen Peng and Jeffrey Quesnelle. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

Jiezongh Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Oguzhan Topsakal and Tahir Cetin Akinci. 2023. Creating large language model applications utilizing langchain: A primer on developing llm apps fast. In International Conference on Applied Engineering and Natural Sciences, volume 1, pages 1050–1056.

Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2024. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36.

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309.10691.

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489.

Zhuosheng Zhang, Jiangtong Li, Peng Fei Zhu, Zhao Hai, and Gongshen Liu. 2018a. Modeling multi-turn conversation with deep utterance aggregation. ArXiv, abs/1806.09102.

Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018b. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3740–3752, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Longagent: Scaling language models to 128k context through multi-agent collaboration. arXiv preprint arXiv:2402.11550.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107.
A Data Details

A.1 Data Collection

We collected a large number of English reading comprehension passages from publicly available datasets of domestic and international large-scale exams. For each question, we also obtained additional data, such as the accuracy rate of test-takers. The annotators are three undergraduate students in computer science who are familiar with reading comprehension tasks and exam question types. The annotation process involved three independent annotators labeling questions based on the original exam materials. Conflicts in labeling were resolved through discussions with two senior researchers. All exam passages and questions are publicly available on official educational websites, and the annotation work was conducted by our research team to ensure alignment with task requirements. Detailed information about the raw dataset is provided in Table 6. The abbreviations in Table 6 are defined as follows: NMET refers to the National Matriculation Entrance Test, CET denotes the College English Test, PGEE stands for the Post-graduate Entrance Examination, and TPO represents the TOEFL Practice Online.

<table>
  <tr>
    <th>Category</th>
    <th>#Passages</th>
    <th>#Words</th>
    <th>#Questions</th>
  </tr>
  <tr>
    <td>NMET</td>
    <td>118</td>
    <td>50k</td>
    <td>446</td>
  </tr>
  <tr>
    <td>CET</td>
    <td>150</td>
    <td>92k</td>
    <td>750</td>
  </tr>
  <tr>
    <td>PGEE</td>
    <td>97</td>
    <td>57k</td>
    <td>478</td>
  </tr>
  <tr>
    <td>TPO</td>
    <td>207</td>
    <td>192k</td>
    <td>2197</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>572</th>
    <th>391k</th>
    <th>3871</th>
  </tr>
</table>

Table 6: Raw data statistics of MTQA-ELC

A.2 Construction Methodology

Preventing Data Leakage. To prevent "data leakage," where test data may overlap with training data, we randomized the paragraph order and shuffled multiple-choice question options. This minimizes the likelihood of LLMs generating answers based on prior exposure, ensuring a more accurate assessment of their understanding and reasoning capabilities in novel contexts.

Ensuring Fair Evaluation Across Different Lengths. To fairly evaluate model performance across varying text lengths without being influenced by data quality, we constructed three distinct "test papers" for each length. Each length's final score is the average accuracy rates and reasoning times across the three test sets.

Assessing the Gap Between LLMs and Human Performance. To evaluate the performance gap between LLMs and humans, we used the accuracy rates of test-takers for each question as the "human performance score," reflecting the real-world difficulty of the questions. The performance gap was then calculated using a series of formulas, detailed in Section 4.3.

B Additional Experiments

B.1 Single-Turn QA Evaluation

L-Eval Evaluation Setting For closed-ended tasks, we selected four datasets: Coursera, QuALITY, TOEFL, and SFciton. CSTree-SRI employed gpt-3.5-turbo for SE/IE modules while testing three AE configurations: Llama2-7B-chat, Llama2-13B-chat and Chatglm2-6b-8k. Baseline models used these vanilla models.

Table 7 demonstrates that our method achieves an average improvement of 13.6% when applied to different base models on the L-Eval benchmark. This indicates that our method is broadly applicable

<table>
  <tr>
    <th>Model</th>
    <th>Crsr.</th>
    <th>QuA.</th>
    <th>TOEFL</th>
    <th>SF</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Llama2-7B-chat</td>
    <td>29.21</td>
    <td>37.62</td>
    <td>51.67</td>
    <td>60.15</td>
    <td>44.66</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>35.47↑</td>
    <td>43.07↑</td>
    <td>61.34↑</td>
    <td>67.19↑</td>
    <td>51.77↑</td>
  </tr>
  <tr>
    <td>Llama2-13B-chat</td>
    <td>35.75</td>
    <td>42.57</td>
    <td>60.96</td>
    <td>54.68</td>
    <td>48.49</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>40.26↑</td>
    <td>46.53↑</td>
    <td>67.82↑</td>
    <td>64.06↑</td>
    <td>54.67↑</td>
  </tr>
  <tr>
    <td>Chatglm2-6b-8k</td>
    <td>43.75</td>
    <td>40.59</td>
    <td>53.90</td>
    <td>54.68</td>
    <td>48.23</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>48.21↑</td>
    <td>45.84↑</td>
    <td>63.19↑</td>
    <td>59.34↑</td>
    <td>54.15↑</td>
  </tr>
</table>

Table 7: The results of CSTree-SRI based on different model on L-Eval. The experimental data for the original models are sourced from the results reported in the L-Eval paper.
Figure 3: The radar chart represents the performance differences between models across task types. The bar chart represents the performance improvements of CSTree-SRI across task types.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">DU</th>
    <th colspan="2">MI</th>
    <th colspan="2">IJ</th>
    <th colspan="2">SR</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>ACC(%)</th>
    <th>ETS</th>
    <th>ACC(%)</th>
    <th>ETS</th>
    <th>ACC(%)</th>
    <th>ETS</th>
    <th>ACC(%)</th>
    <th>ETS</th>
  </tr>
  <tr>
    <th colspan="10">32k Length</th>
  </tr>
  <tr>
    <td>Llama-3.1-7B-Instruct</td>
    <td>67.33</td>
    <td>61.53</td>
    <td>50.94</td>
    <td>46.68</td>
    <td>54.67</td>
    <td>50.00</td>
    <td>88.67</td>
    <td>80.31</td>
    <td>180.86</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>79.00</td>
    <td>67.33</td>
    <td>78.87</td>
    <td>69.67</td>
    <td>72.00</td>
    <td>65.97</td>
    <td>89.33</td>
    <td>82.26</td>
    <td>210.21</td>
  </tr>
  <tr>
    <td>GLM-4-9B-Chat</td>
    <td>65.67</td>
    <td>57.62</td>
    <td>75.85</td>
    <td>66.76</td>
    <td>63.00</td>
    <td>55.34</td>
    <td>88.67</td>
    <td>76.92</td>
    <td>196.83</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>82.00</td>
    <td>71.90</td>
    <td>80.38</td>
    <td>72.40</td>
    <td><b>80.67</b></td>
    <td><b>73.94</b></td>
    <td><b>92.33</b></td>
    <td><b>86.13</b></td>
    <td>218.42</td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B-Instruct</td>
    <td>75.33</td>
    <td>68.28</td>
    <td>69.06</td>
    <td>63.81</td>
    <td>67.00</td>
    <td>58.19</td>
    <td>90.00</td>
    <td>82.30</td>
    <td>201.08</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>85.33</td>
    <td>69.41</td>
    <td>81.89</td>
    <td>74.50</td>
    <td>78.67</td>
    <td>61.77</td>
    <td>92.00</td>
    <td>85.86</td>
    <td>219.71</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td><b>86.33</b></td>
    <td><b>85.29</b></td>
    <td><b>84.91</b></td>
    <td><b>82.95</b></td>
    <td>80.00</td>
    <td><b>79.12</b></td>
    <td><b>95.67</b></td>
    <td><b>94.42</b></td>
    <td><b>224.40</b></td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-16k</td>
    <td>28.00</td>
    <td>27.80</td>
    <td>22.64</td>
    <td>22.49</td>
    <td>30.67</td>
    <td>30.39</td>
    <td>16.67</td>
    <td>16.62</td>
    <td>93.74</td>
  </tr>
  <tr>
    <th colspan="10">256k Length</th>
  </tr>
  <tr>
    <td>Llama-3.1-7B-Instruct</td>
    <td>45.33</td>
    <td>40.87</td>
    <td>42.33</td>
    <td>38.17</td>
    <td>43.67</td>
    <td>39.40</td>
    <td>84.00</td>
    <td>75.58</td>
    <td>156.35</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>78.67</td>
    <td>67.05</td>
    <td>71.00</td>
    <td>61.44</td>
    <td>71.00</td>
    <td>63.29</td>
    <td>89.33</td>
    <td>83.65</td>
    <td>204.8</td>
  </tr>
  <tr>
    <td>GLM-4-9B-Chat</td>
    <td>55.00</td>
    <td>47.76</td>
    <td>67.17</td>
    <td>58.10</td>
    <td>47.33</td>
    <td>40.72</td>
    <td>77.67</td>
    <td>67.37</td>
    <td>199.14</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td><b>86.00</b></td>
    <td><b>75.98</b></td>
    <td><b>79.67</b></td>
    <td><b>68.79</b></td>
    <td><b>81.67</b></td>
    <td><b>71.73</b></td>
    <td><b>93.33</b></td>
    <td><b>87.09</b></td>
    <td><b>220.46</b></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B-Instruct</td>
    <td>61.00</td>
    <td>55.63</td>
    <td>54.67</td>
    <td>49.80</td>
    <td>56.33</td>
    <td>51.41</td>
    <td>87.67</td>
    <td>79.67</td>
    <td>179.05</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>84.00</td>
    <td>72.99</td>
    <td>76.00</td>
    <td>66.73</td>
    <td>78.67</td>
    <td>68.86</td>
    <td>90.33</td>
    <td>84.43</td>
    <td>214.56</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>69.33</td>
    <td>67.71</td>
    <td>62.33</td>
    <td>61.26</td>
    <td>65.33</td>
    <td>63.98</td>
    <td>90.00</td>
    <td><b>88.33</b></td>
    <td>192.96</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-16k</td>
    <td>18.00</td>
    <td>17.93</td>
    <td>23.67</td>
    <td>23.47</td>
    <td>27.00</td>
    <td>26.77</td>
    <td>25.67</td>
    <td>25.55</td>
    <td>96.46</td>
  </tr>
</table>

Table 8: Experimental Results of MTQA for Different Types of Tasks. The best performance is shown in bold, while the second best performance is represented with an underline.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">NMET</th>
    <th colspan="3">CET</th>
    <th colspan="3">PGEE</th>
    <th colspan="3">TPO</th>
    <th rowspan="2">avg.</th>
  </tr>
  <tr>
    <th>ACC(%) ETS</th>
    <th>Overall</th>
    <th>ACC(%) ETS</th>
    <th>Overall</th>
    <th>ACC(%) ETS</th>
    <th>Overall</th>
    <th>ACC(%) ETS</th>
    <th>Overall</th>
    <th>ACC(%) ETS</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td colspan="12"><b>32k Length</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-7B-Instruct</td>
    <td>66.33</td>
    <td>61.66</td>
    <td>182.31</td>
    <td>58.67</td>
    <td>54.61</td>
    <td>170.38</td>
    <td>69.00</td>
    <td>64.17</td>
    <td>197.41</td>
    <td>47.33</td>
    <td>42.38</td>
    <td>139.59</td>
    <td>172.42</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>74.33</td>
    <td>67.43</td>
    <td>198.72</td>
    <td>77.00</td>
    <td>68.19</td>
    <td>207.54</td>
    <td>75.67</td>
    <td>68.93</td>
    <td>210.91</td>
    <td>82.00</td>
    <td>75.43</td>
    <td>211.30</td>
    <td>207.12</td>
  </tr>
  <tr>
    <td>GLM-4-9B-Chat</td>
    <td>64.33</td>
    <td>58.59</td>
    <td>178.27</td>
    <td>61.67</td>
    <td>55.39</td>
    <td>176.50</td>
    <td>55.67</td>
    <td>50.75</td>
    <td>170.27</td>
    <td>73.33</td>
    <td>62.29</td>
    <td>193.23</td>
    <td>179.57</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>83.33</td>
    <td>76.24</td>
    <td>217.01</td>
    <td>79.67</td>
    <td>73.14</td>
    <td>212.89</td>
    <td>82.00</td>
    <td>73.92</td>
    <td>223.76</td>
    <td>89.67</td>
    <td>83.09</td>
    <td>227.02</td>
    <td>220.17</td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B-Instruct</td>
    <td>75.67</td>
    <td>70.69</td>
    <td>201.42</td>
    <td>80.67</td>
    <td>74.83</td>
    <td>215.17</td>
    <td>77.67</td>
    <td>72.87</td>
    <td>214.93</td>
    <td>77.67</td>
    <td>70.37</td>
    <td>202.29</td>
    <td>208.45</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>79.67</td>
    <td>73.18</td>
    <td>209.60</td>
    <td>81.00</td>
    <td>72.31</td>
    <td>215.70</td>
    <td>81.33</td>
    <td>71.92</td>
    <td>222.33</td>
    <td>86.33</td>
    <td>77.08</td>
    <td>220.18</td>
    <td>216.95</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td><b>86.67</b></td>
    <td><b>85.60</b></td>
    <td><b>223.75</b></td>
    <td><b>83.00</b></td>
    <td><b>81.82</b></td>
    <td><b>219.83</b></td>
    <td><b>82.67</b></td>
    <td><b>79.84</b></td>
    <td><b>225.09</b></td>
    <td><b>89.67</b></td>
    <td><b>88.47</b></td>
    <td><b>226.98</b></td>
    <td><b>223.91</b></td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-16k</td>
    <td>37.00</td>
    <td>36.66</td>
    <td>122.43</td>
    <td>31.33</td>
    <td>31.02</td>
    <td>114.82</td>
    <td>26.67</td>
    <td>26.40</td>
    <td>111.52</td>
    <td>12.33</td>
    <td>12.31</td>
    <td>67.36</td>
    <td>104.03</td>
  </tr>
  <tr>
    <td colspan="12"><b>256k Length</b></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-7B-Instruct</td>
    <td>57.33</td>
    <td>51.64</td>
    <td>165.01</td>
    <td>42.00</td>
    <td>37.91</td>
    <td>135.65</td>
    <td>45.00</td>
    <td>40.59</td>
    <td>151.67</td>
    <td>51.67</td>
    <td>46.51</td>
    <td>146.49</td>
    <td>149.71</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>68.67</td>
    <td>60.27</td>
    <td>188.13</td>
    <td>72.33</td>
    <td>62.65</td>
    <td>197.43</td>
    <td>67.33</td>
    <td>58.88</td>
    <td>196.88</td>
    <td>88.00</td>
    <td>79.05</td>
    <td>221.31</td>
    <td>200.94</td>
  </tr>
  <tr>
    <td>GLM-4-9B-Chat</td>
    <td>49.33</td>
    <td>43.93</td>
    <td>151.13</td>
    <td>45.67</td>
    <td>40.44</td>
    <td>148.81</td>
    <td>43.67</td>
    <td>39.26</td>
    <td>159.18</td>
    <td>59.67</td>
    <td>50.17</td>
    <td>152.53</td>
    <td>152.91</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td><b>79.00</b></td>
    <td><b>68.20</b></td>
    <td><b>208.97</b></td>
    <td><b>77.66</b></td>
    <td><b>66.06</b></td>
    <td><b>208.37</b></td>
    <td><b>78.00</b></td>
    <td><b>70.00</b></td>
    <td><b>218.43</b></td>
    <td>86.33</td>
    <td>78.19</td>
    <td>217.81</td>
    <td><b>213.40</b></td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B-Instruct</td>
    <td>70.33</td>
    <td>63.95</td>
    <td>191.36</td>
    <td>56.00</td>
    <td>51.00</td>
    <td>164.13</td>
    <td>58.67</td>
    <td>53.45</td>
    <td>179.38</td>
    <td>74.67</td>
    <td>67.87</td>
    <td>193.92</td>
    <td>182.20</td>
  </tr>
  <tr>
    <td>-CSTree-SRI</td>
    <td>76.67</td>
    <td><b>68.94</b></td>
    <td><b>204.34</b></td>
    <td><b>78.67</b></td>
    <td><b>68.40</b></td>
    <td><b>210.30</b></td>
    <td>71.33</td>
    <td>64.81</td>
    <td>205.05</td>
    <td><b>88.33</b></td>
    <td><b>79.82</b></td>
    <td><b>221.94</b></td>
    <td><b>210.41</b></td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>69.00</td>
    <td>67.48</td>
    <td>188.65</td>
    <td>65.67</td>
    <td>64.55</td>
    <td>183.80</td>
    <td>62.33</td>
    <td>60.95</td>
    <td>186.67</td>
    <td>81.67</td>
    <td><b>80.15</b></td>
    <td>208.43</td>
    <td>191.89</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-16k</td>
    <td>22.33</td>
    <td>22.18</td>
    <td>93.68</td>
    <td>28.33</td>
    <td>28.00</td>
    <td>107.86</td>
    <td>30.33</td>
    <td>30.09</td>
    <td>121.96</td>
    <td>31.67</td>
    <td>31.49</td>
    <td>104.89</td>
    <td>107.10</td>
  </tr>
</table>

Table 9: Experimental Results of MTQA for Tasks of Different Difficulty Levels. The best performance is shown in bold, while the second best performance is represented with an underline.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Avg. token consumption</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>#Input</th>
    <th>#Output</th>
    <th>#Interaction Count</th>
  </tr>
  <tr>
    <td colspan="4"><b>CSTree-SRI (A Pure Open-Source Configuration)</b></td>
  </tr>
  <tr>
    <td>Ours (Purely Llama-3.1)</td>
    <td>23570.11</td>
    <td>1883.58</td>
    <td>37.71</td>
  </tr>
  <tr>
    <td>Ours (Purely GLM-4)</td>
    <td>22442.74</td>
    <td>949.00</td>
    <td>18.98</td>
  </tr>
  <tr>
    <td>Ours (Purely Qwen-2.5)</td>
    <td>18717.58</td>
    <td>777.20</td>
    <td>15.95</td>
  </tr>
  <tr>
    <td colspan="4"><b>CSTree-SRI (The Open-Source Model Only as Answer Expert)</b></td>
  </tr>
  <tr>
    <td>Ours (Llama-3.1 as AE)</td>
    <td>14922.58</td>
    <td>1211.91</td>
    <td>18.14</td>
  </tr>
  <tr>
    <td>Ours (GLM-4 as AE)</td>
    <td>11437.21</td>
    <td>796.38</td>
    <td>13.77</td>
  </tr>
  <tr>
    <td>Ours (Qwen-2.5 as AE)</td>
    <td><b>9938.34</b></td>
    <td><b>471.49</b></td>
    <td><b>11.81</b></td>
  </tr>
</table>

Table 10: Resource Consumption Study on MTQA-ELC (256k-length). The best performance is shown in bold.

across various models while significantly enhancing their QA capabilities.

B.2 Multi-Turn QA Evaluation

Due to DeepSeek’s widespread recognition, access to its API has become challenging, and therefore, related models were not evaluated in the experiments presented in the appendix. In future work, we plan to supplement the evaluation of its models.

Fig. 3 visualizes the experimental results of different models across various task types in multi-turn question answering, while Table 8 provides the detailed experimental data for this study. Most LLMs achieve higher accuracy and ETS scores on DU and SR tasks, indicating their inherent strength in single-paragraph retrieval. However, performance degrades significantly on MI and IJ tasks, revealing limitations in multi-paragraph retrieval and cross-context reasoning. The CSTree-SRI framework mitigates these weaknesses, demonstrating substantial improvements across all task types—particularly for MI (31.30%↑) and IJ (39.77%↑) on ETScore.

Table 9 evaluates MTQA performance on tasks of different difficulty levels. On the 32k-length task, gpt-4o-mini still achieves the highest performance; however, LLMs augmented with our CSTree-SRI demonstrate competitive results across all metrics, narrowing the gap with gpt-4o-mini. Notably, on the ELC task (256k-length), our framework outperforms gpt-4o-mini in tasks of all difficulty levels.

B.3 Resource Consumption Study

Table 10 presents the method cost analysis experiment conducted on 256k-length extra-long contexts from MTQA-ELC. Below, we explain the meaning
of each statistical metric:

• Avg. #Input token consumption: Represents the average number of tokens input into all LLMs (SE/IE/AE) per round in the MTQA process.

• Avg. #Output token consumption: Represents the average number of tokens output by all LLMs (SE/IE/AE) per round in the MTQA process.

• Avg. #Interaction Count: Represents the average number of times each LLM (SE/IE/AE) is invoked per round during MTQA, indicating the interaction frequency between different modules in our framework.

We find that after applying our CSTree-SRI method, even when handling 256k-length extra-long contexts, the token consumption remains exceptionally low—input token consumption is under 24k, and output token consumption is under 2k. Moreover, when using Qwen-2.5-7B-Instruct as AE and GPT-4o-mini as SE/IE, the token consumption is even lower—input token consumption drops below 10k, and output token consumption falls below 0.5k, while the average interaction count among modules is only 11.81.

In table 10, we did not include the token consumption data for the Locally Deployed Models, because these models directly feed the entire 256k-length text into the LLM for every query. As a result, the average token consumption per round in multi-turn question answering is guaranteed to exceed 256k tokens, which further highlights the significant efficiency advantage of our method in terms of token consumption.
C  Prompts

C.1  The Prompts for Introspective Question
C.1.1  Relative

Instruction: Please evaluate the relevance of the provided evidence to the question from the following aspects.
    1. If the evidence relate to the same article as the question, respond with [Relevant]
    2. If the evidence relate to the same topic, or theme as the question, respond with [Relevant]
    3. If the evidence provide background knowledge or context that may help in understanding the question or related concepts, respond with [Relevant]
    4. If the evidence include information could offer relevant context or serve as a contrast that helps clarify the question, respond with [Relevant]
    Please judge whether the evidence is relevant to the question in order according to my standards. If it meets the standards, please return directly to the [Relevant]. Otherwise, respond with [Irrelevant].
    I will provide you with multiple pieces of evidence and a question. Please indicate whether each piece of evidence is relevant to the question, separated by an @ sign. The output example is [Relevant] @ [Irrelevant] @ [Irrelevant]

    Instruction: {instruction}{question}
    Evidence: {retrieval_content}
    Judgment:

Figure 4: The Prompt for <Relative> Introspection

C.1.2  ExtraRetr

Instruction: Based on the multiple retrieval text I found regarding this question, do you think I should continue searching for more text?
    If you believe the existing text is insufficient to answer the question, please respond with [Yes] otherwise respond with [No].

    Retrieval Text: {retrieval_content}
    Question: {question}
    Judgment:

Figure 5: The Prompt for <ExtraRetr> Introspection
C.1.3 NodeRetr

Instruction: You are an intelligent information retrieval assistant. You will be provided an instruction and a summary of an article. Your task is to determine whether it is necessary to retrieve the full content of the article based on the provided summary. There are three cases:
  - If the summary relate to the same article as the question, respond with [Yes].
  - If the summary suggests some similarity to the question or indicates that the article may potentially answer the question, respond with [Yes].
  - If the summary already sufficiently answers the question, respond with [Yes].
    If the information in the [Summary] is likely to be useful for any of these cases, please respond with [Yes]. Otherwise, respond with [No].

Summary: {retrieval_summary}
Instruction: {instruction}{question}
Judgment:

Figure 6: The Prompt for <NodeRetr> Introspection

C.1.4 Support

Instruction: You will receive an instruction, evidence, and output. Your task is to evaluate whether the output is supported by the information provided in the evidence. There are three cases:
  [3-Fully supported] - All information in output is supported by the evidence, or extractions from the evidence. This is only applicable when the output and part of the evidence are almost identical.
  [2-Partially supported] - The output is supported by the evidence to some extent, but there is some information in the output that is not discussed in the evidence. For instance, if the output covers multiple concepts and the evidence only discusses some of them, it should be considered a [Partially supported].
  [1-No support] - The output completely ignores evidence, is unrelated to the evidence, or contradicts the evidence.
    Please select from the following three options [3], [2], [1].

Instruction: {instruction}{question}
Evidence: {retrieval_content}
Output: {answers}
Judgment:

Figure 7: The Prompt for <Support> Introspection
C.1.5 Useful

Instruction: You are a teacher. You will receive an instruction and an output. Your task is to evaluate the student’s output based on the provided instruction. You should score it according to the criteria outlined below.

Scoring Criteria:
[1-Unrelated answer]: Serious errors, confusing, Unclear and worthless.
[2-Partially related]: weak response, Multiple inaccuracies, misleading. Confusing, lacks logic.
[3-Somewhat related]: partial answer, some mistakes, Moderate clarity, includes vague parts.
[4-Relevant and mostly complete]: Generally accurate, no major errors, Clear and logical, easy to understand.
[5-Fully relevant and comprehensive answer]: Highly accurate, rich information, Very clear, logical, and valuable.

Additional Suggestions:
For higher scores, it is best to include examples and explanations that help illustrate key points. Meanwhile, Encourage thoroughness and critical thinking in responses. Please select from the following five options [5], [4], [3], [2], [1].

Instruction: {instruction}{question}
Output: {answers}
Score:

Figure 8: The Prompt for <Useful> Introspection

C.2 The Prompts for Expert Modules

C.2.1 AE Module

Instruction: {content}{query}. Provide the answers directly, without any introductory phrases or explanations.
Your Answer:

Figure 9: The Prompt for AE Module

Instruction: {content}{query}. This answer is wrong [{preanswer}]. Don't apologize, only provide the answers directly, without any introductory phrases or explanations.

Figure 10: The Prompt for AE Module to Regenerate Response
C.2.2 SE Module

Instruction: Please summarize the content in concise sentences, while retaining logical locators (such as unique IDs that represent paragraphs) and key information.
Content: {}

Figure 11: The Prompt for SE Module

C.3 The Prompts for Dataset Evaluation

C.3.1 Mixed Tasks

Instruction: Please answer the following questions based on the following information.
The content within the angle brackets <> represents paragraph IDs from various articles. These IDs are used to identify specific sections of text within different articles.
Information: {ELC}{queries}
Provide the answers directly, without any introductory phrases or explanations.
Your Answer:

Figure 12: The Prompt for Mixed Tasks

C.3.2 DU Task

Instruction: Please answer the following questions based on the following information.
The content within the angle brackets <> represents paragraph IDs from various articles. These IDs are used to identify specific sections of text within different articles.
The following questions are about understanding the details of paragraphs.
Information: {ELC}{queries}
Provide the answers directly, without any introductory phrases or explanations.
Your Answer:

Figure 13: The Prompt for DU Task
C.3.3 MI Task

Instruction: Please answer the following questions based on the following information.
    The content within the angle brackets <> represents paragraph IDs from various articles. These IDs are used to identify specific sections of text within different articles.
    The following questions require you to grasp the main idea of the entire article.
    Information: {ELC}{queries}
    Provide the answers directly, without any introductory phrases or explanations.
    Your Answer:

Figure 14: The Prompt for MI Task

C.3.4 IJ Task

Instruction: Please answer the following questions based on the following information.
    The content within the angle brackets <> represents paragraph IDs from various articles. These IDs are used to identify specific sections of text within different articles.
    The following questions require you to pay attention to the logical relationship of the information in the paragraph, testing your reasoning ability.
    Information: {ELC}{queries}
    Provide the answers directly, without any introductory phrases or explanations.
    Your Answer:

Figure 15: The Prompt for IJ Task
C.3.5 SR Task

Instruction: Please answer the following questions based on the following information.
    The content within the angle brackets <> represents paragraph IDs from various articles. These IDs are used to identify specific sections of text within different articles.
    The following questions require you to understand the meaning of phrases, sentences, or demonstrative pronouns, testing your comprehension of the entire article.
    Information: {ELC}{queries}
    Provide the answers directly, without any introductory phrases or explanations.
    Your Answer:

Figure 16: The Prompt for SR Task
D  Examples For Multi-turn QA over Extra-Long Context

D.1 Example For DU Task

Example of DU Task

Extra-Long Context: <article NMET 66 paragraph 1> In 1916, two girls of wealthy families, best friends from ...(84 words)... Dorothy Woodruff's granddaughter. </article NMET 66 paragraph 1>
...(256k words)...
<article TOEFL TPO 6 paragraph 5> In one example of organizing the allocation ...(117 words)... will receive insufficient moisture. </article TOEFL TPO 6 paragraph 5>

Question1: This is a question about article TOEFL TPO 120. Please choose the correct answer from options A, B, C, and D below to answer the question. According to paragraph 5, Hubbell and Johnson determined:
A. the level of aggressiveness of each of the nine species
B. the number of colonies of each of the nine species
C. the order in which the colonies in the study area had been established
D. the distribution pattern of the nests of five of the nine species
Ground Truth: D ...

Question2: This is a question about article TOEFL TPO 120. Please choose the correct answer from options A, B, C, and D below to answer the question. According to paragraph 2, some species of stingless bees are aggressive mainly toward
A. Bees from their own colony
B. Bees of their own species from different colonies
C. Nonaggressive bees that forage on the same flowers
D. Aggressive bees of other species
Ground Truth: B ...

Question3: This is a question about article CET 119. Please choose the correct answer from options A, B, C, and D below to answer the question. What makes Chris Cocalis believe there is a greater opportunity for ebike sales?
A. The younger generation's pursuit of comfortable riding.
B. The increasing interest in mountain climbing.
C. The public's concern for their health.
D. The further lowering of ebike prices.
Ground Truth: A ...

Question4: This is a question about article CET 119. Please choose the correct answer from options A, B, C, and D below to answer the question. What is the prospect of the bike industry according to Ryan Rzepecki ?
A. It will profit from ebike sharing.
B. More will be invested in bike battery research.
C. The sales of ebikes will increase.
D. It will make a difference in people's daily lives.
Ground Truth: D ...

……

Figure 17: The Example for DU Task
D.2 Example For MI Task

Example of MI Task

Extra-Long Context: <article NMET 67 paragraph 1> Can a small group of ...(54 words)... on a 24/7 basis. </article NMET 67 paragraph 1>
...(256k words)
<article PGEE 68 paragraph 7> The sharp hit to growth predicted around the ...(44 words)... may even see progress. </article PGEE 68 paragraph 7>

Question1: This is a question about article PGEE 82. Please choose the correct answer from options A, B, C, and D below to answer the question. Van Oosten believes that certain plastic objects are
A. complex in structure.
B. immune to decay.
C. inherently flawed.
D. improperly shaped.
Ground Truth: C ...

Question2: This is a question about article PGEE 82. Please choose the correct answer from options A, B, C, and D below to answer the question. The author thinks that preservation of plastics is
A. unpopular.
B. challenging.
C. costly.
D. unworthy.
Ground Truth: B ...

Question3: This is a question about article CET 113. Please choose the correct answer from options A, B, C, and D below to answer the question. What does Maryanne Taylor think of self-imposed sleeplessness ?
A. It may symbolise one's importance and success.
B. It may be practiced only by certain tech heads.
C. It may well serve as a measure of self-discipline.
D. It may turn out to be key to a successful career.
Ground Truth: A ...

Question4: This is a question about article CET 113. Please choose the correct answer from options A, B, C, and D below to answer the question. How does Dr. Sophie Bostock look at the 20-hour daily work schedule?
A. One should not adopt it without consulting a sleep expert.
B. One must be duly self-disciplined to adhere to it.
C. The general public should not be encouraged to follow it.
D. The majority must adjust their body clock for it.
Ground Truth: C ...

......
Example of IJ Task

Extra-Long Context: <article CET 143 paragraph 1> Have you ever wondered ...(35 words)... in interpersonal relationships. </article CET 143 paragraph 1>
...(256k words)
<article CET 77 paragraph 11> "We're learning that student success requires ...(43 words)... feedback loops." </article CET 77 paragraph 11>

Question1: This is a question about article TOEFL TPO 193. Please choose the correct answer from options A, B, C, and D below to answer the question. Why does the author mention “Indian mustard”?
A. To warn about possible risks involved in phytoremediation
B. To explain how zinc contamination can be reduced
C. To show that hyperaccumulating plants grow in many regions of the world
D. To help illustrate the potential of phytoremediation
Ground Truth: D ...

Question2: This is a question about article TOEFL TPO 193. Please choose the correct answer from options A, B, C, and D below to answer the question. It can be inferred from paragraph 6 that compared with standard practices for remediation of contaminated soils, phytoremediation
A. is less suitable for soils that need to be used within a short period of time
B. does not allow for the use of the removed minerals for industrial purposes
C. can be faster to implement
D. is equally friendly to the environment
Ground Truth: A ...

Question3: This is a question about article PGEE 22. Please choose the correct answer from options A, B, C, and D below to answer the question. The text suggests that immigrants now in the U.s.
A. are hardly a threat to the common culture.
B. constitute the majority of the population.
C. exert a great influence on American culture.
D. are resistant to homogenization.
Ground Truth: A ...

Question4: This is a question about article PGEE 22. Please choose the correct answer from options A, B, C, and D below to answer the question. Why are Arnold Schwarzenegger and Garth Brooks mentioned in Paragraph 5?
A. To prove their popularity around the world.
B. To show the powerful influence of American culture.
C. To reveal the public's fear of immigrants.
D. To give examples of successful immigrants.
Ground Truth: B ...

……
D.4 Example For SR Task

Example of SR Task

Extra-Long Context: <article TOEFL TPO 154 paragraph 1> While some European countries ...(75 words)... to understand the sources of their success. </article TOEFL TPO 154 paragraph 1>
...(256k words)
<article TOEFL TPO 166 paragraph 11> Regarding the appearance of celebrities ...(83 words)... like the celebrity in question. </article TOEFL TPO 166 paragraph 11>

Question1: This is a question about article TOEFL TPO 111. Please choose the correct answer from options A, B, C, and D below to answer the question. The word "simultaneously" in the passage is closest in meaning to
A. merely
B. spontaneously
C. at the same time
D. without limits
Ground Truth: C ...

Question2: This is a question about article TOEFL TPO 111. Please choose the correct answer from options A, B, C, and D below to answer the question. The word "differing" in the passage is closest in meaning to
A. increasing
B. varying
C. high
D. necessary
Ground Truth: B ...

Question3: This is a question about article TOEFL TPO 67. Please choose the correct answer from options A, B, C, and D below to answer the question. The word "comprising" in the passage 4 is closest in meaning to
A. made up of
B. covering
C. taken from
D. suggesting
Ground Truth: A ...

Question4: This is a question about article TOEFL TPO 67. Please choose the correct answer from options A, B, C, and D below to answer the question. The word "crucial" in the passage is closest in meaning to
A. established
B. understood
C. important
D. interesting
Ground Truth: C ...

......

Reviewer paper summaries: This draft attempts to propose what looks like a basic agentic framework with RAG for long document multi-turn QA. I found the draft confusing and very poorly written, see my comments below.
Summary of strengths: Some use of open-source models BUT they are coupled with proprietary LLMs. I would like to see a pure open-source configuration without reliance on prompting ChatGPT.

Evaluation on LongBench is provided but it's limited (the scores are also poor). I would like to see more established benchmarks used such as MT-Eval to evaluate on more reliable datasets.
Summary of weaknesses: Poorly explained motivation and in particular, methodology. There is no way anyone would be able to reproduce this method from the paper. Important details are vaguely described using buzzwords and newly introduced non-technical phrases. I don't see any novelty here. This draft would be better suited as a technical blog, for example. It does not meet the scientific rigor that is expected of ARR papers.

I doubt the quality of the proposed dataset and in particular, the strange 'new' metrics that are very poorly motivated. How is it possible that the MAX and AVE # of words in your benchmark is basically the same? This seems very artificial as a benchmark. Therefore, I can only trust the existing evaluation datasets to get a sense for how this agentic framework (with some basic RAG) actually works.
Comments, suggestions and typos: What is the metric used in Table 3? Accuracy? Why is this not in the caption?
Overall assessment scores: 2.5