Document Segmentation Matters for Retrieval-Augmented Generation
Authors: Zhitong Wang, Cheng Gao, Yufei Huang, Shuzheng Si, Kangyang Luo, Yuzhuo Bai, Wenhao Li, Tangjian Duan, Chuancheng Lv, Guoshan Lu, Gang Chen, Fanchao Qi, Chaojun Xiao, Maosong Sun
Accepted at: acl25

Document Segmentation Matters for Retrieval-Augmented Generation

Zhitong Wang*, Cheng Gao*, Chaojun Xiao*♣, Yufei Huang♣, Shuzheng Si♣, Kangyang Luo♣, Yuzhuo Bai♣, Wenhao Li♣, Tangjian Duan♡, Chuancheng Lv♡, Guoshan Lu♡, Gang Chen♡, Fanchao Qi*♣ Maosong Sun*♣♣♡,
♣Department of Computer Science and Technology, Tsinghua University
♡DeepLang AI ♣Institute for AI, Tsinghua University
♢Jiangsu Collaborative Innovation Center for Language Ability
wangzt23@mails.tsinghua.edu.cn, xcjthueg@gmail.com

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge. A critical yet underexplored challenge in RAG is document segmentation, also known as document chunking. Existing widely-used rule-based chunking methods usually lead to suboptimal splits, where overly large chunks introduce irrelevant information and small chunks lack semantic coherence. Existing semantic-based approaches either require costly LLM calls or fail to adaptively group contextually related sentences. To address these limitations, we propose PIC (Pseudo-Instruction for document Chunking), a simple yet effective method that leverages document summaries as pseudo-instructions to guide chunking. By computing semantic similarity between sentences and the summary, PIC dynamically groups sentences into chunks that align with the document’s key themes, ensuring semantic completeness and relevance to potential user instructions. Experiments on multiple open-domain question-answering benchmarks demonstrate that PIC can significantly improve retrieval accuracy (Hits@k) and end-to-end QA performance (Exact Match) without any additional training.

1 Introduction

Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with large-scale textual knowledge to enhance response quality and mitigate hallucinations in LLMs (Lewis et al., 2020; Karpukhin et al., 2020; Borgeaud et al., 2022; Guu et al., 2020; Edge et al., 2024; Sarthi et al., 2024; Yan et al., 2024; Si et al., 2025a,b). By dynamically retrieving real-time information from databases such as Wikipedia, RAG enables more relevant and accurate responses. RAG has been widely applied across domains such as law (Louis et al., 2023), medicine (Wu et al., 2024), and finance (Zhang et al., 2023), making it an essential component in real-world applications.

A standard RAG pipeline comprises three key stages: (1) Document segmentation into manageable units, named chunks; (2) Knowledge retrieval for input instructions; (3) Response generation based on retrieved chunks. While existing research mainly focuses on retriever improvements (Shi et al., 2024; Rubin et al., 2022) and system architecture design (Edge et al., 2024; Zhang et al., 2024), the foundational challenge of document segmentation remains critically underexplored. Effective document segmentation is crucial, as lengthy source documents in knowledge bases like Wikipedia exceed the processing capacities of both retrievers and LLMs.

The majority of chunking methods employed in RAG systems are rule-based, typically relying on heuristics such as fixed-length segmentation (Lewis et al., 2020; Karpukhin et al., 2020; Borgeaud et al., 2022). However, these methods usually suffer from inherent limitations in selecting appropriate chunk sizes. Larger chunks may include irrelevant information, which can both interfere with retrieval and generation, leading to hallucinations. Conversely, smaller chunks may lack sufficient semantic information, making it challenging for the LLM to generate accurate and coherent responses.

To address the above issues, certain studies have explored semantic-based chunking methods. Some research attempts to rewrite documents. For example, Chen et al. (2024) tries to transform the original text into independent, contextualized, and self-contained segments. This approach can integrate contextual knowledge, but it still relies on rule-based context window segmentation and fails to consider the adaptive combination of sentences. Other studies attempt to identify semantic breakpoints. Greg (2024) seperates adjacent sentences with the least semantic similarity, yet it lacks a global consideration of the entire document’s con-
text, leading to suboptimal chunking. Duarte et al. (2024) and Zhao et al. (2024) leverage LLMs’ understanding of semantics for segmentation, but these models require extensive calls to large models, making them costly, time-consuming, and difficult to apply to large-scale text knowledge bases.

We aim to explore a new perspective on what constitutes good chunking. When a user inputs an instruction, our goal is to ensure that sentences relevant to the instruction are grouped within the same chunk, while unrelated sentences are placed in other chunks. This approach improves retrieval accuracy and prevents unrelated information from affecting LLM generation. However, in practical RAG applications, predefining chunks based on specific instructions is infeasible, as the instruction distribution is unknown when constructing chunks. Moreover, dynamically resegmenting a document for each instruction would incur unacceptable computational overhead.

To address the challenge of document segmentation without prior knowledge of real user instruction distributions, we propose PIC (Pseudo-Instruction for document Chunking), which utilizes document summaries as pseudo-instructions to guide segmentation. By generating a summary that preserves key document information (often central to real-world user queries), we compute semantic similarity scores between each sentence and the summary, then group sentences with scores above/below the average similarity threshold into coherent chunks. This approach ensures semantic completeness within chunks while isolating irrelevant content, thereby improving retrieval relevance and reducing hallucinations in LLMs. The method aligns chunk distributions with potential user instructions by prioritizing summary-aligned content aggregation and minimizing unrelated information within individual chunks.

To evaluate the effectiveness of different chunking methods, we process the latest English WikiDump, segmenting each document into chunks to serve as the knowledge base for RAG. We conduct experiments on multiple open-domain question-answering benchmarks, where our proposed method outperforms existing chunking approaches in terms of retrieval performance (Hits@k) and end-to-end QA performance (Exact Match). The results demonstrate that PIC achieves promising performance improvements across various experimental settings, proving to be an effective and generalizable chunking method. Our main contributions are summarized as follows:

• We propose a simple yet effective chunking method, PIC, which leverages an LLM to generate a pseudo-instruction and groups sentences into chunks based on their semantic relationship with the pseudo-instruction.
• We introduce a knowledge corpora PICWiki, derived from Wikipedia. This dataset consists of the original text of each document, the generated pseudo-instruction, and the chunks processed by our method. Serving as a text knowledge base for RAG, this dataset can significantly improve performance on knowledge-intensive tasks without requiring model retraining.
• We conduct extensive experiments on multiple QA benchmarks, demonstrating that our method achieves significant performance improvements in both retrieval accuracy and end-to-end QA system performance.

2 Related Works

Retrieval-Augmented Generation. RAG has been proven to be an effective approach for addressing knowledge-intensive NLP tasks (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022; Huang et al., 2024; Guu et al., 2020). Existing RAG methods mainly focus on improving the retriever performance and designing the whole RAG pipeline. As for the retriever improvement, Karpukhin et al. (2020) proposes training a dense passage retriever to replace sparse retrieval methods like BM25 (Robertson et al., 1995); Yu et al. (2024) introduce reranking mechanisms after retrieval; Sawarkar et al. (2024) and Juvekar and Purwar (2024) integrate multiple retrieval strategies; and Edge et al. (2024) incorporates knowledge graphs, refining retrieval through graph traversal. On the other hand, some works focus on enhancing the reliability of language model generation. For example, Xu et al. (2023) compresses retrieved documents to extract essential information; Yan et al. (2024) corrects and rewrites retrieved knowledge; Asai et al. (2023) introduces self-evaluation and reflection to selectively apply knowledge during generation; and Zhang et al. (2024) trains the generation model with distractor documents to improve its robustness against misleading information. These approaches effectively integrate up-to-date information and reduce model hallucinations, making them indispensable for modern generative models. However, most of these methods primarily focus
Summary: The document primarily explores the historical origins of Chinese New Year and the integration of traditional customs with modern evolution, focusing on the role of family reunions, cultural heritage, and the festival’s impact on Chinese communities worldwide.

Figure 1: Illustration of our proposed PIC, where the document is dynamically segmented based on the similarity between sentences and the pseudo-instruction.

on improving the retriever and generation mechanisms within RAG while overlooking the impact of chunk quality on downstream task performance.

Retrieval Units in RAG. The RAG process begins by splitting long documents into smaller units, which are stored in a knowledge base. Relevant units are then retrieved based on the user’s instructions. This retrieval unit can be defined at various natural text granularities such as sentence, paragraph or document-level (Gao et al., 2023). Some specific rules are also used to manually segment documents into chunks by fixed-size (Borgeaud et al., 2022), punctuation (Liu, 2022), or phrase (Lan et al., 2023). Several advanced chunking approaches further incorporate semantic relationships to enhance retrieval effectiveness. For instance, Greg (2024) suggests computing embedding similarity between adjacent sentences to determine chunking points. Since LLMs capture semantic dependencies better, many model-based methods have been proposed. Chen et al. (2024) uses LLMs to rewrite sentences into contextualized segments with pronouns restored to their corresponding entities. Duarte et al. (2024) leverages LLMs to detect semantic transition points in a document, treating them as chunk boundaries. Zhao et al. (2024) uses perplexity (PPL) to measure semantic coherence, identifying local PPL minima as optimal chunk boundaries.

3 Methodology

In the context of RAG, an effective chunking strategy is critical for both retrieval and generation performance. Given a document \( D = \{ s_1, s_2, ..., s_n \} \) containing n sentences, we aim to segment \( D \) into a set of semantically coherent chunks \( C = \{ c_1, c_2, ..., c_m \} \), where each chunk \( c_i \) consists of one or more combined sentences, and m represents the total number of chunks.

Our method consists of two key steps: 1) Generating pseudo-instructions. 2) Performing semantic chunking based on pseudo-instructions, as shown in Figure 1. We will provide a detailed explanation of both steps below.

3.1 Pseudo-Instruction Generation

In this paper, we propose using pseudo-instructions to guide chunking without knowing the real distribution of user-instructions. Specifically, we leverage document summaries as pseudo-instructions to construct semantically coherent chunks. Summaries can preserve key document information, allowing us to group adjacent sentences that share semantic alignment with the summary while separating irrelevant content.

Specifically, given a document \( D \), we use a summarizer \( S \) to generate a summary \( s_D \) for \( D \). The generated summary should be as concise and informative as possible to effectively serve as a pseudo-instruction.
3.2 Chunking Based on Pseudo-Instructions

For a given document \( D \), after obtaining the summary \( s_D \), we perform chunking by computing the semantic similarity between each sentence and the summary. Adjacent sentences with above-average similarity are grouped into coherent chunks, ensuring semantic completeness within chunks while minimizing unrelated information. Likewise, adjacent sentences with below-average similarity are also grouped together.

Specifically, we first use an embedding model to encode both the summary and sentences. Let \( \phi \) denote the embedding function, then the embeddings for the sentences and the summary are computed as: \( e_{s_i} = \phi(s_i), e_{s_D} = \phi(s_D) \). Next, we compute the cosine similarity between each sentence embedding and the summary embedding as follows:

\[
r_i = sim(s_i, s_D) = \frac{e_{s_i} \cdot e_{s_D}}{\|e_{s_i}\| \cdot \|e_{s_D}\|}.
\] (1)

Based on the similarity between each sentence and the summary, we group adjacent sentences into the same chunk if their similarity scores exceed a predefined threshold. Similarly, sentences with similarity scores below the threshold are also placed in the same chunks. The chunking rules are as follows:

\[
C_{rel} = \left\{ [s_i : s_j] \mid \begin{array}{l}
r_i, r_{i+1}, ..., r_j \geq \tau, \\
r_{i-1}, r_{j+1} < \tau
\end{array} \right\}, \tag{2}
\]
\[
C_{irr} = \left\{ [s_i : s_j] \mid \begin{array}{l}
r_i, r_{i+1}, ..., r_j < \tau, \\
r_{i-1}, r_{j+1} \geq \tau
\end{array} \right\}, \tag{3}
\]
\[
C = C_{rel} + C_{irr}. \tag{4}
\]

Here, \( \tau \) is the threshold that controls the chunking condition, \([s_i : s_j]\) represents a chunk consisting of sentences from \( s_i \) to \( s_j \) (\( i \leq j \)), and \( C_{rel}, C_{irr} \) represent chunks that are relevant and irrelevant with the pseudo-instruction, respectively. The final result, \( C \), is the segmented chunks for the document \( D \) using our method.

For the selection of \( \tau \), we use a dynamic threshold, as a fixed one would result in varying similarity levels across documents, making manual selection difficult. Specifically, we set the threshold \( \tau \) as the mean similarity \( \mu \) between all sentences in document \( D \) and its summary, which is calculated as \( \mu = \frac{1}{n} \sum_{i=1}^n r_i \). We find this mean value performing best among all values through experiments described in Section 6.2.

4 Experimental Settings

To evaluate the performance of different document chunking methods in RAG scenarios, we adopt Wikipedia as the knowledge corpus and conduct experiments on several knowledge-intensive tasks using two different LLMs. In this section, we present the details and results of our extensive experiments.

4.1 Baselines

Following previous work (Gao et al., 2023), we mainly choose the following chunking methods as our baseline methods: 1) Rule-based Chunking. Most existing RAG systems utilize rule-based chunking methods for processing the corpus. In our experiments, we evaluate three widely-used rule-based chunking approaches. Specifically, the documents in the corpus are segmented in three different ways: sentence-by-sentence (Sentence), paragraph-by-paragraph (Paragraph), and fixed-size windows (Fixed-size). In these methods, each chunk corresponds to a single sentence, a single paragraph, or a span of fixed length, respectively. 2) Semantic-based Chunking: These methods segment documents based on the semantic consistency of adjacent sentences. We present the results of the semantic chunking method proposed by Greg, 2024 (Semantic), which computes the similarity between adjacent sentence embeddings and segments when the similarity falls below a certain threshold. 3) Generation-based Chunking: These methods attempt to generate text chunks based on given documents. Chen et al. (2024) (Proposition) rewrites the sentences into self-contained atomic expressions called propositions, which will be used for further retrieval.

4.2 Knowledge Corpus

We use the latest English Wikipedia dump (2024-12-01), containing over 6 million documents, as our textual knowledge base. We extract the cleaned text using WikiExtractor\footnote{https://github.com/attardi/wikiextractor}. Then we apply different processing techniques based on the specific chunking methods: 1) As for the rule-based chunking method, we adopt the NLTK package\footnote{https://www.nltk.org/api/nltk.tokenize.sent_tokenize.html} to split document text to sentences; we split text to paragraphs by the new line character "\n" in the original text; and we split the document into fixed-size chunks of 100 words each, while preserving sentence boundaries. 2) As for the semantic-based
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">NQ</th>
    <th colspan="4">TriviaQA</th>
    <th colspan="4">WebQ</th>
    <th colspan="4">SQuAD</th>
    <th colspan="4">EQ</th>
    <th colspan="4">PopQA</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th>
  </tr>
  <tr>
    <td>Sentence</td><td>33.0</td><td>48.4</td><td>36.9</td><td>44.7</td><td>34.4</td><td>50.8</td><td>39.9</td><td>52.6</td><td>53.4</td><td>65.0</td><td>44.5</td><td>58.4</td><td>40.4</td><td>53.3</td>
  </tr>
  <tr>
    <td>Paragraph</td><td>48.3</td><td>62.3</td><td>40.3</td><td>47.6</td><td>43.3</td><td>58.9</td><td>44.4</td><td>57.9</td><td>61.5</td><td>73.3</td><td>50.3</td><td>64.8</td><td>48.0</td><td>60.8</td>
  </tr>
  <tr>
    <td>Fixed-size</td><td>60.7</td><td>71.7</td><td>40.7</td><td>48.2</td><td>49.1</td><td>63.7</td><td>47.1</td><td>63.1</td><td>66.0</td><td>76.8</td><td>63.2</td><td>77.3</td><td>54.5</td><td>66.8</td>
  </tr>
  <tr>
    <td>Semantic</td><td>59.4</td><td>71.2</td><td>41.2</td><td>48.9</td><td>53.5</td><td>64.4</td><td>46.8</td><td>61.2</td><td>69.9</td><td>79.2</td><td>65.1</td><td>79.4</td><td>56.0</td><td>67.4</td>
  </tr>
  <tr>
    <td>Proposition</td><td>60.0</td><td>71.5</td><td>40.8</td><td>48.1</td><td>52.5</td><td>65.3</td><td>56.0</td><td>68.5</td><td>69.9</td><td>78.8</td><td>67.1</td><td>81.3</td><td>57.7</td><td>68.9</td>
  </tr>
  <tr>
    <td>PIC</td><td>61.0</td><td>71.6</td><td>42.3</td><td>49.2</td><td>54.5</td><td>66.9</td><td>53.0</td><td>67.0</td><td>71.1</td><td>80.3</td><td>68.7</td><td>81.9</td><td>58.4</td><td>69.5</td>
  </tr>
</table>

Table 1: Retrieval performance (Hits@k) across different methods. The proposed PIC method achieves the highest average performance across all datasets, outperforming baseline methods such as semantic and proposition-based chunking in both Top-5 and Top-20 retrieval accuracy.

chunking method, we first build embeddings for each sentence using bge-large-en-v1.5 (Xiao et al., 2024), and calculate the similarity between every two adjacent sentences. Then, we compute the 20th percentile of all similarity scores and use it as a threshold\footnote{We experimented with thresholds at the 5th, 20th, 35th, and 50th percentiles and selected the 20th percentile, which yielded the best performance, as the baseline for comparison.}. Any pair of adjacent sentences with a similarity score below this threshold is split, which is defined as the chunk boundaries. 3) As for the proposition method, we use the open-source Propositionizer model\footnote{https://huggingface.co/chentong00/propositionizer-wiki-flan-t5-large}, first segmenting text into 100-word passages and then apply the model to rewrite passages into propositions.

Notably, we utilize PIC to process the same Wikipedia corpus, generating the resulting datasets named PICWiki. The dataset can be seamlessly integrated to enhance the performance of RAG systems in a plug-and-play manner. We will release PICWiki to facilitate the progress of RAG.

\subsection*{4.3 Evaluation Benchmarks}

We evaluate the chunking methods on the following six OpenQA benchmarks, including Natural Questions (NQ, Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), WebQuestions (WebQ, Bordes et al., 2014), SQuAD (Rajpurkar et al., 2016), EntityQuestions (EQ, Scialvino et al., 2021) and PopQA (Mallen et al., 2023), which focus on reading comprehension and answering natural language queries using Wikipedia or structured knowledge bases.

To fairly compare different chunking methods on these benchmarks, we mainly evaluate the performance of the methods from these two aspects: 1) \textbf{Retrieval Performance}: We evaluate the retrieval performance using Hits@k, measuring whether the top-k retrieved chunks contain the correct answer. 2) \textbf{End-to-end QA Performance}: To demonstrate the impact of different chunking methods on the question-answering abilities of the RAG system, we also evaluate the end-to-end QA performance by Exact Match (EM), which is calculated as the percentage of questions for which the predicted answer exactly matches the ground truth.

\subsection*{4.4 Implementation Details}

For the model implementation, we use GPT-4o-mini (OpenAI, 2024) as the summarizer to generate summaries for all documents with the prompt provided in Appendix C. We use bge-en-large-v1.5 as both the embedding model for chunking and the retriever model for RAG, and evaluate the end-to-end QA performance with two LLMs: Qwen2.5-7B-Instruct (Team, 2024) and Meta-Llama-3-8B-Instruct (Grattafiori et al., 2024). We retrieve the top 5 and top 20 chunks separately and evaluate both their retrieval performance and end-to-end QA performance. The retrieved chunks are then concatenated with the input instructions using the prompt template provided in Appendix D. We use the greedy sampling strategy for answer generation and the maximum length is set as 512.

\section*{5 Main Results}

\subsection*{5.1 Retrieval Performance}

Table 1 presents the retrieval performance of PIC compared to other baseline chunking methods. PIC achieves the highest average retrieval performance across six datasets, with the highest 2-point increase on WebQ. This suggests that our PIC method captures semantic information more effectively, leading to superior retrieval outcomes. We also conduct a case study in Appendix B.

Despite its effectiveness on most datasets, PIC exhibits suboptimal performance on SQuAD. One
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">NQ</th>
    <th colspan="4">TriviaQA</th>
    <th colspan="4">WebQ</th>
    <th colspan="4">SQuAD</th>
    <th colspan="4">EQ</th>
    <th colspan="4">PopQA</th>
    <th colspan="4">Avg.</th>
  </tr>
  <tr>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Avg.</th><th>Top5</th><th>Top20</th>
  </tr>
  <tr>
    <td colspan="26">Qwen2.5-7B-Instruct</td>
  </tr>
  <tr>
    <td>Sentence</td><td>29.8</td><td>35.0</td><td>55.4</td><td>60.2</td><td>30.0</td><td>34.3</td><td>34.8</td><td>39.6</td><td>42.6</td><td>46.4</td><td>36.9</td><td>41.2</td><td>38.2</td><td>42.8</td>
  </tr>
  <tr>
    <td>Paragraph</td><td>38.6</td><td>40.9</td><td>60.9</td><td>64.1</td><td>34.3</td><td>37.9</td><td>35.7</td><td>39.6</td><td>49.0</td><td>52.4</td><td>42.4</td><td>46.2</td><td>43.5</td><td>46.8</td>
  </tr>
  <tr>
    <td>Fixed-size</td><td>42.9</td><td>44.8</td><td>61.2</td><td>64.8</td><td>35.9</td><td>40.3</td><td>33.8</td><td>38.3</td><td>51.8</td><td>55.0</td><td>47.5</td><td>49.7</td><td>45.5</td><td>48.8</td>
  </tr>
  <tr>
    <td>Semantic</td><td>42.5</td><td>44.5</td><td>61.7</td><td>63.5</td><td>36.8</td><td>40.4</td><td>28.3</td><td>32.2</td><td>53.7</td><td>55.8</td><td>47.8</td><td>48.7</td><td>45.1</td><td>47.5</td>
  </tr>
  <tr>
    <td>Proposition</td><td>42.5</td><td>44.3</td><td>61.5</td><td>64.6</td><td>36.9</td><td>41.0</td><td>42.7</td><td><b>45.4</b></td><td>52.7</td><td>54.6</td><td>51.2</td><td>51.1</td><td>47.9</td><td>50.2</td>
  </tr>
  <tr>
    <td>PIC</td><td><b>45.7</b></td><td><b>48.0</b></td><td><b>63.3</b></td><td><b>66.1</b></td><td><b>40.1</b></td><td><b>42.7</b></td><td>39.4</td><td>43.0</td><td><b>58.7</b></td><td><b>61.5</b></td><td><b>52.2</b></td><td><b>53.8</b></td><td><b>49.9</b></td><td><b>52.5</b></td>
  </tr>
  <tr>
    <td colspan="26">Meta-Llama-3-8B-Instruct</td>
  </tr>
  <tr>
    <td>Sentence</td><td>32.8</td><td>37.0</td><td>59.1</td><td>63.0</td><td>30.6</td><td>35.5</td><td>35.3</td><td>40.5</td><td>42.9</td><td>47.5</td><td>38.8</td><td>43.9</td><td>39.9</td><td>44.6</td>
  </tr>
  <tr>
    <td>Paragraph</td><td>40.1</td><td>42.4</td><td>63.0</td><td>65.9</td><td>36.4</td><td>38.9</td><td>36.8</td><td>41.3</td><td>50.3</td><td>53.8</td><td>44.5</td><td>49.0</td><td>45.2</td><td>48.5</td>
  </tr>
  <tr>
    <td>Fixed-size</td><td>44.6</td><td>46.1</td><td>63.2</td><td>67.0</td><td>36.0</td><td>39.2</td><td>34.6</td><td>39.8</td><td>52.4</td><td>55.4</td><td>50.7</td><td>52.2</td><td>46.9</td><td>50.0</td>
  </tr>
  <tr>
    <td>Semantic</td><td>43.9</td><td>46.0</td><td>63.3</td><td>65.4</td><td>37.1</td><td>40.4</td><td>29.2</td><td>33.6</td><td>54.7</td><td>56.1</td><td>50.9</td><td>51.8</td><td>46.5</td><td>48.9</td>
  </tr>
  <tr>
    <td>Proposition</td><td>44.4</td><td>46.7</td><td>63.9</td><td>67.1</td><td>38.0</td><td>40.7</td><td><b>44.8</b></td><td><b>47.8</b></td><td>55.1</td><td>56.3</td><td>54.5</td><td>55.1</td><td>50.1</td><td>52.3</td>
  </tr>
  <tr>
    <td>PIC</td><td><b>45.6</b></td><td><b>48.3</b></td><td><b>64.6</b></td><td><b>67.2</b></td><td><b>39.4</b></td><td><b>41.5</b></td><td>40.5</td><td>44.2</td><td><b>58.8</b></td><td><b>61.5</b></td><td><b>56.9</b></td><td><b>58.5</b></td><td><b>51.0</b></td><td><b>53.5</b></td>
  </tr>
</table>

Table 2: End-to-end question-answering performance across different document segmentation methods. We present results with two different LLMs as the generation models, evaluated by Exact Match (EM).

potential reason for this gap is that SQuAD is more akin to a reading comprehension dataset, with 100k questions derived from only 536 documents. This makes it difficult for a retriever trained on general data to capture the relationship between instructions and chunks. In contrast, Proposition eliminates this bias by rewriting the original text, leading to improved performance. Similar phenomenon is also observed by Lee et al., 2019.

5.2 End-to-end QA Performance

Similar conclusions can be drawn regarding QA performance. As shown in Table 2, experiments with two different LLMs demonstrate that PIC consistently achieves the best results. When using Qwen as the LLM, PIC shows an average improvement of 2 points across six datasets, with a remarkable 5.7-point increase on the EQ dataset. When using Llama-3, PIC also demonstrates an average improvement of nearly 1 point.

It can be observed that the accuracy of the QA task is significantly lower than that of retrieval, indicating that sometimes, even when the correct relevant text is retrieved, the model still fails to provide the correct answer. This may be due to the retrieved text containing too much irrelevant information, making it difficult for the LLM to effectively capture the answer. However, among all methods, our approach exhibits the smallest performance drop. Taking the results on Qwen as an example, under the top-5 setting, our method decreases by 8.5 points (58.4 → 49.9), while the proposition method decreases by 9.8 points (57.7 → 47.9) and the Semantic method by 10.9 points (56.0 → 45.1). We attribute this to the reduced irrelevant information in the chunks, making the LLM less prone to hallucinations.

6 Further Analysis

6.1 Impacts of the Pseudo-Instruction

The core idea of our method is to construct pseudo-instruction (PI) to guide chunking, ensuring that the segmented chunks better align with the possible real instruction distribution. In our implementation, we choose summary as the PI because it encapsulates the main information of the document, allowing the document to be segmented based on its relevance to the central theme. To better understand the importance of PI, we conduct an ablation study by modifying its implementation. Specifically, we implement the following variations of PI:
1) Random: For each document, we use the summary of a randomly selected different document as its PI instead of its own summary.
2) Sent: For each document, we use a randomly chosen sentence from the document as its PI.
3) Mean: For each document, we compute the mean value of all the sentence embeddings and use it as the PI embedding for chunking.

We follow the experimental setup described in Section 4 and evaluate the retrieval and QA performance under different PI implementations. The results are presented in Table 3. It can be seen that using summaries as the PI achieves the best performance. PIC-Random performs the worst, as
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">NQ</th>
    <th colspan="4">TriviaQA</th>
    <th colspan="4">WebQ</th>
    <th colspan="4">SQuAD</th>
    <th colspan="4">EQ</th>
    <th colspan="4">PopQA</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th><th>Top5</th><th>Top20</th>
    <th>Top5</th><th>Top20</th>
  </tr>
  <tr>
    <td colspan="26">Retrieval Performance (Hits@k)</td>
  </tr>
  <tr>
    <td>PIC</td><td>61.0</td><td>71.6</td><td>42.3</td><td>49.2</td><td>54.5</td><td>66.9</td><td>53.0</td><td>67.0</td><td>71.1</td><td>80.3</td><td>68.7</td><td>81.9</td><td>58.4</td><td>69.5</td>
  </tr>
  <tr>
    <td>Random</td><td>56.6</td><td>68.8</td><td>38.7</td><td>45.5</td><td>50.2</td><td>62.5</td><td>49.1</td><td>62.4</td><td>67.5</td><td>76.6</td><td>65.4</td><td>78.6</td><td>54.6</td><td>65.7</td>
  </tr>
  <tr>
    <td>Sent</td><td>56.8</td><td>69.1</td><td>38.7</td><td>45.2</td><td>50.8</td><td>63.4</td><td>49.2</td><td>62.2</td><td>67.9</td><td>76.8</td><td>66.3</td><td>78.7</td><td>54.9</td><td>65.9</td>
  </tr>
  <tr>
    <td>Mean</td><td>57.7</td><td>69.4</td><td>39.6</td><td>46.5</td><td>51.7</td><td>63.9</td><td>50.3</td><td>64.3</td><td>68.4</td><td>78.0</td><td>67.3</td><td>80.0</td><td>55.8</td><td>67.0</td>
  </tr>
  <tr>
    <td colspan="26">Question-Answering Performance (EM)</td>
  </tr>
  <tr>
    <td>PIC</td><td>45.7</td><td>48.0</td><td>63.3</td><td>66.1</td><td>40.1</td><td>42.7</td><td>39.4</td><td>43.0</td><td>58.7</td><td>61.5</td><td>52.2</td><td>53.8</td><td>49.9</td><td>52.5</td>
  </tr>
  <tr>
    <td>Random</td><td>39.6</td><td>43.1</td><td>57.3</td><td>60.5</td><td>35.1</td><td>38.0</td><td>34.6</td><td>38.4</td><td>53.6</td><td>55.6</td><td>48.8</td><td>50.6</td><td>44.8</td><td>47.7</td>
  </tr>
  <tr>
    <td>Sent</td><td>40.6</td><td>43.0</td><td>58.3</td><td>60.9</td><td>34.7</td><td>38.4</td><td>35.4</td><td>38.4</td><td>54.0</td><td>57.4</td><td>49.4</td><td>51.4</td><td>45.4</td><td>48.3</td>
  </tr>
  <tr>
    <td>Mean</td><td>40.8</td><td>43.2</td><td>59.5</td><td>62.4</td><td>35.9</td><td>40.8</td><td>36.3</td><td>40.1</td><td>54.8</td><td>58.5</td><td>50.7</td><td>51.8</td><td>46.3</td><td>49.5</td>
  </tr>
</table>

Table 3: Performance comparison of different pseudo-instruction implementations. For question-answering performance evaluation, we adopt Qwen2.5-7B-Instruct as the backbone model.

the random-selected summary does not contain any information from the document itself, resulting in chunks with little semantic coherence. PIC-Sent contains document-specific information, but its information capacity is too limited, and the selection of a single sentence is highly random, leading to suboptimal chunking. PIC-Mean performs relatively well, likely because it integrates information from all sentences in the document. However, averaging embeddings weakens its ability to capture semantic structures, making it less effective than the summary-based approach.

6.2 Impacts of Chunking Threshold

In Section 3, our PIC algorithm selects the mean similarity between sentences and the summary as a threshold for chunking. However, this threshold requires further consideration. Our method aims to distinguish between sentences that are relevant to the pseudo-instruction and those that are not. A higher threshold results in shorter and fewer relevant chunks, and vice versa.

To explore this, we select five different thresholds ranging from \( \mu - 2\sigma \) to \( \mu + 2\sigma \) and conduct experiments under the same setting as Section 4. The results are shown in the figure 2. Here, \( \mu \) represents the mean, and \( \sigma \) represents the standard deviation.

The results indicate that setting the threshold at \( \mu \) yields the best retrieval and QA performance. As the threshold deviates in either direction (becoming too high or too low), performance gradually declines. We attribute this to the increasing mixture of relevant and irrelevant sentences within chunks as the threshold shifts. This reduces the distinction between chunks and, in extreme cases (when the threshold approaches 0 or 1), degrades the process into no chunking at all, ultimately leading to performance deterioration.

6.3 Real-Instruction for Document Chunking

<table>
  <tr>
    <th>Method</th>
    <th>NQ</th>
    <th>Qasper</th>
  </tr>
  <tr>
    <td>Semantic</td><td>37.8</td><td>14.9</td>
  </tr>
  <tr>
    <td>Proposition</td><td>38.3</td><td>14.8</td>
  </tr>
  <tr>
    <td>PIC</td><td>38.8</td><td>15.6</td>
  </tr>
  <tr>
    <td>RIC</td><td>39.2</td><td>16.2</td>
  </tr>
</table>

Table 4: Performance comparison of Real-Instruction Chunking with other methods. We retrieve the top-5 chunks, and evaluate the QA performance using Qwen-2.5-7B-Instruct. We use Exatch Match for evaluation on NQ, and F1 for Qasper.

When constructing chunks, we typically lack access to the actual user instruction distribution. Therefore, we propose the PIC method, using pseudo-instructions as an approximation of real user instructions. But if the actual instructions are known in advance and used for chunking, it may yield optimal results. To validate this hypothesis, we conducted experiments on the NaturalQuestions (Kwiatkowski et al., 2019) and Qasper (Dasigi et al., 2021) datasets. The NaturalQuestions (NQ) dataset comprises real anonymized queries issued to the Google search engine, paired with corresponding Wikipedia pages. The Qasper dataset consists of human-written QA pairs based on NLP papers. For each dataset, we use the documents it provides as the corpus, instead of Wikipedia. For the Real-Instruction Chunking (RIC), we use the query of each document to calculate similarities with the sentences. The results are shown in Table 4. It can be observed
Figure 2: The influence of different chunking thresholds. Figure (a) represents the average end-to-end QA performance, and figure (b) represents the average retrieval performance.

<table>
  <tr>
    <th>Dataset</th>
    <th>Top-5</th>
    <th>Top-20</th>
    <th>Golden</th>
  </tr>
  <tr>
    <td>NQ</td>
    <td>70.1</td>
    <td>65.7</td>
    <td>66.7</td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>73.1</td>
    <td>67.7</td>
    <td>68.4</td>
  </tr>
  <tr>
    <td>WebQ</td>
    <td>77.0</td>
    <td>72.7</td>
    <td>66.7</td>
  </tr>
  <tr>
    <td>SQuAD</td>
    <td>60.7</td>
    <td>57.9</td>
    <td>65.8</td>
  </tr>
  <tr>
    <td>EQ</td>
    <td>67.2</td>
    <td>63.7</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>PopQA</td>
    <td>64.3</td>
    <td>61.5</td>
    <td>91.0</td>
  </tr>
</table>

Table 5: Proportion of relevant chunks in retrieved PIC Chunks. Top-5 and Top-20 refer to the top 5 and top 20 retrieved chunks, respectively, and Golden represents the retrieved chunks that contain the correct answer.

Figure 3: Length Distribution of Different Chunking Methods.

that chunking with real instructions yields the optimal results, which aligns with the viewpoint we presented in Section 1.

6.4 Chunk Analysis

Distribution of Chunk Lengths. Different chunking methods yield varying length distributions, which may influence chunking effectiveness. To explore this, we visualized the length distribution for each method, as shown in Figure 3, with detailed statistics provided in Appendix A. The overall distribution patterns suggest that chunk length itself does not directly correlate with retrieval or QA performance in RAG. Instead, the primary factor affecting downstream performance is how the text is structured within each chunk. Notably, our proposed PIC method maintains a relatively shorter average chunk length, which helps mitigate the input length burden on the LLM.

Distribution of Retrieved PIC Chunks. The relevant chunks (\( C_{rel} \)) in PIC are expected to contain more key information from the document, making them easier to be retrieved. To evaluate this, we calculate the proportion of relevant chunks retrieved by the PIC method across different datasets, as shown in Table 5. The results indicate that in all cases, relevant chunks account for more than 50% of the retrieved chunks, demonstrating the effectiveness of the PIC method in categorizing chunks and making topic-related chunks more retrievable. Moreover, on the EQ and PopQA datasets, nearly all retrieved golden chunks are relevant chunks. We attribute this to the entity-centric nature of the datasets, which focus on less popular, long-tail entities. In this scenario, the PIC-relevant chunks overlap significantly with user instructions, which further reinforces our conclusion.

7 Conclusions

In this paper, we identify document segmentation as a critical yet underexplored challenge in retrieval-augmented generation (RAG) systems. We propose PIC, a novel document chunking method that leverages document summaries as
pseudo-instructions to guide semantic chunking. By dynamically grouping sentences based on their relevance to document topic, PIC creates chunks that better align with potential user instructions while maintaining semantic coherence. Extensive experiments across six QA benchmarks demonstrate that our method achieves significant improvements in both retrieval accuracy and end-to-end QA performance compared to existing approaches, without requiring additional model retraining.

Limitations

Our work focuses on creating chunks that better align with potential user instructions while maintaining semantic coherence. One limitation of our approach is the computational overhead introduced during summary generation. While this overhead is significantly lower than methods requiring multiple LLM calls per document, future work should explore more efficient strategies for generating high-quality pseudo-instructions. Additionally, our current evaluation is limited to general-domain tasks, leaving the exploration of specialized scenarios requiring cross-document reasoning for future investigation.

Acknowledgements

This work is supported by the National Science and Technology Major Project (2020AAA0106502), the National Natural Science Foundation of China (No. 62236011) and a grant from the Guoqiang Institute, Tsinghua University.

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Preprint, arXiv:2310.11511.

Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 615–620. Doha, Qatar. Association for Computational Linguistics.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milligan, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR.

Tong Chen, Hongwei Wang, Sihaio Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. 2024. Dense X retrieval: What retrieval granularity should we use? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15159–15177, Miami, Florida, USA. Association for Computational Linguistics.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics.

André V. Duarte, João DS Marques, Miguel Graça, Miguel Freire, Lei Li, and Arlindo L. Oliveira. 2024. LumberChunker: Long-form narrative document segmentation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6473–6486, Miami, Florida, USA. Association for Computational Linguistics.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.

Aaron Grattafiore et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.

Kamradt Greg. 2024. semantic chunking.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.

Yufei Huang, Xu Han, and Maosong Sun. 2024. Fast-FID: Improve inference efficiency of open domain question answering via sentence selection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6262–6276, Bangkok, Thailand. Association for Computational Linguistics.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane
Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models. Preprint, arXiv:2208.03299.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.

Kush Juvekar and Anupam Purwar. 2024. Cos-mix: Cosine similarity and distance fusion for improved information retrieval. Preprint, arXiv:2406.00638.

Vladimir Karpukhin, Barlas Ouzug, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Online. Association for Computational Linguistics.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466.

Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023. Copy is all you need. Preprint, arXiv:2307.06962.

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.

Jerry Liu. 2022. LlamaIndex.

Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2023. Interpretable long-form legal question answering with retrieval-augmented large language models. Preprint, arXiv:2309.17050.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.

OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. OpenAI.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.

Stephen E. Robertson, Susan Walker, Steve Jones, Micheline M. Hancock-Beaulieu, and Mike Gatford. 1995. Okapi at trec-3. Proceedings of the Third Text REtrieval Conference (TREC-3), pages 109–126.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.

Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval.

Kunal Sawarkar, Abhilasha Mangal, and Shivam Raj Solanki. 2024. Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), pages 155–161.

Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Empirical Methods in Natural Language Processing (EMNLP).

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. REPLUG: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8371–8384, Mexico City, Mexico. Association for Computational Linguistics.

Shuzheng Si, Haozhe Zhao, Gang Chen, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Kaikai An, Kangyang Luo, Chen Qian, Fanchao Qi, Baobao Chang, and Maosong Sun. 2025a. Aligning large language models to follow instructions and hallucinate less via effective data filtering. Preprint, arXiv:2502.07340.

Shuzheng Si, Haozhe Zhao, Cheng Gao, Yuzhuo Bai, Zhitong Wang, Bofei Gao, Kangyang Luo, Wenhao
Li, Yufei Huang, Gang Chen, Fanchao Qi, Minjia Zhang, Baobao Chang, and Maosong Sun. 2025b. Teaching large language models to maintain contextual faithfulness via synthetic tasks and reinforcement learning. Preprint, arXiv:2505.16483.

Qwen Team. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115.

Junde Wu, Jiayuan Zhu, Yunli Qi, Jingkun Chen, Min Xu, Filippo Menolascina, and Vicente Grau. 2024. Medical graph rag: Towards safe medical large language model via graph retrieval-augmented generation.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. Preprint, arXiv:2309.07597.

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. Preprint, arXiv:2310.04408.

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation.

Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. Preprint, arXiv:2407.02485.

Boyu Zhang, Hongyang Yang, Tianyu Zhou, Muhammad Ali Babar, and Xiao-Yang Liu. 2023. Enhancing financial sentiment analysis via retrieval augmented large language models. In Proceedings of the Fourth ACM International Conference on AI in Finance, ICAIF ’23, page 349–356, New York, NY, USA. Association for Computing Machinery.

Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. 2024. Raft: Adapting language model to domain specific rag. Preprint, arXiv:2403.10131.

Jihao Zhao, Zhiyuan Ji, Yuchen Feng, Pengnian Qi, Simin Niu, Bo Tang, Feiyu Xiong, and Zhiyu Li. 2024. Meta-chunking: Learning efficient text segmentation via logical perception.
Appendix

A  Statistics of Chunk Lengths

<table>
  <tr>
    <th></th>
    <th>Mean</th>
    <th>Std</th>
  </tr>
  <tr>
    <td>Sentence</td>
    <td>18.8</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>Paragraph</td>
    <td>42.2</td>
    <td>48.0</td>
  </tr>
  <tr>
    <td>Fixed-size</td>
    <td>87.8</td>
    <td>26.6</td>
  </tr>
  <tr>
    <td>Semantic</td>
    <td>78.6</td>
    <td>88.9</td>
  </tr>
  <tr>
    <td>Proposition</td>
    <td>58.1</td>
    <td>32.0</td>
  </tr>
  <tr>
    <td>PIC</td>
    <td>42.8</td>
    <td>38.3</td>
  </tr>
</table>

Table 6: Length Statistics of Different Chunking Methods, counted by words.

The detailed statistics of the length of the chunks under different aggregation methods are shown in Table 6.

B  Case Study

We demonstrate the difference of our chunking method with semantic chunking in Table 9. We select a query "where was the statue of liberty originally built" from the NQ dataset with its answer as "France". To retrieve all the relevant information, the retrieved chunk must both include "built in France" and "the Statue of Liberty", in sentence 3 and 5, respectively. However, as we can see from the table, the semantic chunking method fails to group sentence 3 and sentence 5 in one chunk. Instead, it splits sentence 3 and 4 due to the relatively low similarity between them (In the table, \( sim_{next} \) represents the similarity of one sentence with the next sentence). This results in a low similarity (\( sim_{red-chunk-with-query} \) in the table) between the query and the chunk of sentence 3, where the answer "in France" lies, and leads to a retrieval failure. The PIC method treats sentence 3 to 5 as relevant sentences, and groups them into one chunk based on \( sim_{pi} \), which represents the similarity between the pseudo-instruction (PI) and the sentence. This chunk, with both "built in France" and "the Statue of Liberty" in it, has a higher similarity score with the query (\( r_{green-chunk-with-query} \)), and can be easily retrieved by the retriever.

C  Prompt For Summary

The prompt we use for writing summary for each document is shown in Table 7.

Summarizer Prompt

Read this passage:
{context}
Write a summary for this passage. Your summary should be concise and informative. Give the summary directly, without any other text.

Table 7: Prompt used when generating summaries for wiki documents.

RAG Prompt

Refer to the passages below and answer the following question with just a few words.
{chunk 1}
{chunk 2}
...
Refer to the context above and answer the following question with just a few words.
Question: {Question}
The answer is

Table 8: Prompt used when concatenating queries with retrieved chunks.

D  Prompt For Question Answering

The prompt we use for generating answers in QA tasks is shown in Table 8.
Query: where was the statue of liberty originally built?
Answer: France.
Semantic:
...
Fundraising proved difficult, especially for the Americans, and by 1885 work on the pedestal was threatened by lack of funds. (\( sim_{next} = 0.566 \))
Publisher Joseph Pulitzer, of the "New York World", started a drive for donations to finish the project and attracted more than 120,000 contributors, most of whom gave less than a dollar. (\( sim_{next} = 0.4314 \), cut!)
The statue was built in France, shipped overseas in crates, and assembled on the completed pedestal on what was then called Bedloe’s Island. (\( sim_{next} = 0.4958 \), cut!)
The statue’s completion was marked by New York’s first ticker-tape parade and a dedication ceremony presided over by President Grover Cleveland. (\( sim_{next} = 0.538 \))
The statue was administered by the United States Lighthouse Board until 1901 and then by the Department of War; since 1933, it has been maintained by the National Park Service as part of the Statue of Liberty National Monument, and is a major tourist attraction.
...
\( r_{red-chunk-with-query}=0.5713 \)
PIC:
...
Fundraising proved difficult, especially for the Americans, and by 1885 work on the pedestal was threatened by lack of funds. (\( sim_{pi} = 0.493 \), irrelevant)
Publisher Joseph Pulitzer, of the "New York World", started a drive for donations to finish the project and attracted more than 120,000 contributors, most of whom gave less than a dollar. (\( sim_{pi} = 0.445 \), irrelevant)
The statue was built in France, shipped overseas in crates, and assembled on the completed pedestal on what was then called Bedloe’s Island. (\( sim_{pi} = 0.572 \), relevant)
The statue’s completion was marked by New York’s first ticker-tape parade and a dedication ceremony presided over by President Grover Cleveland. (\( sim_{pi} = 0.557 \), relevant)
The statue was administered by the United States Lighthouse Board until 1901 and then by the Department of War; since 1933, it has been maintained by the National Park Service as part of the Statue of Liberty National Monument, and is a major tourist attraction. (\( sim_{pi} = 0.703 \), relevant)
...
\( r_{green-chunk-with-query}=0.73 \)
Pseudo-Instruction: The Statue of Liberty, a colossal neoclassical sculpture designed by Frrric Auguste Bartholdi with a metal framework by Gustave Eiffel, was dedicated in New York Harbor on October 28, 1886, as a gift from France to the United States. Inspired by the Roman goddess Libertas, the statue symbolizes freedom and welcome to immigrants. Conceived in 1865 by French historian douard de Laboulaye, its construction was delayed by the Franco-Prussian War until fundraising efforts resumed in the 1870s. Joseph Pulitzer’s 1885 fundraising campaign in the U.S. attracted over 120,000 donors, enabling the pedestal’s completion. Assembled on Liberty Island, the statue oxidized to its green color within twenty years and became an iconic landmark associated with immigration, especially linked to nearby Ellis Island. Managed initially by the Lighthouse Board and the Department of War, it has been maintained by the National Park Service since 1933. The statue has undergone several restorations, notably for its centennial in 1986, and has been closed and reopened multiple times due to events like the September 11 attacks and Hurricane Sandy.

Reviewer paper summaries: This paper investigates the often overlooked problem of document segmentation (chunking) in retrieval-augmented generation (RAG). The authors argue that how documents are split into chunks can significantly affect downstream retrieval and QA performance. They propose a new chunking method called PIC (Pseudo-Instruction for document Chunking), which uses an LLM-generated summary of a document as a “pseudo-instruction” to guide segmentation. By computing the similarity of each sentence to the summary, PIC groups sentences into semantically coherent chunks that align with the document’s main themes and filter out irrelevant content. The paper introduces PICWiki, a segmented Wikipedia corpus created with this method, and demonstrates through extensive experiments on multiple open-domain QA benchmarks (NQ, TriviaQA, WebQuestions, SQuAD, EQ, PopQA) that PIC yields improved retrieval accuracy (Hits@5, Hits@20) and end-to-end QA performance (Exact Match) compared to standard fixed-size, sentence, paragraph, and other semantic chunking baselines. The approach is simple and requires no retriever or generator retraining, yet achieves consistent gains, highlighting that document segmentation indeed matters for RAG.
Summary of strengths: 1. Novel Focus on Chunking Problem: The paper identifies an important but underexplored challenge in RAG – how to optimally segment documents. This focus is novel, as most prior work focused on retrievers or generators and used simplistic chunking. Highlighting and tackling chunking as a first-class problem is a valuable contribution.

2. Simple and Effective Method (PIC): The proposed PIC method is conceptually simple (using a summary as a proxy for likely queries) yet effective. It cleverly leverages an LLM to create a pseudo-instruction (summary) and uses it to form semantically coherent chunks. This approach adapts chunk boundaries to content meaning, which is more principled than fixed-length or purely local heuristics. The method requires no additional training and can be applied in a plug-and-play manner.

3. Strong Empirical Results: The paper presents extensive experiments across six QA benchmarks, showing that PIC outperforms multiple baseline chunking strategies in both retrieval and QA metrics. For example, PIC achieves the highest average Hits@5/20 retrieval accuracy across all datasets, outperforming fixed-size, sentence-level, and a recent proposition-based segmentation baseline. End-to-end QA exact-match scores also improve (on average +2 EM with one LLM and +1 EM with another), indicating the improved chunks lead to better answers. These consistent gains substantiate the claim that better segmentation improves RAG performance.

4. Thorough Analysis and Ablations: The authors conduct insightful analyses to strengthen the work. They compare different pseudo-instruction choices (actual summary vs random summary vs single sentence vs mean embedding) in Table 3, confirming that using the document’s own summary yields the best performance (other choices degrade results). They also vary the chunking threshold (Figure 2) and show that their choice of using the mean similarity as threshold is optimal. Additionally, the paper examines chunk length distributions (Figure 3) and the proportion of relevant chunks retrieved (Table 5), providing a deep understanding of why PIC works (it creates shorter, on-topic chunks that are easier to retrieve).

5. Resource Contribution (PICWiki): The paper introduces PICWiki, a large-scale dataset of Wikipedia articles segmented by the PIC method (including each article’s original text, generated summary, and resulting chunks). This is a useful artifact for the community – it can serve as a ready-to-use knowledge base for RAG and facilitate further research on chunking and retrieval. Sharing this dataset (and presumably code) enhances the work’s impact and reproducibility.

6. Clarity and Organization: Overall, the paper is clearly written and well-structured. The problem is well-motivated in the introduction, related work properly contextualizes how prior chunking methods operate (e.g. fixed, semantic, proposition-based), and the methodology is explained with an illustrative example (Figure 1) that makes the chunking procedure easy to understand. The experiments are organized logically (retrieval results, QA results, analyses), and the authors include a limitations section and appendix for additional details, indicating a thorough presentation.
Summary of weaknesses: 1. Methodological Novelty: While effective, the PIC method is relatively straightforward – it combines known ideas of using a summary (essentially an LLM-generated gist of the document) and clustering sentences by similarity. Some might view this as an incremental improvement rather than a fundamentally new algorithm. The approach leverages an LLM for summarization and a fixed embedding model for similarity; these tools exist, so the novelty is mainly in the specific application to chunking. The contribution is more in pipeline design than in developing new theory or models.

2. Computational Overhead: The approach requires generating a summary for each document using an LLM, which could be expensive at scale. The paper does acknowledge this in its limitations – PIC’s summarization step is an overhead compared to naive chunking. Although they note it’s cheaper than more exhaustive LLM-based chunking methods (which might call an LLM for every segment), it still might be non-trivial if the knowledge corpus is very large. The impact on preprocessing time and cost for a large corpus (e.g., the whole of Wikipedia or enterprise documents) could be a concern for practical deployment.

3. Dependence on Summary Quality: The success of PIC hinges significantly on the quality of generated summaries. The paper does not discuss potential robustness issues arising from poor-quality summaries or edge cases involving complex documents.

4. Scope of Evaluation: The experiments focus on open-domain QA tasks where queries typically align with a document’s main content. It’s less clear how PIC would perform in scenarios where user queries target more obscure or fine-grained details that might not be emphasized in the document summary. In such cases, PIC’s summary-guided chunks might inadvertently de-prioritize or isolate those details. The current evaluation did not include multi-hop QA or specialized domain tasks (the authors leave cross-document reasoning tasks for future work). Thus, the generality of PIC beyond the evaluated benchmarks remains to be proven. For instance, on SQuAD (which has a different distribution of questions), PIC’s improvement was minimal or slightly worse than a proposition-based method, indicating it may not universally dominate all chunking strategies.

5. Limited Analysis of Failure Cases: While the paper provides an explanation for the weaker performance on SQuAD (attributing it to the nature of the dataset and retriever bias), the analysis of when PIC might fail could be expanded. For example, if an LLM summary is low-quality or contains hallucinations, how robust is the chunking? The paper does not discuss summary errors or variability. Also, a brief discussion of queries that might not benefit from summary-based chunking (e.g., queries on very minor points) would strengthen understanding of the method’s limits. These aspects are touched upon but could use deeper analysis.

6. Clarity Issues: There are a few clarity issues. The description of the baseline methods is sometimes terse – e.g., “Semantic” and “Proposition” chunking are referenced, and while related work covers them at a high level, some readers might appreciate a bit more detail or an example of how those baselines work (though the footnotes and citations help). Also, the name “GPT-4o-mini” used for the summarization model is a bit unclear – presumably an anonymized reference to an OpenAI model – but readers might not know what this corresponds to.
Comments, suggestions and typos: 1. Clarify Summarization Details: It would be helpful to provide more details on the pseudo-instruction generation process. For example, specifying the exact prompt used to obtain the summary (the prompt is said to be in Appendix C) and a brief mention of the summary length or style (one sentence vs. few sentences) would give readers a better sense of the pseudo-instructions’ nature. Also, clarifying what “GPT-4o-mini (OpenAI, 2024)” refers to (if possible, use the real model name or a description) would avoid confusion.

2. Discuss Query Types: The paper could discuss how PIC handles queries that focus on minor details not covered well in the summary. Since PIC optimizes chunks for the document’s main themes, one might wonder if information not reflected in those themes becomes less retrievable. The authors could add a short discussion or analysis of this scenario (perhaps in the limitations or experiments). For instance, were there cases where a baseline chunking retrieved an obscure fact better than PIC because PIC had isolated it as “irrelevant”? This would highlight when one might not want summary-based chunking.

3. Multi-hop or Cross-Document Scenarios: As future work (or even in analysis), the authors mention exploring tasks requiring cross-document reasoning. It would strengthen the paper to hypothesize how PIC might perform in multi-hop QA or retrieval of information spread across documents. Would using summaries per document still help, or might it merge chunks that prevent retrieving pieces from separate docs? Even if an experiment is out of scope, a discussion could be insightful for readers interested in applying PIC to such tasks.

4. Efficiency Improvements: Since the main limitation is the cost of generating summaries, the authors could discuss possible optimizations. For example, could a smaller fine-tuned model generate adequate pseudo-instructions instead of a large LLM? Or could chunking be done iteratively in a coarse-to-fine manner to reduce the number of LLM calls? Any thoughts here would be useful for practitioners who may not have access to powerful APIs for millions of documents. This doesn’t need new experiments, just a commentary on feasible optimizations.

5. Baseline Tuning: One suggestion for completeness: ensure that all baseline chunking methods are as strong as possible. For example, the semantic baseline uses a fixed 20th-percentile similarity threshold – the paper notes this was chosen as best among a few values. It might be worth confirming in the appendix if each baseline’s parameters (like fixed chunk size, etc.) were tuned for fairness. If not already done, authors could clarify this to preempt any criticism that PIC benefited from under-tuned baselines.

6. Typos/Grammar: One wording issue: “we introduce a knowledge corpora PICWiki” – here “corpora” should likely be singular (“corpus”) since PICWiki itself is one collection. Also, check for any minor formatting issues (in the PDF text, “a simple yet e!ective method” looked like a small encoding issue for “effective” ).
Overall assessment scores: 3