Grounding Language Model with Chunking-Free In-Context Retrieval (2024)

Abstract: This paper presents a novel Chunking-Free InContext (CFIC) retrieval approach, specifically tailored for Retrieval-Augmented Generation (RAG) systems. Traditional RAG systems often struggle with grounding responses using precise evidence text due to the challenges of processing lengthy documents and filtering out irrelevant content. Commonly employed solutions, such as document chunking and adapting language models to handle longer contexts, have their limitations. These methods either disrupt the semantic coherence of the text or fail to effectively address the issues of noise and inaccuracy in evidence retrieval. CFIC addresses these challenges by circumventing the conventional chunking process. It utilizes the encoded hidden states of documents for in-context retrieval, employing autoaggressive decoding to accurately identify the specific evidence text required for user queries, eliminating the need for chunking. CFIC is further enhanced by incorporating two decoding strategies, namely Constrained Sentence Prefix Decoding and Skip Decoding. These strategies not only improve the efficiency of the retrieval process but also ensure that the fidelity of the generated grounding text evidence is maintained. Our evaluations of CFIC on a range of open QA datasets demonstrate its superiority in retrieving relevant and accurate evidence, offering a significant improvement over traditional methods. By doing away with the need for document chunking, CFIC presents a more streamlined, effective, and efficient retrieval solution, making it a valuable advancement in the field of RAG systems. The codes will be released in this repository.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1298–1311 August 11-16, 2024 ©2024 Association for Computational Linguistics
CFIC addresses these challenges by circumventing the conventional chunking process. It utilizes the encoded hidden states of documents for in-context retrieval, employing autoaggressive decoding to accurately identify the specific evidence text required for user queries, eliminating the need for chunking. CFIC is further enhanced by incorporating two decoding strategies, namely Constrained Sentence Prefix Decoding and Skip Decoding. These strategies not only improve the efficiency of the retrieval process but also ensure that the fidelity of the generated grounding text evidence is maintained. Our evaluations of CFIC on a range of open QA datasets demonstrate its superiority in retrieving relevant and accurate evidence, offering a significant improvement over traditional methods. By doing away with the need for document chunking, CFIC presents a more streamlined, effective, and efficient retrieval solution, making it a valuable advancement in the field of RAG systems. The codes will be released in this repository.

heading: 1 Introduction
text: Recently, retrieval-augmented generation (RAG) has marked a significant advancement in the field
∗Corresponding author.
of natural language processing (NLP). This technique has demonstrated remarkable effectiveness in reducing hallucination in text generation (Ji et al., 2023), particularly in knowledge-intensive tasks like open-domain question answering (Wang et al., 2019; Lewis et al., 2020; Shuster et al., 2021; Komeili et al., 2022). An RAG system typically consists of two components: the retriever and the generator. Given an input query, the retriever first identifies relevant evidence text, upon which the generator then generates the answer.
The generator’s output should be grounded by precise evidence text obtained by the retriever. However, this poses challenges for most retrieval systems, as they often retrieve lengthy documents such as web pages. In practice, we only need specific grounding text from these documents to help answer user queries. Using lengthy documents directly in the RAG system presents two difficulties. First, generation models may struggle to handle the extensive length of these documents. Second, irrelevant or distracting content within the documents can lead the model astray from the main query, resulting in inaccurate response generation (Gao et al., 2024).
To address this issue, common approaches involve chunking documents into smaller passages and employing strategies like reranking for relevance (Nogueira and Cho, 2020; Mao et al., 2021; Gao et al., 2024), or selecting passages based on other measurements (Asai et al., 2022; Jiang et al., 2023). However, the chunking process is often suboptimal, as determining the granularity of the passage chunking is challenging. Improper chunking can disrupt the semantics and result in incomplete and incoherent retrieved information (Dong et al., 2023). Another method involves adapting large language models (LLMs) to process longer contexts by training them on long contexts or implementing a sliding context window (Ratner et al., 2022; Chen et al., 2023). While these methods enable LLMs
1298
Search Engine
LLMChunking
Sub-Optimal Evidence
Semantic Cut-off
Long Grounding Article
Input Query What significant change occurred in Bach's style in the last decade of his life?
Decoding
Chunking-Based Chunking-Free
Optimal Evidence
Hidden States
Evidence Selection
Figure 1: Comparison of Chunking-Based and Chunking-Free Methods. The left panel illustrates the chunking-based method, involving chunking a lengthy document into smaller passages followed by refinement through passage ranking. The right panel depicts the chunking-free method proposed in this paper, where grounding text is directly decoded by LLMs without the need for document chunking.
to handle longer texts, they do not fully address the issue of noise in the lengthy documents and cannot output the grounding text for the generated response (Kaddour et al., 2023).
In this paper, we propose a Chunking-Free InContext (CFIC) retrieval approach aimed at helping the RAG system mitigate information bias introduced by document chunking and irrelevant noisy text. Specifically, given an input query and a long grounding document, instead of refining the long documents with a chunking-based method, we leverage the document’s encoded hidden states to perform Chunking-free In-Context Retrieval. It circumvents the traditional chunking process, allowing the retrieval system to auto-aggressively decode and pinpoint the precise evidence text to ground the response generation to a query. Figure 1 shows the comparison between the chunking-based method and the chunking-free method for grounding text retrieval. The chunking-free method demonstrates a superior ability to identify optimal evidence text, as it considers the entire document for a comprehensive perspective.
Concretely, CFIC involves encoding a document into transformer hidden states. When a user query is input, CFIC continues to encode the query alongside task instructions following the hidden states, subsequently generating grounding text. In practice, we can cache the documents’ hidden states to further reduce computation1. Given the expectation
1In a single-sided transformer model, the forward side is auto-regressive; once an output token’s hidden state is computed, it remains unchanged for subsequent forward steps, allowing us to use these encoded states as a cache.
for CFIC to process lengthy documents, it becomes imperative to adapt CFIC for handling long contexts. Considering the trade-off between efficiency and effectiveness, in this paper, we adapt CFIC to accommodate a 32k context, utilizing LLAMA27B-chat as the foundational model. To achieve this, we construct a dataset containing long document, user query and precise text evidence to training the foundation model via Supervised FineTuning (SFT).
Despite its promise, CFIC encounters two major challenges: (1) Efficiency issue: the autoaggressive generation process involves executing attention interactions for generating each new token, a procedure that becomes particularly timeconsuming with longer contexts due to the management of exponentially larger attention matrices. This process requires substantial computational resources (Kaplan et al., 2020), and (2) Faithfulness issue: it is challenging to ensure the generation model’s output remains faithful to the original input context, given its open-ended decision boundary (Li et al., 2022b). To address these, we propose two decoding strategies that accelerate inference and ensure that generated text evidence originates from the corpus. These include: (1) utilizing sentence prefixes as decoding candidates to shift the model’s decision boundary from open-ended to document-dependent generation and (2) upon locating the appropriate sentence prefix, bypassing the decoding of intermediate tokens and directly selecting sentence ends with the highest likelihood of the [eos] token, thereby terminating the generation. Furthermore, to retrieve multiple text spans as evidence, we sample several sentence prefixes with the best likelihood as candidates and rank them by sequence likelihood. By this means, CFIC not only enhances the relevance and accuracy of retrieved evidence text but also preserves the semantic integrity of the information, effectively addressing major drawbacks of current retrieval systems.
We tested CFIF on the LongBench tasks (Bai et al., 2023) including: (1) single-document question answering with datasets like NarrativeQA, Qasper, MulitfieldQA, and (2) multi-document QA with datasets like Musqus and HotpotQA. The experiment results verify the effectiveness of our method. In summary, our contributions are as follows: (1) we propose a chunking-free in-context retrieval method dedicated to the RAG system, aiding in locating precise text evidence to answer user
queries; (2) we propose the CFIC model of which the ability to find text evidence from long context is enhanced via Supervised Fine-Tuning with selfconstructed dataset; (3) we design two decoding strategies that significantly improve the efficiency and accuracy of the CFIC’s decoding process.

heading: 2 Related Work
text: The RAG framework, initially introduced in the works of Lewis et al. (2020), aimed to enhance language models’ capacity for generating knowledgebased responses(Chung et al., 2022; Yang et al., 2023). Subsequent research primarily focus on refining the RAG’s two core components. On the retrieval front, significant strides have been made towards more efficient and precise retrieval methods (Khandelwal et al., 2019; Nishikawa et al., 2022; Mao et al., 2023; Guo et al., 2022; Kang et al., 2023). For example, the arise of Dense Passage Retrieval significantly surpasses traditional sparse dense (Karpukhin et al., 2020). Parallel efforts on the generation side have concentrated on fine-tuning generative models to better harmonize with retrieved information, a notable example being the work of Izacard and Grave (2021b) in optimizing external knowledge utilization (Izacard and Grave, 2021a; Chung et al., 2022; Kamalloo et al., 2023; Qian et al., 2023b).
Nevertheless, RAG encounters specific challenges, especially in managing lengthy and complex retrieved documents. Researchers, including Mao et al. (2021), have developed chunking and reranking techniques to enhance passage relevance. Furthermore, Guu et al. (2020) introduced methods for jointly learning retriever and generator models, thereby improving the coherence and relevance of outputs. Addressing the issue of lengthy contexts in RAG has involved either refining contexts (Li et al., 2022a; Jiang et al., 2023) or adapting generation models to handle extended contexts (Ratner et al., 2022; Chen et al., 2023).
Recent advancements in RAG predominantly incorporate large-scale language models (LLMs), such as GPT-3 and GPT-4, to augment language processing capabilities (Brown et al., 2020; OpenAI, 2023; Google, 2023). The integration of LLMs has paved the way for more contextually rich and nuanced generation, especially in aligning generated responses with human preferences (Ram et al., 2023; Zhou et al., 2024; Liu et al., 2023b). In RAG systems employing LLMs, the accuracy
of retrieved textual evidence is crucial for reducing hallucinations and incorporating external knowledge (Zhang et al., 2023b; Yao et al., 2023; Bang et al., 2023; Qian et al., 2023a). However, the challenge of processing long and noisy contexts persists (Liu et al., 2023a; Li et al., 2022a; Xu et al., 2023). This paper introduces a chunkingfree in-context retrieval approach that leverages transformer hidden states to generate grounding text evidence, treating evidence retrieval as a generative process. This method represents a more streamlined and efficient retrieval solution for RAG systems, marking a significant advancement over previous retrieval methodologies.

heading: 3 Method

heading: 3.1 Preliminary
text: In a RAG system, the system takes a user query q as input, retrieves text evidence A from a text corpus C using a retriever θ(·) as external knowledge, and utilizes a generation model ϕ(·) to produce the final response T . This pipeline can be formalized as:
A = θ(q, C), T = ϕ(q,A). (1)
The retriever θ(·) can be either a standalone retriever (e.g., DPR (Karpukhin et al., 2020)) or a commercial search engine (e.g., Google), and the generation model ϕ(·) is usually a trained LM. Based on Eq. (1), the quality of the generated text T is bounded by the accuracy of the evidence A, emphasizing the importance of accurately finding the accurate text evidence.
In practice, most RAG systems’ retrievers cannot accurately find exact text evidences, but only retrieve lengthy documents (e.g., web pages or preindexed articles) that contain the evidences. As mentioned in Section 1, such lengthy documents might bias the generated content. Thus, given the retrieved evidence A, we usually select a few useful text spans, called supporting text evidence P = {p1, · · · , pk} ∈ A, to support the answer generation for the input query q in a RAG system.
We define the process of finding supporting passages as a mapping function f(·):
P = {p1, · · · , pk} = f(A). (2)
The mapping function f(·) can take various forms, such as chunking the text evidence A and prioritizing relevant chunks through re-ranking. In this paper, we define the mapping function f(·) as a
generation process in which we directly generate the supporting text evidence P conditioned on the transformer hidden-states h = Trans(A) of the lengthy document:
P = f(A) ∼ Generator(P|h, q). (3)
Compared to regular auto-regressive decoding, the above process is characterized by the fact that the generation target P contains text sourced from A. This means that once we determine the decoding prefix, we can skip the intermediate tokens and directly find the terminating position by computing the probability of inserting [eos] token. This greatly improves inference efficiency while ensuring that the generated text accurately represents the source text. Additionally, a single supporting passage may not always be sufficient for question answering. Therefore, we can obtain multiple sentence prefixes as top-k candidates using sampling decoding. In this paper, our proposed model CFIC applies these ideas to generate the top-k supporting text evidence P , which are further discussed in the following sections.

heading: 3.2 The Proposed Model: CFIC
text: Figure 2 presents an overview of our proposed model, CFIC. The process begins with CFIC receiving a user query. It then retrieves a long article as grounding evidence through a search engine (e.g., Google). Subsequently, CFIC combines the long document and the query into an input prompt, following the format outlined in Table 2. This input prompt is encoded into hidden states. Based on these hidden states, CFIC first identifies the top-k sentence prefix candidates using the Constrained Sentence Prefix Decoding strategy. This strategy
ranks the sentence prefixes considering the generation score (accumulated token log probabilities normalized by token length) of each sentence prefix. CFIC then skips the decoding of intermediate tokens and terminates the generation process by locating the [eos] token position with the highest likelihood (Skip Decoding). Consequently, we obtain k grounding evidence texts that can aid in supporting downstream tasks. It is important to note that this paper primarily focuses on pinpointing precise grounding text evidence within the long document, rather than on the retrieval of the long document. Therefore, we assess our CFIC and all baseline models using the LongBech benchmark, which provides pre-prepared long documents. In the subsequent sections, we will introduce the two proposed decoding strategies and then discuss the training and inference processes of CFIC.
Constrained Sentence Prefix Decoding Normally, the generation process of an auto-aggressive decoding model is as:
wn ∼ |w|∏
n=1
p(wn ∈ V|w<n,h), (4)
where h represents the hidden states of previous tokens. The current token, denoted by wn, is selected from the entire vocabulary V of the generation model. In the case of CFIC, the generation target P consists of text spans that originate directly from the source context. Consequently, it is possible to define a more constrained generation space to ensure the faithfulness of the text produced. Specifically, we suggest employing the prefix of each sentence within the source context as generation constraints. This approach guarantees that the
text generated by CFIC can be traced back to the input context. Thus, Eq. (4) can be modified as:
wn ∼ |w|∏
n=1
p(wn ∈ V̄|w<n,h), (5)
where V̄ denotes a token set contains each sentence’s prefix.
The sentence prefix serves as an position identifier to facilitate the identification of the starting point of a supporting passage within the source context. To select the top-k candidate passages, it is essential to differentiate k distinct sentence prefixes. This is achieved through the constrained top-k sampling decoding, a process that entails selecting the next token wn from the top-k most likely tokens V̄k ∈ V̄ based on the token’s probability, p(wn|w<n). The sampling process terminate once the generated sentence prefixes are capable of uniquely identifying positions in the source context. The number of decoding steps required until termination is denoted by β, resulting in up to kβ prefix candidates after β steps. We denote the generated sentence prefix by b. Subsequently, these prefix candidates are ranked according to the prefix sequence score s, which calculates the normalized accumulated log probability of tokens as follows:
s = 1
|w|
|w|∑
n=1
log p(wn|w<n). (6)
Finally, the k sentence prefixes with the highest scores are selected.
Referring to Figure 2 for illustration, the decoding process initiates by sampling k tokens, such as [Bach, In, ..., Throughout], to represent the first set of candidate tokens. Given that multiple sentences in the long article begin with the tokens [Bach, In], the decoding of subsequent tokens is necessary. For sentences that start with "Bach", the decoding terminates at step β = 2. And for sentences beginning with "In", the decoding ends at step β = 3. Following this, we retain k = 2 sentence prefixes to identify the supporting passages.
Skip Decoding Similarly, since the generation target originates exactly from the source text, once the generation prefix is determined, we can use the generated prefix as a position identifier to locate the original text in the source text. Subsequently, we can bypass decoding the intermediate tokens and directly compute the token probability p([eos])
for the [eos] token after each sentence following the generated prefix. We select the position with the highest probability as the termination point. In practice, we calculate p[eos] after each sentence within a predefined token distance d. Formally, given a generated prefix b, we determine the termination position as follows:
w∗[eos] = argmax l∈L p[eos](b⊕ l), |l| ≤ d, (7)
where l represents the token sequence following the prefix b with a maximum length of d.
Training and Inference As previously discussed, we define the task of identifying supporting passages from a long source text for grounding downstream tasks as evidence generation. To this end, it is crucial to enhance the generation model with the capability to pinpoint precise textual evidence within extensive texts. In this study, CFIC achieves this through Supervised Fine-Tuning (SFT). We employ a prompt, formed using the pair (q,A) as outlined in Table 2, as the input, and use the text evidence P as the target for generation. The model is trained using the negative log-likelihood (NLL) loss function:
L(q,A,P∗) = − |P∗|∑
n=1
log p(P∗n|P∗<n, q,A). (8)
The training dataset is introduced in Section 4.1. During the inference stage, given the input (q,A), we apply Constrained Sentence Prefix Decoding and Skip Decoding strategies to extract k supporting passages. Should these passages exhibit overlapping sections, we amalgamate such intersecting passages into a single cohesive passage. Subsequently, these collated supporting passages are utilized to ground downstream tasks.

heading: 4 Experiment

heading: 4.1 Datasets and Evaluation Metric
text: As mentioned above, we train the CFIC model using data that contains (q,A,P) triplets via SFT. Most current datasets cannot provide such data format. Thus, we use self-constructed SFT data to train the CFIC model, and evaluate all baselines on the LongBench benchmark (Bai et al., 2023). Specifically, to construct the SFT training data, we first collect a corpus of lengthy articles, including Wikipedia articles, novels, and news articles. Subsequently, we randomly select
text spans from these articles and ask ChatGPT to generate a query that can be answered by each text span. As for evaluation, we choose five datasets from LongBench including NarrativeQA (Kočiský et al., 2017), Qasper (Dasigi et al., 2021), MultiFieldQA (Bai et al., 2023)), HotpotQA (Yang et al., 2018) and MuSiQue (Trivedi et al., 2022). Following the LongBench benchmark, we use F1-score as the evaluation metric. For further details of LongBench, please refer to Bai et al. (2023). We show the statistical information of all datasets in Table 1.

heading: 4.2 Baseline Settings
text: In this study, we focus on in-context retrieval within the Retrieval-Augmented Generation (RAG) system. As such, we employ stand-alone LLMs as generators. Specifically, we utilize Llama2-7Bchat-4k (Touvron et al., 2023) and Vicuna-v1.57B-16k (Zheng et al., 2023) as our generators. To assess our chunking-free approach against the traditional chunking-based methods, the baseline model settings are as follow:
Chunking-Base Method Chunking-based methods generally commence by segmenting a lengthy document into smaller passages using heuristic strategies, followed by reranking these passages with a ranking model. In our research, we investigate two prevalent chunking strategies: (1). Sliding Window Chunking (SW): This strategy involves dividing the document into sentences and then grouping these sentences into passages. Each passage is designed not to exceed a predefined maximum length of 256 words, with a stride of one sentence. (2). Paragraph-based Chunking (Para): Here, the
document is split by paragraph markers (e.g., \n). We employ “bge-large-en-v1.5” (Xiao et al., 2023) and “llm-embedder” (Zhang et al., 2023a) as the ranking models. We utilize the SW and Para strategies to divide the document into passages, which are then reranked by the ranking models. The highest-ranking passages are chosen as the input context for the generators to support the QA tasks.
Chunking-Free Method For the chunking-free models, we present the outcomes using Vicunav1.5-7B-16k (Zheng et al., 2023), LongChat7B-32k (Li et al., 2023), and LongAlpaca-7B32k (Chen et al., 2023) as baseline models. These models refine lengthy documents into concise text evidence, which then serves as context for generator to support QA tasks. To ensure a fair comparison, all baseline models provide a comparable volume of textual evidence for downstream tasks, maintaining consistency in the number of passages or token length. We also explore the effectiveness of feeding full articles into generators.

heading: 4.3 Implementation Detail
text: To train CFIC, we employed the “LLAMA2-7Bchat” as the foundation model for our CFIC. During the training, we set the batch size to 1 per GPU and the learning rate to 1e-5. We set the gradient accumulation step as 8 and utilized the AdamW optimizer with an epsilon value of 1e-8. The model’s maximum length parameter was set to 32768. We train the model for 600 steps on 8 * Nvidia A800 80GB GPUs. For CFIC, We set the number of sampled sentence prefixes as k = 3 and the maximum decoding length as d = 256 (refers to Eq. (7)). Besides, we use a warm-up strategy to adjust the learning rate. To save GPU memory, we employed DeepSpeed’s Stage 2 zero optimization to save GPU memory.

heading: 4.4 Main Results
text: Table 3 shows the main experiment results which are the performance across different QA tasks using various refined text evidence as context. From the results we have the following findings: First, CFIC
significantly outperforms other LLMs in chunkingfree in-context retrieval tasks as CFIC is specifically optimized to select precise text evidence crucial for grounding QA tasks. This underscores the necessity and effectiveness of supervised finetuning (SFT) in adpting the foundation model into the in-context retrieval task. Second, Chunkingbased methods serve as strong baselines due to their ability to extract passages directly from the source context, whereas LLMs lacking SFT tend to generate content that may not always align faithfully with the source material. CFIC, however, consistently surpass all chunking-based baselines across all datasets, indicating the potentiality of the chunking-free in-context retrieval paradigm. Last, Compared to using the entire article as context, our CFIC model significantly improves the performance of QA tasks across most datasets, except for the NarrativeQA dataset. This improvement evidences the critical role of identifying and utilizing the right and precise context in optimizing QA task performance, demonstrating the CFIC model’s efficiency in context filtering and utilization. As for the NarrativeQA dataset, we find that NarrativeQA’s precise text evidence frequently appears at the start of lengthy articles, a location that LLMs tend to prioritize their attention (Liu et al., 2023a). This might explain why CFIC does not perform as well on this dataset, given that its approach to identifying precise evidence could inadvertently introduce errors, thereby diminishing its accuracy. In practice, however, that precise text evidence can be located throughout the entire length of an article, not just at the beginning.

heading: 4.5 Discussion
text: Ablation Study To assess the effectiveness of the design of CFIC, we conduct an ablation study by removing key components of the model, including: (1). Removal of Sentence Prefix Decoding Strategy (w/o prefix): we remove the constraint of limiting the decoding space to sentence prefixes. Instead, a beam search algorithm was employed to sample short sequences (each comprising 8 tokens) based on the input article. Subsequently, the top-k short sequences were matched back to the input article to identify starting prefixes. (2). Removal of Skip Decoding (w/o skip): we dispensed with the practice of bypassing intermediate tokens following the sentence prefix decoding. The model continued to decode the remaining tokens up to a maximum length of 256 tokens. (3). Removal of Both Decoding Strategies (w/o both): the CFIC model was tasked to decode outputs using a greedy search algorithm, devoid of both the sentence prefix and skip decoding strategies. (4). Absence of SFT (LongAlpaca7B): LongAlpaca-7B is a context-extended version of LLAMA2-7B-chat. We utilized LongAlpaca-7B as the base model, representing the variant of CFIC without task-specific SFT.
The results of the ablation experiments are presented in Table 4. Our findings can be summarized as follows: (1). The removal of any of the CFIC model components resulted in a notable degradation in performance, underscoring the collective contribution of these elements to the model’s effectiveness. (2). The most substantial decrease in performance was observed when SFT was omitted. This suggests that the vanilla LLM struggles to accurately locate precise grounding text from lengthy documents, despite its enhanced capability
for processing extended contexts. (3). Removing either the sentence prefix decoding or the skip decoding strategies led to an obvious reduction in performance. This finding verifies our hypothesis that these decoding strategies not only curtail decoding computational demands but also improve the fedelity of the generated grounding text.
Choice of Decoding Length In our CFIC model, as defined in Eq. (7), the generation process is terminated upon locating the position of the [eos] token within a predetermined distance d. This distance is analogous to the maximum generation length typically set in standard text generation tasks, which governs the length of the decoded text. The selection of d involves a careful balance: too small a value may lead to excessively brief output grounding text, offering scant information for substantiating downstream tasks. Conversely, a larger d may result in longer output texts, potentially introducing additional textual noise and necessitating increased computational resources to process the extended sequences.
To investigate the optimal choice of decoding length d in CFIC, we conducted experiments with various settings of this parameter. The results of these experiments are depicted in Figure 3. Our findings substantiate the initial hypotheses: the performance across all tasks progressively improves and reaches its zenith at a d value of 256. Beyond this point, performance begins to wane, suggesting that a setting of d = 256 strikes an effective balance for these tasks. This observation aligns with the intuition that a span of 256 tokens typically suffices to encapsulate a semantically complete and coherent unit of information.
Case Study: CFIC v.s. GPTs OpenAI’s model APIs, including GPT-3.5 and GPT-4, serve as robust baselines in the domain of LLM. However, they were excluded from the primary model comparisons in our experiments for two primary rea-
sons: (1). these APIs lack of control over decoding process resulting in the inability to manipulate their decoding mechanisms to align with our methodological requirements. (2). The foundational models of GPT-3.5 and GPT-4 are characterized by their vast parameter sizes (e.g., 175 billion parameters), endowing them with exceptional language modeling capabilities, especially in handling extended contexts. However, our focus with CFIC is on applying LLMs with comparatively smaller parameter sizes. This approach ensures more manageable computational resource requirements and enhances model scalability.
Despite these exclusions, we conducted a comparative case study, the results of which are presented in Table 5. This study reveals that our CFIC7B model consistently provided complete and relevant grounding text evidence in response to queries. In contrast, the other models exhibited limitations: (1). LongAlpaca-7B failed to accurately locate appropriate grounding text, resulting in the generation of information irrelevant to the downstream tasks. (2). GPT-3.5 is able to directly respond to queries, it did not successfully identify precise grounding text from the original source material. (3). Although GPT-4 managed to retrieve grounding text pertinent to the query, the information provided was incomplete, lacking the necessary comprehensiveness to fully support the response logically.

heading: 5 Conclusion
text: This study introduces a Chunking-Free In-Context (CFIC) retrieval method for the RAG system, addressing the challenges of processing lengthy documents and refining evidence retrieval. Unlike traditional chunking-based methods that either compromise textual coherence or struggle with noise and inaccuracies, CFIC leverages auto-aggressive decoding to pinpoint precise evidence directly, eliminating the reliance on chunking. CFIC incorpo-
rates Constrained Sentence Prefix Decoding and Skip Decoding strategies to further enhances retrieval efficiency and accuracy. Through comprehensive evaluations on various open QA datasets, CFIC has demonstrated remarkable improvements in sourcing relevant and precise evidence to ground language models.
Limitations
This paper introduces a novel approach for Retrieval-Augmented Generation systems through the Chunking-Free In-Context (CFIC) retrieval method. Despite its advancements and effectiveness, there are certain limitations that warrant discussion.
One of the primary limitations stems from the training data used to develop our models. The dataset, self-constructed and annotated using ChatGPT, may harbor annotation biases. Such biases could affect the model’s performance, particularly in its ability to generalize across different types of data or domains. While our approach excels in tasks requiring precise text evidence, it may offer limited assistance in scenarios demanding a highlevel understanding of context, such as summarization tasks. This limitation is due to the model’s focused capability on specific evidence retrieval rather than broader context comprehension.
Additionally, in this study, we have set the maximum length that CFIC can handle to 32k tokens. While this threshold accommodates a wide range of documents, it may not suffice for longer texts, such as novels, which exceed this limit. This constraint is primarily dictated by the available computational resources, highlighting a need for more efficient processing methods or greater computational power to extend CFIC’s applicability to longer documents. With the increase in computational re-
sources and advancements in model acceleration algorithms, we envision the future possibility of enabling CFIC to handle even longer contexts. This could potentially extend to encoding the entire corpus, facilitating corpus-level in-context retrieval for each query.
Ethical Impact
The development of CFIC builds upon existing Large Language Models (LLMs), which are trained on vast, diverse text corpora. This foundation introduces potential risks associated with biases inherent in the original training data. These biases can manifest in the model’s outputs, influencing the quality and impartiality of the retrieved evidence.
Furthermore, the long documents processed by CFIC are sourced from the web, a domain rife with its biases. The web’s text content reflects a wide array of perspectives, some of which may be skewed or unrepresentative of broader viewpoints. Given that CFIC is designed to process and retrieve information from these documents, there is a risk that the model might inadvertently perpetuate or amplify these biases without the capacity to discern or mitigate them.

heading: Acknowledgement
text: This work was supported by National Science and Technology Major Project (2023ZD0121504) and Public Computing Cloud, Renmin University of China. The work was partially done at the Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE.

heading: A Computational Expense
text: To explore the computational efficiency of our method, we conduct experiments on two datasets. The results, shown in the Table 6, indicate that our decoding strategies enable CFIC to achieve a threefold increase in inference efficiency compared to without the decoding stategies. Furthermore, although we set the maximum decoding length to 256 tokens, our analysis on the MuSiQue and Qasper datasets reveals that the Constrained Sentence Prefix Decoding strategy typically stops decoding after an average of 2.7 and 3.1 tokens, respectively. This is because most sentence prefixes can be distinguished after decoding just three tokens, preventing the need to decode the full 256 tokens until reaching the [eos] token.
In practice, the Constrained Sentence Prefix Decoding operates rapidly due to the minimal number of tokens required for decoding. The Skip Decoding, while more time-consuming due to its computation of the probability for the [eos] token after each sentence via a for loop, which can be significantly optimized with parallel computing techniques. We are confident that further engineering efforts will enhance CFIC’s inference time.
B In-Depth Evaluation with ChatGPT and Human
We conducted additional evaluations to assess the faithfulness and effectiveness of the generated text evidence. We defined faithfulness as the measure of how accurately the generated text evidence reflects the original long documents, and effectiveness as how well the generated text evidence supports the query. We randomly selected 50 samples from the test sets for two evaluation approaches. Firstly, we tasked ChatGPT with determining the faithfulness of the generated text evidence to the long documents and assessing the extent to which the evidence supports the query (supported, partially supported, not supported). Secondly, we had two human annotators blindly rate the quality of text evidence generated by different models. The results, presented below, show that BGE-SW and CFIC-7B are highly faithful to the original documents, directly extracting text spans from them. Notably, CFIC-7B provides more effective text evidence compared to other methods, indicating its superior performance in QA tasks.

heading: C Choice of k in Constrained Sentence Prefix Decoding
text: Regarding the optimal selection of the number of retained distinct sentence prefixes (k) in our decoding process, we explored the impact of varying k on the F1-score and inference latency. These experiments were performed on a single Tesla A800-80G GPU with an inference batch size of 8. Our investigation, summarized in Table 8, demonstrates the influence of k on both performance and efficiency:
(1) We observed that k = 4 represents the performance peak, beyond which the effectiveness declines, indicating that an increase in text evidence beyond a certain point introduces data noise and diminishes returns. Considering both efficiency and accuracy, we chose k = 3 for all experiments detailed in this paper, as it offers an optimal balance.
(2) The decoding strategies significantly improved efficiency. When comparing CFIC-7B with and without decoding strategies, the average latency reduction is evident, demonstrating the strategies’ effectiveness in enhancing processing speed without compromising result quality. For instance, at k = 3, CFIC-7B achieved an average latency of 361 ms, compared to 1,065 ms for CFIC-7B without decoding strategies, underscoring a substantial improvement in inference efficiency.

heading: Future Work
text: With the increase in computational re-
sources and advancements in model acceleration algorithms, we envision the future possibility of enabling CFIC to handle even longer contexts. This could potentially extend to encoding the entire corpus, facilitating corpus-level in-context retrieval for each query. Ethical Impact
The development of CFIC builds upon existing Large Language Models (LLMs), which are trained on vast, diverse text corpora. This foundation introduces potential risks associated with biases inherent in the original training data. These biases can manifest in the model’s outputs, influencing the quality and impartiality of the retrieved evidence. Furthermore, the long documents processed by CFIC are sourced from the web, a domain rife with its biases. The web’s text content reflects a wide array of perspectives, some of which may be skewed or unrepresentative of broader viewpoints. Given that CFIC is designed to process and retrieve information from these documents, there is a risk that the model might inadvertently perpetuate or amplify these biases without the capacity to discern or mitigate them