SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAG
Authors: Xuechen Zhang
Koustava Goswami
Samet Oymak
Jiasi Chen
Nedim Lipka
Venue: ICLR 2026 Poster (2026)
Keywords: Retrieval-Augmented Generation (RAG)
Long-Document QA
Adaptive Chunking
Efficient Information Retrieval
GRPO
Reinforcement learning
Primary area: foundation or frontier models, including LLMs
TL;DR: SmartChunk is a query-adaptive retrieval framework that dynamically adjusts chunk granularity and uses lightweight compression to improve accuracy and efficiency in long-document QA, outperforming state-of-the-art RAG baselines while lowering cost.

Abstract: Retrieval-augmented generation (RAG) has strong potential for producing accurate and factual outputs by combining language models (LMs) with evidence retrieved from large text corpora. However, current pipelines are limited by static chunking and flat retrieval: documents are split into short, predetermined, fixed-size chunks, embeddings are retrieved uniformly, and generation relies on whatever chunks are returned. This design brings challenges, as retrieval quality is highly sensitive to chunk size, often introduces noise from irrelevant or misleading chunks, and scales poorly to large corpora. We present SmartChunk retrieval, a query-adaptive framework for efficient and robust long-document question answering (QA). SmartChunk uses (i) a planner that predicts the optimal chunk abstraction level for each query, and (ii) a lightweight compression module that produces high-level chunk embeddings without repeated summarization. By adapting retrieval granularity on the fly, SmartChunk balances accuracy with efficiency and avoids the drawbacks of fixed strategies. Notably, our planner can reason about chunk abstractions through a novel reinforcement learning scheme, STITCH, which boosts accuracy and generalization. To reflect real-world applications, where users face diverse document types and query styles, we evaluate SmartChunk on five QA benchmarks plus one out-of-domain dataset. Across these evaluations, SmartChunk outperforms state-of-the-art RAG baselines, while reducing cost. Further analysis demonstrates strong scalability with larger corpora and consistent gains on out-of-domain datasets, highlighting its effectiveness as a general framework for adaptive retrieval.

Published as a conference paper at ICLR 2026
SMARTCHUNK RETRIEVAL: QUERY-AWARE CHUNK COMPRESSION WITH PLANNING FOR EFFICIENT Doc-
UMENT RAG
Xuechen Zhang"
Koustava Goswami
Samet Oymak University of Michigan oymakθ ch edu
University of Michigan zxuechen umi ch edu
Adobe Research
koustavag@aadbeecoom com
Jiasi Chen
Nedim Lipka
University of Michigan jiasiQumi edu
Adobe Research 1 dipka@adobe.coom adobe. com
ABSTRACT
Retrieval-augmented generation (RAG) has strong potential for producing accu- rate and factual outputs by combining language models (LMs) with evidence re- trieved from large text corpora. However, current pipelines chunking and flat retrieval: documents are split into short, predetermined, fixed- size chunks, embeddings ever chunks are returned. This design brings challenges, as retrieval quality is highly sensitive chunk size, often introduces noise from irrelevant or mis- leading chunks, and scales poorly to large corpora. We present SmartChunk re- trieval, query-adaptive framework for efficient and robust long-document ques- tion answering (QA). SmartChunk uses (i) a planner that predicts the optimal chunk abstraction level for each query, and (ii) lightweight compression mod- ule that produces high-level chunk embeddings without repeated summarization. By adapting retrieval granularity on the fly, SmartChunk balances accuracy with efficiency and avoids the drawbacks of fixed strategies. Notably, our planner reason about chunk abstractions through novel reinforcement learning scheme, STITCH, which boosts accuracy and generalization. To reflect real-world ap- plications, where users face diverse document types and query styles, we evalu- ate SmartChunk five QA benchmarks plus out-of-domain dataset. Across these evaluations, SmartChunk outperforms state-of-the-art RAG baselines, while reducing cost. Further analysis demonstrates strong scalability with larger corpora and consistent gains on out-of-doman datasets, highlighting its effectiveness as general framework for adaptive retrieval.
limited by static
retrieved uniformly, and generation relies what-
INTRODUCTION
Large language models (LLMs) have achieved remarkable progress in recent years, demonstrating strong performance across a wide range of natural language processing tasks, particularly question answering (QA) (Team et al. (2023): Grattañori et al. (2024). QA is becoming an integral part of different assistance tools (e.g., ChatGPT, Copilot) which users increasingly rely on for informa- tion seeking, decision support, and task completion. While LLMs can act as effective knowledge stores and can be further adapted through fine-tuning, they unavoidably struggle with knowled that changes over time or lies outside their pretraining distribution (Xu et al. (2024); ZZhng et al.) (2023); Huang et al. (2025): [Kandpal al. (2023)). To address these limitations, retrieval- -uggeentd gener- tasks by retrieving relevant information from external corpora and providing it to the LLM additional context (Chen et al.) (2017): RRa et al. (2023); Akyürek et al. (2022); Izacard & Grave (2020).. Despite strong promise, current RAG pipelines break down for long-document QA. Imperfect retrievers introduce
ation (RAG) ) has emerged to enhance generation accuracy in knowledge-intensing
Work done during internship Adobe.

---

Published as a conference paper at ICLR 2026
noisy even misleading evidence (Su et al. (2024); Yoran et al. (0023); Khattab al. (2023 Shi
et al. (,023)). and generators (LLMs) struggle to process very long inputs (An et al. (2024): Liu
al. 2023); Xu et al. (2023): Li et al. (2024b)) As result, answer quality often degrades as the
corpus size increases.
In practice, most multi-document systems follow a static pipeline: documents are split into short, fixed-size chunks, em- beddings are retrieved independently with
0.55
Ours
Best Trade-off!
each treated equally (i.e., non-hien non-hierarchical retrieval), and the top-k chunks are then
QA Accuracy
Structured Expensive!
0.45
passed to the LLM. Controlled exper- iments indicate that this static chunk- ing- ing-retrieval design is the primary bottle- neck as retrieval quality is highly sensi- tive to chunk size, and no single gran- ularity works well across heterogeneous queries and document structures (Bhat) al. (2025)). Recent work attempts improve chunking via recursive chunking, sliding windows, paragraph-base based splits. semantic boundaries or contextual chunk- ing (LangChain team (2023; ; Kamradt
0.40
0.35 -
Late chunking
RAPTOR
Single
-
Low Acc!
GRAG
SMARTCHUNK
0.05 0.10 0.15 0.25 0.30 0.35 0.40
Monetary Cost ($)
Figure 1: QA accuracy vs. Monetary cost across meth- ods. SMARTCHUNK achieves higher accuracy with lower cost compared to state-of-the-art basclines.
2024) Günther et al. (2024): Anthropic
2024 While these methods offer some improvements, they remain constrained by static pre-
processing pipelines, which cannot adapt to diverse queries and often suffer from noisy retrieval
or "lost-in-the-middle"" effects (Liu al. (2023)). More recently, tree- and graph-structuree RAG
methods (Sarthi al. (2024): Zheng al. (2025 ; Chen al. (2023); Zhang al. (2025a): Edge ict al. (2024): Liang et al. (2024); Sun et al. (2023a); Li et al. (2024a); Liu et al. (2025): Fazili
al. (024); Nair et al. (2023) organize text into hierarchical structures to support multi-hop rea- soning, but these systems introduce substantial complexity, computational cost, and dependence on structured representations. Other works also address these issues implicitly by finetuning LMs to predict correct answers despite noisy retrieval. However, such finetuning is computationally ex- pensive, time-consuming, and often reduces interpretability (Chen al. (2024); Cuconasu et al. (224);; Liu et al. (2023); y; Wu et al. (2024). These limitations highlight the need for lighter- weight, query-adaptive chunking frameworks that can dynamically adjust abstraction levels while remaining efficient.
In this paper, we present SMARTCHUNK, a query-aware framework for dynamic chunk generation and retrieval in long-document QA. Our approach addresses both efficiency and adaptability through three key innovations:
Planner for Query-Aware Chunking. We introduce a lightweight planner that decides the small- est and largest chunk sizes required to answer a query, enabling multi-level retrieval that adapts to both query complexity and document structure, reducing unnecessary chunk expansions. The planner is designed to be accurate, low-latency (S 1s), and adaptable across diverse domains and document types.
STITCH for Robust Planner Training, Training the planner is challenging because ground-truth labels unavailable, supervised traces costly to generate, and pseudo-labels often noisy. In addition, multi-objective reinforcement learning(RL), balancing accuracy. monetary cost, and latency, can be unstable. We propose STITCH (Solve with RL, Then Imitate To Close Holes), a sta- ble RL ++ SFT loop that alternates between RL, hinted RL, and imitation learning. To enable this process, we also design a pseudo-labeling and reasoning trace generation pipeline. This frame- work enables stable multi-objective training with improved sample and computational efficiency, delivers strong planning performance.
Compressor for Compact Representations. We introduce a lightweight compression module that produces embeddings capturing the high-level semantics of large document spans, without invoking expensive summarizers based large LLMs (e.g., GPT). This addresses key limita- tion structured chunking: while building hierarchical chunks can effective, is very expen- sive. By providing coarse compressed views alongside fine-gre ane-innieed local context, the compressor
2

---

Published as a conference paper at ICLR 2026
achieves the benefits of structured chunking more efficiently, balancing retrieval accuracy with token and computation cost.
Strong Empirical Results. Across four QA benchmarks and one out-of-domain dataset, SMARTCHUNK outperforms state-of-the-arr RAG baselines while reducing monetary cost(cach query incurs access fees from LLM APIs, often a few cents per call) by 30%, shown in Fig- [ Extensive ablation studies validate the efficiency of our design choices, and detailed evalu- ations show that SMARTCHINK achieves the best trade-off between cost (including monetary cost and latency) and accuracy. Furthermore, we demonstrate that SMARTCHUNK is orthogonal to other advances in RAG pipelines, such as late chunking( Günther al. (2024) and Hybrid Search (Liu & contributors (2022)). When combined, these methods show even greater improvements.
RELATED WORK
Chunking Optimization Chunking strategy plays a central role in retrieval-augmented generation (RAG). While early approaches use fixed-length or sentence-level chunking (Lewis al. (220)), )
recent work shows that static heuristics are often suboptimal (Bhat et al. (2025)). To address the challenge of missing
fragmented context, several methods augment chunking strategies, such as recursive chunking, sliding window chunking, paragraph-basee chunking, and semantic chunk- ing (LangChain team (2023); Kamradi (2024); Günther al. (2024)). Anthropic's 's Contextual Re- trieval (Anthropic| (2024)) proposes generating contextualized chunk embeddings using a secondary LLM, improving semantic fidelity the cost of greater computational overhead. However, these techniques rely on static preprocessing, which limits their ability to adapt to varying query complex- ity and can lead to noisy retrievals or the "lost-in-the nn-thhemmiddle" issue (Liu et al. (2023)).
Tree & Graph Structured Retrieval Augmented Generation Recent work has explored tree and graph structures as alternatives to vanilla RAG for handling large, noisy corpora and complex rea- soning. Tree-structured methods such as RAPTOR (Sarthi et al. (204)), MAL (Zheng et al. (2025) and Mem Walker (Chen et al. )2023) recursively embed, cluster, and summarize passages into multi- level trees. SiReRAG (Zhang et al. (2025a)) constructs both similarity and relatedness trees. PG- RAG (Liang et al. (2024)) GraphRAG (Edge et al. (2024) HopRAG (Lu et al. (2025)) DALK (Li et al. 2024a)) and ToG (Sun et al. (2023ab) prompts LLMs organize knowledge into mindmaps.
While tree & graph- based RAG effectively models hierarchical logic, they often incur higher cost, complexity, and reliance on structure rather than plain text. These trade-offs motivate lighter-weight query-adaptive alternatives that can flexibly combine multi-level abstraction with efficiency.
Finetuning efficient and controllable SLMs There is growing evidence that small language mod- els (SLMs) can power practical agents when finetuned with the right signals and constraints (Belcak) al. p0255). Recent works focus on methods that can make LLMs effective reasoners. Error-driven approaches such as self- eepooimproveeent loops (Shinn et al. (2023) show good performance. With the emergence of (Guo et al. (2025) ; Shao et al. (2024); Team al. (2025 :
Qin & Springenberg (225 ).
more and more reinforcement finetuned reasoning models demonstrate the importance of RL YYu
et al. (2025): Lin et al. (025);; Xu et al. 2025)) in reasoning tasks. Distillation works (Muennighoff et al. 2025 Guo et al. (2025) prove that SFT can help SL.Ms acquire reasoning capability compa- rable to the expert/teacher models. However, reliably finetuning SLMs for reasoning, especially still want to keep cost/latency low, remains challenging (Zhang et al. (2025b cy: Li et al. (2025); Qu et al. (225); Ma et al. (2025), ; Chu et al. (2025)). ). Pursuing controllability (e.g.. constraining rea- soning length) also remains difficult problem: recent work explores objective design (Zhang al. (202cc); Aggarwal & Welleck (2025); Shrivastava et al. (2025)) ), but it often requires high compu- tational cost and complex training pipelines. In contrast, we propose STITCH, which achieves both strong performance and concise reasoning traces in a more efficient manner.
3 PROBLEM STATEMENT
We study the RAG problem. where the goal is to provide accurate responses to user queries from documents. Our focus is on long-document QA, a particularly challenging task due to variation in document length, domain, and structure, well as diversity in query types ranging from abstract to detailed and mopt
The problem is to answer query q Q, where the correct answer à depends retrieving subset of useful chunks Ō C D, where D is the reference corpus segmented into chunks C.
3

---

Published as a conference paper at ICLR 2026
STITCH for Robust Planner MoannerTraiinn Training
Comprrsniio Eocadert
Genemders F
Step MiVnllall rolous
Queeton TM
Qusazon
Foley
Plamnr 2 (.42)
Step Hirted ming
Lomp
faleddii ats cala strl 2011 pa
Chunk Compression E (8433)
Step Imsttation heaming
Gosstion
Figure 2:
Left: Overview of the SMAKTCHUNK framework. Compared to vanilla RAG, which uses fixed chunking and flat retrieval, SmarChunk introduces two key modules: (1) a planner P that predicts the smallest and largest chunk sizes per query, enabling adaptive multi-level retrieval, and (2) Chunk Compression Encoder that produces compact, high-level embeddings for aggregated chunks, lowering the cost of the multi- representation. These additions allow SMARTCHUNK to adapt to different query complexity and document structure, balancing accuracy and efficiency. Modules added by SmartChunk are shown in bluc, while modules from vanilla RAG are shown in black. The figure distinguishes between text (represented by blocks with horizontal lines) and embeddings (shown solid-colored blocks). Right: The STITCH method for Planner training.
The retrieval task is to identify all and only the useful chunks Ö from C, and supply them to a downstream generator G, typically a large language model (LLM) such as GPT-4o Hurst et al. 2024), to produce the final response a = Ggg. C). The key difficulty lies in retrieving the right chunks from large corpora while avoiding noise, fragmentation, and excessive token usage, so that the generator can produce an accurate answer.
The goal to maximize response accuracy while minimizing the costs associated with retrieval and generation. We formulate this as joint optimization problem that balances two objectives:
Answer accuracy: probability that = G(q, π (q. D)) produces the correct answer å.
Cost Efficiency: minimizing the overall cost 7 (T, which includes the token budget (correlating to monetary cost) and the latency incurred by embedding, retrieval, and generation.
The goal of our framework is learn a policy ı that optimizes the trade-off between accuracy and cost efficiency: max, E(g.D.a)~> Acc(G(g, (g.. C))), a) AT((G, q, D))] where À > 0 controls the trade-off between accuracy and efficiency. We consider two types of costs this work, monetary price and latency. The details are shown in Section 5.1
4 PROPOSED FRAMEWORK: SMARTCHUNK RETRIEVAL
4.1 OVERVIEW OF SMARTCHUNK RETRIEVAL
We propose the SMARTCIIUNK framework, depicted in Figure When new question q E Q arrives, and there is a reference corpus of documents that segmented into a set of chunks C, the planner P predicts the smallest and largest chunk sizes consider, using both document meta- data and the query. In parallel, the Chunk Compression Encoder & builds a multi-level hierarchy by clustering smaller chunks and aggregating them into higher-level chunks. A high-level chunk embedding contains representation of the text summary of that cluster. The Chunk Compression Encoder & produces multi-level embeddings directly. Each high-level chunk thus encodes the se- mantics of cluster without requiring explicit text summarization. This design eliminates repeated summary generation and enables efficient construction of embeddings for high-level chunks in the hierarchy. The retriever R and generator G follow the vanilla RAG system (Liu & contributors (2022
Multi-level Chunking We assume a multi-level chunk hierarchy H(D), where a document is rep- resented at multiple granularities (e.g., sentence, paragraph, section, document or token spans such as 128, 256, 512, 1024 tokens) in a hierarchical structure. Higher-level chunks are constructed by aggregating lower-level ones through a summarizer (-)) In practice, LLMS such as GPT can serve as S to generate summarized text that forms a higher-level chunk. While raw fine-grained chunks preserve full semantic fidelity, they may fragment higher-level structure. Conversely, compressed chunks capture compact semantic summaries but may lose fine- ane-ggained details. This design ables the system to balance accuracy versus efficiency by adaptively mixing raw and compressed representations. We provide an example of the Multi-level chunk hierarchy in Section
4

---

Published as a conference paper at ICLR 2026
Planner The planner. denoted by P. is model that predicts the appropriate chunking granularity for a given query. Formally, P(g. ()) = (levelmin, levelmax), where levelmin levelmax denote the smallest and largest chunk levels sufficient to answer q while minimizing overall cost (including multi-level chunk construction, retrieval, and generation), and MetaData( D) denotes the metadata of documents. These predictions restrict the candidate retrieval set to C {c H(DD) (c) E [levelmin: levellax] We provide examples of the planner's ' inputs and outputs
in Section F
Chunk Compression Encoder Beyond text-level summarization, we introduce an embedding-level compression module, denoted E()) Unlike S(-), which produces summarized text chunks, E-) directly maps set of fine-graiin fine-grained chunks into single compressed embedding without generating intermediate text. Formally, given group of lower-level chunks (c1, 02,,-... C4 the module out- puts e(cs) C2,,,....... C4) Rd. where e(cs) is high-level semantic embedding that captures the aggregate meaning of the input chunks. In practice, can be a pretrained embedding model such as SBERT encoder imulti-gaa-mpne--ase-c-oo embedding model (jin--mmeeddings-33 Sturua et al. (2024)), or a finetuned Chunk Compression Encoder.
Reimers & Gurevych (2019)), long-context
4.2 PLANNER: TRAINED WITH SYNTHETIC DATA PIPELINE + STITCH
The planner must satisfy several requirements be effective in retrieval-augmeent essential, as errors in selecting chunk levels directly propagate to retrieval and degrade downstream answer quality. The planner should also operate under low latency, with a target reasoning length of fewer than 128 tokens (roughly one second on commodity hardware), ensuring practical deployment in interactive settings. Finally, the planner must exhibit adaptability, performing robustly across diverse domains, query types, and document structures. To address the challenges of training a planner, including unavailable ground-truth, noisy pscudo-labels, costly data collection, and unstable multi-objective optimization, we propose STTTCH (Solve with RI., Then Imitate To Close Holes), robust and sample-efficient training framework. As illustrated in Figure rright) and Algorithm P STITCH combines reinforcement learning (RL) and supervised fine-tuning (SFT) in stable loop.
QA. Accuracy is
In SMARTCHUNK, for each question paired with the documents and answer a, the workflow of SMARTCHUNK is shown in proceeds as follows:
1. Vanilla RL rollout: In Step 1, we optimize the planner using vanilla RL. Given a task (g, D). the planner generates a group of G rollouts ] P(g. MetaData( D)), where each rollout predicts chunking granularity plan based on the query and document metadata. For each rollout, we extract the predicted chunk sizes (levelmins levelmax) and execute the corresponding eetrenal-augmeented QA. If any rollout leads to a correct answer and satisfies the planning latency constraint, we mark the case as solvable and use it for policy updates.
Hinted RL rollout: If the question remains unsolved, we enter Step 2 (Hinted RL rollout). We generate an expert trace that solves the question while satisfying all constraints, and extract short hint from this trace. This hint is appended to the question (4. p) and used to condition a new RL rollout. If the hinted rollout succeeds, the planner is updated accordingly.
3. Imitation learning: However, for hard cases that remain unsolved even with hints, move Step (imitation learning). These examples are stored and periodically used to fine-tune the model via supervised learning. using full expert traces and answers.
This training loop continues, alternating between RL and SFT, until convergence. STITCH enables stable learning under multi-objective reward, improves sample and computational efficiency, and produces planner that generalizes across domains and document types with low latency.
Solvable cases from either stage are used to update the policy with the following objective, following GRPO Shao et al. (2024). The update balances reward, policy divergence, and planning efficiency:
Policy updates:
JSmrCH (0)
(D ( (mal(-K9.)))
Joal
min clip(rrr)))))) +5) Art)) -8Dra (Tol(tt
(1)
1=1
where
Te p. Oscc))
ALs = R, - mean( (Ri (E))
(0)
(2)
(oi P. oiee))
std (R:)
5

---

Published as a conference paper at ICLR 2026
RL Reward Components in STITCH To train the planner effectively, design multi-objective reward R that balances several requirements. It includes correctness of the QA answer, a penalty for excessive chunk usage, a penalty for overly long reasoning traces, and a reward for producing outputs in the desired format. We also introduce a pseudo-label alignment reward, which provides stable guidance in early training by aligning the planner's s outputs with precomputed labels before it fully explores the performance-eficienny trade-off. Details of the formulations are provided in Section G
Synthetic data pipeline To support this loop, we design a reasoning trace generation pipeline with four stages:
1. Hierarchy Construction. We first construct the complete chunk hierarchy, with the smallest unit corre- sponding to sentence and the largest corresponding the full document. Initial Retrieval. Given query, we perform top-k retrieval and use the retrieved chunks to generate an answer with a backbone model.
3. Pseudo-Labe Assignment. If the generated answer is correct, we record the smallest and largest chunk levels among the retrieved chunks pscudo-label. We emphasize that these pscudo-labels not guaran- teed to be optimal. If the answer is incorrect, expand retrieval with additional chunks check whether the model can then recover the correct answer.
4. Reasoning Trace Generation, We use the collected pseudo-lahels to generate reasoning traces. To improve trace diversity and reduce overfitting, we sample traces from diverse large language model families, covering parameter scales from 1.5B to 671B. The decision choice on the same has been covered in the Appendix. This pipeline enables automatic construction of supervision signals that are both diverse and scal- able, providing foundation for robust planner training. We shown ablation experiments of sample traces from multiple large language models or single LLM and the example of generated trace in Section
4.3 COMPRESSOR
A straightforward approach to obtaining high-level chunk embeddings is to first generate a sum- mary with a LLM and then encode that summary using an embedding model. Formally, given group of fine-grained chunks (ci....,Con we obtain summarized text chunk via an LLM: á = S(ci,..., .m), and its embedding using a pretrained encoder e: egt = <($) E Rd This provides a ground-truth compressed embedding Egt- However, this pipeline expensive because it requires repeated calls to the LLM S.
To reduce cost, we train a lightweight compression model that maps a set of lower-level chunk embeddings directly into high-level compressed embedding: Ccomp S(c(c)).... <(mm)) The compression model trained to minimize the discrepancy between its output Ecomp and the ground-
truth embedding Eg.. Leommp (S) (comp Cg. 1| Once trained, S can directly produce high-level embeddings without requiring LLM summarization, yielding comparable retrieval accuracy at much lower cost in both compute and latency.
We further discuss our design choices and their justification in Section |
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Datasets We extensively evaluate the effectiveness and efficiency of SMARTCHUNK across five QA benchmarks that span diverse document types and question formats, aiming to reflect realistic generation (RAG) scenarios. The datasets are shown in Table Following prior work (Sarthi et al. (2024); Bhat et al. (2025)), we include four in-domain datasets, which cover narrative comprehension, academic question answering, long-form multiple-choice QA, and structured table/text QA respectively. To assess generalization, we additionally include an out-of- distribution benchmark, NewsQA, which is grounded in CNN news articles and introduces a new domain with diverse topics and more open-ended -occmmett patterns, providing a challenging testbed for cross-domain robustness.
Evaluation We evaluate both the effectiveness and efficiency of our system using standard metrics established in prior work (Sarthi et al. (2024); [Bhat et al. (2025): (2025):(Zeea [Zeng et al. (2025). Our evaluation focuses three key dimensions: retrieval quality, end-to-end answer quality, and system efficiency.

---

Published as a conference paper at ICLR 2026
Dutaset
Does
Doe Length
# QA Pairs
Docureett Type
Query' Type
NarrativeQA Kočisky er al 2018
1.6k
62k
46k
Fictional stoniss
Absrmattvv namnlivv comprehension
QASPER Danirre mi
1.5k
NLP papers
Extractive buct-baee
QuALITY ang qpal. 20211
381
5.0k
6.7k
Informative articles
Long-fore multiple-choide
Natural Quiaines K Bwarggedki et 2019)
25k
6.9k
25k
Wikipedia articles
Span based open-domain
NewsQA Trischler
658
8.5k
120k
News articles (CNN)
Wh- and yes'no factual
Table I: Dataset statistics used in our evaluation. NewsQA serving as an out-of-domain domain benchmark.
The datasets span diverse domains and query types, with
Type
Method
QA Ace
Retrieval recall
Monetary cost(
Lateney
Fixed-size chunking (sentence)
0.251
0.517
0.007
1.16
Single
Fixed-size chunking (512)
0.285
0.648
0.006
1.119
Late chunking Günther et
2024)
0.363
0.661
0.007
1.26
RAPTOR Sarthi et 2024
0.526
0,714
0.398
3.21
Multi
MAL RAG Zheng
0.561
0.842
0.301
4.14
GRAG
FFage et al.
0.547
0,806
0.269
4.20
w/o P
0.539
0,773
0.096
1,94
w/o Eldirectly encode) w/o E(summarize) SMARTCHUNK
0.427
0,723
0.079
2.00
Ours
0.582
0.861
0.204
3.85
0,564
0.829
0.078
3.62
Comparison of different chunking and retrieval strategies. Our method achieves competitive QA accuracy and retrieval recall while significantly reducing monetary cost and latency.
Table 2:
Retrieval Quality. We report Recall K. a widely used metric in RAG tasks, which measures whether the top-SK$ retrieved chunks contain the gold answer span, We set K = 5 in our experiments. This assesses the planner and retriever's ability to select relevant content under various chunking strategies. Answer Quality. To assess downstream QA performance, nuanced evaluation, where a strong GPT-4o model judges whether the predicted answer is semantically correct with respect to the reference answer. This enables unified evaluation across datasets, accommodating variations in phrasing and the presence of multiple valid answers. Additionally, we report standard dataset- specific metrics (e.g., FI, ROUGE) in Appendix Table 1 for completeness.
adopt the I.LM-as-a-judge approach for more
System Efficiency. To evaluate efficiency, we report two metries: monetary cost and latency. Token-based prices are computed using provider rates (c.g., OpenAI, Together.Al Al TTgeeher Al (2024); (OpenAA (2024); details are in Scction E Notably, the cost of generator/summarizer models (c.g., GPT-4o) dominates, while small models like the planner and embedder are far cheaper and can even run locally. For consistency, we still include their cost using Together.Al pricing, offering conservative estimate. While some applications prioritize monetary efficiency, latency equally critical in real-time settings. SMART CHAAKCCCUKKKC delivers strong performance across both cost and latency, adapting well to diverse deployment constraints.
Baselines We also compare against standard and state-of-the-art RAG systems.
Sivele-Level Chunking Baselines. These methods use fixed chunk sizes without dynamic planning: Fixed- size chunking (sentence) (Kamradi) (2024) : LangChain team (203)) Liu & contributors (2022) splits doc- uments sentence boundaries. Fixed-size chunking (512) (Lewis al. (2020)) splits documents into
overlapping 512-token chunks. Late chunking (Günther et al, (2024) ) defers chunking until embeddings are computed, enabling better context preservation hut still operates at single granularity level.
Multi-Level Chunking without Planner or Compression. These methods build multi-scale document hi- erarchies but lack adaptive planning query-aware compression: MAI. RAG (Zheng et al. (202)) leverages chunks multiple abstraction levels, but lacks both planner and Chunk Compression Encoder. As sult, constructs
full chunk hierarchy for every document and relies on expensive GPT calls to generate summaries for each chunk. GRAG (Edge et al. (2024) integrates graph-based representations for multi-level retrieval and reasoning. RAPTOR (Sarthi et al. (2024)) builds multi-level chunk tree by recursively em- bedding, clustering, and summarizing text from the bottom up, and performs retricval over this hierarchical structure.
Additional experimental details are shown in Section
5.2
MAIN RESULTS
The main results are presented in Table 2] with per-dataset breakdowns provided in Table 9 SMARTCHUNK consistently outperforms all baselines across key metrics. While single-level chunk- ing baselines are low cost, their QA accuracy is substantially lower. SMARTCHUNK achieves up to about 30% improvement accuracy over these methods. Compared to recent state-of-the-an tree- graph-structured RAG methods that lack our proposed planner and chunk compression encoder, SMARTCHUNK delivers comparable or better results, showing an average gain of 1.7% in QA accu- racy and 4.0% in retrieval recall. Most importantly, it does with far greater efficiency, requiring

---

Published as a conference paper at ICLR 2026
Smantshork BewineNi
MAPTOR
MAL PAG
GMAG
Aeg Chunk Size (tokers))
1900
1000
500
MarrativeAA
QASPER
QuALITY
Natural Questonns
Averaga
NorraiveO
QASVER
QuALITY
Niturll Queseors
(a) QA quality among datasets.
(b) Chunk sizes among datasets.
Figure 4: (a) Performance gaps of SmartChunk over competing methods on four benchmarks Naarrative (ROUGE), QASPER (FI). QuALITY (Accuracy). and Natural Questions (FI); positive bars mean SmartChhnk outperforms the baseline. (b) Average chunk sizes (tokens) selected by our planner across datasets, illustrating httpt//quuer--appptiiiee bchavior.
less than 30% of the GPT cost and achieving lower latency. This efficiency comes from our plan- ner, which reduces the number of chunks that need to be built, and our compression module, which avoids repeated calls to GPT for summarization when constructing the chunk hierarchy.
To further assess the contributions of each module, we conduct ablation studies. Removing the plan- ner leads to always building the full chunk tree, while removing the compression encoder requires GPT summarization for each chunk. Both of them result in higher cost and latency. Nevertheless, even these ablated versions SMARTCHUNK remain competitive with existing baselines, con- firming the effectiveness of design. We also evaluate variant that removes the compression encoder and directly encodes chunks without summarization. This leads to a noticeable drop in retrieval and QA performance, highlighting the importance of providing condensed representations to the retriever. These findings emphasize the value of our lightweight compression encoder, which produces effective summaries with far lower cost than querying large LMs.
Total Cost of SMARTCHUNK While SMARTCHUNK intro- duces additional training cost, emphasize that this cost incurred only once. The learned planner and compression en- coder are universal components that can be directly applied across different datasets and domains, and we further demon- strate strong generalization out-of-distribution data. re- alistic RAG deployments, where the system serves millions of user queries, the one-time training cost is minor compared to the continuously accumulating inference cost of baseline mete than times compare to ours). Also, we only fine-tune lightweight 1.5B -Sr-amareettrr model (Qwen2.5-1.55 Instruct), while the generator and summarizer in the RAG pipeline is much larger GPT-class model whose inference dominates the overall cost. In Figure 3 we visualize the total end-to-end cost of SmartChunk and MAL RAG as the number of questions increases. The total cost including both the training(data collection + training cost of compressor and planner) and test-time cost. While SmartChunk in- curs small fixed cost upfront due to planner and encoder training, test-time cost grows very slowly. In contrast, MAL RAG begins zero but scales fasr with the number of questions, surpass- ing SmartChunk at around 2000 queries. This illustrates that the training cost is negligible relative to the savings obtained through more efficient, query-adaptive chunking. Consequently, once amor- tized over a realistic query volume, SmartChunk becomes substantially more cost-effective.
SmartChurk NAL RAG
1200
1000
Total
600
400
1000
2000
3000
5000
4. questions
Figure Total cost including train- ing and test-time.
5.3 ADDITIONAL OBSERVATIONS
Robustness Across Document Types. Our SMARTCHUNK framework demonstrates strong gener- alization across diverse document types, shown in Figure 4a which visualizes performance gaps relative to state-of-the-ar baselines (positive values indicate SMARTCHUNK superiority). Specifi- cally, the "performance gaps" refer to the accuracy improvement of SmartChunk over each state-of- the-art (SOTA) baseline, A Performanceemaatchukk www.enuuuunePerraaaaaaeeaaa Performanceeaettee Consistent with results in the main table, SMARTCHUNK outperforms all baselines on average. GRAG performs well on knowledge-centrr datasets like QuALITY but underperforms on narrativedrrvven tasks such as Nar- rativeQA, where SMARTCHUNK achieves nearly 8% higher accuracy. Similarly, RAPTOR struggles
8

---

fixed retrieval Published as a conference paper at ICLR 2026
NarrativeQA
QUALITY
Min Chunk
Proportion of Sammles
0.4
Max Chunk
02-
01
Chunk Level
Chunk Level
Figure 5: Distribution of Min-Max Chunk Sizes Across Datasets. chunk sizes (ranging from sentence-level to document-level).
Higher chunk levels correspond larger
Method
F1
Retrieval recall
Monetary cost(S)
Latency
Fixed-size chunking
0.846
0.895
0.006
0.98
MAI. RAG (Zheng et al. 20255)
0.907
0.971
0.147
1.32
SMARTCHUNK
0.875
0.931
0.026
1.29
SMARTCHUNK + few shot
0.906
0.967
0.032
1.34
Table 3: Performance on out-of-domain dataset NewsQA.
with temporal reasoning, leading to a 3% gap on story-centric benchmarks Narrative MAL RAG shows stronger generalization document structure changes but consistently incurs high cost due to exhaustive tree construction and summarization.
Query and Dataset-A Adaptie Chunking Behavior. We analyze the chunk size decisions made by the planner across different datasets. As shown in Figure 4b STITCH dynamically adjusts chunk sizes based on both document and query type. For instance, in NarrativeQA, which features long- form fictional stories paired with abstractive narrative comprehension questions, the planner selects larger chunks (avg. size: 1725) to capture broader contextual ares necessary for narrative under- standing. In contrast, scientific datasets like QASPER, which require factual multi-hop reasoning over dense, technical papers, STITCH favors smaller chunks (avg. size: 230) to maintain precision and avoid context dilution. As shown in Figure 5 this adaptive behavior also manifests in the plan- ner's predicted range of chunk levels. In NarrativeQA, both the minimum and maximum chunk levels are distributed toward the higher end (levels 4-6), reflecting the need for broader contextual integration to capture long-range narrative dependencies. In contrast, QuALITY exhibits a skew toward smaller chunk levels (1-3), its questions typically target specific facts or short reasoning chains within individual paragraphs. The divergence between minimum and maximum chunk levels further suggests that STITCH learns maintain flexible granularity. Starting from smaller, fine- grained chunks for grounding, then expanding to larger contextual spans when broader evidence aggregation is beneficial. This distributional pattern underscores the planner's ability to tailor chunk selection to dataset characteristics and query complexity, rather than relying granularity.
Out-of- Out-of-Domain Generalization. We evaluate the generalization capability of SMARTCHUNK on the out-of dittr---iitiitttion NewsQA (Trischler et al. (2017).. Results are shown in Table 3 Even with- out any in-domain fine-tuning, the planner trained on other datasets significantly outperforms the fixed-size chunking baseline in both QA F1 score (+2.8% and retrieval recall (+3.7%), while main- taining low monetary cost and latency. This indicates that the planner effectively transfers to new domains with different document structures and query distributions. Furthermore, when using few-shot prompting with only three in-context examples, SHARTCHUNK matches the performance of MAL RAG, the best-performin basclines in main experiments, while using only 25% of the monetary cost and comparable latency. This result highlights the robustness and adaptability of our proposed planner and compression encoder, even under domain shift, and demonstrates the practi- cality of our method real-world applications where training data may be limited or distribution shifts are common.
On the Necessity of Reasoning and Finetuning for Planner As shown Table 4 frozen models perform poorly, with low planning accuracy (42.6% even with few-shot context) and high latency due to the lack of training for length control. Finetuned classifiers also remain only moderately accurate. In contrast, 9.4% over the strongest baseline while maintaining reasonable latency. These results demonstrate the necessity of combining finetuning and reasoning: pretrained L.LMs fail to adapt without task-
fine-tuned planner achieves the best performance, improving accuracy by

---

Published as a conference paper at ICLR 2026
Type
Method
Planning Accuracy
Planning Latency(s)
LLM
0.407
2.472
no Finetuning
LLM
Few shot
0.426
4.036
SLM
0.382
3.058
MLP
0.609
0.0003
no Reasoning
SLM
0.724
0.019
Finetuning Reasoning
Ours
0.820
0.848
Table 4: Ablation study showing the effect of planner, finetuning, and reasoning components on planning
accuracy, cost, QA accuracy, and latency. Our full system achieves the hest performance across all metrics.
Method
Training Tokens
Planning Ace
Planning Latency
Q4 Ace
Monetary cost(S)
SFT RL SFT+RL
795k
0.740
1.162
0.491
0.085
0.356
0.365
0.427
0.066
795k
0.763
0.831
0.538
0.073
SFT+RL STITCH
418k
0.544
0.693
0.467
0.081
418k
0.820
0.848
0.564
0.078
Table 5: Comparison of training strategies for the planner. The details of baselines are shown in Section
specific training, while classifiers without reasoning cannot capture the structured decision-making required. Our planner instead learns to reason over metadata and query intent, producing latency- aware plans tailored the task.
Effectiveness and Efficiency of STITCH. We evaluate the effectiveness and efficiency of STITCH in Table STITCH achieves nearly 5% higher accuracy than the strongest SFT+RL baseline while using only half the supervised tokens. This is an important advantage given the high cost of labeled data. It also reduces monetary cost, RL can explore beyond suboptimal pseudo-labels. Standard RL, however, often fails due to weak base models and instability under multi multi-objective rewards. In contrast, STTTCH balances accuracy, latency, and cost effectively, with gains driven by solving first with RL to cut reliance on supervision, and targeted imitation learning to stabilize training where RL struggles. We further analyze the training dynamics of STITCH Appendix Figure to demonstrate the STITCH's stability. Beyond these gains, we also demonstrate STITCH's ability to generalize to new domains such as mathematical reasoning shown Section
Orthogonality to Other RAG Advances We demon- strate that SMARTCHUNK orthogonal to recent ad- vances in RAG pipelines, such late chunking (Günther
50
+SmartChunk
QA Accuracy(5)
et al. (2024) and hybrid search (Liu & contribu- tors (2022)). Hybrid search combines both keyword- based methods (BM25 Roberts et al.) (2020)) and vec- (embedding) search techniques. As shown in Fig- ure 6 integrating SMARTCHUNK with these methods gets further gains in QA accuracy. This indicates that SMARTCHUNK can complement complementary im- provements in chunking and retrieval, amplifying their effectiveness when combined.
30
LateChunking
Hybrid
Figure 6: QA accuracy when combining SMARTCHUNK with other RAG improve- ments.
CONCLUSIONS
We introduced SmarCChunk Retrieval, a query-adaptive framework for long-document QA that dy- namically selects chunk granularity and compresses embeddings to balance accuracy and efficiency. Across diverse benchmarks, SmartChunk consistently outperforms static baselines while reducing retrieval noise and cost. Our results highlight the importance of flexible retrieval strategies that adapt to both document structure and query demands. Beyond QA, this framework points toward general direction for controllable, resource-aware reasoning in language models.
Future work In future work, we plan to explore SmartChun''s application to deep research and other open-book QA tasks. Beyond QA, STITCH also points toward a broader direction for con- trollable, resource-aware reasoning in language models. Another future research direction we would like to explore image-text understanding using STITCH the shared embedding space to perform multimodal document retrieval.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was conducted during an internship at Adobe. It is also supported in part by the Na- tional Science Foundation grants CCF-2550179, CCF-2403075, CCF-2212426, the Office of Naval Research grant N000142412289,
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sébastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark. Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese Tarun Gogineni, Adam P. Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Do- minik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh Mc- Grath, Scott McKinney, Aidan McLaughlin, Song Mei. Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang. Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whit- ney, Hannah Wong, Lin Yang. Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang. Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card. CoRR, abs/2508. 1092500...10..... 2025. doi: 10.48550/ARXIV.25500 10925. URL httpss:/doi... oro.i.orrg/100 10. 48550/ arwia. 2508. 10925
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXhv:2503.04697 2025.
Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing factual knowledge in language models back to the training data. arXiv preprint arXiv:2205. 1482, 2022.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your Ilm fully utilize the context. Advances in Neural Information Processing Systems, 37:62160- 62188, 2024.
Anthropic.
Introducing contextual retrieval, 2024. URL
heeke/ccontextual--eezzzzzzz
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. 2024.
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02 153, 2025.
Kevin Bello and Jean Honorio. Learning latent variable structured prediction models with gaussian perturbations. Advances in Neural Information Processing Systems, 31, 2018.
Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, and Nicolas Flores-Herr. Re- thinking chunk size for long-document retrieval: A multi-dataset analysis. arXiv preprint arXiv:2505.21700 2025.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. arXiv preprint 1704v 2017.

---

Published as a conference paper at ICLR 2026
Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the mem- ory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029, 2023.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17754-17762, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint
(tt:::2501.1116.1 2025.
Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining re- trieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 719-729, 2024.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-soeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky. Robert Osazuwa Ness, and Jonathan Larson. From local to global: A query-focused
graph rag approach query- focused summarization. arXiv preprint WrXiv:2404.161300 2024.
Barah Fazili, Koustava Goswami, Natwar Modani, and Inderjeet Nair. Gensco: Can question decom- position based passage alignment improve question answering? CoRR, abs/2407.10245.. 2024. doi: 00.48550/ARXIV.2400 10245. URL
4 2407.
10245
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint 2rX 2024.
Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. Late chunk- ing: contextual chunk embeddings using long-context embedding models. arXiv preprint arXiv:2409.04701, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501.122 2025.
Lei Huang, Weijiang Yu, Weitao Ma. Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 13(2):1-55, 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. arXiv preprint arXiv:20/2.0458. 2020.
Greg Kamradt. 5 levels of text splitting, 2024. URL https::/github. com/
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to lear long-tail knowledge. In International conference on machine learning, pp. 15696-15707. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Omar Khattab, Amay Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-inmgo pipelines. arXiv preprint arXiv:2310.037/4 14, 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Trunsactions of the Association for Computational Linguistics, 6:317-328, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wor Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Lc, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl a 0.276. URL
http:s////clanntooooyyooooo
LangChain team. Parent document retriever, 2023, URL
Com/ doc///occ//////// http//owwtttttteeeooee coouctooo/aaeeeettccoeetttttttt /tt.r//eeerrcccmmmttteeerrreee// retriever/
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augment gener- ation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Dawei Li, Shu Yang, Zhen Tan, Jae Young Baik, Sukwon Yun, Joseph Lee, Aaron Chacko, Bojian Hou, Duy Duong-Tran, Ying Ding, et al. Dalk: Dynamic co-augmentation of ilms and kg to answer alzheimer's disease questions with scientific literature. arXiv preprint arXiv:2405.048111 2024a.
Shaojie Li and Yong Liu. Towards sharper generalization bounds for structured prediction. Advances in Neural Information Processing Systems, 34:26844-268577 2021.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-conter iims struggle with long in-context learning. arXiv preprint arXiv:2404.0206) 2024b.
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143, 2025.
Xun Liang, Simin Niu, Sensen Zhang, Shichao Song, Hanyu Wang, Jiawei Yang, Feiyu Xiong, Bo Tang, Chenyang Xi, et al. Empowering large language models set up knowledge retrieval indexer via self-learning. arXiv preprint 2405::2405.169/ 2024.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-basee reasoning models. arXiv preprint arXiv:2503.2234 2025.
Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. Ho-
PRAG: Multi-hop reasoning for logic-aware -arievel-augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Asso- ciation for Computational Linguistics: ACL 2025, pp. 1897-1913, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-256-5 doi: 10. 10.18653/1/2025,
findings-acla ac1.97. URL hattps://aclantholooyy htt.a///////attt////a/aaaaoooooo htt.://ooo......o0.222.....aaaa.
Jerry Liu and Llamalndex contributors. Llamaindex.
Jmma/
1laaa_innen idex 2022.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. 2023.
13

---

Published as a conference paper at ICLR 2026
Ben London, Bert Huang, and Lise Getoor. Stability and generalization in structured prediction. Journal of Machine Learning Research, 17(221):1-52, 2016.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: In- terleaved online fine-tuning for hardest questions. arXiv preprint orXiv:2506.0752) 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. /9393, 2025.
Inderject Nair, Shwetha Somasundaram, Apoorv Saxena, and Koustava Goswami. Drilling down into the discourse structure with Ilms for long document question answering. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 15593-14606. Association for Compu- tational Linguistics, 2023. doi: 10.18655/1/2223.FNDDDD EMMDNN99 URL https:
wtt.//........11///// hig.://o22..indnnnns-nnnnsaeee wmg..ggs-een....m 972
OpenAL. Openai api pricing, 2024. URL https://platfomn openai. wtt../ccccccc..o/.o
pricing Accessed: September 18, 2025.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.088008 2021.
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning on curated data is reinforcement learning (and can be improved). arXiv preprini arXiv:2507. 2025.
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov. and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning. arXiv preprint arXiv:2503.07577 2025.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin eyton-Brown, and Yoav Shoham. In-context retrieval- language models. Transactions of the Association for Computational Linguistics, 11:1116-331,, 2023.
Nils Reimers and Iryna Gurevych. Sentence-BERT: networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 111/663////D19.... 1410. URL https:::: mocelannhhlogge wwt.:/9-g/DD111191110
Sentence embeddings using Siamese BERT-
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910 2020.
Clayton Sanford, Bahare Fatemi, Ethan Hall, Anton Tsitsulin, Mchran Kazemi, Jonathan Halcrow, Bryan Perozzi, and Vahab Mirrokni. Understanding transformer reason- ing capabilities via graph algorithms.
A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.). Advances in Neural In- formation Processing Systems, volume 37, pp. 78320-78370. Curran Associates, Inc.. http://pper.ffleespppppp///////a
2024. URL htttss:/proceeedingg wtwp:eeeedddnng....niieeee nwurips wwtpriipppccpppppr.peeee File/ E511
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher Man- ning. Raptor: Recursive abstractive processing for tree-organized retrieval. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.174 17416, 2025.
14

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19255 2024.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.126553 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 0188634-655,, 2023.
Vaishnavi Shrivastava, Ahmed Awadallah. Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arXiv:2508.0977 2025.
Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krim- mel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina-embeddings- v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409. 10173, 2024.
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S Siegel. Michael Tang, al. Bright: realistic and challenging bench- mark for reasoning-in -intoinngintensivee retrieval. arXiv preprint arXiv:2407. 12883, 2024.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang. Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeurg Yeung Shum, and Jian Guo. Think-on-graphe Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.001 0769777 2023a.
Philip Sun, David Simcha, Dave Dopson, Ruiqi Guo, and Sanjiv Kumar. Soar: Improved indexing for approximate nearest neighbor search. In Neural Information Processing Systems, 2023b. URL
atttps://arxii org abs /0gg//////244.00777
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint vrXiv:2501.15500 2025.
Together AI. Together ai pricing, 2024. URL http::///ww.ttogtheeea/////eeeee
Ac-
cessed: September 18, 2025.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. In Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih (eds.), Proceedings of the 2nd Workshop on Rep- resentation Learning for NLP, pp. 191-200, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 10.18653/1/WW17-22222 URL
oor/// 2Tr//11-26237
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furn Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:22/2.0355. 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of tf-thuuhttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 22-837, 2022.
15

---

Published as a conference paper at ICLR 2026
Kevin Wu, Eric Wu, and James Y Zou. Clasheval: Quantifying the tug-of-war between Ilm's internal prior and external evidence. Advances in Neural Information Processing Systems, 37: 33402-33422. 2024.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025. 2023.
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- nown-sampling rollouts in IIm reinforcement learning. arXiv preprint arXiv:2504.1388 138188 2025.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0rXiv:2505.09388 2025.
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmennte language models robust to irrelevant context. arXiv preprint arXiv:331000/558, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan. Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534, 2023.
Nan Zhang, Prafulla Kumar Choubey, Alexander Fabbri, Gabriel Bernadett- Shapiroo, Rui Zhang, Prasenjit Mitra, Caiming Xiong, and Chien-Sheng Wu. SireRAG: Indexing similar and related information for multihop reasoning. In The Thirteenth International Conference on Learning Representations, 2025a. URL
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025b.
Xuechen Zhang, Zijian Huang, Chenshun Ni, Ziyang Xiong, Jiasi Chen, and Samet Oymak. Mak- ing small language models efficient reasoners: Intervention, supervision, reinforcement. arXiv preprint wrXiv:2505.07961 2025c.
Zheng Zheng, Xinyi Ni, and Pengyu Hong. Multiple abstraction level retrieve augment generation. arXiv preprint arXiv:2501.16952 2025.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
The Supplementary Material is organized as follows:
Section
we describe the role of LLMs in this paper.
In Section
we provide ethics statement.
In Section
we provide Reproducibility statement.
In Section
explain the additional experimental details.
In Section conduct additional ablation experiments show the effective of each mod-
ule.
In Section
we show the pseudo code of STTTCH.
In Section we discuss why we need different chunk size and indicate that static chunk- ing- ing-retrieval design is the primary bottleneck retrieval quality highly sensitive to chunk size, and no single granularity works across heterogeneous queries and document structure.
In Section I discuss the design choices and justifications.
In Section discuss the limitation.
B THE USE OF LLMS
In this paper, leverage large language models (LLMs) assist with writing and refinement. Specifically, LLMs are used to modify, restructure, and improve the clarity and fluency of our text.
C ETHICS STATEMENT
Our study does not involve human subjects, personally identifiable information, sensitive user data. All datasets used are publicly available and widely adopted in the research community. As for the potential risks of misuse, since method focuses improving the efficiency and accuracy of -ugmened QA systems, we believe will not lead direct harmful applications.
D REPRODUCIBILIITY STATEMENT
The full details of model architecture, training setup, hyperparameters, and evaluation protocols are provided in the main text and appendix. All datasets used publicly accessible, and we describe the preprocessing pipeline in Section
E EXPERIMENTAL DETAILS
Models We use GPT-4o Hurst et al. 2024 as the end answer generation and summarization model (Yang al.) (2025)
across all method. The planner fine-tuned from the Qwen2.5-1.5B-Imm model. For retrieval, adopt standard vector-based retrieval setup. We use SBERT (multi-qa- mpnet-base-cos-vl) (Reimers & Gurevych (2019) as the baseline embedding model. Our Chunk Compression Encoder is initialized from the same SBERT model and fine-tuned to produce com- pressed representations for high-level chunks.
Hardware For the hardware requirements, all of experiments done with 80G H100 GPUs.
Training settings. We adopt the verl framework (Sheng et al. (2024)) for training the planner. We utilize the Adam optimizer (Kingma (2014)) with constant learning rate of 10-5 For rollout, the prompt batch size is 256 and we sample 8 responses for each prompt. For training, the mini- batch size is 64. To find the appropriate hint length in the hinted RL rollout step, split the expert trace into sentences.
Monetary cost. We report the API price used to compute the monetary code for each model.
17

---

Published as a conference paper at ICLR 2026
Platform
Model
Input
Output
OpenAl
GPT-4o
2.5
10
OpenAl
Emhedding
2.5
10
Together.Al
Qwen2.5-7B
0.3
0.3
Together.Al
Llama-3.1 8b
0.18
0.18
Together.Al Al
GPT-oss-1 120b
0.15
0.15
Together.Al
GPT-oss-20b
0.05
0.05
Table 6: The monetary cost of each model.
Dataet train/test split. If the dataset is already partitioned into non-overlapping training, validation, and test portions, we will use Or, the dataset is split into 80% for training and 20% for evaluation. The training portion is used to train both the planner and the chunk compression encoder modules in our system.
Planner training baseline We provide the baseline details in Table For RL, adopt DAPO Yu et al. )2025)), an improved GRPO variant. SFT is trained for one epoch, following standard practice in prior work (Muennighoff et al. (2025)), while RL is trained for 120 steps with the same configura- tion as STITCH for fair comparison. During rollout, we use a prompt batch size of 256 and sample 8 responses per prompt; for training, the mini-batch size to 64.
Prompt We also list the prompt we use below to provide better reproducibility.
Summarizer:
"Summarize the following text in at most n tokens: [paragraph]." Here, is hyperparameter we can get. We use 128 in our experiments as this is the commonly used working length of the embedding model. [paragraph] is the content we want summarize.
Planner: "You
expert in analyzing document structure for retrieval.
Input: A document (up to 1000 tokens) A question about the document
Task: Predict two values: The **largest chunk size** that still provides sufficient context to answer the question. 2. The **smallest chunk size that still preserves the necessary information for the answer.
Valid chunk sizes: ("document", "section". ", "subsection" ". "paragraph", "sentence") J. Output: Please analyze the context, then return your prediction in JSON format with the
fields: { "largest chunk chuunssize"" "..." "smallest chunk size": "...""
Context: **Document (first 1000 tokens):** [document text] * **uesttonn*** ** [question text]"
F ABLATION RESULTS
Planner input and output Here is an example in QASPER (Dasigi et al. (202I)).
Input Input: First 1000 tokens of the document A question about the document Task: Predict two values: I. The **largest chunk * that still provides sufficient context to answer the question. 2. The **smallest chunk **ze** that still preserves the necessary information for the answer. Valid chunk sizes: ["document", "section", "subsection", "paragraph", ", "sentence") Outpul: Please analyze the context, then return your predic- tion in JSON formal with the fields: { "largest "hunk_sizee"
"smallest "smallest_chunkksiizz ciunk_size": "..." Context: * **Doumment (first 1000 tokens):** ** [It has been shown that word em- beddings derived from large corpora tend to incorporate biases present in their training data. Various methods for mitigating these biases have been proposed, but recent work has demonstrated that these methods hide but fail to truly remove the biases, which can still be observed in word nearest-neighbor statistics. In this work we propose a probabilis- tic view of word embedding bias. We leverage this framework to present a novel method for mitigating bias which relies on probabilistic observations to yield a more robust bias mitigation algorithm. We demonstrate that this method effectively reduces bias according to three separate measures of bias while maintaining embedding quality across various popular benchmark semantic tasks] ** **Question=*** ** [How is embedding quality assessed?)
18

---

Published as a conference paper at ICLR 2026
0.8
0.9
Trest planning accuracy
0.8
0.7
Rewerds
0.7
0.6
0.5
0.4
0,4
25
50
75
100
125
150
175
200
25
50
100
125
150
175
200
Steps
Steps
(a) Training curve.
(b) Test accuracy over training steps
Figure 7: Training dynamics of STITCH. The blue curves represent the training reward (left) and test planning accuracy (right) over training steps. In the left figure, the orange curve shows time-weighted EMA of the training reward, highlighting the overall trend. Red dashed lines show the points where targeted SFT is applied.
Output Question seeks how embedding quality is measured. Document states quality is maintained "across various popular benchmark semantic tasks." This key detail ap- pears in one sentence, smallest chunk is sentence. However, understanding the context of bias mitigation requires the surrounding discussion, making section the largest rele- vant unit. Retrieval at document level too broad; section provides enough scope. "largest chunk size": "section", "smallest chunk size", ; "sentence"
Training curve We further analyze the training dynamics STITCH in Figure The results demonstrate that STITCH is both stable and effecttie. Training reward increases steadily and con- verges to a high level. Test accuracy follows similar trajectory. showing consistent improvement over training and eventually plateauing near 80%. Importantly, the alternating ensures robust progress: RL explores beyond pseudo-labels, while SFT injections provide stability critical points, preventing collapse. Overall, the curves confirm that STITCH achieves strong and stable learning dynamics.
of RL and SFT
Trace diversity We also demonstrate the importance of using diverse reasoning trace for finetuning planner.
Method
# of samples
# of tokens
Planning Acc
Base
41.7%
SFT (single model, small)
2000
931k
53.1%
SFT (single model, large)
2000
783k
45.6%
SFT (6 models)
2000
794k
74.3%
SFT models)
1000
401k
69.2%
STITCH
421k
81.8%
Table 7: Comparison of planner training methods. STITCH achieves the highest planning accuracy while using fewer tokens.
Hint Extraction in STITCH. Step of STITCH, we extract hint from the expert-generated reasoning trace to guide hinted rollouts. We evaluate several strategies for determining the hint length and find that estimating based on the average success rate among all samples achieves the best balance between performance and computational cost. The results shown in Figure 8 This strategy provides strong guidance while minimizing overhead, and is therefore adopted across all experiments, including those the main results.
STITCH generalizes to other domains such as mathematical reasoning. STITCH outperforms all the baselines, the resukts presented in Table 8 Let discuss each baseline in turn. STITCH improves final accuracy by almost 20% over vanilla GRPO. SFT can offer a stronger starting point for RI,, but STITCH is still better.
19

---

Published as a conference paper at ICLR 2026
Hint Extraction Strategies in STITCH
67.
63.0%
Planning Accurscy (88)
60
-0
41.7%
80
30
20
10
Basc Random
EAS
Estimate
Figure 8: Comparison of hint extraction strategies used in Step 2 of the STITCH framework. Evaluation is conducted under fixed training budget 204,800 rollouts rollouts, 200 steps, batch 128 using vanilla GRPO). The Estimate strategy. which adaptively determines hint length based on rollout success rates, achieves the highest planning accuracy while maintaining low computational cost. This strategy is adopted for all remaining experiments.
Model
Dataset
GRPO
SFT+GRPO
STITCH
DeepSeck-Distin
Owen-1.5B-Instee
NuminaMath-CoT
0.347
0.348
0.366
0.368
Qwen-1.588 1.5.-L.5--nsstreet( buffer size T= 64)
NuminaMath-
0.347
0.348
0.367
0.368
Qwen-3B-Instrus
Nu
0.475
0.503
0.649
Table SMARTCHUNK outperforms other baselines and nearly reaches the accuracy DeepSeek- Ri-Dsstlll which has the benefit vast training data. There no DeepSeek-RR RI-Distill for Qwen-3B provided by (Guo ct al. (025)), ), so its cells are left blank.
STITCH succeeds on the hard questions, where other baselines fail. To understand the gains of STITCH, we train and evaluate each method exclusively on very difficult problems. The goal is to investigate whether expert hints in STITCH can help SLMs learn new information particularly from these hard questions. We conduct the experiment on the NuminaMath-Col dataset and the Qwen2.5- 3B-Instruct as the base model. To build the hard dataset, we first run ordinary SFT and from training dataset, we select the 500 questions for which three independent generations produce no correct trace (pass
0). These tasks are unsolved by small model, and their expert traces proved too complex for SFT to learn. The 500 samples were split into 80/20 train/test subsets. We then again used Qwen2.5-33B Instruct as the base model and trained each method on this hard subset. The results shown in Figure Pa] All baselines show little set after training on the hard questions. In contrast, STITCH achieves a clear upward performance with continued training, demonstrating that its partial expert guidance and branched rollout strategy can provide learnable information, even when standard SFT and vanilla GRPO fail. We argue that STITCH succeeds because it adaptively reduce problem difficulty and densifies the rewards. As shown in Figure 9b STITCH sharply lowers the solve-none ratio, getting more informative samples and richer feedback. This enables STITCH to learn effectively even from very hard questions and complex reasoning traces.
improvement the test
Scalability SmartChunk While our current evaluation focuses on moderate-sizee corpora for controlled comparison with prior RAG baselines, SmartChunk is inherently scalable because the planner operates at the chunk-level index rather than the token level. This design allows efficient retrieval decisions even when the underlying corpus grows by orders of magnitude.
Type
Method
NurrativeAAROOGEE
OASPERIFI)
QuALITY(Ace)
Natural Questions(P1)
Avg (Aceuracy)
Single
Fised-sioo chunking (S12)
0.409
0.496
0.695
0.727
0.285
Late chunking
Gümterr
20241
0.421
0.503
0.738
0.756
0.363
RAPTOR
Santh 2024
0.442
0.584
0.824
0.758
0.526
Mulii
MAI.
0.468
0.582
0.009
0.561
GRAG
Eags cral
0.742
0.547
Ours
SMARTCHUNK
0.474
0.583
0.819
0.806
0.561
Table 9: QA performance for each dataset separately.
20

---

Published as a conference paper at ICLR 2026
0.16
Smni
Teat aczurai
0.12
Solw pone natin
0.08
0.6
04
0.8
0.0
0.00
25
Training step
Training step
(a) Test accuracy over training steps
(h) Ratio of questions whose rollouts are all wrong
Figure 9: (a) Test accuracy on very hard questions over training steps. SMARTCHUNK outperforms other baselines significantly while the traditional methods (SFT, GRPO, SPT+GRPO) learn littl.. (b) Proportion of questions for which every rollout fails (solve-none ratio). In vanilla GRPO, the ratio stays persistently high, signalling that training stalls under sparse rewards. SMARTCHUNK. solve-none ratio regular rollout starts similarly high, but hinted RL injects expert traces and densifies the reward. This rate continuously decreases, confirming that the model is learning.
Algorithm SMARTCHUNK: SmartChunk Test-Time Inference Pipeline
Require: Query q E Q, reference corpus D, base chunk set C, planner P, Chunk Compression Encoder ε, retriever R, generator G
Ensure: Answer à
// Step 1: Plan chunk-size range
2: (Smin: Smax) &- (gg metadata D))
> predict smallest and largest chunk sizes to consider 3: // Step Build multi-level hierarchy (in parallel with planning)
4: H + ECC, 8min; 8max)
H is hierarchy of chunks with multi-level embeddings > smaller chunks are clustered and aggregated into higher-level chunks each high-level chunk embedding encodes the semantics of its cluster 7: // Step 3: Retrieve relevant chunks (yanilla RAG over multi-level embeddings)
5:
6:
retrieve top-k chunks using multi-level embeddings 9: // Step 4: Generate answer (vanilla RAG)
8: S < Rg. top k)
à + ((g,
11: return
In practice. SmartChunk be seamlessly integrated with sharded vector databases (e.g., FAISS Douze et al. (2024). ScaNN Sunn et al.] (2023b)) and supports parallel retrieval across chunk hierarchies, enabling real-time performance. The adaptive planner also reduces the number of re- trieval calls per query, lowering cost and latency as corpus size increases.
G ALGO
G.1 SMARTCHUNK
Test-time pseudo-code
G.2 STITCH
RL Reward Components in STITCH train the planner effectively, we design multi-objective reward R that balances several desiderata:
R Rox + Reast + RFormat + Ri.nggh
Each defined as follows:
+1, if generated answer a = golld
QA Answer Correctness Roa: RoA
0, otherwise
Chunk Budget Penalty Ress: To encourage fewer, more informative chunks, we penalize the cost of building multi-level chunks Rcm ChunkCost
21

---

Published as a conference paper at ICLR 2026
Trace Length Penalty Reenguh We penalize overly long traces beyond target token budget Lmax- Let the number of generated tokens the reasoning trace: ReLU(e- m
Reength = -
Format Rooraat: To encourage structured reasoning and ensure the answer is extractable, include a reward term that promotes outputs matching a predefined format. This is standard
+7, matches format 0, otherwise
in reasoning-baad based models training. Rioeraal
Pseudo-Label Alignment AplanningAcc: As alternative to optimizing QA accuracy and efficiency tradeof(( + RCas), we introduce a reward that aligns the planner's 's output with pseudo-labels. This bypasses retrieval and answer generation during training, enabling more stable and fast training. This term particularly useful during early training to guide the model before begins fully exploring the cefformannc-efffciinn trade-off on its own.
Training objective Our training objective follows the formulation of Shao et al. (2024), and use the same formulation. For each question 9. STITCH samples a group of outputs for o2, } from the old policy Told, and then optimizes the policy model by maximizing the following objective:
[smrCH())
G
lo.1
min (re:(0)A..,, (r..().. cuippro - + A.s) -BDx (mo (Dou(rollmat))
(3)
01
where
R. mean( [ ALt
(0,
(4)
Tou 9, oie))
R )
where and hyper-parameters, and A.,, is the advantage calculated based relative rewards of the outputs inside each group only. Tp and Told are the current and old policy models. = is sapping-related typer-parameter.
STITCH regularizes by directly adding the KL divergence between the trained policy and the refer- ence policy to the loss. We estimate the KL. divergence with the following unbiased estimator:
Tree(Or.1 q,00et)) log Te(o,,t g, (1,e)) To (o q, 01,e))
Tret(Or, q, gi,0t) 1.
DKL[TO
Teec]
(5)
The key difference from GRPO lies in the input and reward function: GRPO takes only the query input, whereas our method uses the document along with related questions.
STITCH pseudo code We also show the pseudo code in Algorithm 2
22

---

Published as a conference paper at ICLR 2026
Algorithm 2 STITCH: Solve with RL, Then Imitate To Close Holes
Require: Dataset D. planner policy Po. batch size N, GRPO optimizer, imitation learning (IL) buffer size T 1: Initialize Bu. +
2: for each training step do
Sample batch {(g,, D,, 0z ( Initialize RI. buffer BRL. <-
5:
for each (g. D2 a) in batch do
6:
Extract metadata
<- METADAAA
7:
# Step I: Vanilla RL rollout # Sample vanilla rollouts {0:) ~ Pa(: q. if ISSOLVABL
9:
then
10:
Add (q. m, {o, }) to BRL
11:
continue
12:
end
13:
# Expert trace generation #
14:
Get pseudo-label f- GETCHUNKLABEL.G D.a)
15:
Generate reasoning trace <- GENTRACE(g, D.y)
16:
# Step Hinted RL rollout # Generate hint p f EXTRACTHINTTRACEE(T
18:
Sample hinted rollouts (o Pol. g p.m) if ISSSOLVABLEE(c)
19:
then
20:
Add (4 + p. m, {o[]) to BR.
21:
else
22:
Add (q. D, a, 7) to Bu. end if
23:
24:
end for
25:
Policy update GRPO # if BRI. # 0 then
26:
27:
UPDATEPOLICYGRPO(fr BaI.)
28:
end if
29:
Step 3: Imittaton learning on unsolved cases #
30:
if [Ba.] 2 T then
31:
EMITATIONLAANNNGSTTER
32:
Clear Bn.
33:
end if
34:
end for
H MULTI-LEVEL CHUNK HIERARCHY
We provide an example of the Multi-level chunk hierarchy. The comparison between multi- sentence-level and paragraph-level summarization highlights the trade-off's in chunk granularity. Multi-sentence chunks preserve finer detail, capturing multiple aspects of the original text. This granularity leads richer reasoning traces and ensures that subtle arguments like the distinction between geometric and probabilistic treatments are not lost. However, it also produces longer outputs and potentially introduces redundancy across chunks.
In contrast, paragraph-level summarization yields more concise results by collapsing related content into a single compact representation. While this improves efficiency and reduces redundancy, some nuance sacrificed: details experimental setup and fairness principles are compressed, making the summary less expressive about specific mechanisms.
Taken together, the results illustrate why static chunking at single level is suboptimal. Finer chunks capture more information but may be inefficient, whereas coarser chunks are efficient but risk oversimplification SmartChunk's 's query-adaptive planning balances these extremes, allocat- ing finer granularity when detailed reasoning is necessary and coarser granularity when efficiency suffices, leading both accurate and cost-effective retrieval.
Whole paragraph paragraphs": [ "Word embeddings, or vector representations of words, are an portant component of Natural Language Processing (NLP) models and necessary for many downstream tasks. However, word embeddings, including embeddings commonly deployed for for public use, have been shown to ex- hibit unwanted societal stereotypes and biases, raising concerns about disparate impact on axes of gender, race, ethnicity, and religion BIBREFO, BIBREFI. The impact of this bias has manifested range of downstream
23

---

Published as a conference paper at ICLR 2026
lasks, ranging from autocomplete suggestions BIBREF2 to advertisement delivery BIBREF3, increasing the likelihood of ammliffynng harmful biases through the use of these models.". "The most well-establisked method thus far for mitigating bias relies on projecting target words onto a bias subspace (such as a gender subspace) and subtracting out the difference between the resulting distances BIBREFO. On the other hand, the most pop- ular metric for measuring bias is the WEAT statistic BIBREFI, which compares the cosine similarities between groups of words. However, WEAT has been recently shown to overestimate bias as result of implicitly rely- ing on similar frequencies for the target words BIBREF4, and BIBREFS demonstrated that evidence of bias can still be recovered after geometric bias mitigation by examining the neighborhood of a target word among socially- biiaed words.". "In response to this, we propose an alternative framework for bias mitigation in word embeddings that approaches this problem from probabilistic perspective. The motivation for this approach two-fold. First, most popular word embedding algorithus are probabilistic at their core -.e. they are trained (explicitly or implicitly BIBREF6) to minimize some form of word co-occurrence probabilities. Thus, we argue that a framework for measuring and treating bias in these embeddings should take into account, in addition to their geometric aspect, their probubilistic nature too. On the other hund, the issue of bias has also been approuched (albeit in different contexts) in the fairness literature, where various intuitive notions of equity such as equalized odds have been formalized through probubilistic criteria. By considering analogous criteria for the word embedding setting, we seek to draw connections between these two bodies of work." "We present experiments on various bias mitigation benchmarks and show that our framework is comparable to state-of of the-art alternatives according to measures of geometric bias mitigation and that it performs far better according to measures of neighborhood bias. For fair comparison, we focus on mitigating a binary gender bias in pre-trained word embeddings using SGNS okippgrram with brgative-sampling though we note that this framework and methods could be extended to other types of bias and word embedding alg algorithms."' J.
Multi-sentence level summarization "Word embeddings are vital for NLP tasks but have been shown to encode stereotypes and biases across gender, race, ethnicity, and religion BIBREFO, BIBREFI. Such biases can propagate into downstream applications, including autocomplete BIBREF2 and ad delivery BIBREF3, amplifying harmful social inequalities and raising concerns about fairness and disparate impact in real-world systems."
"Bias in word embeddings is often miligated by projecting words onto bias subspace (e.g.. gender) and removing residual components BIBREFO. Biax is typically measured with the WEAT statistic BIBREFI, which compares cosine similarities between groups. However, WEAT may overestimate bias due to frequency effects BIBREF4, and studies show that even after geometric debiasing, bias remains detectable in neighborhoods of socially-biased words BIBREFS."
"We propose probabilistic frumework, for mitigating bias in word embeddings, motivated by two factors. First, embedddingy are inherently probabilistic, trained model word co-occurrence probubilities BIBREF6, bias treatment should reflect both geometric and probabilistic aspects. Second, fairness literature often formalizes equity using probabilistic criteria, such as equalized odds. By adapting analogous principles, our approach connects embedding bias mitigation with established fairbes concepts, offering unified and principled per- spective."
"Experiments on bias mitigation benchmarks show our framework matches state-of-the-ar methods in geo- metric debiasing while significantly outprrformigg them in reducing neighborhood bias. For evaluation, we target binary gender bias in pre-trained SGNS sskip-gram with negative sampling) embeddings. Importantl, the framework is general and can be extended to address other forms of bias and applied across different word embedding algorithms."
Paragraph level summarization Word embeddings are essential for NLP but encode societal biases that prop- agate into applications like autocomplete and ad delivery, raising fairness concerns BIBREFO BIBREF.. Tra- ditional debiasing projects words onto a bias subspace and measures with WEAT BIBREFO, BIBREFI, though WEAT may overestimate bias and residual signals persist BIBREF4, BIBREFS. We introduce a probabilistic framework that accounts for embeddings' probabilistic nature and adapts fairness notions like equalized odds BIBREF6. Experiments show our method performs comparably geometric debiasing and substantially better in mitigating neighborhood bias, evaluated on binary gender bias in SGNS but extensible to other biases and embedding algorithms.
This also help us explain why we need different chunk size. In short, contiguous spans rarely suffice for long- range reasoning. For example, typical question in NarrativeQA QA (Kočiský et al. (2018) ) is "What motivates the main character's 's change of heart by the end?". This is not answerable by extracting localized text. requires integrating information spread widely across the entire text. While oversized chunks introduce a diffcrent problem: important fine-grained details are blurred or lost, making it harder to ground accurate answers. At the same time, embedding models have limited effective context and degrade on very long inputs, while L.MSs remain inefficient leveraging long context. Not all queries require long context in the first place. For example, the QASPER dataset of questions about NLP research papers (Dasigi et al. (2021)). many answers are extractive and contined to a single paragraph. A question such as "What other political events are included in the database?" can typically be answered from a sentence- or paragraph-Ievel chunk in the methods section,
24

---

Published as a conference paper at ICLR 2026
SmartChunk Retrieval = Planner + Chunk Compression Encoder
Chunk
Coompression
Encoder
Multi-level
Retriever
Retrieved
Generator
11
Answer
Documents
Pianner P
Planner P 4.2)
Predicts min/max chunk sizes per query
Query
Encoder E
Trained with synthetic data pipeline STITCH (bridge RL & SFT)
Adapts to query complexity dacument
structure
Documents D
Vanilla
Chunk Compression Encoder & (8 4.3)
Encoder
Retriever R
Generates summarized chunk embedcling Lowers the of building multi-level embeddings
Query
Embedding
Text
Figure 10: Overview of the SMARTCHUNK framework. Compared to vanilla RAG, which uses fixed chunking and flat retrieval, SmartChunk introduces two key modules: (1) planner P that predicts the smallest and largest chunk sizes per query, enabling adaptive multi-level retrieval, and (2) Chunk Compression Encoder & that produces compact, high-level embeddings for aggregated spans, lowering the cost of multi-level representation. These additions allow SMARTCHUNK to adapt chunking to query complexity and document structure, balancing accuracy and efficiency. Modules added by SmartChunk are shown in blue, while baseline modules from vanilla RAG are shown in black. The figure distinguishes between text (represented as blocks with horizontal lines) and embeddings (shown as solid-colored blocks).
without needing the entire paper context. Using overly large chunks for such questions can retrieve the relevant information with surrounding noise, hurting retrieval accuracy and answer quality.
I DESIGN CHOICES AND JUSTIFICATIONS.
Planner
(I) Planning as More Than Classification. At first glance, predicting planning signals such as smallest or largest chunk sizes may seem like a straightforward classification problem. But simple classification ignores the structured and sequential nature of planning (London al. (2016), which involves balancing trade-off between retrieval cost and answer quality. Unlike classification, reasoning model capture this ambiguity through explicit traces that encnde conditional logic and deep thinking (e.g., factual queries favor finer chunks, while synthesis tasks require larger spans). Theoretical work in structured prediction demonstrates that intro- ducing intermediate latent variables reduces sample complexity and improves generalization under non-linear mappings (Bello & Honorio (2018): Li & Liu (202T). suggesting that reasoning traces serve scaffolding stabilize learning
improve robustness to distribution shift. Our ablation experiments further confirm this: simple MLP routers or LLMs without reasoning underperform compared to a finetuned SLM, showing the necessity of explicit reasoning for planning. Most importantly, reasoning-bai based planners exhibit stronger trans- fer across tasks and domains, consistent with insights from compositional generalization (Wei et al.) (2022)
Sanford al. (2024).
(2) Add hint to build curriculum learning Reasoning problems are inherently compositional hence need a chain-of- th-uhh (Wei et al. (2022) to solve such problems. Suppose the problem contains T' steps/subtasks to solve, cach with successful completion probability of for the base student model. Then, we receive a final trajectory-level correctness reward with probability of z" assuming steps are independently completed. STITCH only needs to complete the
In contrast, by branching out from the expert trace at step T downstream steps, increasing the success e". Notably, by increasing т from to Т, STITCH facilitates curriculum learning by tackling individual problem steps one at a time requiring only ((1/c) samples to attain success per step.
(3) If a question is hard, start by memorizing RI.-hased training relies on preference correctness rewards, which can be sparse and unstable. In contrast, SFT provides strong fallback: the student memorizes canonical
25

---

Published as a conference paper at ICLR 2026
traces, ensuring coherence and baseline ability even hard questions. This aligns with cognitive intuition: humans first memorize worked examples before refining their skills through exploration. Thus, SFT not merely an initialization step but an essential safeguard against RL instability.
(4) Sample diverse trace to avoid SFT overfitting A key risk of pure SFT is overfitting to a narrow distribution of traces, which can cause models to fail on semantically valid but stylistically different reasoning patterns. It is also known that L-times deeper LLM can internally simulate L chain-of-thought steps of smaller LLM (Saunshi al. (2025) ). Consequently, expert models often generate long and dense traces that are pro- hibitively difficult for small student models to digest in one stage of supervised tinettuing (SFT). Without proper strategy, the student may fail to align with the expert. Recent work (Li et al. (2025) ) provides empirical evidence that small language models indeed struggle to learn directly from strong reasoners, To mitigate this, deliberately sample traces from diverse expert models. We sample traces from Ilama3 (Grattafiori al. (2024), Qwen3 (Yang al. (2025) directly where as we rewrite sample traces from DeepSeek-RI (Guo et al. (2025)) and GPT-OSS (Agarwal et al. (2025) due to their longer trace generations. Thus we do not directly use the traces from DeepSeek-R RI or GPT-OSS but are rewritten with our planner model. Theoretically, this expands the student's support over reasoning space, reducing variance in downstream generalization error. Empirically, ensures robustness varied reasoning styles, thereby preventing over-specialization single teacher.
Compressor
(I) Why we need summarization. Transformer-basee encoders (e.g.. SBERT Reimers Gurevychi (2019), ES Wang et al. (2022)) degrade when applied to very long spans. Summarization condenses essential seman- ties into a shorter sequence before encoding, yielding embeddings that better capture document-leve mcaning. Works like RAPTOR (Sarthi al. (2024)) and MALRAG (Zheng ct (2025) show that hicrarchical RAG ben- efits from summarized nodes: summarics act as "semantic bottlenecks, discarding redundancy while keeping high-level signals. Without summarization, embeddings of concatenated raw text tend to overweight surface- level lexical overlap rather than conceptual structure. Recent retrieval benchmarks (e.g.. Kamradi (2024): Asai al. 2024)) report that embeddings of summaries achieve higher recall and downstream QA accuracy than embeddings raw large chunks, especially for multi-hop queries requiring abstraction.
(2) Summarization changes the representation. Summarization alters the text distribution. which tum changes its embedding representation. As a result, embeddings of summaries are not equivalent to embeddings of the original content, motivating the need for dedicated compression model.
J LIMITATION
SmartChunk's 's adaptive planner performs best average. but for certain type of datasets, some methods indeed perform better. For examplc, GRAG shows strong performance on fact-based corpora such as QuALITY, where queries primarily require direct entity or fact retricval rather than hicrarchical reasoning.
26


Reviews:
[{"review_id": "VYswEBpex3", "reviewer": "Reviewer_DoNt", "summary": "The paper presents SmartChunk, a RAG framework that overcomes the limitations of static chunking. The method has two components, a Planner that predicts the chunk granularity for a given query, and a Chunk Compression Encoder that generates high-level embeddings directly from lower-level chunks without summarization. To train the planner without ground-truth labels, the authors introduce STITCH, a training method combining RL, expert hints, and imitation learning. Experiments across five benchmarks show SmartChunk having comparable performance with strong baseline while reducing costs.", "questions": "Beyond the average length of the retrieved documents, what is the distrubution of actaully the levels being chosen both across datasets and within the same dataset?\nHow does the method compare to other methods in terms of training efficiency and time?\nHow would the model compare to the baselines mentioned in the weaknesses section?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the method is well ablated with each part showing the tradeoffs. The method has comparable performance to methods like RAPTOR, GRAG, MAL RAG. The Chunk Compression Encoder is specfically interesting, It is a surprising result how it can boost results compared to directly just embedding the document. The method also shows some generalization.", "weaknesses": "Compared to some previous methods like RAPTOR, SmartChunk requires training, for the planner and the compression encoder. A few important baselines that are currently mssing is just retrieving from the database with differing chunk levels i.e. the model can retireve from all the chunks together (at different token levels) where the tokens can be embedded normally and also via the chunk compression encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QK3LTfnKQ4", "reviewer": "Reviewer_zCxf", "summary": "The paper aims to address the trade-off dilemma between accuracy and cost in existing RAG systems. SmartChunk introduces two modules: The planner dynamically predicts the minimum and maximum chunk levels required to answer a user query upon receiving it. The chunk compression encoder generates high-level summary embeddings from embeddings of low-level chunks. The paper also proposes a sophisticated hybrid training scheme.", "questions": "1. Is such extremely high training complexity a necessary condition for achieving adaptive planning, or is it a design choice made to compensate for deficiencies?\n2. Given the extremely long, multi-stage execution chain from planning to the final answer, how do the authors address the long-range credit assignment problem?\n3. Is the continuous interval assumption scientifically reasonable? In real-world scenarios, answering a complex question may require a non-continuous, cross-granularity combination of information. Could this rigid interval prediction paradigm become a fundamental bottleneck for the system in handling multi-hop or complex reasoning tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper identifies the most significant pain point of current advanced RAG systems: high costs.\n2. In response to the exorbitant costs of LLM-based summarization, the paper proposes the chunk compression encoder.\n3. The paper implements a dynamic RAG system, claiming to achieve a trade-off between accuracy and cost.", "weaknesses": "1. The motivation behind the planner is extremely difficult to comprehend. The core assumption of the paper is that a query-based planner can predict, prior to retrieval, the minimum and maximum chunk levels required to answer a question. However, given that the information distribution within documents is unknown, such predictions lack sufficient informational support and exhibit low scientific rigor.\n2. Why must the information required to answer a question be precisely distributed across a continuous range of chunk levels? For instance, a user may need a document summary to grasp the global context while simultaneously requiring a few precise sentences to extract key facts. The rigid interval design proposed in the paper is disconnected from the nonlinear information needs encountered in real-world scenarios.\n3. The paper attempts to use a query-aware model to plan a problem that should ideally be decided based on document structure during indexing. Although the paper claims to be selecting from pre-existing hierarchical layers, this constitutes a guess made on the basis of severely inadequate information.\n4. STITCH combines SFT, RL, Prompt-based RL, and Imitation Learning. This complexity exposes the poorly defined nature of the planner's task itself. If SFT + RL is effective, why is imitation learning still necessary? The paper employs extremely high training complexity and engineering techniques to forcibly fit a task with questionable motivation.\n5. Reward signals in reinforcement learning are challenging to define. The RL reward is derived from the final QA accuracy, which is an extremely sparse and significantly delayed signal. The planner predicts intervals, the retriever fetches relevant information, and the generator determines answer correctness. RL simply attributes the final error to the initial planner, which is methodologically untenable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfETOQF8Tk", "reviewer": "Reviewer_iEV9", "summary": "This paper presents SmartChunk, a query-adaptive framework for retrieval-augmented generation (RAG) that addresses the limitations of static chunking strategies in long-document question answering. The key innovation lies in dynamically selecting optimal chunk granularity based on query characteristics rather than using fixed-size chunks.\n\nThe framework introduces three main components: (1) A planner that predicts the smallest and largest chunk sizes needed to answer a query, trained using a novel reinforcement learning approach called STITCH (Solve with RL, Then Imitate To Close Holes); (2) A chunk compression encoder that produces high-level embeddings without expensive LLM-based summarization; and (3) A multi-level chunking hierarchy that balances fine-grained detail with computational efficiency.\n\nThe authors evaluate SmartChunk on five QA benchmarks (NarrativeQA, QASPER, QuALITY, Natural Questions, and NewsQA) and demonstrate consistent improvements over state-of-the-art RAG baselines, achieving higher accuracy while reducing monetary costs by approximately 30%. The STITCH training methodology combines reinforcement learning with supervised fine-tuning in a stable loop, addressing challenges of noisy pseudo-labels and multi-objective optimization.\n\nThis paper presents a solid contribution to the RAG literature with a practical solution to an important problem. The query-adaptive chunking approach is intuitive and well-executed, with comprehensive experimental validation. The STITCH training methodology, while complex, addresses real challenges in this domain.\n\nHowever, the contribution is somewhat incremental, and the added complexity may limit practical adoption. The improvements, while consistent, are not dramatic enough to represent a major breakthrough. The work would benefit from stronger theoretical foundations and more analysis of limitations.\n\nThe paper is above the acceptance threshold due to its practical relevance, solid experimental work, and novel training methodology, but it falls short of being a strong accept due to the incremental nature of the contribution and complexity concerns.", "questions": "1. Training data requirements: How much training data is needed for the planner to achieve good performance? How does performance degrade with limited training data?\n2. Computational overhead: What is the actual computational overhead of the planner during inference? How does this compare to the savings from more efficient chunking?\n3. Hyperparameter sensitivity: How sensitive is the STITCH training procedure to hyperparameter choices? Are there guidelines for setting these parameters for new domains?\n4. Failure mode analysis: Can you provide more analysis of when the adaptive chunking fails? Are there query types or document structures where fixed chunking performs better?\n5. Real-world deployment: Have you tested this approach in production settings? What are the practical challenges in deploying the full pipeline?\n6. Comparison with simpler alternatives: How does the approach compare to simpler adaptive strategies, such as using query length or complexity as heuristics for chunk size selection?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Practical relevance: Addresses a real bottleneck in current RAG systems where fixed chunking strategies perform poorly across diverse queries and documents.\n2. Technical innovation: The STITCH training methodology is novel and addresses genuine challenges in training planners with noisy pseudo-labels and multi-objective rewards.\n3. Comprehensive evaluation: Thorough experimental validation across multiple datasets with different characteristics, including out-of-domain evaluation.\n4. Efficiency gains: Demonstrates both accuracy improvements and cost reductions, which is crucial for practical deployment.\n5. Ablation studies: Systematic analysis of each component's contribution validates the design choices.\n6. Orthogonality: Shows that the approach can be combined with other RAG improvements for additional gains.", "weaknesses": "1. Complexity vs. gains: The STITCH training procedure adds significant complexity to achieve what appears to be modest improvements over simpler baselines. The cost-benefit trade-off may not justify the added complexity in all scenarios.\n2. Limited theoretical analysis: While the empirical results are strong, the paper lacks theoretical analysis of when and why the approach should work better than alternatives.\n3. Reproducibility concerns: The STITCH training involves multiple stages with various hyperparameters and design choices that may make reproduction challenging.\n4. Scalability questions: The evaluation is limited to relatively small corpora. It's unclear how the approach scales to very large document collections or real-time applications.\n5. Planner generalization: While out-of-domain results are promising, more analysis is needed on how the planner generalizes to truly novel domains or document types not seen during training.\n6. Limited error analysis: The paper doesn't provide sufficient analysis of failure modes or cases where the adaptive chunking performs poorly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fy0ykmWEyw", "reviewer": "Reviewer_yPFE", "summary": "This paper introduces SmartChunk, a retrieval-augmented generation (RAG) framework designed to address limitations of static chunking in long-document question answering. The core contributions include a lightweight planner that dynamically selects chunk sizes based on query complexity and document structure, a compression module that generates high-level chunk embeddings without expensive summarization, and STITCH, a reinforcement learning-based training method that combines RL and supervised fine-tuning to train the planner efficiently. The authors evaluate SmartChunk on five QA benchmarks and one out-of-domain dataset, demonstrating superior accuracy and efficiency compared to state-of-the-art baselines.", "questions": "Could you comment on the generalizability of SmartChunk to other RAG applications beyond QA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces a well-motivated approach to adaptive chunking in RAG systems. \n\n- The paper thoughtfully addresses both accuracy and cost (monetary and latency), making the method highly relevant for real-world deployment.", "weaknesses": "- Several key implementation details appear to be omitted. Specifically:\n    - The synthetic data pipeline lacks description of how chunks are merged or how the hierarchy is adjusted when initial answers are incorrect.\n    - The test-time workflow is not explicitly outlined, leaving the inference process from query to final answer unclear.\n- The fonts in Figure 2 are too small for comfortable reading. Furthermore, Figure 3a is challenging to interpret due to insufficient explanation of how the \"performance gaps\" are calculated and which specific \"SOTA baselines\" are being compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)