Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models
Authors: Yujie Feng
Jian Li
Zhihan Zhou
Pengfei Xu
Yujia Zhang
xiaoyu li
Xiaohui Zhou
Alan Zhao
Xi Chen
Xiao-Ming Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucination
Long-form Hallucination
Large Language Models
Primary area: generative models

Abstract: Large Language Models (LLMs) achieve impressive performance across many tasks but remain prone to hallucination, especially in long-form generation where redundant retrieved contexts and lengthy reasoning chains amplify factual errors. Recent studies highlight a critical phenomenon: the closer key information appears to the model outputs, the higher the factual accuracy. However, existing retrieval-augmented language models (RALMs) lack effective mechanisms to ensure this proximity — external evidence is injected into reasoning via multi-turn retrieval, but this cannot ensure key information stays close to the outputs. We propose Micro–Macro Retrieval ($M^2R$), a novel retrieve-while-generate framework to fill this gap. At the macro level, $M^2R$ retrieves coarse-grained evidence from external sources; at the micro level, it extracts essential results from a key information repository built during reasoning and reuses them while generating answers. This design directly addresses the key-information–to-output proximity bottleneck, effectively reducing hallucination in long-form tasks. $M^2R$ is trained with a curriculum learning–based reinforcement learning strategy using customized rule-based rewards, enabling stable acquisition of retrieval and grounding skills.  Extensive experiments across different benchmarks demonstrate the effectiveness of $M^2R$, especially in lengthy-context settings.

Published as a conference paper at ICLR 2026
MICRO-MACRO RETRIEVAL: REDUCING LONG-FORM HALLUCINATION IN LARGE LANGUAGE MODELS
Yujie Feng 1.-- Jian Li Zhihan Zhou², Pengfei Xu', Yujia Zhang' Xiaoyu Li', Xiaohui Zhou', Alan Zhao', Xi Chen't Xiao-Ming Wu²t
Solar System of OVB, Tencent, China
"The Hong Kong Polytechnic University, Hong Kong S.A.R. JJilin University, China
ABSTRACT
Large Language Models (LLMs) achieve impressive performance across many tasks but remain prone to hallucination, especially in long-form generation where redundant retrieved contexts and lengthy reasoning chains amplify factual errors. Recent studies highlight critical phenomenon: the closer key information ap- pears to the model outputs, the higher the factual accuracy. However, existing retrieval-augmennte language models (RALMs) lack effective mechanisms en- sure this proximity - external evidence is injected into reasoning via multi-turn retrieval, but this cannot ensure key information stays close to the outputs. We pro- pose Hicro-Macri Retrieval (M²R), a novel retrieve-while-- wiriive--wil--ggeerrt generate framework fill this gap. At the macro level, M retrieves coarse- coarse-grained evidence from external sources; at the micro level, it extracts essential results from a key informa- tion repository built during reasoning and reuses them while generating answers. This design directly addresses the neck, effectively reducing hallucination in long-form tasks. M is trained with a curriculum learning- -baiid based reinforcement learning strategy using customized rule- based rewards, enabling stable acquisition of retrieval and grounding skills. Ex- tensive experiments across different benchmarks demonstrate the effectiveness of M especially in lengthy-context settings.
proximity bottle-
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities across a wide spectrum of tasks, from question answering to complex reasoning and generation (Feng et al. 2023 Liu al., 2024 Zhang et al. 2025 Jin et al. 2025). Despite such impressive progress, even the most capable I.MMs, such as OpenAT-ol (Achiam et al. 2023) and DeepSeek-R1 (Guo et al.) 2025). still suffer from knowledge hallucination, i.c.. producing factually incorrect yet seemingly plausible content. Recent advances in aassoning-orienten LLMs suggest that explicit reasoning processes can partially mitigate hallucination by enforcing more faithful intermediate steps. Nevertheless, in long-form tasks that require generating multiple sentences or paragraphs, hallucination tends to be further
exacerbated (He al. 2023 Xu et al. 2023 Wu al. 2024 Cheng al. 2025a).
To alleviate hallucination, netrieval-augmenttee promising paradigm (Vu et al.] 2023 Yu al. (023). By incorporating external knowledge in plug-and-play fashion, RALMs are able to complement the parametric memory of LLMs with accu- rate and up-to-date information. A growing body of work has demonstrated their effectiveness and this mechanism significantly reduces the reliance on potentially outdated or incomplete parametric knowledge, thereby mitigating hallucination (Gao al. 2023 Wang et al. 2024
language models (RALMs) have recently emerged
However, RALMs are far from solving hallucination in long-form generation (Liu et al. 2025b:
Chang et al.) 2025b). key challenge. which we refer to as Lost in Lengthy Contexts, arises when key evidence is obscured in long contexts. This challenge manifests in two aspects. First, retrieved results are often lengthy, and the redundant information makes it difficult for the model to capture
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Macro Retrieval
C>
Simplified Output Rollout
Input Question
<tinkk> answer these questions <innero tool call> When was mucro results
C
founded
LLM Policy
Scarch
<think- Now, I need to find out sanacro tool call> Who was the macro resule> President
>>>
<think> I know the answer 6. lames and store it sey inlo key Jan
Reusoning Phase
Key Information Repository
Micro Rerrieval
Far
smicero tool call> smicro resule> value for kok name <answer> The answer James Close
James
C
LLM Policy
Search
(Red circles) Micv retrieval brings key to the answer tokens relucing hallecintion on effect supported by the tnst-in-th--mdddle plinnoeeooe.
Answer Phase
Figure 1: Overview of the M²R framework. During the reasoning phase, MPR performs macro retrieval and stores answer-aligned facts into internal key-information repository. During the answer phase, the model invokes micro retrieval to fetch the stored facts and place them close to the generated answer tokens.
the key information (Limitation f). Second, long reasoning chains often cause the model forget earlier intermediate results, leading to errors in the final answer (Limitation 2).
Recent studies highlight that the proximity of key evidence to final output is crucial for factual reliability: the closer the evidence appears to the final answer, the more likely the model is to remain faithful (Liu et al. 2023 Zhang et al. 2024). Additional empirical results and theoretical analysis of this phenomenon are provided in Appendix B However, existing RALMs lack effective mechanisms to guarantee such proximity external knowledge injected into the reasoning process via multi- turn retrieval, but this strategy cannot ensure that essential evidence is retained near the outputs.
To overcome these limitations, we propose a Micro-Macro Retrieval (M'R) framework. As shown in Fig. [ M has components. The first is macro retrieval, which follows the traditional paradigm of retrieving relevant passages from external sources during the reasoning phase. Cru- cially, whenever the reasoning process yields answer-aligned evidence, is preserved into struc- tured key-value repository, forming the key information repository. and the detection and storage of such key information are performed directly by the model during the <think> phase. The second is a novel micro retrieval mechanism introduced in the answer phase, which extracts essential results from this repository to ground the final output. By storing key information dedicated repository, the model avoids forgetting earlier intermediate results (addressing Limitation 1) while establishing a bridge that links macro retrieval with micro retrieval. During answer generation, the model can re-access the saved results and insert them directly before producing the corresponding output to- kens. this way, the proximity between key information and generated outputs is ensured, keeping key information tightly coupled with the answer (addressing Limitation 2). Finally, by adopting the retrieve- setrieve-while-ger while-generaeee paradigm, MPR effectively alleviates hallucination in long-form tasks.
In terms of implementation, forcement learning (RL) strategy (i.e., GRPO (Shao et al. 2024)) to train the model to perform the entire micro-macro retrieval process. Customized rule-based rewards are designed encourage accurate evidence saving and consistent grounding, allowing the model to gradually acquire the re- trieval reasoning skills in a stable manner. We train M from scratch on Owen2.5-3BBInsstru (Hui et al. 2024) and Qwen2.5-78 I-sttuutt and conduct extensive experiments on long-form question answering and retrieval-augee augmenttd generation benchmarks. Results demonstrate that MºR yields substantial improvements over strong baselines, with particularly pronounced gains under lengthy- context settings. Our contributions are summarized follows:
employ curriculum learning- based (Bengio et al.; 2009) rein-
By grounding generation on position-aware key information, we propose the M²R framework. M introduces terrieve-while-generate mechanism during the answer phase, where
2

---

Published as a conference paper at ICLR 2026
trieval is performed over model-generaee key information, and answer generation is constrained by enforcing proximity between the retrieved evidence and the generated tokens.
By employing a curriculum learning- based reinforcement learning strategy with customized rule- based rewards, M'R gradually acquires the ability progress from macro retrieval to key infor- mation saving and finally to micro retrieval in a stable manner.
By conducting extensive experiments on different open-source benchmarks, M demonstrates substantial improvements over strong baselines in terms of factual consistency and hallucination reduction, with particularly pronounced gains under lengthy-context settings.
2 RELATED WORK
LLMs have demonstrated outstanding performance across various tasks. However, in certain spe- cialised domains or wewwledg-inttensiv tasks, LLMs are prone to hallucinations. Regarding this problem, many approaches focus on detecting hallucinations in LLMs Wei et al. (2024): Kim et al.
(2224b); Chuang et al. (2024); Luo et al. (2025): [Zhong & Litman (025). Recently, numerous methods for detecting hallucinations
LLMs have emerged, specifically targeting scenarios with long context (Feng et al., 2024]] Shi et al. 2024a Qin et al.] (025). Liu et al.] (2025a) employed
self-generated thoughts derived from preceding utterances as expressions to induce intrinsic knowl- edge and comprehend long-context semantics. Park et al. (2025) achieve hallucination detection by
incorporatinng incorporating learnable lightweight and flexible steering vectors within LLMs.
Existing approaches to mitigating hallucinations in large language models can broadly be divided into two categories. One category comprises retrieval-auggmented generation (RAG) (Izacard & Grave 202T; Yu et al. 2024a; Xu al., 2025 Izacard al.] 2023; Shi et al., 2024b] Li et al., 2024). which direct models to retrieve external knowledge, thereby enhancing response accuracy and reducing hallucinations. Numerous approaches have been developed optimise the retrieval process for LLMs, thereby enhancing their performance. For instance, approaches such as [Triedii al.] (2023b), Shao al. (2023b), and Yu et al.. (2024b) introduce iterative retrieval-generation cycles, enabling LLMs to dynamically refine their retrieval strategies. Xu et al. (2024) and Kim et al. (2024a) enhance the utilisation of external information, reduce information overload, and improve factual consistency by optimising LLM generation through summarisation retrieval. Another class of approaches focuses on stimulating the LLM's capacity to utilise its internal knowledge. For instance, Li al. (2023) and Chen et al. (2024) employ probes or learnable parameters to optimise feature representations within the LLM. Chang et al. (2025a) imposes constraints on the generative process of LLMs. Cheng et al. (2025b) implemented flow-thinking generation process for LLMs through a see-search-baasee algorithm, thereby reducing hallucinations during the reasoning process.
Prior multi-turn retrieval frameworks such as ReAct (Yao et al. 2023) and Self-RAG (Asai al. 2023) interlcave retrieval with generation, but they operate only over external documents and cannot access model-generated intermediate reasoning. In contrast, 2 R retrieves from internal key- information repository constructed during the reasoning phase, enabling reuse model-generrted evidence. Moreover, R explicitly enforces evidence proximity by placing retrieved key facts im- mediately before answer tokens, mitigating long-context drift, constraint absent in prior methods.
METHOD
Our framework performs macro retrieval during reasoning to gather coarse evidence, and micro retrieval during answering to query key-information repository at generation time. With GRPO- based RL training, the model learns to maintain crucial evidence close to the produced outputs, improving factual reliability in lengthy contexts.
In this section, we first introduce reinforcement learning with integrated micro macro retrieval 1)..I. We then detail the micro-mace retrieval process itself, including the design of the train- ing template and the rule-based reward modeling ($3.2 $3.3). Finally, we present currculum learning-basee training schedule that stabilizes M training ($3.4).
3

---

Published as a conference paper at ICLR 2026
3.1 REINFORCEMENT LEARNING WITH MICRO-MACRO RETRIEVAL
We formulate the RL objective under the proposed micro-macro retrieval framework as follows:
max Egw D.w.DiD.--llloooo Ram Rmiee) (o((x - DKL (e Rmmero Rmicro)
T6
Ter(y [x; Rmaacr Rmiscr)]
where Te is the policy LLM, Tie is the reference LLM, To is the rule-based reward function, and DKL is the KL-divergence regularizer. Here, denotes input samples from the dataset D, and represents generated outputs conditioned on macro retrieval results Rmacro from external sources and micro retrieval results Ruicro from the key information repository constructed during reasoning.
Unlike prior retrieval -aggeeetee RL approaches (Chen et al., [025 Jin et al. 2025), , our framework integrates two-level retrieval directly into the policy with fixed macro -micro order:
To (.1 x; Remacros R Reiiero)
= 50 (.
x, M; Remiero) think (. Rmarr))
M SaveKey (arnKeyyyiiiig (Tthiok(. (- x;; Rmacro))
where o denotes sequential (staged) composition: the policy first executes the <think> phase with macro retrieval to collect coarse-grained evidence and build the key-information repository M, and then runs the <answer> phase with micro retrieval over M. This staged policy lever- ages external evidence while keeping key information proximal to the final answer, leading more reliable long-form generation.
GRPO with Micro-Macro Retrieval. We adopt Group Relative Policy Optimization (GRPO) as our RL algorithm. Unlike Proximal Policy Optimization (PPO), which typically trains an auxiliary value critic, GRPO estimates the baseline from group of rollouts and therefore avoids explicit critic. Given a current policy Tuw and a fixed reference T0 GRPO draws G rollouts { per input ~ D. The objective is:
T(0) = E (.:.D, ~o
1
To (o(()
[o(y)< To (H/[z)) 6, 1 A.) -302x. (Nol (Tar))
(3)
min
A,, clip
G
To (Hx(z)
where A, (r: - m-am( (ry (r:);= /std (1)))( denotes the normalized advantage of the i-th rollout within the group, is the clipping threshold, and ,3 is the coefficient for the KI. regularization term. The additional KL penalty prevents the updated policy from drifting too far from the reference model, stabilizing training and maintaining alignment with the base LLM. In our micro-macro trieval framework, all policy terms Eq, equation 3 are evaluated under retrieval conditioning: con- cretely, replace every occurrence of .T (- (z) with (- 2;; Rmacro Rmicro) (for {0, Bold, ).
Rollout with Macro and Micro Retrieval. Unlike conventional rollouts that contain text-only reasoning, rollouts in M²R are staged as macro->micro. During the <think> phase, the policy may issue multiple macro retrieval calls (e.g., <macro_tool saves answer-aligned key information (i.c., the answer to specific question) into a structured key- value repository M using the <key_ing policy performs miero retrieval calls (c.g., save> tag. In the subsequent <answer> phase, the toooooooaaaaaooo (ttl__oolllai)____ by querying M and conditions decoding on the returned values so that key information remains proximal to the output tokens and reduces hallucination in long-form generation.
1>) to external sources. Crucially,
Retrieval Result Masking. In standard GRPO, the policy loss is computed over all tokens a rollout. In our setting, however, rollouts contain retrieval results that are injected by the environment (external tools) rather than produced by the policy. To avoid assigning credit to tokens the policy did not generate, exclude retrieval-resee -rsultt spans when computing the loss. Concretely, in Eq. we update gradients only on tokens corresponding to text-basee reasoning and the model's 's own retrieval queries, while tokens inside retrieval results masked out.
4

---

Published as a conference paper at ICLR 2026
For implementation, let my {0, 1) be binary mask (1 for generated tokens; 0 for retrieval results). We replace sequence peg-pobb terms with a masked sum,
log To 1-)
Σ my log no (yt Met,, ) max(1, Σ me).
(4)
and analogously form the (masked) log-ratio in Eq. This preserves correct credit assignment, prevents spurious gradients from environment-ineected text, and stabilizes training.
What GRPO Optimizes in M It is important clarify that GRPO in M²R does not optimize
the retrieval module itself. Instead, GRPO supervises the model's generation behavior, teaching it (i) when to invoke macro- and micro-retricval, (ii) how to compose and sequence tool calls, (ii) what key information should be written into the repository during the reasoning process, and (iv) how retrieved information should be incorporated into the final answer. Since the retrieval component is not modified by GRPO, M remains agnostic to the underlying retrieval system and is compatible with future improvements in retrieval quality.
3.2 TRAINING TEMPLATE
We describe the training template for both macro and micro retrieval within our framework. The training process is organized into two stages: macro retrieval in the <think> phase and micro retrieval in the <answer> phase. The complete prompt template is shown in T6
Macro Retrieval and Key Information Saving. During the <think> phase, the model issues multi-turn macro retrieval calls enclosed within macro calls is to gather coarse-grained evidence from external sources. After retrieving the relevant information, the model saves the results as key-value pairs using the skey_info_save tag, storing them in structured repository M, which is accessed later during the <answer> phase.
tags. The purpose of these
Micro Retrieval for Final Answer Generation. In the <answer> phase, the model queries tag. The retrieved results are returned within the tags, which are then used form the final response. The final answer must be directly grounded on the results of micro retrieval. This ensures that the answer is based solely the key information retrieved, rather than independent reasoning.
M using the mnnicro_too__oee
3.3 REWARD MODELING
Since there is no supervised reasoning data available, we design rule-based reward function to optimize the policy through reinforcement learning. Our reward modeling consists of two primary components: format reward and answer reward.
The format reward ensures that the model adheres to the predefined structure specified in the prompt templates for both macro and micro retrievals. Specifically, it checks the correctness of tag usage (e.g., valid <macro_ cesacr__too__ee ttol_colllcllo caccll>> and <key_info_see _av_eve>>>>>ees during reasoning, and <micro_toor ttt__tooo_lllllat in the answer phase) It also ensures that every key value in the final answer is enclosed in \boxed }.
Format Reward:
Answer Reward: The answer reward is a combination of three sub-rewards, all of which are computed using the F1 score:
Final Answer Correctnsss (Sgmat): This evaluates the agreement between the model's final output (values extracted from \boxed }) and the ground-truth answer.
Key Information Correctness ): This measures whether the key information stored in the key-value repository M aligns with the ground-truth answer, ensuring that only the most relevant evidence is retained.
Consistency Score (Scons): This assesses the alignment between the stored key information and the final output, ensuring that the answer is grounded in the relevant retrieved evidence.
The total answer reward is computed as:
T Tans Stinal
Skey + B Sconss
(5)
5

---

Published as a conference paper at ICLR 2026
Specifically, for the final reward of rollout:
Fans F1 score is not and answer is correct,
0.1, if F1 score is 0 but format is correct,
(6)
0, if F1 score is 0 and format is incorrect.
3.4 STABILIZING TRAINING WITH CURRICULUM LEARNING
Training a model to integrate macro retrieval, key information saving, and micro retrieval is chal- lenging. In our initial experiments, found that directly optimizing all components at once often leads to unstable rollouts and poor convergence. To address this issue, we employ a curriculum learning approach, dividing the training into two stages. In the first stage, the model focuses ex- clusively on macro retrieval and key information saving, learning to correctly identify relevant in- formation and store in the predefined structure. In the second stage, we introduce miero retrieval and fine-gres grained answer grounding. enabling the model to leverage the stored key information when generating the final response.
This staged training strategy offers several advantages. First, it simplifies the learning process by reducing the complexity at each stage, which leads to improved training stability. Second, allows the model to progressively build the necessary skills, ensuring that the later micro retrieval steps are built upon a strong foundation of accurate macro retrieval and evidence saving. Finally, this approach encourages the model to generate answers that are not only factually accurate but also grounded in the retrieved evidence, maintaining consistency throughout the reasoning process.
This staged progression mirrors human reasoning, where individuals typically gather and organize broad information first, and then refine it into precise and reliable answers.
4 EXPERIMENT
To assess the effectiveness of MRR, conduct extensive experiments multi-hop question swering benchmarks that demand multi-step reasoning and repeated retrieval (Feng et al. 2025b). These settings naturally induce Lost in Lengthy Contexts scenarios. Our method is instantiated on Qwen2.5-3B 5-B--B--nsttt and Qwen2.5-7B 5-ren..-7B-Instrruct. Following ReSearch (Chen et al. 2025). we train
only on the MuSiQue (Trivedi et al.) (2022) training split, which offers diverse multi-hop ques- tions curated with fine-grained quality control. Our code is available at atthtps::/ / /gihhbb......... com/
WoodScene / /M2R
We evaluate M on four standard multi-hop QA benchmarks: HotpotQA YYang et al.] 2018). 2WikiMuliHHoPAA (Ho et al., 2020), MuSiQue (Trivedi et al.) 2022). and Bam- boogle (Press et al. 2023). HotpotQA, 2WikiMultiHopQA, and MuSiQue are automatically con- structed from Wikipedia or Wikidata (Vrandecic & Krötzsch 2014) with different multi-hop mining strategies and crowd-sourced validation, while Bamboogle is a manually curated set of challenging two-hop questions. For standard evaluation, 2WikiMullHHopQA (12,576), MuSiQue (2,417), and the test set of Bamboogle (125). For the first three benchmarks, we discard the original contexts and only retain question-answer pairs, with trieval performed from a shared Wikipedia corpus.
Benchmarks
the full development sets of HotpotQA (7,405),
Baselines We compare RR against several baselines: (1) No RAG: directly using the instruction- tuned model to generate answers without retrieval augmentation: (2) Naive RAG: standard retrieval-augmented setup where the retrieved documents are concatenated with the question be- fore generation; (3) Iter-RetGen (Shao et al. 2023ah: an iterative method that interleaves retrieval and generation; (4) IRCoT (Trivedi et al. 2023a): : an iterleaving method, which use retrieval and the chain-of ooainn-f-thoughh (CoT) guide each other. (5) COFT (Lv et al. 2024): coarse-to-fine frame- work that highlights key reference contexts to mitigate the problem of getting lost in lengthy inputs. (6) SURE (Kim et al.] (024b): generates summaries of retrieved passages for multiple answer can- didates, and then selects the most plausible answer by evaluating and ranking these summaries. (7) ReSearch (Chen et al. 2025): a reinforcement learning basneng-bbsed framework that trains LLMs to reason with multi-turn search, serving strong baseline.

---

Published as a conference paper at ICLR 2026
Evaluation Metries To assess the correctness of the final answers, adopt two complementary metrics. First, we report Exact Match (EM), which considers prediction correct only exactly matches the ground-truth answer. While straightforward. EM is often too rigid for setting, since the retrieval environment is open-ended and the generated answers are expressed in natural lan- further employ LLM-aa--ajjude (I.)) metric. Specifically,
guage. To address this limitation, we use spt -0 do-mini with a tailored judging prompt to evaluate whether predicted answer is semantically consistent with the ground truth. The full judge prompt provided in Appendi
RESEARCH QUESTIONS
RQ 1: ANSWER CORRECTNESS
Does MCR improve answer correctness compared to existing RAG methods on multi-hop QA tasks?
The main results of baselines and ReSearch are demonstrated in Table [ and show the meth- ods based on LLMs with different sizes respectively. Compared with all baseline methods, MrR consistently achieves superior performance across multi-hop QA benchmarks, demonstrating the effectiveness of integrating both macro and micro retrieval. Specifically, MPR significantly outper- forms the strongest baseline, ReSearch, which performs retrieval solely during the <think> phase.
These results verify that explicitly re-accessing key information through micro retrieval not only improves factual grounding but also enhances the correctness of the final output.
Table Exact Match (EM, %) and LLM--a-a-Judge (IJJ %) results on multi-hop question answer- ing benchmarks. The best results are highlighted in bold.
HotpotQA
2Wiki
MuSiQue
Bamboogle
Model
EM
IJ
EM
EM
Dwen2.5-3B-Instreee
Naive Generation
12.05
18.45
10.09
19.79
2.62
6.08
5.01
8.50
Naive RAG
21.04
37.23
13.82
23.08
4.14
10.32
14.43
20.00
Iter-RetGen
24.63
42.22
14.75
28.86
6.91
13.43
16.03
22.61
IRCoT
23.63
40.60
12.50
23.54
4.39
10.83
17.60
26.13
COFT
36.17
50.88
32.82
39.76
13.49
20.11
25.30
31.38
SURE
35.44
51.23
36.20
42.38
17.24
27.61
31.20
39.95
ReSearch
38.78
55.70
38.90
47.41
19.40
31.56
38.11
48.12
M'R-Qwen-3B-Instee
38.70
56.46
40.07
48.34
20.87
32.97
39.58
47.20
Qwen2.5-7B-Instruue
Naive Generation
19.18
30.64
25.76
27.87
3.76
10.38
10.40
22.40
Naive RAG
31.90
49.59
25.78
29.52
6.21
12.78
20.80
32,00
Iter-RetGen
34.36
52.22
27.92
31.86
8.69
16.14
21.60
35.20
IRCoT
30.33
52.06
21.57
30.65
6.99
14.19
24.80
36.80
COFT
41.08
61.71
41.86
48.70
17.12
26.28
35.71
49.23
SURE
39.56
60.16
45.65
53.93
20.87
32.24
39.58
52.81
ReSearch
43.52
63.62
47.59
54.22
22.30
33.43
42.40
54.40
MrR-Qwen-7B--nstree
44.11
65.98
48.89
57.01
24.12
35.44
44.56
56.89
RQ HALLUCINATION REDUCTION
Can M²R effectively reduce hallucinations under more challenging long-context scenarios?
To further stress-test the ability of M²R, we move beyond standard single-questinn inference and construct harder evaluation settings the HotpotQA dataset. Specifically, we concatenate mul- tiple questions into a single inference instance instance-denoted denoted as HotpotQA-2Q, (two questions) and HotpotQA-3Q (three
the model to answer them jointly within rollout.
7

---

Published as a conference paper at ICLR 2026
This setting substantially increases reasoning depth, retrieval calls, and contextual redundancy, thereby amplifying the difficulty of maintaining factual consistency.
We evaluate this setting with Qwen2.5-3B-Ih I-Innttuct,, and results are shown in Figure 2 M sistently outperforms all baselines as the number of questions increases. While naive RAG and ReSearch suffer from rapidly rising hallucination rates, M'R maintains stable accuracy and substan- tially lower hallucinations. This robustness comes from its miero retrieval continually re-anchors key evidence close the outputs, making M²R especially effective in bigh-redundancy, long-context scenarios.
paradigm, where
31A
SURE
Native Generetion
COFT
Native RAG
(S1
22:1
0.3
HotpoiQA-10
Hotpo1QA-2Q
HoCoOQAA-QQ
Figure 2: Performance on HotpotQA under multi-question inference settings. Here, HotpotQA-2Q denotes concatenating two questions into a single inference instance.
RQ 3: ABLATION STUDY
How critical the retrieve-while- generate design in M²?
To validate the contribution of micro retrieval, we compare our framework with a simplified variant that removes the retrieve-while -eeieregenertee mecha- nism. this baseline, when the model enters the <answer> phase, all saved key information from the repository M is provided to the model once. The model then generates the full answer based on this one-shot grounding, without in- voking micro retrieval during generation. In contrast, M performs on-demand micro retrieval: at each step of answer generation, the model can selectively fetch only the relevant key information and re-anchor it immediately before producing the corresponding output tokens.
Table Ablation study on M
Variant
EM (%)
LJ (%)
Full M'R
24.12
35.44
One-shot Grounding
23.38
34.72
Results in Table 2 show that one-shot grounding yields weaker factual consistency, as injected evidence can be diluted by redundant reasoning tokens. In contrast, the retrieve-while-geneee paradigm achieves more stable performance by inserting evidence precisely where needed. These results confirm that on-demand grounding is crucial for mitigating hallucination in long-form tasks.
RQ 4: REWARD DYNAMICS
How does Md R evolve in terms of reward during reinforcement learning?
To further understand the training dynamics of M²R, we analyze the reward curves during reinforce- ment learning, as shown Figure 3 the initial stage, the Qwen2.5-7B-Inee
model exhibits much sharper increase in reward compared to the 1-t.2..533--nstrrue model, demonstrating its stronger capacity to quickly adapt to the strreve-whiile miile-ggenerate paradigm. However, as training progresses, the reward growth of the Qwen2.5-3B Swem2.5-3B-Instruce model gradually catches up, and both models eventually converge to a similar level. This suggests that while larger models can accelerate early adaptation, the long-term reward dynamics between different scales tend to align.
8

---

Published as a conference paper at ICLR 2026
10
Revard
Touning Reward (ne)
Revend lemoothed)
Traiing Roward (snothee))
08
Thaining Rewand
Training Reward
0.1
0.2
0.2
0.0
0.0
Training Step
Training Step
(a) Reward curve M'R with backbone.
(b) Reward curve of M'R with backbone.
Figure Reward dynamics of M2R during reinforcement learning.
Dataset
Think
Answer
Total
Min
Max
HotpotQA
3.7
1.4
5.1
6
2Wiki
4.5
1.7
6.2
3
10
MuSiQue
5.7
1.9
7.6
9
Bamboogle
3.5
1.3
4.8
2
6
Table Average number of model invocations per query.
RQ 5: CASE STUDY
To provide a clearer view of how MRR operates in practice, Table 15 presents simpli- fied case study drawn from the evaluation set. This example demonstrates reasoning and under our framework. The text within <think> tags reflects the model's intermediate reasoning, while macro retrieval operations are invoked tags. Key evidence, directly aligned with the target answer, is tags. In the final <answer> phase, the
retrieval process of Qwen2.5-7BBInsttuee
via smacro_tool_call explicitly preserved using <key_info_save model invokes <micro cer__rool_llloooeeeeeeeeeeelllle mool_cal>>> to retrieve the stored key values, which are returned in cecccrrrrrooseeeeeee__eeeeeeeeee and faithfully incorporated into the output. This case illustrates how M²R decomposes question into manageable steps, preserves essential evidence, and grounds the final prediction through micro retrieval. By positioning the supporting evidence close to the generated answer, the framework effectively reduces hallucinations and strengthens factual consistency.
RQ INFERENCE CoST AND EFFICIENCY
What is the inference-time overhead of M²R, R, and how efficient is the micro-macro retrieval frame- work in practice?
To understand the computational cost of M we measure (1) the number of model invocations per query, and (2) the end-to-end latency under standard inference settings. We separate the analysis (macro retrieval) and <answer>> (micro retrieval) phases.
into the <think>
Model Invocations. Table 3 reports the average number of model calls using Qwen2.5-3B. Instruct. Most invocations originate from the <think> phase-a cost shared by all multi-turn tool-based RAG frameworks. The additional overhead introduced by MPR is only 1-2 micro- retrieval calls, corresponding to roughly 20-30% relative increase. Micro retrieval itself is ex- tremely lightweight, as it performs rulc-based lookup over a small, local repository.
End-to-End Latency. We benchmark real inference time, including all tool-calling and retrieval overhead, shown in Table 4 M'R increases latency by less than 10% on average compared to ReSearch, while delivering significantly higher answer accuracy.

---

Published as a conference paper at ICLR 2026
Dataset
Avg Invocations
Inference Time (s)
HotpotQA
5.1
R=4.7
2Wiki
6.2
~5.2
MuSiQue
7.6
R6688
Bamboogle
4.8
2:4.6
Table 4: Measured inference time of M²R (Qwen2.5-3B + SGLang, 4xA100).
CONCLUSION AND FUTURE WORK
This work introduced Micro-Macro Retrieval (MFR), novel Strreee-whiiee-generate framework that integrates macro retrieval during reasoning with micro retrieval during answering. By explic- itly preserving and reusing key evidence close to the outputs, M'R directly addresses the "Lost in Lengthy Contexts" problem, leading to substantial gains in factual consistency and reduced hal- lucination over strong baselines. For future work, direction is move beyond simple rule- based rewards and incorporate learned reward models that better capture factuality, coherence, and grounding. Another is further refine micro retrieval, for example by dynamically optimizing the proximity between evidence and output tokens. Finally, extending M²R with richer tool use, diverse external sources, and multimodal capabilities would broaden its applicability and robustness.
ETHICS STATEMENT
Our work focuses improving the factual consistency and reliability of LLMs in multi- hop ques- tion answering and eetreval-augmmented generation. All experiments are conducted on publicly available datasets (HotpotQA, 2Wik, MuSiQue, and Bamboogle), which do not tain personally identifiable information or sensitive human-subject data. We do not introduce new data collection involving human participants. Our research complies with the ICLR Code of Ethics, and we see no direct risks regarding privacy, discrimination, or legal compliance.
REPRODUCIBILITY STATEMENT
We make extensive efforts to ensure the reproducibility of our results.
Model and Training: We describe the reinforcement learning setup (GRPO with micro- macro retrieval), training templates, and reward modeling details in Section 3 with addi- tional hyperparameters Appendix D
Datasets: All datasets used (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) are publicly available; we detail preprocessing and evaluation protocols in Section 4
Code and Implementation: To facilitate reproducibilityy we provide anonymous link to the source code and experimental scripts in the supplementary material.
Together, these measures allow independent researchers reproduce our results and verify the claims made in this paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint vrXiv:2303.08774 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL WRRLhhtts:////xxv....
abs 0 2210. 11511
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
10

---

Published as a conference paper at ICLR 2026
Yurui Chang, Bochuan Cao, and Lu Lin. Monitoring decoding: Mitigating hallucination via eval- uating the factuality of partial response during generation. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 14574-14587, July 2025a.
Yurui Chang, Bochuan Cao, and Lu Lin. Monitoring decoding: Mitigating hallucination via evaluat- ing the factuality of partial response during generation. arXiv preprini arXiv:2503.00 03.0003.1110 106, 2025b.
Mingyang Chen, Tianpeng Li. Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang. Jeff Z Pan, Wen Zhang. Huajun Chen, Fan Yang, et al. Learning to reason with search for Ilms via reinforce- ment learning. arXiv preprint arXiv:2503. 19470, 2025.
Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Cheng- Zhong Xu. Truth forest: toward multi-scale truthfulness in large language models through in- tervention without tuning. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Artificial Intelligence, 2024.
Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Think more, hallucinate less: Miti- gating hallucinations via dual process of fast and slow thinking. arXiv preprint arXiv:2501.03300 2025a.
Xiaoxue Cheng, Junyi Li, Xin Zhao, and Ji-Rong Wen. Think more, hallucinate less: Mitigating hallucinations via dual process of fast and slow thinking. In Findings of the Association for Computational Linguisties: ACI. 2025, pp. 7979-7990, July 2025b.
Yung-Sun Lung-Sung Chuang, Linlu Qiu, Cheng- Yu Hsich, Ranjay Krishna, Yoon Kim, and James R. Glass. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 1419-1436, November 2024.
Zhihao Ding, Jieming Shi, Shiqi Shen, Xuequn Shang, Jiannong Cao, Zhipeng Wang. and Zhi Gong. Sgood: hubstructure-enhancee graph-level outtof-disttibution detection. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 467-476, 2024.
Zhihao Ding, Yongkang Sun, and Jieming Shi. Betrieve-and-verine A table context selection frame- work for accurate column annotations. Proceedings of the ACM on Management of Data, 3(6): 1-27, 2025.
Xiaoyu Dong, Yujie Feng, Zexin Lu, Guangyuan Shi, and Xiao-Ming Wu. Zero-shot cross-domain dialogue state tracking via context-aware auto-prompting and cdtructioo-followiin contrastive decoding. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8527-8540, 2024.
Yujie Feng. Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. Towards Ilm-driven dialogue state tracking. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 739-755, 2023.
Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual dialog state tracking via task skill localization and consolidation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1266-1279, 2024.
Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S Yu, Xu Chu, and Xiao-Ming Wu. Recurrent knowledge identification and fusion for language model continual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27396-27413, 2025a.
Yujie Feng, Li-Ming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang. Jiannong Cao, Philip Yu, and Xiao-Ming Wu. Geoedit: Geometric knowledge editing for large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp- 13401-13416, 2025b.
11

---

Published as a conference paper at ICLR 2026
Yujie Feng, Hao Wang, Jian Li, Xu Chu, Zhaolu Kang, Yiran Liu, Yasha Wang, Philip S Yu, and Xiao-Ming Wu. Forever: Forgetting surve-inspired memory replay for language model continual learning. arXiv preprint arXiv:2601.03938, 2026.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmmente generation for large language models: A survey. XrXiv preprint arXiv:2312.10997, 2(1), 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12900 2025.
Junqing He, Kunhao Pan, Xiaoqun Dong. Zhuoyang Song, Yibo Liu, Qianguo Sun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. Never lost in the middle: Mastering long- context question answering with position-agnostic decompositional training. arXiv preprint arXiv:2311.09198, 2023.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In COLING pp. 6609-6625. International Committee on Computational Linguistics, 2020.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 1866, 2024.
Jie Hui, Wenyu Mi, Jianji Wang, Yuanyang Cao, Ziyi Zhou, and Nanning Zheng. Afsiformer: Adap- tive frequency-spate interaction attention mechanism for aerial image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing, 63:1-19, 2025. doi: 10. 109//GR. 2025.3599214.
Jie Hui, Zhenxiang Zhang, Wenyu Mi, and Jianji Wang. Enabling true global perception in state space models for visual tasks. In The Fourteenth International Conference on Learning Repre- sentations, 2026. URL https:
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, April 2021.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yo Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: few-shot learning with retrieval augmented language models. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmeened generation research. CoRR, abs/2405. 13576, 2024.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang, Ziwen Wang, Huaxuan Ding, Zhuo Cheng, Wenhao Cao, Zhiyuan Feng, et al. Hssbench: Benchmarking humanities and social sciences ability for multimodal large language models. arXiv preprint arXiv:2506.03922. 2025.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S.
H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781. Association for Computational Linguistics, 2020.
Jaebyung Kim, Jachyun Nam, Sangwoo Mo, Jongjin Park, Sang- Woo Lee, Minjoon Seo, Jung- Woo Ha, and Jinwoo Shin. Sure: Summarizing retrievals using answer candidates for open-domain qa of llms. In International Conference on Representation Learning, volume 2024, pp. 15213- 15245, 2024a.
12

---

Published as a conference paper at ICLR 2026
Jacbyung Kim, Jachyun Nam, Sangwoo Mo, Jongjin Park, Sang- Woo Lee, Minjoon Seo, Jung- Woo Ha, and Jinwoo Shin. Sure: Summarizing retrievals using answer candidates for open open-domain qa of llms. arXiv preprint arXiv:2404.1308 2024b.
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 10879-10899, August 2024.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In Advances in Neural 2023.
Information Processing Systems, volume 36, pp.
Bo Liu, Li-Ming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. How good are Ilms at out-of-distribution detection? In Proceedings of the 2024 Joint International Conference on Com- putational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 8211- 8222, 2024.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023.
Zihang Liu, Jiawei Guo, Hao Zhang, Hongyang Chen, Jiajun Bu, and Haishuai Wang. Long-form hallucination detection with self-elicitation. Findings of the Association for Computational Linguistics: ACL 2025, pp. 4082-4100, July 2025a.
Zihang Liu, Jiawei Guo, Hao Zhang, Hongyang Chen, Jiajun Bu, and Haishuai Wang. Long-form hallucination detection with self-elicitation. In Findings of the Association for Computational Linguistics; ACL 2025, pp. 4082-4100, 2025b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yan- song Tang, and Weizhu Chen. Arena learning: Build data flywheel for lims post-training via simulated chatbot arena. arXiv preprint arXiv:2407. 10627, 2024.
Haipeng Luo, Huawen Feng, Qingfeng Sun, Can Xu, Kai Zheng, Yufei Wang, Tao Yang, Han Hu, Yansong Tang, and Di Wang. Agentmath: Empowering mathematical reasoning for large language models via tool-augmented agent. arXiv preprint arXiv:2512.20745 2025.
Qitan Lv, Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu, Coarse-to-fine high- lighting: Reducing knowledge hallucination in large language models, 2024. URL https:
Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, and Yixuan Li. Steer Ilm latents for hallucination detection. In Forty-second International Conference on Machine Learning, ICML 2025, 2025.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In EMNLP (Findings), pp. 5687- 5711. Association for Computational Linguistics, 2023.
Chengwei Qin, Wenxuan Zhou, Karthik Abinav Sankararaman, Nanshu Wang, Tengyu Xu, Alexan- der Radovie, Eryk Hclenowski, Arya Talebzadch, Aditya Tayade, Sinong Wang, Shafiq Joty, Han Fang, and Hao Ma. Learning auxiliary tasks improves reference-free hallucination detection in open-domain long-form generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1173-1 182, July 2025.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang. Nan Duan, and Weizhu Chen. Enhancing retrieeal-augmennted large language models with iterative retrreval-generatioon synergy. In EMNLP (Findings), pp. 2248-9274- Association for Computational Linguistics, 2023a.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang. Nan Duan, and Weizhu Chen. Enhancing attrivall-uugeente large language models with iterative retrieval-gence generattinnn synergy. In Find- ings of the Association for Computational Linguistics: EMNLP 2023, pp. 9248-9271 December 2023b.
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint mxXi:22402.033000 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. CoRR, abs/2409. 19256, 2024.
Guangyuan Shi, Zexin Lu, Xiaoyu Dong, Wenlong Zhang, Xuanyu Zhang, Yujie Feng, and Xiao- Ming Wu. Understanding layer significance in Ilm alignment. arXiv preprint Std1:24100178773 2024a.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval -uggeette black-box language models. In Proceedings of the 2024 Conference of the North American Chupter of the Association for Com- putational Linguistics: Human Language Technologies (Volume Lonng Papers), pp. 8371-8384, June 2024b.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2021. httphh/sss//aii..r////////aa//
2104. 09864
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguisties, 10:539-554, 2022.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with cfain-of-thought reasoning for knowledge-intensive multi-step questions. In (C), pp. 10014-10037. Association for Computational Linguistics, 2023a.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving trieval with thouugtt reasoning for knowledge-intente multi-step questions. In Pro- ceedings of the st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 10014-10037, July 2023b.
Denny Vrandecic and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Commun. ACM, 57(10):78 85, 2014.
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214. 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by supervss contrastive pre-training. CORR, abs/2212.035333, 2022.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.05313 2024.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, et al. Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:80756 008277, 2024.
Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao WU, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Dettoolchain: A new prompting paradigm to unleash detection ability of milm. In European Conference on Computer Vision, pp. 164-182. Springer, 2024.
Yixuan Wu, Yang Zhang, Jian Wu, Philip Torr, and Jindong Gu. Postalign: Multimodal grounding as a corrective lens for milms. arXiv preprint arXiv:2506.1790 2025.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval- augmented Ims with compression and selective augmentation. arXiv preprint arXi::23/0.04408, 2023.
14

---

Published as a conference paper at ICLR 2026
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval- -uumeetee Ims with context compression and selective augmentation. In International Conference on Representation Learning. volume 2024, pp. 43478 43502, 2024.
Yongxin Xu, Ruizhe Zhang, Xinke Jiang, Yujie Feng, Yuzhen Xiao, Xinyu Ma, Runchuan Zhu, Xu Chu, Junfeng Zhao, and Yasha Wang. Parenting: Optimizing knowledge selection of retrieval- augmented language models with parameter decoupling and tailored tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 11643-11662, 2025.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pp. 2369-2380. Association for Computational Linguistics, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023. URL
org/abs/2210, 03622
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval- generation for large language models. arXiv preprint arXiv:2411.19443, 2024a.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonmmous wal-rievllllaggentee generation for hrtp:///orrabb///1..... 19443
large language models, 2024b.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang. and Dong Yu.
Chain-of-note: Enhancing robustness in retrieeal-augmented language models. arXiv preprint arXiv:2311.09211 2023.
Li-Ming Zhan, LIU Bo, Zexin Lu, Yujie Feng, Chengqiang Xie, Jiannong Cao. and Xiao-Ming Wu. Deal: Disentangling transformer head activations for Ilm steering. In Knowledgeable Foundation Models at ACL 2025, 2025.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, pp. 1-46, 2025.
Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang, et Found in the middle: How language models use long contexts better via aad--la positional encoding, Advances in Neural Information Processing Systems, 37: 10755-60777 2024.
Yang Zhong and Diane Litman. Discourse-driven evaluation: Unveiling factual inconsistency in long document summarization. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2050-2073, April 2025.
Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wa, Yu Cheng, and Hehe Fan. Dropping experts, recombining neurons: Retraining- pruning for sparse mixture- of-experts 1lms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 15169-15186, 2025.
Yixiao Zhou, Yang Li, Dongzhou Cheng, Hehe Fan, and Yu Cheng. Look inward to explore out- ward: Learning temperature policy from Ilm internal states via hierarchical rl. arXiv preprint arXiv:2602.13035, 2026.
A USE OF LARGE LANGUAGE MODELS (LLMs)
In preparing this work, we used large language models (LLMs) solely for improving the clarity and readability of the writing. Specifically, LLMs were employed as an assistant for language polish- ing, grammar checking, and style refinement. All research ideas, methodology design, experiments, analyses, and conclusions were conceived, implemented, and validated entirely by the authors with- out the involvement of L.Mss. We take full responsibility for the content of this paper.
15

---

Published as a conference paper at ICLR 2026
B A POSITIONAL ENCODING PERSPECTIVE ON KEY-INFORMATION PROXIMITY
Empirical studies have shown that large language models often struggle to effectively use informa- tion located far from the prediction site, a phenomenon sometimes referred to as "lost in the middle" (Liu et al. 2023)- As shown in Figure the accuracy of GPT-3.5 QA tasks decreases markedly when the answer- answer-bearing document is positioned in the middle of the context. This observation underscores that the proximity of key information output tokens plays critical role in ensuring factual reliability (Dong et al., 2024) |024) (Zoou et al. (0026). Motivated by this observation, while the po- sition of early imputs largely fixed, we propose actively adjust the placement of critical evidence closer to the output tokens, which directly inspired the design of our micro retrieval mechanism.
20 Total Retrieved Documents (~4K tokens)
75
70
Accuracy
60
55
1st
5th
10th
15th
20th
Position of Document with the Answer
gpt-3.5-turbo-0613
gpt-3.5-turbo-0613 (closed-book)
Figure 4: Effeet of answer position model accuracy (figure taken from Liu et al. (2023). Accu- racy declines sharply as the mnwerr-bearin evidence appears in the middle of the context.
We also provide theoretical explanation of this effect from a positional encoding perspective.
RoPE and Relative Position Encoding. Rotary Position Embeddings (RoPE) (Su al. 2021 encode relative positions by rotating query and key vectors in the complex plane. For a query at position m and key at position n, the inner product is:
Pm Ra.m W am = Ra,ml ki = Ro.n WK.K = Ro,nki
(7)
On hnp = (Ro. (Rg,nk) = q Romm-nki
(8)
where Ro,m is block diagonal rotation matrix with components
cos(m):)) - sin(mo:)] R sin(m
cos(m)))
0, = b
(9)
Spectral Decomposition. with distinct frequencies 0;:
The dot product be decomposed into d/2 sinusoidal components
d/2-1
Qm knn
Σ
(g2shir (0,)) + q244 k2041 sin( (0;));
(10)
where A is the relative distance. High-frequeng components oscillate rapidly and cancel out when is large, while low- oowfrrquency components dominate at shorter distances. This spectral bias makes attention contributions stronger for nearby tokens than for distant ones.
16

---

Published as a conference paper at ICLR 2026
Table 5: Prompt for L.M--as-a-Judee
Prompt Template
You will be given a question and its ground truth answer list where each item can be ground truth answer. Provided a pred answer, you need judge the pred answer correctly answers the question based on the ground truth answer list. You should first give your rationale for the judgement, and then give your judgement result (i.e., correct or incorrect..
Here is the criteria for the judgement:
1. The pred_ane answer doesn't 't need to be exactly the same any of the ground truth answers, but should be semantically same for the question.
2. Each item in the ground truth answer list can be viewed ground truth answer for the question, and the pred answer should semantically same to at least one of them. question: {question)
ground truth answers: [gt_answer) rre__answer: prre__anwwer)
The output should in the following json format: 'json
"rationale": "your rationale for the judgement, as a text", "judgement": "your judgement result, can only be 'correct or 'incoreet
Your output:
Proposition B.1. For RoPE-based attention, the expected contribution of evidence tokens decreases monotonically with their distance to the output position. Hence, key information placed closer the outputs more likely be faithfully incorporated into generation, providing theoretical justification for the effectiveness of key-information proximity.
Discussion. This analysis shows that proximity is not only empirical observation but also theoretical consequence of how positional encoding interacts with attention. Placing key information closer output tokens mitigates the risk of dilution by redundant context and reduces the chance of being forgotten in long reasoning chains. This provides a formal foundation for the design of our micro-macro retrieval framework, which explicitly manages evidence placement improve factual consistency.
C PROMPT FOR LLM-AS-A-JUDE
Table 5 presents the exact prompt we used to evaluate model responses under the LLM-as-a-Judge setting, ensuring consistency and reproducibility of the evaluation process.
D IMPLEMENTATION DETAILS
Implementation Details. We build our reinforcement learning framework upon verl (Sheng et al. 2024 Zhou et al., 2025). For training. we use the MuSiQue dataset, restricting to the training split (19,988 samples), and train the models for two epochs. The retrieval environment is imple- mented with FlashRAG (Jin et 2024 Hui al. 2025), a standard toolkit for sxtrieall-augmente generation. Following ReSearch, we adopt b5-base-v2 (Wang et al., 2022) as the dense retriever and use the December 2018 Wikipedia snapshot as the underlying knowledge base (Karpukhin
let al. 2020 Ding et al. 2024
2025). All document embeddings and indexes are preprocessed
2024 Hui al. 2026). During both training and evaluation rollouts, retrieve the top-3 passages for each query. For baseline systems, we directly use the implementations
by FlashRAG (Luo al.
provided by FlashRAG ensure fairness. In Eq. set and B To these values were
17

---

Published as a conference paper at ICLR 2026
found to provide good balance between final answer correctness, key information preservation and consistency after empirical validation in preliminary experiments (Kang et al. 2025).
To further improve reproducibility and transparency, provide additional implementation details regarding hardware, environment configuration, and experimental settings. These specifications will also be included in the publicly released code.
Hardware Requirements. All models were trained on A10000 40GB GPUs. All inference ex- periments were conducted on 4x 100 40GB GPUs using the SGI ang serving framework eWu 2025).
Random Seeds. All experiments were run with a fixed random seed of 42 to ensure determinism and reproducibility where possible.
These details ensure that future researchers can reliably reproduce both the training and inference pipelines of M We show some important parameter settings during training in Table
Table 6: Implementation details of M
Parameter
Value
Learning Rate
le-6
Train Batch Size
256
Number of Training Epochs
Number of Rollout
5
Rollout Temperature
1.0
KL. Loss Coefficient
0.001
Clip Ratio
0.2
E ADDITIONAL EXPERIMENTS
E. EXTENDED MODEL FAMILIES AND MULTI-QUESTION REASONING BENCIMARRS
To further assess the empirical coverage of M we conduct additional experiments on (1) larger and different model families, and (2) more challenging long-form settings. These results comple- ment the main experiments and address concerns regarding generalizability and training sufficiency.
Different Model Families. We evaluate M²R on two additional models, Llama-3. 1.1-a-3.1-BB- and Fiitrall-B--nstrruc As shown in Table 7 R consistently outperforms ReSearch across both model families, with an average improvement of 1.03%.
Long-Form Multi-Question Benchmarks. To evaluate the effectiveness of M under longer reasoning chains, we extend each dataset by concatenating multiple questions (3Q and 5Q). As shown in Table g. M²R achieves the largest gains under these extended settings, validating the benefit of micro retrieval in maintaining localized evidence for deeper reasoning.
These additional results demonstrate that M²R generalizes robustly across model families and re- mains effective under substantially longer reasoning chains.
E.2 ADDITIONAL ANALYSIS OF FLASHRAG CONFIGURATION AND RETRIEVAL ABLATIONS
This section provides additional details of the FlashRAG configuration, ablations on retrieval hyper- parameters, and token statistics during inference. These analyses complement the main results and demonstrate that M is robust to retrieval settings.
FlashRAG Configuration. To ensure fair comparison and avoid introducing retrieval-side advan- tages, strictly follow the official FlashRAG and Re-Search configuration without modification:
18

---

Published as a conference paper at ICLR 2026
Method
HotpotQA
2Wiki
MuSiQue
Bamboogle
Naive Generation
18.1
24.7
4.4
11.6
Naive RAG
34.2
COFT
38.4
39.5
15.8
32.5
SURE
39.3
42.0
18.7
37.8
ReSearch
42.2
45.8
20.9
43.1
M²R
43.0
47.2
22.1
42.9
Mistral-7B-Instruue
Method
HotpotQA
2Wiki
MuSiQue
Bamboogle
Naive Generation
21.9
28.8
7.7
12.5
Naive RAG
34.5
31.2
COFT
43.5
45.5
18.5
40.3
SURE
42.6
47.7
21.2
42.8
ReSearch
45.0
49.1
23.7
45.5
M'R
45.6
50.0
25.5
46.0
Table 7: Exact Match results on additional model families.
Method
HotporQA-3Q
2Wiki-3Q
MuSiQue-3Q
Bamboogle-3Q
Naive Generation
13.1
17.5
4.3
7.1
Naive RAG
20.6
16.9
5.1
15.2
COFT
24.5
27.2
12.6
22.0
SURE
28.3
31.5
11.3
25.5
ReSearch
30.2
33,9
14.2
28.0
M'R
32.0
35.8
17.9
30.6
Method
HotpoiQA-5Q
2Wiki-5Q
MuSiQue-5Q
Bamboogle-5Q
Naive Generation
5.5
4.5
0.7
1.8
Naive RAG
8.2
7.7
2.3
4.5
COFT
9.5
11.1
3.5
SURE
13.1
14.8
ReSearch
13.9
17.0
5.7
12.8
M²R
15.4
18.5
8.4
14.9
Table 8: Performance under multi-question reasoning (3Q and 5Q).
Knowledge Base: FlashRAG's December 2018 Wikipedia snapshot.
Chunk Size: ~100-word passages (default).
Retriever: E5-base-v2 dense retriever (Feng et al. 2025a).
This ensures that improvements from M stem from its generation-side retrieval mechanism rather than retrieval tuning.
Ablations on Retrieval Settings. We ablate two key FlashRAG parameters--rttiree retrieve--oo--- and chunk size-on 2Wiki using Qwen2.5-3B. As shown in Table M²R consistently outperforms ReSearch across all configurations, demonstrating strong robustness to retrieval hyperparameters.
Token Statistics During Inference. To analyze whether M alleviates long-form reasoning constraints, report input and output token statistics in Table 10 "Input Tokens" repre- sent question tokens only, whereas "Output Tokens" include both reasoning chains and final answers (excluding retrieved passages). Overall, these analyses indicate that M is robust to retrieval configurations and benefits long-form reasoning by injecting concise, model gonel-ggeerated key information near the answer generation step.
19

---

Published as a conference paper at ICLR 2026
Setrieve-Top--
Naive RAG
ReSearch
M²R
13.5
37.2
38.3
5 (default)
13.8
38.9
40.1
8
13.6
38.0
39.4
Chunk Size
Naive RAG
ReSearch
M
50
13.4
38.1
39.4
100 (default)
13,8
38.9
40.1
150
13.9
38.4
39.7
Table 9: Ablations on retrieve-top-k and chunk size for FlashRAG.
Dataset
Input Tokens
Output Tokens (ReSearch)
Output Tokens (M'R) R)
HotpotQA 25
416
432
MuSiQue
31
483
505
2Wiki
37
440
478
Bamboogle 21
376
389
Table 10: Token statisties during inference. M-R produces slightly longer outputs due micro
retrieval, but the inserted key facts are compact and answer-aligned, improving grounding and final accuracy.
E.3 INFERENCE COST AND STORAGE ANALYSIS
We provide additional analysis of the inference latency and storage cost of the key-information repository in M These results complement the main experiments and demonstrate that the micro- macro retrieval pipeline introduces only minimal overhead.
Inference Latency. The macro- macro-retrieval stage M²R follows the same workflow as standard RAG (Zhan et al. 2025 Feng et al. 2026), and the key-information saving step stores only handful of ansner-aligned facts (typically 3-10 items), making its cost negligible. Micro retrieval is also lightweight, as it performs a simple dictionary-style lookup over small local repository.
To quantify the overhead, we report real inference time using Qwen2.5-3B with SGL on 4 A100 40GB GPUs. As shown in Table 11 MPR increases inference time by less than 10% on average compared to ReSearch, while offering substantially larger accuracy improvements.
Inference Time (s)
HotpotQA
2Wiki
MuSiQue
Bamboogle
ReSearch M'R
Rr4.3
44.8
R26.3 206.8
R,4.2
#44.7
R,5.2
4.6.
Table 11: Measured inference latency of M compared to ReSearch.
Scaling With Input Complexity. We also evaluate multi-question setting by concatenating 2-3 HotpotQA questions into a single input [2) Latency grows approximately linearly with reasoning complexity, consistent with multi-turn tool-use systems. Overall, these results show that M introduces only minimal overhead beyond standard RAG systems. The micro-maero retrieval pipeline remains efficient even under long-form reasoning.
Storage Cost of the Key-mnnormntion Repository. The key-information repository stores a very small number of atomic facts produced during the <think> phase. Table 13 reports the measured token counts, Across all datasets, the repository remains small (50-150 tokens), which is negligible compared with the retrieved passages themselves.
Overall, these results show that M introduces minimal inference overhead and negligible storage cost, while providing substantial improvements in grounding and answer correctness.
20

---

Published as a conference paper at ICLR 2026
Setting
Avg Invocations
Inference Time (s)
1Q
7.6
66.8
2Q
13.8
14.1
3Q
19.7
x22.3
Table 12: Latency scaling under multi-question reasoning.
Dataset
Avg Tokens
Min
Max
HotpoIQA
62
108
2Wiki
73
28
121
MuSiQue
88
32
139
Bamboogle
55
18
95
Table 13: Size of the key-information repository measured tokens.
E.4 ABLATION STUDY: IMPORTANCE OF CURRICULUM LEARNING
To further analyze the role of curriculum learning in M*R, we compare our two-stage training strat- egy with direct joint optimization of macro and micro retrieval. This experiment complements the main results and provides insight into training stability and optimization difficulty.
Accuracy Comparison. Table 14 reports the Exact Match scores under three training strategies. Direct optimization performs poorly across all datasets- even worse than Naive RAG- because the model must simultancously learn macro retrieval, key-information saving, and micro retrieval. This substantially increases optimization difficulty and prevents the model from obtaining meaningful rewards.
Training Strategy
HotpotQA
2Wiki
MuSiQue
Bamboogle
Naive RAG
21.0
13.8
4.1
14.3
Direct Optimization
13.2
8.4
3.9
10.8
Curriculum Learning (ours)
38.7
40.1
20.9
39.6
Table 14: Comparison of training strategies, showing the importance curriculum learning.
Discussion. In contrast to direct optimization, curriculum learning decomposes training into two tractable stages, allowing the model first master macro retrieval before learning micro retrieval. This staged formulation dramatically stabilizes training, reduces reward sparsity, and yields signifi- cantly better end-to-end performance. These findings validate the necessity of curriculum learning in effectively training M²R.
F
CASE STUDY
Table 15
presents simplified case study drawn from the evaluation set.
G PROMPT TEMPLATE FOR M²R
The complete prompt template is shown in Table 16
21

---

Published as a conference paper at ICLR 2026
Table 15: A simplified case study of MRR on a test instance.
Question: Guest 101 wants to book any available suite room and needs to know both the total price after VIP discount and the room number with the lowest payable amount.
Answer: 180.0, Room
<think> First, need to find an available suite room. Then will calculate the price and apply the VIP discount for Guest 101. </think> seacro_tool_cat
"name": aeet__vvallble_p_
"room type": "suite"
</macro_tool_call>>>>>>
<macro amacro_result>> "number": 301, "type": "suite", "price": 200.0, "avail- able": True; "number": 302, "type": "suite", "price": 220.0, "available": True </macro_results
<think> Rooms 301 and 302 are available. Their prices are 200.0 and 220.0, re- spectively. With a 10% VIP discount, the payable amounts would be 180.0 and 198.0. </think>
smacro_tocl_calll
"name": "get_guest, vip_status", "guest_id" id": 101
</macro_tool_cal>>
<macro_result>>> True s/macro_resul
<think> Guest 101 is confirmed as VIP. The lowest final payable amount 180.0
for Room 301. </think>
besy__nfo_savee_
"fnnaPayableeAmount: "180.0", "RoomNumber": "301"
likey_info_saver
<think> I have obtained both the final payable amount and the room number with the lowest cost. will now proceed to the answering phase. </think>
<answer>
Saicrr_ttol_calll
"query":
"fnalPayyaleAAmount
<mi amiccrorrssposse
The requested final payable amount is \boxed (180.0)
smicroo_ool__aat call> "query": "RoomNumber" </micro_tool_call
"30"
The suite room with the lowest payable amount (301).
</ /nsser>>>>
22

---

Published as a conference paper at ICLR 2026
Table 16: Template for M²R.
Prompt Template For M
You are helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. During the reasoning process, you have access the user query, referred to as macro retrievals. These macro retrievals are enclosed within <macro, tool call></macro, tool_oooooooooo holl_clll> tags. You may conduct multiple rounds of function calls, and in each round, you can call one or more functions.
of tools you
assist with
The results of the macro function calls will be given back to you after execution, and you can continue to call functions until you get the final answer for the user's 's question. Additionally, during the reasoning process, whenever you obtain the answer the user's question, you must store it as key-value pair in key information dictionary using the <key info save></key info save> tag.
The format must strictly follow JSON, e.g.: ("target_value". "value")
The stored key-value pairs must be directly relevant to the final answer.
Finally, once you have obtained the answer and stored the key information, proceed to the answering phase. At this stage, do not call any further functions. Before writing the final answer sentence, you must first perform micro retrieval fetch the answer from the key information dictionary. The final answer must be based only on the results of micro retrieval, rather than answering independently.
Notes for micro retrieval:
Micro retrieval must be enclosed within <micro smicro_tool cooo__cll</m/rrooo
tags.
You may query multiple items at once or issue requests in batches.
The results of micro retrieval will be provided after execution.
If the miero retrieval fails, you must simply state that the result could not be retrieved, and must not fabricate an answer independently.
Every value from the key information dictionary that appears in the final answer must enclosed in \boxed[].
23


Reviews:
[{"review_id": "k4c4m42D23", "reviewer": "Reviewer_Jaoq", "summary": "A \"retrieve-while-generate\" framework that performs macro retrieval (external sources during reasoning) and micro retrieval (from key-information repository during answer generation) to ensure evidence proximity and reduce hallucination.", "questions": "1. How many model invocations does M²R require per query on average?\n\nPlease report separately for: (a) think phase, (b) answer phase, (c) total\nBreak down by dataset (HotpotQA, MuSiQue, etc.)\nWhat is the range (min/max)?\n\n\n\n2. What is the end-to-end latency in realistic deployment?\n\nTable 2 shows batch inference time (0.67s), but what about non-batched API calls?\nAssuming 100ms per forward pass: how long does a typical query take?\nHow does this scale with question complexity?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Strong Empirical MotivationLost in the Middle Phenomenon (Liu et al. 2023):\n\nGPT-3.5 accuracy drops from 75% → 55% when answer-bearing evidence moves from context start to middle (Figure 3)\nEmpirically validated across multiple models and tasks\nTheoretical Grounding (Appendix B):\n\nAnalyzes RoPE positional encoding: attention ∝ q^T R_{θ,m-n} k\nHigh-frequency components cancel at large distances Δ\nFormal claim: \"Evidence contribution decreases monotonically with distance\"\nThis foundation is solid — problem is real and well-documented.\n\n2. Explicit Key Information ManagementKey-Value Repository Design:\n\nAdvantages:\n\n- Explicitly separates \"what to remember\" from \"how to answer\"\n- Forces model to extract atomic facts rather than rely on context attention\n- Reduces cognitive load during answer generation\nThis is cleaner than implicit reasoning traces (e.g., ReSearch where key info is buried in <think> text).", "weaknesses": "1. \"Retrieve-While-Generate\" might be Misleading\n\nWhat the Paper Claims:\n\n\"M²R is the first framework to introduce a retrieve-while-generate paradigm during the answer phase.\"\nThis suggests a novel generation mechanism — e.g., retrieval happening during the forward pass.What Actually Happens:Multi-Turn Generation with Tool Calls:\n\nThis is identical to:\n\nOpenAI's function calling\nAnthropic's tool use\nReAct (Yao et al. 2022)\nSelf-RAG (Asai et al. 2023) — which also retrieves during answer generation!\nM²R requires 5-10 sequential model invocations per query.\n\n2. Cost Analysis Completely Absent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0wUrAma7E", "reviewer": "Reviewer_Sm15", "summary": "This paper tackles hallucination in large language models (LLMs), especially in long-form generation where redundant contexts and extended reasoning amplify factual errors. The authors observe that factual accuracy improves when key information appears closer to the generated output, yet existing retrieval-augmented LMs (RALMs) lack mechanisms to ensure this proximity.\nTo address this, they propose Micro–Macro Retrieval (M2R), a retrieve-while-generate framework combining macro-level external retrieval with micro-level key information reuse from an internal repository built during reasoning. M2R directly maintains evidence proximity to outputs and reduces hallucination in long-context tasks.\nTrained via curriculum-based reinforcement learning with rule-based rewards, M2R achieves consistent gains in factual accuracy and grounding across multiple benchmarks, showing good effectiveness in lengthy-context scenarios.", "questions": "Please see the weaknesses,", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The topic is valuable, especially for mitigating hallucinations in long-form reasoning models.\n\n2. The method is straightforward and conceptually sound.\n\n3. The paper is well written and easy to follow.", "weaknesses": "1. The core methodological details are underspecified. If I understand correctly, the approach hinges on constructing and maintaining a key-information repository, and then:\n\n- (1) For macro retrieval: per Lines 060–061, how is “the reasoning process yields answer-aligned evidence” detected, and how exactly is it inserted into the repository(i.e., what constitutes the key and the value)?\n\n- (2) For micro retrieval: what query is used to retrieve answer-related information from the repository?\n\n- (3) Does GRPO merely teach the LLM when/how to invoke the macro- and micro-retrieval tools, rather than optimizing the retrieved content?\n\n2. Empirical coverage is limited. Training is conducted only on Qwen2.5-3B/7B and evaluated on four relatively simple QA datasets. How does the method perform on other model sizes/families and on more challenging reasoning benchmarks? Moreover, Figure 2 shows substantial reward oscillation and a low mean (~0.4), suggesting training stability or sufficiency may be a concern.\n\n3. FlashRAG details and ablations require clarification, including the knowledge base size, and the effects of chunk size and retrieve number on performance. Reporting token statistics of input/output during inference would further clarify whether the approach practically alleviates long-form reasoning constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pp3iDQ6LT6", "reviewer": "Reviewer_KBYP", "summary": "This paper proposes Micro–Macro Retrieval (M2R), a two-level retrieve-while-generate framework designed to reduce hallucination in long-form generation. By combining macro retrieval of coarse evidence during reasoning with micro retrieval of key information during answer generation, M2R ensures that essential evidence remains proximal to output tokens. Trained with a curriculum-based reinforcement learning strategy using rule-based rewards, the method achieves significant improvements in factual consistency and robustness across multi-hop QA and long-context benchmarks compared to strong RAG baselines.", "questions": "See above, I am warmly welcome to discuss further detailed on this paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The idea of Micro–Macro Retrieval surprisingly natural and well-motivated — it addresses one of the most persistent issues in RAG systems (long-form hallucination) with a solution that feels both principled and minimal. The “retrieve-while-generate” framing elegantly captures how reasoning and retrieval should co-evolve.\n\n2. The paper is very carefully written. I particularly like how the authors formalize the two retrieval levels and the transition between \\<macro_tool_call>, \\<key_info_save>, and \\<micro_tool_call>. It feels like reading a well-designed system that could actually be implemented in production without hidden tricks.\n\n3. I really appreciate that the system is interpretable by design: it shows that we can literally see the reasoning flow: what it retrieved, what it saved, what it reused. That’s a refreshing contrast to the black-box nature of most retrieval-augmented models. It also feels cognitively aligned with how humans solve tasks: note things down, then recall them precisely.\n\n4. Compared with Self-RAG[1] this work feels like a thoughtful continuation rather than simple imitation. Self-RAG let the model decide when to retrieve; M2R turns that spark into a full reasoning routine. It not only detects when retrieval is needed but also explicitly manages what to keep and reuse, maintaining a long-term internal memory grounded in already verified facts. I find this progression deeply satisfying—the model isn’t merely “asking for help” anymore; it’s learning to remember what it already knows to be true. That evolution from reactive retrieval to proactive self-memory feels like a genuine step forward.\n\n[1] Asai, Akari, et al. \"Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.\" The Twelfth International Conference on Learning Representations.", "weaknesses": "I think this paper has no particularly obvious weaknesses. Unlike the naïve combination of RAG with RL or GRPO, this work takes a much more principled approach—from memory to retrieval to the overall training strategy, making it a significant step forward in improving RAG performance. However, I do have a few questions that I’d like to raise briefly.\n\n1. In your gradient computation implementation, how did you mask out the information from the retrieval part? Is the positional information handled relative to the retrieval step, or do you directly remove the masked portion?\n2. When invoking retrieval, are macro-retrieval and micro-retrieval mutually exclusive, or can they be used jointly?\n3. When is **key_info_save** called? In your experiments, is it triggered only when the macro retrieval is considered contextually relevant, and thus the macro information is saved? If there are identical or similar **micro_tool_calls**, does the model jointly retrieve them?\n4. If the model chooses to invoke **micro**, but there is no stored memory or relevant content, does that lead to hallucination? How do you handle cases where micro-retrieval fails or retrieves incorrect results?\n5. Are there situations where neither **micro** nor **macro** is used?\n6. What are the overall training costs and latency characteristics? If the database is large or the query is long, does it lead to bottlenecks?\n7. If provides a main figure, it will help more readers to fastly grasp your methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wK0EbahLpA", "reviewer": "Reviewer_vzvj", "summary": "This paper aims to reduce LLM hallucinations in long-context settings, especially in the context of RAG.\nThe key limitation of current RAG approaches is that, either the long context makes it hard for the model to accurately identify key info, or model fails to retrieve intermediate results from its own reasoning chains.\nTo address this limitation, the paper introduces **micro-macro retrieval (M$^2$R)**, a retrieve-while-generate method.\nM$^2$R hinges on the positive correlation between key information proximity to model outputs and factual accuracy, which was reported previous works.\nM$^2$R directly enforces this proximity mechanism into the LLM through curriculum-learning with GRPO, such that *macro retrieval* (also `<think>` phase) implements traditional RAG and maintains a key info repo while *micro retrieval* (also `<answer>` phase) extracts key info from the established repo to ground model outputs.\n\nExperiment evaluation on multi-hop QA benchmarks show that M$^2$R generally outperforms most baselines and its comparative advantage is more evident in challenging scenarios (HotpotQA-2/3Q).", "questions": "**Major questions (that could affect rating)**\n1. **Question 1**: Limited model family\n\n   M$^2$R is only tested on two models of different sizes (3B, 7B) but the same model family (Qwen2.5). This limitation does *not* directly undermine the central claims of the paper, but results on more diverse model families could greatly enhance the general utility of the proposed method.\n2. **Question 2**: Cost analysis\n\n   Inference efficiency is a critical concern in RAG applications. The proposed method requires maintaining a key info repo during macro retrieval and retrieving key infos during micro retrieval. Therefore, a natural concern arises as to whether the performance boost is worth the cost:\n\n   *How does this micro-macro framework affect inference latency, and what are the storage costs of the key info repo?*\n\n   A detailed analysis (either theoretical or empirical) could be useful in deciding whether M$^2$R is usable in practice.\n\n**Minor questions and suggestions (that are not considered to affect rating)**\n1. **Minor question 1**: Additional details for reproducibility\n\n   Although the authors have provided source code and some experiment details in the paper, additional details such as hardware requirements and seeds could be useful for reproductions of results.\n2. **Suggestion 1**: Notations\n\n   The method name, M$^2$R, is not consistently presented: it is usually written in normal font but sometimes written in italics (lines 206-261, 267, etc).\n3. **Suggestion 2**: Paper organization\n\n   Related Work section could help readers set the context, but currently it is placed in the Appendix. Therefore I recommend move Table 1 (M$^2$R prompt template) to the Appendix and move Related Work section to the main body instead.\n4. **Suggestion 3**: Details on training stability\n\n   The paper mentions at lines 242-243 that, directly optimizing macro/micro retrieval leads to poor convergence. Detailed results (preferably placed in the Appendix) could help future researchers gain in-depth understandings regarding the significance of curriculum learning.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Originality\n\n   This paper is novel in that it alleviates hallucinations in RAG based directly on empirical insights from [1-2] that key information position methods in long-form generation.\n   The application of GRPO + curriculum learning to realize the insight above also makes sense.\n2. Quality\n\n   The experiment design is sound and directly supports central claims of this paper.\n   The authors have also done an excellent job by releasing the source code with clear documentation.\n3. Clarity\n\n   The paper clearly explains its motivation, key insights (lines 53-55, Appendix B), as well as discussions on the rationale of key designs (lines 120-122, 136-137, 161, 249-256, etc).\n   The case study is useful for readers to understand the method intuitively.\n4. Significance\n\n   This paper could be of significance to the field by grounding in empirical findings of previous works, combining GRPO and curriculum learning and achieve a compelling performance boost in challenging multi-hop QA tasks.\n\nReferences:\n\n[1] Lost in the middle: How language models use long contexts. (2023)  \n[2] Found in the middle: How language models use long contexts better via plug-and-play positional encoding. (2024)", "weaknesses": "(Authors do **not** need to refer to points raised in this section since the main points are already mentioned in *\"Questions\" section*.)\n1. W1: Limited model family\n\n   The paper only involves experiments on Qwen-2.5-3B/7B models; results on more diverse model families could strengthen the paper's arguments.\n2. W2: Missing discussions on costs\n\n   Since the core method introduces additional storage requirements (key info repo) and requires retrieving key info during answer phase, there are concerns regarding whether these components induce heavy storage/time costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)