MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAG
Authors: Xuebing Liu
Shanbao Qiao
Roseline Nyange
Dongwook Min
Hyun Kim
Seung-Hoon Na
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-hop reasoning
Knowledge enhancement
Retrieval-augmented generation
Hypernetwork-based expert generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large language models (LLMs) can be enhanced with external knowledge through two dominant approaches: (1) **retrieval-augmented generation (RAG)**, which supplements LLMs with in-context retrieved passages, and (2) **parametric knowledge adaptation (PKA)**, which directly updates model parameters with new domain knowledge. Recently, parametric RAG (PRAG) has emerged as a promising framework, extending RAG by translating retrieved passages into parameter updates, thereby mitigating inefficiency and noise sensitivity inherent to RAG. However, existing PRAG methods remain limited to single-pass retrieval, falling short of the **multi-hop RAG** setting that requires iterative retrieval and reasoning. 
We propose **MergePRAG**(*Orthogonal Merging of Passage-experts for Multi-hop PRAG*), a novel framework that sequentially integrates retrieved passages into LLM parameters through a continual merging mechanism, which is advanced by two key proposals: (1) **orthogonal merging** using the Gram–Schmidt process to minimize conflicts between "passage experts", and (2) **critical-layer parameterization** to efficiently encode in-context passages. Experiments on multi-hop open-domain QA and reasoning-aware knowledge editing show that MergePRAG consistently outperforms both standard and state-of-the-art RAGs as well as existing parametric adaptation methods, achieving superior effectiveness and efficiency. All datasets and code will be released at https://github.com/Liu-Xuebing/MhQA_hypernetwork.

Published as a conference paper at ICLR 2026
MERGEPRAG: ORTHOGONAL MERGING OF PASSAGE- EXPERTS FOR MULTI-HOP PARAMETRIC RAG
Xuebing Liu', Shanbao Qiao', Roseline Nyange', Dongwook Min". Hyun Kim", K Seung- Na
DDeaarmment of Computer Science and Artificial Intelligence, Jeonbuk National University Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology Ellctronicc and Telecommunications Research Institute
(liuxuebing, joe, roselinenyange jbnu. ac.k. mi ndd9600n st..s...... ac. h. hikimeetrs .eeeetireeeeeeee.... re.kr, nash@unist. ac .c.
ABSTRACT
Large language models (LLMs) can be enhanced with external knowledge through two dominant approaches: (1) auumenel-augmeented generation (RAG), which supplements LLMs with in-context retrieved passages, and (2) parametric knowledge adaptation (PKA), which directly updates model parameters with new domain knowledge. Recently, parametric RAG (PRAG) has emerged a promising framework, extending RAG by translating retrieved passages into parameter updates, thereby mitigating inefficiency and noise sensitivity inher- ent to RAG. However, existing PRAG methods remain limited to single-pass re- trieval, falling short of the multi-hop RAG setting that requires iterative retrieval and reasoning. We propose MergePRAG (Orthogonal Merging of Passage- experts for Multi-hop PRAG), novel framework that sequentially integrates re- trieved passages into LLM parameters through a continual merging mechanism, which advanced by two key proposals: (1) orthogonal merging using the Gram-Schmidt process to minimize conflicts between "passage experts", and (2) critical-layer parameterization to efficiently encode in-context passages. Ex- periments multi-hop open-domain QA and reasoning aware knowledge edit- ing show that MergePRAG consistently outperforms both standard and state-of- the-art RAGG well existing parametric adaptation methods, achieving perior effectiveness and efficiency. All datasets and code will be released at attps://githur: com Liu welluu-uebinnnnnne /MhOA hypernetwork
INTRODUCTION
Large language models (DuMs) al. 2024 Mesnard et al. 2024 Team 2024 DeepSeek- AI 2024) have achieved strong performance on a wide range of knowledge-intensive tasks, driven by billions of parameters and large-scale pretraining corpora. However, their parametric knowl- edge remains static, making them ill-suited for evolving world knowledge or emerging domains. Retrieval-augmentee generation (RAG) has become popular remedy, injecting retrieved passages into the input context at inference time. While effective, RAG faces challenging issues: (1) knowl-
edge conflict between parametric and retrieved information (Xie et al., 2023 Kortukov al. 2024
Zhang et al. 22255 Bi et al. 2025), (2) inference inefficiency from processing long retrieval-hean
contexts (Leng et 2204: Jin al. 2024 Chen al. (2025), and (3) noise sensitivity, where ir-
relevant or erroneous passages degrade performance (Cuconasu et al. 2024 Wu et al. 2024
Fang
al. 2024).
Alternatively, Parametric RAG (PRAG), along with its dynamic variant (Su et al. 2025 Tan et al.) 2025a), has recently emerged as a promising direction. PRAG translates retrieved passages into LoRA parameter updates via "hypernetwork", enabling L.LMs to internalize external knowledge
"Corresponding author

---

Published as a conference paper at ICLR 2026
beyond mere in-context conditioning Notably, PRAG has been shown consistently outperform standard RAG, both when applied independently and when combined with retrieval- -ased methods.
Despite its promise. PRAG has thus far been investigated only in simplified RAG settings, typ- ically limited to a single retrieval step rather than the more challenging multi-hop RAG scenario Yuu et al., 2024b Li al. 20256). In multi-hop RAG, complex query is decomposed into sub- questions, each requiring iterative retrieval and sub-answer generation, such that retrieved passages are incrementally provided during the question answering (QA) process. A central research ques- tion, therefore, is how to effectively extend PRAG to multi-hop seltings- the internalization of retrieved passages must continuously progress across hops without necessitating the retraining or rebuilding of hypernetwork originally designed for single-hop RAG. This extension of PRAG to multi-hop RAG represents an important milestone, as provides a natural bridge toward recent reasoning- erasoning-enhanced RAG frameworks (e.g.. IRCo'T, Self-RAG, DeepRAG, and RAG-R1 (Trivedi
et al. 2023 Asai et al.
2024 Guan et al.
2025 Tan et al. 20256))
We propose MergePRAG (Orthogonal Merging Passage-experts for Multi-hop PRAG), general- ized framework that scales PRAG to multi-hop RAG. At each stage, retrieved passages are trans- lated into expert parameters by hypernetwork and merged with the previously accumulated ex- perts through a continual merging mechanism. thus enabling effective accumulation of knowledge across iterative retrievals (Figure D. For effective continual merging, (1) orthogonal merging using the Gram-Schmidt process to minimize conflicts between newly in- troduced and existing experts by eliminating redundant information and promoting complementary, non-overlapping representations, and (2) a critical-layer parameterization module that updates only the preselected critical layer to efficiently encode in-context passages, while keeping the re- maining layers frozen to reduce computation and stabilize reasoning. Together, these techniques allow MergePRAG to reuse single passage-level hypernetwork across hops, avoiding the redesign or retraining of additional hypernetworks to support multi-hop RAG.
propose two advances:
Our contributions are threefold: (1) We introduce MergePRAG, the first generalized PRAG frame- work for multi-hop RAG. (2) We propose continual merging mechanism that sequentially inte- grates retrieved passages into LLM parameters, enabled by two advances: orthogonal merging and critical-layer parameterization. (3) We conduct extensive experiments across multiple LLM back- bones and benchmark datasets, showing that MergePRAG consistently outperforms existing RAG and PRAG baselines in both effectiveness and efficiency.
2 RELATED WORKS
2.1 PARAMETRIC KNOWLEDGE ENHANCEMENT
Parametric knowledge enhancement methods aim to increase the knowledge capacity of language models by adjusting their parameters to better encode new information. The most direct approach is full fine-tuning, but this quickly becomes impractical as model sizes grow. To address scala- bility, parameter-effiiient fine-tuning (PEFT) techniques, such as LoRA (Hu et al. 202I) and its variants (Valipour et al.) 2022: Yu et al.) 2024a Liu et al. 2024), update only small set of low- rank matrices, achieving performance comparable to full fine-tuning at a fraction of the cost. PEFT methods can be attached or removed with minimal disruption, enabling efficient adaptation across domains and tasks. Moreover, they reduce training-time memory and compute requirements.
With the rise of model editing, more targeted approaches have been developed that directly locate and modify knowledge representations within the model. Methods such as ROME (Meng et al.) 2022ah, MEMIT (Tan al. 2023), and PMET (Li et al. 2024) update critical layers to encode new facts, while MEND (Mitchell et al. 2021) and MALMEN (Tan et al. (023) employ hypernetworks to inject knowledge into specific layers, effectively fusing edits with existing parameters. To mitigate catastrophic forgetting and preserve general-purposs purpose capabilities, approaches like T-Patcher (Huang jet al. 2202)) and MEMoE (Wang & Li| (2024) introduce external memory modules that store edits separately from the core model. These designs are particularly valuable in dynamic settings where
In this paper, we use PRAG broad term encompassing the original PRAG Su et al. 2025 and its variants, including DyPRAG (Tan al. 2025a).
2

---

Published as a conference paper at ICLR 2026
Retrieval
Parameterization Inner Merging
Orthogonal Continual Merging
Answer current Sub-questions
Relevant Passages
C₁
80g
SP
SP.
Pi
P2
Pm
H.B
(1))
(ps)
H (Pm)
(SP)
Plug in Critical
Memory Attentoin
Pascags
(pm)
7d (1p:)
H (SP)
FisP...)
Cullectio
Layer
C--1
Retriever
H( ps 1m
Fish
Critical layer
Merge ((AA) (pm))
Mergo FiSP ((P))
8qJ=
1 New Sub-question Generation
Figure 1: Overview MergePRAG for multi-hop QA. complex query is decomposed into sub- questions, and retrieved passages sequentially incorporated through parameterization and contin- ual merging. At each timestep t: (1) sub-question sq: is generated from the reasoning chain C,-1 (Eq. [1 Section (2) the retriever returns top-ranked passages S.P. : R; (3) given SP, [p,)]==>> j=1- each passage is parameterized by the hypernetwork to produce { (pt) which are combined into (SP)) via the inner-merging mechanism (Eq. 6 Section 3.2); (4) orthogonal continual merg- ing updates the accumulated parameters F(SP with (STP)) to obtain F(SPLa) (Eq, [1 Section 3.2.2); and (5) the merged expert F(SP 1:)) is injected into the base LLM Ma, at the criti- cal layer R to generate the sub-answer (Eqs, 5 This process repeats until no further sub-questions
are produced, after which the final answer generated.
facts evolve time and updates must be applied repeatedly. They also support controllable interventions by localizing changes to a small subset of parameters or auxiliary components.
Overall, parametric enhancement methods differ in where and how they modify parameters- ranging from full updates to low-rank adapters, targeted edits, or external memory-yel they share the goal of augmenting LLMs with new knowledge while retaining general abilities.
2.2
RETRIEVAL. AUGMENTED GENERATION
Borgeaud et al,, 2022 train language models jointly with top-retrieved documents, enabling the model to incorporate external knowledge sources when generating answers. To further improve performance, subsequent approaches expanded the knowledge sources, incorporated query rewriting, jointly trained the retriever and the generator to achieve tighter integration. To mitigate the computational overhead of fully parameterized RAG training, methods such as PRAG (Su et al. (2025) and DyRAG (Tan al.) 2025a) have been proposed, which enhance the model's internal knowledge by learning mappings from retrieved documents model parameters.
Early RAG methods (Lewis et al. 2020 Guu et al. 2020 Izacard & Grave 2021
Recent advances increasingly emphasize the importance of reasoning over retrieved facts. For in- stance, FLARE (Jiang al. (023), ,, MeLLo (Zhong et al., 2023). IRCoT (Trivedi al. 2023) and SELF-REASONING (Xia et al.) 2025) employ iterative cycles of reasoning, retrieval, and eтror correction refine responses. DeepRAG (Guan et al.) 2025) formulates reasoning as a Markov De- cision Process (MDP) to enable adaptive retrieval, while R3-RAG (Li et al.) 2025b) leverages large models to construct trajectories and applies reinforcement learning teach L.LMs stepwise rea- soning and retrieval strategies. Collectively, these works highlight the effectiveness of constructing chain-of-thought reasoning processes for complex tasks.
Building these insights, present MergePRAG, which extends PRAG to the multi-hop RAG setting and serves as a critical stepping stone toward crasoning-enhanee RAG systems. In contrast to prior PRAG methods (Su et al. 2025) that rely simple arithmetic merging, MergePRAG in- troduces a merging module with orthogonal merging, enabling more effective integration of passage experts across hops.

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
In this section, we present MergePRAG, illustrated in Figure | We first provide a brief background multi-hop RAG, and then deseribe MergePRAG and its two main components: orthogonal merg- ing with the Gram-Schmidt process and critical-layer parameterization.
We define two language models and retrieval module. Ma, denotes general-purpos. LLM for sub-answer generation, also referred to as the base LM, Mag a sub-question generator, based a smaller LLM,, and R. the retriever, which returns of top-ranked passages for each query q, denoted as RQ)).
3.1 MULTI-HOP RAG
Let q be the original complex query. the multi-hop RAG setting, each step involves sub-question generation, retrieval, and response generation. At step t, given Ct-1, the accumulated context so far, the next sub-question sq; and sub-answer sa, are obtained as
8ft = Ma (C.-1)) sa, = Ma. (sge: SPt), SP, S R(sq),
where SP, denotes the retrieved passages at step Task-specific instruction prompts for Mag and Ma, are described in Appendix H
The newly obtained tuple (sq,, sa,) is appended to the context: C. = [C--1, 8q,, sa,) The final answer to the original query q is then generated
MMCC where sQr Ma (Cr-1)
(EOS).
In the single-hop setting, RAG produces the answer in step: -Me (q. SP1), SP. S R(g), and the process terminates immediately.
3.2 MERGEPRAG
To present MergePRAG, we first review PRAG in the single-hop RAG setting.
PRAG. As in DyPRAG TTan et al., eterization module. Let Hg denote the hypernetwork, which maps retrieved passage p set of passage-specific LoRA parameters:0, = Hos (p). The hypernetwork is trained to efficiently translate an in-context passage into its corresponding parameters.
2025a PRAG employs a hypernetwork-bbseed passage param-
Let Ma. denote the base LLM with parameters fg. PRAG augments the model by injecting passage- specific parameters (p) generated from the hypernetwork Hs, such that for a passage p.
0' 0 @ Hp()),
where @ denotes the purameter-injection operation. Unlike RAG, which conditions on the passage p explicitly in the input prompt, PRAG generates the answer under the eessage-injecter model as
MergePRAG. MergePRAG extends PRAG to the multi-hop RAG setting, where passages arrive sequentially through iterative retrieval. By timestep f, the accumulated passages are SPI: = (SP,,...,SP. To inject all context passages into the LLM parameters, let F denote mapping from the sequence SP to the parameter space. Instead of directly "training" F over datasets with varying numbers of passages MergePRAG introduces continual merging mechanism that induces F by reusing the passage-level hypernetwork Hgg which maps single passage to its parameter representation.
Sequence-merginn The sequence merging, denoted as Mergesegi is recursive operation that com- bines the previously accumulated parameters F(SPP with the new passage-specific parameters (SP.))
F(SP...) Merge seq (FisPre-1): Merge..o(FisP..u.) Hs(P))
Using the "merged" parameter representation, MergePRAG generates a candidate answer at timestep t without relying in-context passages:
804
4

---

Published as a conference paper at ICLR 2026
At the final timestep T, MergePRAG generates the final answer (err (g).
MergePRAG+. Similar to PRAG-Combine (Su al. 2025). MergePRAG+ integrates RAG and PRAG in a complementary manner, yielding:
sūg = (a (SPt, sq.), t t< < T,
a = M t=
(5)
Inner-merging, We introduce inner-merging mechanism to induce H(SP) from individual pas- sage parameters, for |SP| > Formally, given a list of passages SP = [pi..... .Pm). H(SP) is obtained by applying an inner merging operation Merge
H(p))] Merge (P1)............ Hp(pm))
Merge (H([p-)(=T) H (pm))
(6)
3.2.1 HYPERNETWORK-BASE KEY-VALUE MEMORY PARAMETERIZATION FOR H.g.
For passage parameterization, MergePRAG adopts a key-value memory parameterization scheme, where the hypernetwrk generates k key and value vectors for each passage, which serve as a "com- pressed" passage-specific memory. The passage-specific memory inserted into the feed-forward network (FFN) at the critical layer l" via additional attention mechanism, referred to the mem- ory attention mechanism.
Formally, the hypernetwork (p) first produces the passage-specifie memory for passage p as:
Ha(p) (Kp, Vp )
where Kp, Vp E Rkxda are the key and value matrices, respectively.
Suppose that the original FFN module at layer is denoted as a function MLP Rdd tm parameterized by Oo. The passage-specific FFN expert (HL(p) is then obtained for input E Rd using a memory attention mechanism, i.e., standard attention applied to the passage-specific memory (Kp, Vp) with the base FFN output MLPe, (x) used the query. Formally,
EHo(p) (x)
InrentionMMPP. (x), Kp, Vp),
qK
Attention(q. Kp. vn) softmax
Vp.
(8)
Vanaa
The passage-specific FFN expert injected into the original FFN layer at 1*, yielding:
MLPo,sHa(p) MLP (x) + (x)(
3.2.2 ORTHOGONAL, CONTINUAL. MERGING MECHANISM (Merge) FOR
Once the parameterization module (SP.)) produces passage vectors (Kp, Vp) as in Eq. 7 the continual merging mechanism operates each parameter independently. To form merged expert without overwriting previously acquired knowledge, we propose an orthogonal merging method based Gram-Schmidt projection, inspired by recent studies (Xu al.) 2025)_ For- mally, let { {W.)i= denote the set of key or value memory matrices (i.c., K, or Vp) obtained from {Hs where W, E gexdaur.
Let w be the merged parameter obtained from (W.) up to step 1- The Gram- Soam-Schmidtt or- thogonalization procedure first computes the projection matrix onto the subspace spanned by F
p = w (WW w "
(10)
The new parameter W, is then merged by adding only its orthogonal component with respect the subspace spanned by W
W
(I-P'-l)V
(11)
where 1 is the projection matrix defined in Eq. 10 A detailed discussion of orthogonal merging using the Gram-Schie Schmidt procedure is provided in Appendix B
5

---

Published as a conference paper at ICLR 2026
3.2.3 HYPERNETWORK ARCHITECTURE: SEQUENCE-TE -OO-ENOOR-O-
The hypernetwork is designed to take token sequence of a passage and produce its key-value mem- ory. Given a passage as an input sequence of tokens, the hypernetwork Ho first computes a passage embedding via astentive pooling over the token-level embeddings. The resulting passage embed- ding is then passed through two-layer MLP, whose output is transformed by linear projections to generate the passage-specinng specifcc memory, i.e., the key and value matrices.
Formally, given passage p. denote its sentence embedding by Emd(p), obtained as the atten- tively pooled representation from an auxiliary Transformer encoder (Appendix [c. The hypemet- work then transforms Emd(p) into latent representation using MLPhyp as follows:
h MLP hyp (p)) ReLU(V' LN (ReLU(W' ((e(()))
(12)
where LN refers to the layer normalization layer.
Finally, we apply two distinct linear transformations map the latent representation h, into flattened key and value matrices, i.c., the "passage-specific memory" for
Kp WR h, + bx. Vp Wv h, bv:
(13)
where WK. Wv E xdxdds are linear projection tensors and bx. by E R2 are bias terms. With slight abuse of notation, we treat matrix in gxxxx1 as matrix in IR kxd by removing the singleton dimension. More details of the hypernetwork architecture are provided in Appendix
3.2.4 CRITICAL-LAYER PARAMETERIZATION FOR Hg
The critical-layer parameterization applies only single critical layer 1", rather than across all layers, motivated by the locate-and-edit methods of (Meng et al. 2022a Li et al. 2024 Fang
al. 2025).
To identify the critical layer 1", we conduct layer-wise scanning experiments on both models across all datasets. For each layer, we measure the change in perplexity after injecting the corresponding passage vectors, thereby evaluating the effectiveness of the layer-specific hypernetwork (see Ap- pendix: A). As shown in Figure 2 and 3 the early-to-mdddle layers contribute most substantially when used parameterization modules. Based this analysis, the insertion positions for the single-layer passage-vector parameterization are summarized in Table
3.2.5 TRAINING OBJECTIVE
Hypernetwork Ha. To train H. 2 construct dataset DH ((i, Pt, ) where each triple consists of question its relevant passage Pt, and the ground-truth answer a, The hypernetwork trained by minimizing the cross-entropy loss:
Ler(d) = Σ
log PMo lg).
(14)
(s.P.a)EDH
(a q) denotes the probability of generating answer a conditioned on question under the parameters of the passage-injecee model Mo.
where PMogen s
Subquestion generator Meg- Following Li et al. (2025b), we adopt a cold-start stage to train the sub-question generator Mag by constructing a dataset Dag = {(((1) y( M where each target sequence is
(1) اd) = shs saj 882 saz sqyj sa (EOS)
The autoregressive objective on Day is used to train Ma as detailed in Appendix
EXPERIMENTS
4.1 EXPERIMENTS SETTING
Models and Datasets. We employ LLaMA3.1-8B (Dubey et al. 2024) and Qwen2.5-7B (Tam
2024 as research base models. For the multi-hop question answering task, we follow works (Guan
Here, Ho denotes the layer-specific hypernetwork that injects passage knowledge into the FNN at the
critical layer?".

---

Published as a conference paper at ICLR 2026
2025 Li et al. 2025b) and utilize the E5 (Wang et al., 2022) and BM25 (Lü 2024 retriev- et al.
ers.
For the multi mop editing task, we follow work (Zhong et al. 2023) and adopt the Contriever
model (Lei et al. 2023) as the retriever. g
We conduct experiments on multi-hop question answering benchmarks, including HotpotQA YYngg al. 2018). 2WikiMultihopQA (2WikiMhQA) (Ho al., 2020), and MuSiQue (Trivedi et al.,
2022 In addition, we evaluate multi-hop editing on the MQuAKE (Zhong et al. 2023) dataset,
with detailed experimental settings and results reported in Appendix E.
Metrics. We evaluate model performance using Exact Match (EM) and F1 score (F1) ) Kwiatkowski al. 2019). EM measures the strict string-level agreement between predictions and gold answers, while Fl quantifies partial correctness by computing the token-level overlap between predictions and references. For all experiments, we take the model's final response as its predicted answer and compare it against the gold standard.
Baselines. We evaluate our approach against a range of baselines: (i) RAG and RAG-CoT, which retrieve relevant documents to answer queries, (ii) iterative retrieval methods such as IRCoT, FLARE and MeLLo, (iii) parameterized RAG methods including PRAG and DyPRAG and (iv) reasoning- enhanced RAG methods including Auto-RAG, Adaptive-RAGe Deep-RAG, R3-RAG, Search-R1 and Search-ol. The detailed descriptions of these baseline methods can be found in the Appendix
Implementation Details. All experiments were conducted on a workstation with 8 NVIDIA RTX A6000 GPUs The detailed training settings and inference are provided in Appendix D
4.2 MAIN RESULTS AND ANALYSIS
We evaluated MergePRAG multi-hop QA datasets using LLaMA3.1-8B and Qwen2.5-7B, with results summarized Table [ MergePRAG consistently outperforms state-of-the-ar baselines across all three datasets, showing the best performances in most cases, except for the run using LLaMA3.1-8B on MuSiQue. Compared with early passage-injection methods such as PRAG and DyPRAG, MergePRAG+ achieves higher performance. demonstrating that the hypernetwork-basee parameterization framework extends effectively to multi-hop QA. Additional gains are obtained when combined with explicit in-context passages, without sacrificing generalization. The results further indicate that increasing the number of retrieved passages || with Merge provides ad- ditional improvements over using a single passage (SSPl= 1).
To examine the effect of hypernetwork-based parameterization, we include an additional baseline, MultihopRAG (Section 3.1). which directly uses the original LLM On without hypernetwork- based parameterization injection (Algorithm 2 Comparisons with MultihopRAG show that hypernetwork- based passage knowledge injection contributes substantially to performance gains.
4.3 ABLATION STUDY
We conducted series of ablation studies examine the effectiveness of the proposed framework and to identify the contribution of its key components. In addition, we performed efficiency analysis experiments evaluate the computational performance of our approach; the detailed results are presented in Appendix E.2
4.3.1 MERGEPRAG+ VS. MULTIHOPRAG W/ FINETUNING
To compare standard fine-tuning with the proposed hypernetwork- parameterization in MergePRAG, we apply fine-tuning to MultihopRAG: directly adjusting O on the same training data used our framework. We consider two settings: (1) fine-tuning without passages, i.e., [sq - sa). where the model is trained to predict sa from sq alone; and (2) fine-tuning with passages, i.e.. [(Pgold, sq) + sa], where the model is trained to predict sa given and the gold passages, resem- bling the standard RAG training paradigm.
We follow these works fair comparison. Pre- Pre-trained models can obtained from Hugging Face. LLaMA-3, 3.......8BBBB-- hl-B///ttpsss/hgggggggeeeee..
Qwen2.5 http::/:///ggggggoacc/.///////// E5:
/ /ttpa//p////ggggooooooooooo/o/or

---

Published as a conference paper at ICLR 2026
Ho
2WIKMMQAA
MuSiQue
Model
Retriever
Method
EM
F1
EM
F1
FM
FI
E5
RAGisPI=l
21.60
36.67
4.90
17.36
2.00
11.49
E5
RAG
27.80
40.51
5.10
15.80
2.70
11.27
E5
RAG-COT
37.60
45.15
30.90
35.00
5.60
13.38
E5
RAG-COT
43.70
50.41
36.20
40.00
5.90
12.49
E5
IRCoT
39.30
46.00
35.10
37.50
12.00
13.60
E5
FLARE
20.90
10,90
11.40
2.30
R3-RAG
45.60
58.80
52.90
60.90
21.20
32.80
LLaMA3.1-BB
BM25
R3-RAG
44.40
57.60
50.60
58.60
17.20
27.70
BM25
Scarch-o1f
14.80
24.08
22.20
27.10
5.40
11.98
BM25
Auto-RAC
25.80
36.09
23,00
30.09
BM25
DeepRAGE
40.70
51.54
48. 10
53.25
BM25
PRAGE
44.84 84
40.55
BM25
DyPRAG
38.35
50.24
MergePRAG +1sp||1
48.80
55.53
66.30
71.05
14.40
25.04
ES
MergePRAG +15P
52.40
60.67
73.20
79.34
16.70
27.69
BM25
MergePRAG +|sm=1
46.80
53.40
61.10
67.31
17.80
29.39
BM25
MergePRAG
+181-4
52.40
60.58
70.20
76.65
20.30
31,20
RAG|sPI=1
36.60
34.90
37.36
3,20
E5
RAC
45.30
52.08
42.00
44.49
5.80
12.73
ES
PSG-CoT|SPI=1
30.20
36.20
19.10
23.05
4.30
RAG-CoTISPI=
51.28
35.40
37.79
5.20
IRCoT
35.70
41.10
31.10
33.50
9.40
11.20
E5
FLARE
23.40
32.06
21.80
26.51
3.60
4.80
Qwen2.5-7B
E5
R3-RAG
46.40
59.70
54,20
62.70
21.40
34,00
BM25
R3-RAG
44.90
58.20
52.80
61.10
17.60
30.00
BM25
Search-o1f
11.60
16.95
22.00
25.02
2.10
7.48
BM25
DeepRAG
32.10
41.14
40.40
44.87
ES
MergePRAG +|sP|=1
43.40
65,80
69.72
9.70
19.61
E5
MergePRAG +|sp]=
50.80
58.37
77.40
81.49
12.30
21.57
BM25
MergePRAG +1sp|=1
42.00
49.09
59.70
63.05
13.00
23.35
BM25
MergePRAG +1S
51.40
59.33
71.80
76.06
16.70
27.33
Table 1; Overall results on three multi-hop QA tasks. Bold numbers indicate the best perfor-
mance. denotes results reported from the original papers or R3-RAG paper. PRAG and DyPRAG results correspond to the combined setting with in-context passages (i.e., PRAG-Combine and DyPRAG-Combine) In MergePRAG runs, JSP] refers to the number of retrieved passages per hop. MergePRAG applies orthogonal continual merging (Section 3.2.2) for both inner-merging and sequence-merginn i.e., Merge and Merge Additional results obtained using alternative mod- els and methods are provided in Table 12
HolpotQA
2WikiMhQA
Traing type
EM
F1
EM
MultihopRAG |sP|= (w/o finetuning)
37.80
47.56
23.30
35.59
MultihopRAG |sP|= (finetuning: [sq - sal
43.70
50.15
58.10
62.57
MulthhoRAAG ||||= (finetuning: [(Poodd sg) + so])
40.10
46.79
60.30
62.04
MergePRAG =sP]=1
47.40
55.29
65.60
70.54
Table Comparison of MergePRAG+ and MultihopRAG with fine-tuning (without hypernetwork) under LLaMA3.1-8B.
Under the LLaMA3.1-SD model with ISP: 1, Table 2 compares these MultihopRAG variants with MergePRAG. Interestingly, naive fine-tuning with passages ([(Poous sq) + sa)) performs even worse than fine-tuning without passages ([sg sal). These results consistent with prior find- ings (Yang et al.) 2024; Lampinen et al. 2025). which show that directly fine-tuning LLMs on domain-adaptive data may degrade their generalization ability.
8

---

Published as a conference paper at ICLR 2026
HotporQA
2WikiMRQA
Inference type
EM
F1
EM
F1
RAG|SPI-
21.60
36.67
4.90
17.36
MergePRAG [SP|=
28.40
35.52
45.60
50.06
MergePRAG + S||=1
48.80
55.53
66.30
71.05
Table 3: Comparison of MergePRAG and MergePRAG+ under LLaMA3.1 1-0.......B..... |.SP| MergePRAG, which does not use in-context passages as prompts during inference.
0 denotes
4.3.2 MERGEPRAG MERGEPRAG+
Table 3 compares MergePRAG with MergePRAG+. MergePRAG+ exhibits strong generalization and is negatively affected when in-context passages are provided. In contrast, applying fine-tuning methods to MultihopRAGi leads to performance degradation, implying that direct fine- tuning unstable for preserving generalization (Section (43.1]. Overall, these results highlight that MergePRAG preserves the model's ability to perform RAG while benefiting from parameterized knowledge injection, compared with standard fine-tuning methods.
4.3.3 EFFECT OF THE MERGING METHODS
To evaluate the effectiveness of the proposed orthogonal merging method in Section 3.2.2 we con- duct ablation experiments on HotpotQA using the LLaMA3.1 model. Table reportss the results of different merging methods for sequence-level merging under the setting SPT = 1, where each sub-question sq retrieves only single passage. Details of the merging methods provided in Appendix
The results show that the proposed orthogonal merging achieves the best performance, improving by 2.4% over TIES-merging, while arithmetic mean merging also performs comparably. Furthermore, Table presents comparisons using different merging methods for both inner merging Merge and inter- mntermeeggng Mergesa across varying values of [SP]. Although arithmetic merging is compet- itive in most settings, orthogonal merging consistently achieves the best results, often showing an improvement of approximately % EM over arithmetic merging. We expect that orthogonal merging will exhibit greater robustness in scenarios with more severe knowledge conflict.
4.3.4 EFFECT OF THE NUMBER OF PASSAGES PER RETRIEVAL ([SP) >
Table reports the results of MergePRAG+ under different numbers of retrieved passages |SP]. As [SP] increases, MergePRAG+ consistently improves performance without degradation, when longer in-context passages are provided.
4.3.5 EFFECT OF THE NUMBER OF VALUE VECTORS
To examine the impact of the number of key-value vectors used for passage-knowledge parame- terization, we conduct an ablation study on HotpotQA and 2WikiMhQA using LLMMA3.1-82 For each dataset, we train models with different values of k (i.e., nuum)e)) under two retrieval settings:
[SP] = and | 4. The results are summarized Table 7
Overall, increasing the number of KV vectors (k) consistently improves performance across both datasets and retrieval settings, as a larger &s expands memory capacity and preserves more passage-
[SP[ =
SP
ISP
=
ISP
ISP
ISP
12
Mergo
Merge
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
EM
Fl
50.20
58.07
51.40
60.35
51.40
59.82
52.00
60.86
55.00
62.84
54.40
62.76
50.60
58.06
52.00
60.64
52.00
60.21
53.00
61.32
55.40
63.40
54.60
62.64
50.40
58. 14
51.60
60.13
51.40
59.63
52.60
61.46
54,80
62.80
$4.60
62.76
50.80
58.23
$2.40
60.67
52.40
60.67
53.40
61.77
55.60
63.45
55.00
62.93
Table 4: Performance comparison between different merging methods for Merge and Merge
varying |SP|: Arithmetic mean merging, Gram SchmidS orthogonalization merging.

---

Published as a conference paper at ICLR 2026
MeG+ HotporQA
HotporQA
2WikiMhQA
Mergess.
EM
F1
# SPI
EM
F1
EM
F1
36,20
43,47
48.80
55.33
66.30
71.05
48.20
55.04
2
50.80
58.23
71.40
76.94
46.40
54.07
52.00
59.50
73.10
79.06
48.20
54.95
52.40
60.67
73.20
79.34
48.80
55.53
Table 6: Performance of MergePRAG+ on Hot- potQA and 2WikiMHQA with varying numbers of retrieved passages ((SP) per sub-question. Increasing |SP provides broader evidence for
Table 5: Performance comparison between dif- ferent merging methods for Mergeseg under the setting of |SP] 1: Additive merging, Arithmetic mean merging, :
TIES merging, v: Concat merging, Gram-Schmide orthog- onalization merging.
answering each sub-question, which can im- prove overall QA accuracy.
HosporQA
2WikiMQQA
SSP
SSP
4
ISP
|SP
Is (i.e., #nourgg))
EM
F1
EM
F1
EM
F1
EM
F1
45.60
52.67
49.00
58.24
62.40
67.89
69.00
75.21
2
45.60
52.67
51.20
59.20
63.80
69.01
69.00
76.32
4
45.60
52.86
50.80
58.88
64.00
69.37
71.20
77.09
8
46.40
54.25
49.40
58.39
65.90
70.93
72.00
78.09
16
48.80
55.53
52.40
60.67
66.30
71.05
73.20
79.34
Table Ablation study on number of passage vectors & (i.e., numkk)) for LLaMA3. 1-8B.
specific information. By capturing richer passage-level representations and mitigating information loss. larger k boosts both EM and Fl.
CONCLUSION
In this work, we introduced MERGEPRAG, which generalizes the PRAG framework to the multi- hop QA setting- an important milestone toward reasoning-enhanced RAG. We proposed two key technical components: (1) orthogonal continual merging, which incrementally updates passage ex- perts with newly retrieved knowledge during multi-hop inference while avoiding interference; and (2) critical-layer parameterization. which applies passage knowledge injection only to selected critical layer, greatly reducing injection cost. Experimental results on multi-hop QA and reasoning- aware knowledge editing showed that MERGEPRAG consistently outperforms standard and state- of-the-art RAG systems, existing PRAG methods, and fin-tuning-basee parametric adaptation.
For future work, plan to extend the framework more general reasoning-enhanced RAG set- ting to examine whether passage injection also contributes to further performance improvements. We also aim to explore the "pretraining" of hypernetworks, enabling them to be applied and adapted efficiently to new domains without requiring substantial additional training. Finally, we will inves- tigate in depth why standard fine-tuning suffers from stronger performance degradation, whereas hypernetwork-be based parameterization is helpful in boosting the performance. It is also worth ex- ploring alternative hypernetwork architectures, such as memory-augmenter designs, which can pa- rameterize longer contexts more effectively beyond the
setting used in this work.
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation(IIIP) grant funded by the Korea (ovemmenn(MSSTyNoRRS.2220222 -0N.N.R8-2022-1201333 Artifi- cial Intelligence graduate school (UNPO)((UNIST)) and IITP grant funded by the Korea govern- ment(MSIT) (No RRS-2023-002166 Development of artificial complex intelligence for conceptu-
10

---

Published as a conference paper at ICLR 2026
ally understanding and inferring like human). Xuebing Liu and Shanbao Qiao were also supported by China Scholarship Council (CSC).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877, 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang. Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Confer- ence on Leurning Representations, 2024. URL httph::///// Settps::/openreviiee net for hSyW5go0vE
Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. Parameters vs. context: Fine-grained control of knowledge reliance in language models. CoRR, 2025.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference machine learning, pp. 2206-2240. PMLR, 2022.
Guanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li, and Michael Qizhe Shieh. RAPID: Long- context inference with retreeeal-uummented speculative decoding. In Forty-second International Conference on Machine Learning, 2025. URL https::/openreview..et net forum? ?dd
7 3
Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining re- trieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 719-729, 2024.
DeepSeek-AL Deepseek-v3 technical report, 2024. URL 19437
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10028-10039 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang. Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan Hc, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
wwt. htt.
Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojic Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. Deeprag: Thinking to retrieve step by step for large language models. arXiv preprint arXiv:2502.011 142, 2025.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, Bareelona. Spain (On- line), December 2020. International Committee on Computational Linguistics. URL hetps:
7/www. web. org/ h/t.l//eb.rrrgaattooooo///////.. www.:ncaonnnaaan....m.m

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 096800 2021.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong- Transformer-patchanr One mistake worth one neuron. In The Eleventh International Confer- ence on Learning Representations, 2023. URL htttps://openrevieee net fortm/formmddd
4oYUGeGBPm
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pp. 874-880, Online, April 2021. Association for Computational Linguis- tics. doi: 20.18663/11/2211.0 ceacl-main.744
URL https: http://////aaaaaccoooo/o.o/o org.//000..rg/22.1
eacl
747
Soyeong Jeong. Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt ctrieval-augment large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 7029-7043, 2024.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context Ilms meet rag: Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05988 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 11ms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095/6, 2025.
Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Scong Joon Oh. Studying large lan- guage model behaviors under montext-memory conflicts with real documents. arXiv preprint arXiv:2404. /6032, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1162/tucl URL wwwp:/yy.o../10111100090
Andrew K Lampinen, Arslan Chaudhry, Stephamie CY Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, and James L McClelland. On the generalization of language models from in-context learning and finetuning: controlled study. arXiv e-prints, pp. arXiv-2505, 2025.
Yibin Lei, Liang Ding, Yu Cao, Changtong Zan, Andrew Yates, and Dacheng Tao. Unsupervised dense retrieval with relevance-aware contrastive pre-training. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 10932-10940, Toronto, Canada, July 2023. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/2023 10.88633/112023.fniinn--.a- findings-acl.699, URL https: https://aclantthlooy: .rrg/ 2023. findings-acl. 6957
Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, and Michael Carbin. Long context rag performance of large language models. arXiv preprint 2xXi:2411.003534 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmente gener- ation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
12

---

Published as a conference paper at ICLR 2026
Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17); 18564-18572, Mar. 2024. doi:
ht URL
aaa. org/
Xiaoxi Li, Guanting Dong. Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05336 2025a.
Yuan Li, Qi Luo, Xiaonan Li, Bufan Li, Qinyuan Cheng, Bo Wang, Yining Zheng, Yuxin Wang, Zhangyue Yin, and Xipeng Qiu. R3-rag: Learning step-by-step reasoning and retrieval for Ilms via reinforcement learning, arXiv preprint arXiv:2305.23777 2025b.
Shih-Yang Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09355. 2024.
Xing Han Lù. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. arXiv preprint arXhv:2407.036/6 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations gpt. Advances in Neural Information Processing Systems, 15:17359-17372 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory a transformer. arXiv preprint arXiv:2210.07229) 2022b.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, et al. Gemma: Open models based on gemini research and technology. CoRR, 2024.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint arXiv: 10. /300 2021.
Weihang Su, Yichen Tang, Qingyao Ai, Junxi Yan, Changyue Wang. Hongning Wang, Ziyi Ye, Yujia Zhou, and Yiqun Liu. Parametric retrieval augmented generation. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1240-1250, 2025.
Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. arXiv preprint urXiv:2311.04600 2023.
Yuqiao Tan, Shizhu He, Huanxuan Liao, Jun Zhao, and Kang Liu. Dynamic parametric retrieval augmented generation for test-time knowledge enhancement. arXiv preprint arXiv:2503.23895, 2025a.
Zhiwen Tan, Jiaming Huang, Qintong Wu, Hongxuan Zhang. Chenyi Zhuang, and Jinjie Gu. Rag-rl: Incentivize the search and reasoning capabilities of llms through multi-query parallelism. arXiv preprint arXiv:2507.02900 2025b.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10677 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Compu- tational Linguistics, 2022.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with chain-of -f-ouughtt reasoning for Knowledge-intensive multi-step questions. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meet- ing the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014-10037, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023, lol- URL http://aaaaaaataaaaoooa wtt. acl won..557.
13

---

Published as a conference paper at ICLR 2026
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuming of pre-trained models using dynamic search-free free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by veakly-supervise contrastive pre-training. arXiv preprint arXiv:2212.08553 2022.
Renzhi Wang and Piji L.. Memoe: Enhancing model editing with mixture of experts adaptors. arXiv preprint arXiv:2405. XXX1:24055/90066, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 25.24824-24837, 2022.
Jinyang Wu, Shuai Zhang, Feihu Che, Mingkuan Feng, Chuyuan Zhang, Pengpeng Shao, and Jian- hua Tao. Pandora's box or aladdin's lamp: A comprehensive analysis revealing the role of rag noise large language models. arXiv preprint arXiv:2408. 13533, 2024.
Yuan Xia, Jingbo Zhou, Zhenhui Shi, Jun Chen, and Haifeng Huang. Improving retrieval aug- mented language model with self-reasoning. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 99(24)25534-22542 Apr. 2025. doi: 10. 10.1609/aaai 03909///aai.3922434444443 URL
aaai. wwwoorgg/noooo wnnexiii/nn..///ee// whtp:ppppp/aaappp.ae.p httpp//AAAaaaaiicl//////////////
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth:
Revealing the behavior of large language models in knowledge conflicts. In The Twelfth Interna- fional Conference on Learning Representations, 2023.
Haoyu Xu, Pengxiang Lan, Enneng Yang, Guibing Guo, Jianzhe Zhao, Linying Jiang, and Xingwei Wang. Knowledge decoupling via orthogonal projection for lifelong editing of large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers). pp. 13194-13213, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 779-8-89176-25111 doi: 10. 00653/v1/2022 a1122025.alllooo acl-llng...6444444 URL attts://aclaane
.rg. 2025 acl2 -ong. 6467
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: re- solving interference when merging models. In Proceedings the 37th International Conference on Neural Information Processing Systems, pp. 7093-7115, 15, 2023.
Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Am Ann Heng, and Wai Lam. Unveiling the generalization power of fine-tuned large language models. In NAACL-HLT, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotporQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaicr, and Jun' ichi Tsuji (eds.). Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIss 1259.
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron- indexed dynamic lora. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17): 19449-1947,, Mar. 2024a. doi: 10.16099maai..v38817.ooo 2.2996.. URL attps://ooo. aaai.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval-augmmente generation for large language models. arXiv preprint arXiv:2411.19443 2024b.
Qinggang Zhang, Zhishang Xiang, Yilin Xiao, Le Wang. Junhui Li. Xinrun Wang, and Jinsong Su. Faithfulrag: Fact-level conflict modeling for context-faithte retrieval auggeente generation.
arXiv preprint 2506 250688850 08938, 2025.
14

---

Published as a conference paper at ICLR 2026
Zexuan Zhong. Zhengxuan Wu, Christopher Manning. Christopher Potts, and Danqi Chen. Assessing knowledge editing in language models via multi-hop questions. Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference Empiri-
MQUAKE:
cal Methods in Natural Language Processing, pp. 35686-15702, Singapore, December 2023.
Association for Computational Linguisties. doi: 10.18653/v1/2055
URL
emn ma maiin......m......... 971/
A LAYER SCANNING EXPERIMENTS FOR CRITICAL LAYER PARAMETERIZATION
The critical-layer parameterization module applies H only to a single critical layer 1". To identify 1", we perform a layer-wise scanning experiment that evaluates perplexity after adding layer-specific paragraph vector to the /-th layer. For this purpose, we construct small sub-dataset from the dataset used in the experiment.
Formally. let H denote the layer-specific hypernetwork for the 1-th layer, parameterized by sb and defined following Eqs. 13 Given a question we first retrieve relevant passages P C R. Each passage p E P is fed into Hi to obtain its passage expert (p) which are then merged into single expert Ex,, (P) using the mnner-gegging operation in Eq. 6 The merged expert is subsequently incorporated into the [-th layer of the base LLM Mg. via Eq. 9 We train H. by minimizing the cross-entropy loss defined Eq. 14
To measure the importance of each layer 1, we evaluate perplexity after training H. Fig- 3a 3b compare perplexity across layers for LLaMA3.1-8B and Qwen2.5-7B on different experimental dataset, respectively, under the setting of [SP] = 1.
ures 2a 2b 2c
The results show a clear sensitivity pattern: in LLaMA3. 1-BB and Qwen2.5-7B. injecting passage vectors into carly-to-middle layers yields the largest perplexity reduction, indicating that these lay- ers play a central role in integrating external knowledge. Mcanwhile, we observe that the two mod- els exhibit opposite patterns the layers where external knowledge is least efficiently integrated. Specifically. LLaMA3. 1.-...1-BBBB. shows higher perplexity when the passage vector is injected into the shallowest layers, whereas Qwen2.5-7B displays higher perplexity when the injection applied to the deepest layers. This contrast suggests that conducting layer-wise scanning is essential for identifying the optimal injection layer for different model architectures.
Overall, our findings show that both LLaMA3. 1-BB and Qwen2.5-7B exhibit their highest sensitivity to passage-vector injection in the early-to-middle layers, suggesting that these layers are primarily responsible for incorporating external knowledge across model families. Based the layer-wist perplexity analysis conducted on three Motnsets-HotpotQAA.. WikiMhQA, and Musique-we we select the optimal insertion layer I* for each model-datase. pair. The selected layers are summarized in Table g
Model
HotpotQA
2WikiMultihopQA
MuSiQue
1-8
9
|*
Qwen2.5-7B
-"
1* 8
[*
Table Selected critical layers (* for passage-vector insertion based on layer-wise perplexity anal- ysis across datasets.
15

---

Published as a conference paper at ICLR 2026
The training perplexity (PPL) of the HotPot QA dataset each layer of LLaMa3.1-88 model
130
122.15
130
422 A00
421
5.1
Layer Index
(a) HotpotQA: layer-wise perplexity (ISP) = 1)
The training perplexity (PPL) the WikiMhQA dataset at each layer of LLaMa3. 1-BB model
250.
240
270
30
143-4E 3/6
KM
A55
454
JSa
15
20
25
30
Layer Index
(b) 2WikiMhQA: layer-wise perplexity ([SP)
The training perplexity (PPL) the MuSiQue dataset at each layer of LLaMa3.1-88 model
200
250
53.4
PPL
ce
Layer Index
(c) MuSiQue: layer-wise perplexity (ISP) = 1)
Figure 2: Layer-wise perplexity trends of LLaMA3.1-8B with paragraph-vector insertion on Hot- potQA, 2WikiMhQA, and MuSiQue under SPP = 1.
16

---

Published as a conference paper at ICLR 2026
The training perplexity (PPL) of the HotPot QA dataset each layer Qwen2.5-78 model
33 96
PPL
420
444
4.5
1.35
Layer Index
(a) HotpotQA: layer-wise perplexity (ISP) = 1)
The training perplexity (PPL) the WikiMhQA dataset at each layer of Qwen2.5-78 model
8-
522
521
PPL
A21
AM 462
4.00
367
10
25
Layer Index
(b) 2WikiMhQA: layer-wise perplexity ([SP)
The training perplexity (PPL) of the WikIMhQA dataset at each layer of Qwen2.5-78 model
30
13.28
10
0.00
3.00
PPL
5.50
6.27
4.7
4.33
Layer Index
(c) MuSiQue: layer-wise perplexity (ISP) = 1)
Figure 3: Layer-wise perplexity trends of Qwen2.5-7B with paragraph-vector insertion Hot- potQA, 2WikiMhQA, and MuSiQue under |SP] = 1.
17

---

Published as a conference paper at ICLR 2026
B ORTHOGONAL MERGING USING THE GRAM-SCHMIDT PROCEDURE
In multi-hop RAG, set of passages SP, arrives for
each sub-question sq.. This setting naturally mo-
M
tivates the design of a continual merging mecha- nism that combines previously accumulated knowl- edge with newly retrieved passage knowledge, recur- rently updating the current FFN expert by incorpo- rating each new expert.
M
pla-IMO
M
M (
To minimize overwriting previously acquired knowl- edge, MergePRAG adopts orthogonal continual merging based on the Gram-Schmidt process, in- spired by recent orthogonal approaches in model merging and knowledge editing (Xu al. 2025). Specifically, the new parameter matrix is projected onto the span of the previously merged parameters, and only its orthogonal residual added to the current merged expert.
P M
Figure 4: Illustration of orthogonal continual
merging based on Cram--chhmitt procedure.
We apply orthogonal continual merging separately to either key or value matrices, resulting from Hd. Formally, let [M denote the sequence of key or value passage memories, where each M E (SP) corresponds to either K, or Vp-
Suppose that F denotes the merged memory parameter obtained from { { Following Eq. i0 Section 3.2.2 the Gram-Schmidt orthogonalization procedure first computes the projection matrix onto the subspace spanned by F
pla-1= F ( (M
M
(15)
The new parameter ( is then merged by adding only its orthogonal component with respect to the subspace spanned by F
M M + (I- pl:--1111 M
1(
(16)
where 1 is the projection matrix defined in Eq. 15
With slight abuse of notation, the recursion in Eq. 16 is denoted by Merge:
Merge
M - M(
(17)
When the merging procedure Merge is applied independently to the sequences of key and value matrices Kp and Vp, we obtain the merged passage memories for both parts:
K
Merge
F K K(t))
Fia
Merge Fi-- v(O).
(18)
where (t) and v(a) denote the key and value memory matrices at timestep t, respectively. Here, Merge refers to either Merge in Eq. 3 or Merge in Eq. 6
C HYPERNETWORK ARCHITECTURE
As shown in Figure 5 given passage p. the hypernetwork Hg generates the corresponding key-value memory through three stages: (1) Atlentive pooling, which produces a sentence-level embedding (p) for passage Pt (2) MLP, which maps the passage embedding to a latent repre- sentation using a two-layer ReLU-based MLP; and (3) Linear projection, which converts the latent representation into key-value vectors K,, Vp. yielding the passage-specha memory.
We define a lightweight Encoder consisting of 2-layer Transformer encoder layer with attention head. The hidden dimension of the encoder layer is set to be consistent with the LM model's internal representation dimension Specifically, for LLaMA3.1 1-8B, d = 4096, while for Qwen2.5-7B. d = 3584.
18

---

Published as a conference paper at ICLR 2026
HyperNetwork HΦ
X exdd E RXxx
K E RR
ReL.U
W K
IRd
V
LayerNorm
W
Softmax
ReLU
W
w
E R Rddd
E R
Attentive pooling
MLP hyyp
Lincar projection
Memory
Figure 5: The hypernetwork (p) generates passage-specific key-value vectors Kp, V, Rkxddot referred as passage-specific memory. which serve lightweight, plug-in passage-level experts for downstream reasoning. The process consists of three stages: (1) attentive pooling, (2) MLP, and (3) linear projection. 1) Attentive pooling, Given a one-hot token matrix X RTx|
for passage p. the model first converts into sequence of embeddings Xemb E RRxxd via the word embedding layer: Xemb Embedding X) (Eq 19). Attention is then applied over the to- ken embeddings, where learnable vector W. E R" serves as the query: Emd(p)
h R" (Eq. (0). 2) MLP.. The pooled representation h is passed through two-layer feedforward network with ReL.U activations and LayerNorm, producing a latent representation h, (Fq. (T). 3) Linear pro- jection. Two independent linear projection heads map h, into the key and value parameter spaces: R kxduu yielding flattened key-value memory vectors of length k dmmddl for passage p (Eq. The resulting passage-specific memory is subsequently injected into the target model as additional knowledge signals.
Kpp V
Attentive pooling. Given passage p. we apply attention- based aggregation over token-level em- beddings of p: (1) obtaining sequence of its word embeddings, and (2) applying attentive pooling. Formally, let the retrieved passage be represented as a sequence of tokens, denoted by X E '''' where each row X, is one-hot vector over the vocabulary indicating the identity of the token the t-th position, and V denotes the vocabulary set. We apply word embedding layer Embedding to X and obtain its embedded representations, as follows:
(19) where sequence of token embeddings Xemd E Txxd where T is the passage length and d is the embedding dimension. Note that word embedding layer Embedding is obtained from the pretrained I.LM Ma, (e.g., LLaMA3.1-88 Qwen2.5-7B).
eemd Emneedding(X)..
The passage embedding (p) is then obtained via attentive pooling:
Emd(p) h softmax w X Xemd Rd
(20)
where wa E Rd is learnable attention vecto The embedding Emd(p) = h serves as the atten- tively pooled representation of the passage, capturing its global semantic content.
MLP. To increase representational capacity and allow the hypernetwork to perform nonlinear rea- passed through two-layer
soning over the passage summary, the pooled vector Emd(p) feedforward network, denoted MLP hyp. as follows:
h MLP hyp (h) ReLU(V LN (ReLU(W'h))).
(21)
where LN is the layer normalization layer.
We omit additional bias term has negligihle impact.
19

---

Published as a conference paper at ICLR 2026
Linear projection. Finally, two linear transformations map the latent code h, into flattened key and value matrices, i.e., the passage-specine memory:
K Wx ha + bK:
V Wy by,
(22)
where (i.e., num-kv) denotes the number of key-value slots generated per passage and is the model dimension. Each of the k rows corresponds directly attended to by the language model.
independent memory vector that can be
This stage design enables the hypernetwork compress entire retrieved passage into compact set of attention-ready memory vectors, which are efficiently integrated into the model via the memory attention mechanism the designated target layer (Eq. g in Section 3.2.1.
D TRAINING AND INFERENCE PROCEDURE
D.1 TRAINING
Hypernetwork. We prepare a training dataset Dy at the sub-question level from training set in each task train the hypernetwork Hg. Each instance (g, P, E Di consists of a sub-question q. its gold passage p. and the corresponding answer a. The hypernetwork parameters Φ are trained by minimizing the negative log- lilkliiiihood of generating the correct answer a under Mo (Eq. while the base parameters Og remain frozen.
Subquestion generator. To train the sub-question generator Mag. we construct a dataset Deg ((g0), د using GPT-4.1 (Achiam et al. 2023) to generate sub-questions from 4,000 ran- domly sampled examples in the training split of each dataset. The prompt template used for this dataset construction is shown in Table The template specifies the desired output format and in- cludes several illustrative examples. Given an input question q and its associated gold passages, GPT-4.1 refers the examples and decomposes q into sub-questions using the provided passages.
(i) (i) Each target sequence is g(و) = | sa()) 8802 s02 Section
()) () 8qn; 80n, (EOS) as described in
Given a pair (g y))), Mag is trained with supervised fine-tuning by minimizing the negative log-likelihood (NLL):
M ly ((Mag)= = Σ
log PM.,,
1/
19 1
(23)
where of denotes the 1-th token in the target sequence y(t)
D.2 INFERENCE
At inference time, MergePRAG tackles a multi-hop QA task by decomposing the original complex question into sub-questions. For each sub-question sq;, the top-retrieved passages SP, are fed into the hypernetwork H. to produce a sub-expert EH, (SP.). This sub-expert is then merged with the previously accumulated FFN expert (SP....))) using orthogonal continual merging, yielding the updated fused expert Et))) ensuring that knowledge from earlier reasoning steps is preserved without redundancy.
The fused FFN expert is injected into the base LLM Mo, at the critical layer I*. Response gener- ation is then performed under the updated model Mo. either with the current in-context passages SP, (MergePRAG+) or without them (MergePRAG), After all sub-questions are processed at timestep T. the final answer to the original complex question is generated by the fully passage- injected model Ma,FF(SP,.) The complete inference procedure is summarized in Algorithm For comparison, the inference procedure of MultihopRAG without passage knowledge parameten- zation is shown in Algorithm
20

---

Published as a conference paper at ICLR 2026
Your task is to convert given question and its related facts into multi-step reasoning chain.
Requirements: Each step in the reasoning chain must:
Use one fact from the input facts, do not combine, summarize, or fabricate facts; each fact must be used as-is from the input.
Generate "Sub-question" and a short answer "Sub-answer".
The answer "Sub-answer" must be directly derivable from the corresponding Fact.
Examples:
Question: "When did the civilisation start that Desalpar Gunthli was a part of?".
Facts: ["Desalpar Gumthli: Desalpar Gunthli a village and site belonging to Indus Valley Civili-
sation located at Nakhtrana Taluka, Kutch District, Gujarat, India.", "Indus Valley Civilisation: The Indus Valley Civilisation (IVC) or Harappan Civilisation was a Bronze Age civilisation (3300-1300 BCE; mature period 2600-1900 BCE) mainly in the northwestern regions of South Asia, extending from what today is northeast Afghanistan to Pakistan and northwest India."] Output: [
"Sub-questionn": "Which civilisation was Desalpar Gunthli part of?",
"Fact": "Desalpar Gunthli: Desalpar Gunthli is village and site belonging to Indus Valley Civilisation located Nakhtrana Taluka, Kutch District, Gujarat, India.",
"Sub-answer":: "Indus Valley Civilisation"
"Sub-question":
"When did the Indus Valley Civilisation exist?",
"Fact": "Indus Valley Civilisation: The Indus Valley Civilisation (IVC) Harappan Civilisation was a Bronze Age civilisation (3300-1300 130001300 BCE; mature period 2600-1900 BCE) mainly in the northwestern regions of South Asia, extending from what today northeast Afghanistan Pakistan and northwest India.",
"Sub-answer": "3300-1300 BCE"
}
[other examples demonstrations abbreviated]
Question: U
Facts: (
Output:
Table 9: Prompt templates used by GPT-4. to generate sub-questions and sub-answers from the training set. The generated results are used to construct D.g for training the sub-question generator.
E FURTHER EXPERIMENT RESULTS
E. MULTI-HOP EDITING
The MQUAKE (Zhong et al. 2023) benchmark consists of two multi-hop editing datasets: MQuAKE-CF and MQUAKE-T MQLAKE-CF is counterfactual knowledge editing dataset de- signed to evaluate how effectively models can incorporate and generalize from counterfactual mod- ifications existing knowledge. In contrast, MQuAKE-T targets temporal knowledge updates, assessing a model's ability to track and apply changes in real-world facts over time.
As shown in Table 10 MergePRAG achieves superior performance compared with prompt- based methods such as RAG and MeLLo, which rely on explicit prompting. These results further demon- strate the effectiveness of injecting knowledge into the model for multi-hop editing tasks as well.
E.2 EFFICIENCY ANALYSIS
To evaluate the inference cost of MergePRAG, we conduct an efficiency analysis (Table II] focusing on three components of the system: (1) passage-specific memory generation by Hyp (2) sub-question
21

---

Published as a conference paper at ICLR 2026
Algorithm Multi-hop Inference with MergePRAG
Require: Original question 4, sub-question generator Mag. base LLM Ma, hypernetwork Has retriever R
Ensure: Final answer
1: Initialize merged expert F 0
2: Initialize reasoning chain C f- 0
3: while next sub-question exists do
4: Generate sub-question: 8qi <- Msg(g,
5: Retrieve passages: SP: + R(sq;)
6: Parameterize passages: (SP)))) + Merge (Holp) (erl))mee((HH6 1p SP.))
(Eq. 6
7: if > then
8:
Orthogonal continual merge: F t- Merge seg , (SP:))
(Sec. 3.2.2)
9: else
10:
Initialize expert: F + HS(PP))
end if
12: Inject F into the base LLM: Maoa
Generate sub-answer:
(MergePRAG)
Moogr(SP, sq:). (MergePRAG+)
Append (sq; sa;) to reasoning chain C end while Generate final answer:
15:
16:
a MggeF(CC (MergePRAG/ MergePRAG+)
17: return
Algorithm Multi-hop Inference with MultihopRAG
Require: Original question q. sub-question generator Mag. base LLM Ma, retriever R
Ensure:
Final answer a /
1: Initialize reasoning chain C +- 0
2: while next sub-question exists do
3: Generate sub-question: 8Q <- M M,n(g,C)
4: Retrieve passages: SP: + R(sq:)
5: Generate sub-answer: sa; + Mo.
6: Append (sq;, sa;) to reasoning chain C
7: end while
8: Generate final answer: a +- (M,
9: return
response generation, and (3) overall response generation. Thanks the lightweight design of the hypernetwork, the time required to produce passage-specific key-value memory is minimal. The subsequent step of generating sub-questions also incurs only modest overhead.
Although decomposing a complex query into multiple sub-questions increases the number of in- ference steps compared with standard RAG, the overall latency remains within practical range. Notably, the proposed pipeline still requires less time than RAG-CoT methods, which rely on long tfain-of-thoughh prompting, while achieving higher accuracy. These results demonstrate that MergePRAG offers a favorable trade-off between computational efficiency and reasoning effective- ness.
F BASELLINES INTRODUCTION
RAG: (Lewis et al. 2020) For a given query q, the retriever selects the top-k relevant passages. The generator then directly infers the answer based these retrieved passages. To ensure stylistic
22

---

Published as a conference paper at ICLR 2026
MQuAKE-CE CF
MQUAKE-7
Model
Method
EM
F1
EM
F1
RAG
4.48
9.27
27.69
31.92
RAG-CoT
11.7
13.18
45.93
47.28
LLaMA3.1-8B
MeLLo
32.90
34.10
85,40
86.21
MergePRAG+ SPI=
50.30
51.36
96.10
96.10
Table 10: Results on the multi-hop editing task under the MQuAKE datasets.
Generate Passage Memory
Response sa
Average Time
|AG| RAG-CoT |SP|=1
0.712s
6.389s 0.259s
MergePRAGi+ssPIE
0.001s
2.517s
Table 11: Efficiency analysis of MergePRAG using the LLaMA3.1-8B model on the HotpotQA dataset.
consistency of the generated answers, apply task-specific prompt. The RAG prompt template provided in Table I3
(Wei et al. 2022) Building upon RAG, RAG-CoT incorporates chain-of of-thhuugt rea- soning. To guide the model's reasoning process, we employ a one-shot demonstration sampled from the training data, which encourages the model to generate step-by-step explanations before arriving at the final answer. The prompt template is included in Table 14] We evaluate RAG and RAG-CoT by retrieving 1-8 relevant passages, with the accuracy results shown in Figure
RAG-CoT:
MulitihopRAG: MultihopRAG can be viewed as a variant of MergePRAG without H. which itera- tively responds to sub-questions using a pure RAG-style approach (Algorithm 2)
IRCoT:
(Trivedi et al. 2023) Interleaves retrieval with chain-of-thought reasoning, enabling itera- tive evidence retrieval conditioned on intermediate reasoning steps, which enhances multi-hop QA performance and reduces hallucination.
MeLLo: (Zhong al. 2023) MeLLo is system that iteratively decomposes multi-hop questions into subquestions, generates tentative answers, retrieves relevant facts, and updates predictions based on potential contradictions.
FLARE: (Jang et al., 2023) Incorporates adaptive retrieval triggered when the model generates low-confidence tokens, leveraging retrieved evidence to improve response quality.
2024) Adaptive-RAC automatically selects the optimal retrieval and reasoning strategy based on query complexity, ensuring efficient handling of simple queries while improving accuracy complex ones.
Adaptive-RAG: (Jeong al.
(Yu et al. 2024b) It performs iterative reasoning to decide when and what to retrieve, and terminates the process once sufficient external knowledge has been gathered, before generating the final answer.
Auto-RAG:
(Guan al. 2025) It odels attrivall-augment generation Markov decision pro- cess, where the query is iteratively decomposed and the model dynamically decides at each step whether to retrieve external knowledge rely parametric reasoning.
DeepRAG:
R3-RAG:
(Li et al. 2025b) It is a reinforcement learning- baased method that trains LLMs to iter- atively reason and retrieve, enabling them to acquire more comprchensive external knowledge and generate more accurate answers.
Search-ol: (Li al.
2025a Search-ol lets reasoning model dynamically retrieve and analyze external knowledge to fill knowledge gaps during long reasoning.
Search-R1: (Jin et al. 2025) Search-R1 enables LLM to learn, via reinforcement learning, how to autonomously issue effective multi-turn search queries during step-by-step reasoning, thereby substantially improving retrieval-augmented QA performance.
23

---

Published as a conference paper at ICLR 2026
HA
2WikiMhQA
MuSiQue
Model
Method
Retriever
EM
F1
EM
FI
EM
F1
FLAN-T5-XL
Adaptive-RABe
BM25
42.00
53.82
40.60
49.75
23.60
31.80
LLaMA3.I-8B
Search-ol
BM25
14.80
24.08
22.20
27.10
5.40
11.98
Qwen2.5-7B
Search-ol
BM25
11.60
16.95
22.00
25.02
2.10
7.48
Qwen2.5-3B
Search-RI
E5
32.40
31.90
10.30
Qwen2.5-7B
Search-RI
E5
37.00
41.40
14.60
Qwen2.5-7B
MergePRAGyysPP
E5
43,40
50.64
65.80
69.72
9.70
19.61
Qwen2.5-7B
MergePRAGr [SP|=1
BM25
42.00
49.09
59.70
63.05
13.00
23.35
LLaMA3.1-8B
MergePRAG SP|=1
E5
48.80
55.53
66.30
71.05
14.40
25.04
LLaMA3.1-8B
MergePRAG SP|=1
BM25
46.80
53.40
61.60
67.31
17.80
29.39
Table 12:
Performance comparison of MergePRAG+ with other advanced RAG methods on three QA benchmarks Adaptive-RAG, Search-RI Searhhh and Search-ol.
LLaMA3.1-88 HotPot-QA
LLaMA3.1-8B/ 2WikiMultthapAA
LLaMA3.1-88 MuSiQue
Reffrrancc.
Performancce
Number of Retrieval instances Used
Number of Retrieval Instances Used
Number Retrivaal trstances Used
Qwen225-78 HotPot-QA
Qwen2.5-78 2WikiMuitthhoppp
Qwen2.5-78 MuSiQue
Porfrrancee 196)
Performance 196)
Derformande
10
Number of Retrieval instances Used
Number Retrieval Instances Used
Number Recreval instances Used
Figure 6: Results of RAG and RAG-CoT varying the number of retrieved passages three multi- hop QA datasets using LLaMA3.1-8P and Qwen2.5-7B.
PRAG+: (Su al. 2025) By transforming the documents retrieved for query q into parametric representations that are directly integrated into the feed-forward networks of the LLM, parametric retrieval-augmented generation introduced.
DyPRAG+: (Tan et al. 2025a Extends PRAG by employing lightweight parameter transformation module to efficienlly convert documents retrieved for query q into parametric knowledge, which can be directly leveraged to generate the response.
G MERGING METHODS: INTRODUCTION
Arithmetic mean merging. Arithmetic mean merging computes the element-wise mean the task vectors (T, in { where is the number of tasks. This approach assumes that all vectors lie in shared embedding space and produces balanced fusion without introducing additional Iearnable parameters:
Merge (T))
Σ T3
(24)
j=1
Additive merging. Additive merging performs Element-wise summation of task vectors. This oper- ation preserves activation magnitudes and emphasizes consistently high-valued features across task
24

---

Published as a conference paper at ICLR 2026
Follow the format below to answer the following question with a very short phrase, such as "1998", "May 16th, 1931", or "James Bond", to meet the criteria of exact match datasets.
Passage: {}
Question: {}
Answer:
Table 13: Input template used for evaluating multi-hop questions with RAG.
You are a reasoning assistant tasked with answering user questions step by step. Follow the format below to answer the following question with very short phrase, such "1998", "May 16th, 1931", or "James Bond", to meet the criteria exact match datasets.
Since moving to Los Angeles in 2009 he has worked at Disney Television Animation serving as creative director "Fish Hooks" and co-executive producer on "The 7D". Fish Hooks: Fish Hooks an American animated television series created by Noah Z. Jones that originally aired Disney Channel from September 3, 2010 to April 2014.
Passage: Tom Warburton:
Question: What show that Tom Warburton worked on aired from September 3, 2010 to April 4, 2014?
Thoughts: The passage says Tom Warburton worked as creative director on Fish Hooks. The pas- sage also says Fish Hooks aired on Disney Channel from September 3. 2010 to April 4, 2014. The question asks which show that Tom Warburton worked on aired during those dates. So, the answer must be Fish Hooks.
Answer: Fish Hook
Passage: O
Question: (}
Thoughts:
Table 14: Input template used for evaluating multi-hop questions with RAG-CoT.
vectors, without trainable parameters, as follows:
Merge ((T, (T,}}=)
Tj.
(25)
Concat merging. Concat merging first concatenates the task vectors and then applies a learnable linear projection to map the concatenated vector into the merged space:
Merge((T;) = concal (TIcat(T,,..... TT.).
(26)
In our setting, concatenation is equivalent to increasing the number of key-value vectors from k to x n. For example, suppose (11), (2) E Tx are the key memories from two tasks. Concat merging produces:
Merge (1) k(2)) = concat (K(1, K()) E R 122k x
(27)
TIES merging. TIES merging (Yadav et al. 2023) Trim-EElettSiin Merging) fuses task vectors by retaining only the largest-magnitude and sign-consistent components across tasks. This approach preserves salient and mutually aligned activations while suppressing contradictory or noisy features. Given n task vectors, TIES merging proceeds in three stages:
Trim.
Given a task vector Tj, the trimming step applies magnitude- pruning:
t, = tooop
(28)
where tops retains the top 1% of parameters by magnitude and the remaining entries zero. The trimmed vector is decomposed into its sign and magnitude components:
7 7 mij.
(29)
25

---

Published as a conference paper at ICLR 2026
where
9 = sfgn(f)),
m, = -
and @ denotes element-wise multiplication.
2. Elect. The election step performs nagnitude-weightee sign aggregation. The merged sign vector computed by selecting, for each coordinate, the sign with the largest summed magnitude across all trimmed task vectors:
7/m sign Σ
T3
(30)
3. Merge. Given the trimmed task vectors Tj the merging step selectively aggregates only those coordinates whose signs match the elected sign Ym Formally,
aj
I( Ym),
Tm
Σ
Σaj
aj
(31)
1=1
where I(e) is the indicator function that returns if the condition is true and otherwise, and @ denotes element-wise division.
H CASE STUDY
We present a case study to illustrate the decomposition process. As shown in Table 16 the sub- question generator iteratively breaks down the question into sub-questions. For each step in Table the upper part above the dashed line corresponds to the input template used by the sub-question generator, while the lower part shows the retrieval and sub-answer generation process. Green text denotes retrieved content, and red text indicates generated sub-answers. When the sub-question generator produces no further sub-questions, the resulting chain is passed into the merged-expent LM model. The inference process is illustrated in Table 15] answer.
where blue text highlights the final
We further provide an error-case analysis, shown in Table 17 This failure is triggered by an incorrect retrieval result for one of the sub-questions, which leads to an erroneous sub-answer. The mistake then propagates to subsequent steps, causing the next sub-question to deviate from the original problem and ultimately resulting in a chain reaction of compounding errors. As indicated by ablation study the number of retrieved documents (i.e., cases where |SP| 1), increasing retrieval depth helps stabilize sub-question answering accuracy and consequently improves overall performance. This reveals key limitation of our method: incorrect sub-answers may induce ripple effect throughout the iterative decomposition process-a challenge shared by all multi-hop decomposition-basee approaches.
Sub-question: Who starred in Duel at Diablo?
Sub-answer: James Garner
Sub-question: Did James Garner also star in Space Cowboys?
Sub-answer: Yes
Sub-question: What year was James Garner born?
sub-answer: 1928
Question: What year the actor born that starred in both Duel at Diablo and Space Cowboys?
Answer: 1928
Table LM responds directly the original question. When (EOS) is generated Table 16 the inference chain terminates and the resulting context is used as input to the LM.
26

---

Published as a conference paper at ICLR 2026
Initial prompt:
Decompose the following question into sub-questions:
What year was the actor born that starred in both Duel at Diablo and Space Cowboys?
Duel at Diablo: Duel at Diablo is a 1966 western film starring James Garner his first Western since leaving "Maverick" and Sidney Poitier in his first Western. Sub-question: Who starred in Duel at Diablo? Sub-answer: James Garner
2-step prompt:
Decompose the following question into sub-questions:
What year was the actor born that starred in both Duel at Diablo and Space Cowboys? Sub-question: Who starred in Duel Diablo? Sub-answer: James Garner
James Garner: He starred in several television series over more than five decades, including such popular roles as Bret Maverick in the 1950s western comedy series Maverickänd Jim Rockford in The Rockford Files, and played leading roles in more than 50 theatrical films, including The Great Escape( (963) with Steve McQueen, Paddy Chayefsky's The Americanization of Emily(1964), Grand Prix(1966) Blake Edwards' Victor/Victoria( 1982), Murphy's Romance(1985), for which he received an Academy Award nomination, Space Cowboys(2000) with Clint Eastwood, and The Notebook(2004),
Sub-question: Did James Garner also in Space Cowboys?
Sub-answer: Yes
3-step prompt:
Decompose the following question into sub-questions:
What year was the actor born that starred both Duel at Diablo and Space Cowboys?
Sub-question: Who starred in Duel Diablo?
Sub-answer: James Gareer
Sub-question: Did James Garner also star in Space Cowboys?
Sub-answer: Yes
James Garner: James Garner (born James Scott Bumgarner; April 7, 1928 -July 19, 2014) was an American actor, producer, and voice artist. Sub-question: What year was James Garner born?
sub-answer: 1928
4-step prompt:
Decompose the following question into sub-questions:
What year was the actor born that starred in both Duel at Diablo and Space Cowboys?
Sub-question: Who starred in Duel Diablo?
Sub-answer: James Gamer
Sub-question: Did James Garner also star in Space Cowboys?
Sub-answer: Yes
Sub-question: What year was James Garner born?
sub-answer: 1928
(EOS)
Table 16: Case example generated by the sub-question generator. At each step, the input appears above the dotted line, while the black text below shows the output. Red text indicates the answer to the generated sub-question.
27

---

Published as a conference paper at ICLR 2026
Initial prompt:
Decompose the following question into sub-questions: The actress that plays Sarah Hurst Easy Virtueštars with Dervla Kirwan in a 2012 supernatural thriller film written by who?
Charlotte Riley: She is known for her roles as Sarah Hurst in "Easy Virtue" and as Catherine Earn- shaw in ITV's adaptation of "Wuthering Heights". Sub-question: Who plays Sarah Hurst in the 2010 film 'Easy Virtue"?
Sub-answer: Charlotte Riley
2-step prompt:
Decompose the following question into sub-questions:
The actress that plays Sarah Hurst Easy VirtueStars with Dervla Kirwan in 2012 supernatural thriller film written by who?
Sub-question: Who plays Sarah Hurst in the 2010 film 'Easy Virtue"?
Sub-answer: Charlotte Riley
Into the Dark (film): Into the Dark, also known as "I Will Follow You into the Dark", is a 2012 supernatural romance thriller written and directed by Mark Edwin Robinson. Sub-question: Which 2012 supernatural thriller film stars Dervla Kirwan and Charlotte Riley? Sub-answer: Into the Dark
3-step prompt:
Decompose the following question into sub-questions:
The actress that plays Sarah Hurst in Easy VirtueStars with Dervla Kirwan in 2012 supernatural thriller film written by who?
Sub-question: Who plays Sarah Hurst in the 2010 film 'Easy Virtue"?
Sub-answer: Charlotte Riley
Sub-question: Which 2012 supernatural thriller film stars Dervla Kirwan and Charlotte Riley?
Sub-answer: Into the Dark
Into the Dark (film): Into the Dark, also known as "I Will Follow You into the Dark", is a 2012 supernatural romance thriller written and directed by Mark Edwin Robinson. Sub-question: Who wrote the 2012 supernatural thriller film 'Into the Dark"? sub-answer: Mark Edwin Robinson
4-step prompt:
Decompose the following question into sub-questions:
The actress that plays Sarah Hurst Easy Virtueštars with Dervla Kirwan in 2012 supernatural thriller film written by who?
Sub-question: Who plays Sarah Hurst in the 2010 film 'Easy Virtue??
Sub-answer: Charlotte Riley
Sub-question: Which 2012 supernatural thriller film stars Dervla Kirwan and Charlotte Riley? Sub-answer: Into the Dark
Sub-question: Who wrote the 2012 supernatural thriller film 'Into the Dark'?
Sub-answer: Mark Edwin Robinson
(EOS)
Table 17: A failure case induced by sub-question retrieval. An incorrect retrieval result generated from sub- sub-question triggers chain reaction, ultimately resulting in an overall failure.
28


Reviews:
[{"review_id": "gEHqLncyEh", "reviewer": "Reviewer_xajy", "summary": "The paper proposes extending Parametric RAG (PRAG) to handle multi-hop question answering by decomposing questions into atomic sub-questions and recursively merging a hypernetwork with retrieved passages.", "questions": "This paper iteratively merges the hypernetwork using retrievals from each step’s sub-question and generates sub-answers, denoted in Eq. (4) as:\n\n$$\ns_{a_t} = M_{\\theta_0 \\oplus F(SP_{1:t})}(sq_t)\n$$\n\nHowever, why not:\n1. Generate a single hypernetwork using the passages from all steps at once,  \n   $$\n   H(SP_{1:T})\n   $$\n   instead of sequentially?\n2. Or generate sub-answers using the fully merged hypernetwork,  \n   $$\n   s_{a_t} = M_{\\theta_0 + F(SP_{1:T})}(sq_t)\n   $$", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The problem addressed in this paper is well-motivated: adapting Parametric RAG (PRAG) for multi-hop question answering.\n- The paper decomposes a multi-hop question into atomic sub-questions and recursively merges the hypernetwork with each retrieved passage.\n- The proposed method extends prior work, enabling PRAG to effectively handle multi-hop reasoning tasks.", "weaknesses": "- Baseline selection and setup: \n  - The baseline selection is weak. Some competitive baselines are missing, such as Adaptive-RAG [1] for (i) RAG, and Search-o1 [2] and Search-R1 [3] for (ii) iterative retrieval.\n  - The experimental settings for PRAG should be comparable to those of the proposed method. MergePRAG may gain advantages from sub-question decomposition and sub-answer generation. Even though PRAG was not originally designed for multi-hop retrieval, the sub-question decomposition should still be applied to the plain PRAG for fair comparison.\n\n- Missing ablation study:\n  - The paper does not include ablation studies on individual components (e.g., orthogonal merge, critical layer parameterization). Such analyses would help readers understand the necessity and novelty of each component.\n\n- Additional resource considerations:\n  - The proposed method requires sub-question decomposition and sub-answer generation, each followed by a hypernetwork merging step.    However, the inference cost (e.g., number of inferences for sub-answer generation or latency) is not compared. Such an analysis is needed to confirm that the performance gains do not come at the expense of computational efficiency.\n\n[1] Jeong et al., \"Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity\", NAACL 2024.               \\\n[2] Li et al., \"Search-o1: Agentic Search-Enhanced Large Reasoning Models\", ArXiv 2025.    \n[3] Jin et al., \"Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning\", ArXiv 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hSK6gnIgmH", "reviewer": "Reviewer_EL8F", "summary": "This paper proposes MergePRAG, a novel extension of Parametric RAG (PRAG) that enables multi-hop reasoning by orthogonally merging passage-specific experts into a language model’s parameter space. The method introduces an orthogonal merging mechanism, critical-layer parameterization, and a hypernetwork-based low-rank adaptation to efficiently absorb multi-hop retrieved knowledge. The idea is timely and original, addressing an important limitation of current PRAG systems that struggle with multi-hop reasoning. The paper is generally well-written and conceptually interesting, with promising empirical results.", "questions": "none", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of orthogonal merging of passage experts is innovative and aligns well with current trends in retrieval-augmented and parametric reasoning research.\n\nThe approach is conceptually neat, allowing models to accumulate knowledge from multiple retrieval steps while minimizing interference.\n\nThe authors evaluate the method on multiple multi-hop QA and knowledge-editing datasets, showing that MergePRAG achieves competitive or even superior performance compared to prior RAG and PRAG variants.", "weaknesses": "1. Inconsistency and Missing Entries in Table 1\nTable 1 looks quite confusing. Several entries (e.g., DyPRAG’s EM scores) are missing. For Qwen2.5-7B, there are no PRAG or DyPRAG baselines, which makes it unclear whether the claimed improvement of MergePRAG+ is significant or merely due to the absence of strong comparisons. Moreover, Table 1 reports MergePRAG+ results only—there seems to be no result for the base MergePRAG. It is also unclear whether MultiHopRAG corresponds to MergePRAG or another baseline. Overall, the table lacks a fair and consistent setting for comparison, casting doubt on the validity of the experimental conclusion\n\n2. Effectiveness of Knowledge Injection (Table 4) According to Table 4, the performance of MergePRAG appears to drop substantially under certain configurations. This raises concerns about the effectiveness and stability of the proposed parametric knowledge injection. The authors should discuss why this degradation happens and whether it reflects limitations in the merging mechanism or the hypernetwork parameterization.\n\n3. The paper does not include a natural baseline where all retrieved documents are merged into a single LoRA adaptation (without orthogonal decomposition). Such a comparison would be essential to demonstrate that orthogonal merging is the key factor, rather than simply adding more LoRA parameters. Without this, it is hard to isolate the contribution of the proposed mechanism.\n\n4. The choice of “critical layers” is somewhat arbitrary and lacks theoretical or empirical justification. The method appears to require layer-by-layer scanning to find the best injection point, which seems neither elegant nor scalable. Moreover, it is unclear whether different documents or tasks share the same optimal layer, or whether the critical layer should adapt dynamically. This design choice needs clearer motivation and analysis.\n\n5. The method seems tuned for specific architectures (LLaMA-3.1-8B and Qwen-2.5-7B). There is little evidence that the approach generalizes across other model types, sizes, or training regimes. Since the critical layer and merging behavior may differ per model, the generalization ability of MergePRAG remains uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LferWfFGoU", "reviewer": "Reviewer_YcxX", "summary": "This paper introduces MergePRAG, a novel framework that extends Parametric RAG (PRAG) to multi-hop question answering. The key innovation is a continual merging mechanism that sequentially integrates retrieved passages into LLM parameters through orthogonal merging (using Gram-Schmidt process) and critical-layer parameterization. The method shows consistent improvements over existing RAG and PRAG baselines on multiple multi-hop QA datasets.", "questions": "1. Why does lower perplexity indicate higher sensitivity to expert injection?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigates a critical research question: how to inject retrieved knowledge into parameters for multi-hop QA? The motivation is novel, and the proposed methods address the concerns of conflicts during merging and maintaining lightweight.\n2. The experiments show advantages on multi-hop QA datasets, comparing with a wide range of methods.\n3. Some augmentation designs like only modifying the critical layer, and shared basis low-rank parameters make sense.", "weaknesses": "1. The advantages need further clarification. When compared to other PRAG methods like DyPRAG, how does the pipeline work? Is the sub-question generator used consistently across all PRAG baselines? What are the exact architectural and procedural differences that lead to performance gains?\n\n2. The significance of the method lies in the passage merging that avoids knowledge conflict, compared to other PRAG methods. First, the severity of the knowledge conflict may need illustration. The paper doesn't quantify how often or how severely knowledge conflicts occur when passages from different sub-questions are merged. Second, if the gram-schmidt orthonormalization is ablated, will the performance significantly decrease? There is no empirical evidence showing the actual conflict patterns in the retrieved passages\n\n3. There is no analysis on the significance of critical layer selection, and shared basis low-rank parameters. Missing comparison of different merging strategies. Computational overhead comparison with baselines would strengthen the efficiency claims.\n\n4. The paper would benefit from error analysis showing failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hun7nb3AxJ", "reviewer": "Reviewer_5C8J", "summary": "This paper introduces MergePRAG, a novel framework that extends Parametric Retrieval-Augmented Generation (PRAG) to handle multi-hop reasoning, a limitation of previous methods restricted to single-pass retrieval. The proposed approach sequentially integrates knowledge from multiple retrieved passages into the large language model's parameters during inference. This is achieved through two key innovations: (1) orthogonal continual merging, which uses the Gram-Schmidt process to add new \"passage experts\" without conflicting with previously integrated knowledge, and (2) critical-layer parameterization, which efficiently injects this new knowledge by updating only a select critical layer, reducing computational cost. Experiments on multi-hop question answering and reasoning-aware knowledge editing tasks demonstrate that MergePRAG consistently outperforms standard RAG, state-of-the-art RAGs, and other parametric adaptation methods.", "questions": "- Given the limitations of Exact Match and F1 in capturing semantic quality, can the authors provide justification for omitting a model-based evaluation (like LLM-as-a-judge), or discuss how these metrics adequately validate the model's multi-hop reasoning capabilities?\n\n- The experiments are limited to models under 8B parameters. Can the authors provide any analysis or theoretical argument on whether the efficiency and effectiveness benefits of MergePRAG are expected to scale to current, much larger state-of-the-art models?\n\n- The critical-layer parameterization relies on findings from a single model (LLaMA3.1-8B). Have the authors verified that this \"early-to-middle layer\" sensitivity generalizes to other architectures? If not, how does this potential model-specificity impact the method's general applicability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n\n- It introduces MergePRAG, the first framework to successfully generalize Parametric Retrieval-Augmented Generation (PRAG) to the more complex multi-hop reasoning setting, addressing a key limitation of prior PRAG methods.\n- It proposes a novel \"continual merging\" technique that uses orthogonal merging to integrate new knowledge without conflicts and critical-layer parameterization to ensure high efficiency, allowing knowledge to accumulate effectively across retrieval steps.\n- It performs extensive experiments across multiple benchmarks and LLM backbones, demonstrating that MergePRAG consistently outperforms existing RAG and PRAG baselines in both effectiveness and efficiency.", "weaknesses": "This paper has the following weaknesses:\n\n- The evaluation relies solely on token-level metrics (Exact Match and F1), which may not fully capture the semantic quality or factual accuracy of the generated responses. The study would be strengthened by a more holistic, model-based evaluation, such as using an LLM-as-a-judge.\n\n- The experiments are limited to models up to 8B parameters. It remains an open question whether the proposed method's benefits and efficiency gains will scale effectively to current state-of-the-art, much larger models (e.g., 200B+ parameters).\n\n- The analysis identifying optimal \"critical layers\" for expert injection (Figure 2) is conducted on only one model (LLaMA3.1-8B). It is unclear if this finding generalizes across different model architectures, which is a key assumption for the \"critical-layer parameterization\" technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)