C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Authors: Guoxin Chen
Minpeng Liao
Peiying Yu
Dingmin Wang
Zile Qiao
Chao Yang
Xin Zhao
Kai Fan
Venue: ICML 2025 poster (2025)
Keywords: Retrieval-Augmented Generation
Primary area: deep_learning->large_language_models

Abstract: Retrieval-augmented generation (RAG) systems face a fundamental challenge in aligning independently developed retrievers and large language models (LLMs). Existing approaches typically involve modifying either component or introducing simple intermediate modules, resulting in practical limitations and sub-optimal performance. Inspired by human search behavior—typically involving a back-and-forth process of proposing search queries and reviewing documents, we propose C-3PO, a proxy-centric framework that facilitates communication between retrievers and LLMs through a lightweight multi-agent system. Our framework implements three specialized agents that collaboratively optimize the entire RAG pipeline without altering the retriever and LLMs. These agents work together to assess the need for retrieval, generate effective queries, and select information suitable for the LLMs. To enable effective multi-agent coordination, we develop a tree-structured rollout approach for reward credit assignment in reinforcement learning. Extensive experiments in both in-domain and out-of-distribution scenarios demonstrate that C-3PO significantly enhances RAG performance while maintaining plug-and-play flexibility and superior generalization capabilities.

Existing approaches address this misalignment through three main strategies: (1) fine-tuning retrievers to align Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
C-3PO:
Guoxin Chen 12
Minpeng Liao?* Peiying Yu² Dingming Wang" +
Zile Qiao? Chao Yang Wayne Xin Zhao 1 Kai Fan 2'
Abstract
Hurst et al., 2024; Dubey et al., 2024; Yang et al., 2024a;
Mesnard et al., 2024; Asai et al., 2024; Chen et al., 2024a;b; Wei et al., 2025; Sun al., 2025b;a), empowering LLMs to acquire up-to-date or domain-speeithe knowledge while mitigating hallucinations (Gao al., 2023; Fan et al., 2024; Qiao et al., 2024). The effectiveness of RAG systems, how- ever, hinges on the alignment' between the retriever and the LLM-an inherently challenging goal as these components are typically developed independently without co-training. This lack of co-training can result in semantic mismatch and suboptimal interactions: retrievers may fail provide information tailored to the LLM's 's needs, while LLMs may struggle to generate effective queries scamlessly incorpo- rate retrieved content.
Retrieval-augnee -ugmeentdd generation (RAG) systems face a fundamental challenge in aligning inddpen- dently developed retrievers and large language models (LLMs)) Existing approaches typically in- volve modifying either component or introducing simple intermediate modules, resulting in practi- cal limitations and sub-optimal performance. In- spired by human search behavior eehavior-typically in- volving a back-and-forth process of proposing search queries and reviewing documents, we pro- pose C-3PO, a proxy-centric framework that fa- cilitates communication between retrievers and LLMS through lightweight multi-agent sys- tem. Our framework implements three special- ized agents that collaboratively optimize the en- tire RAG pipeline without altering the retriever and LLMs. These agents work together to assess the need for retrieval, generate effective queries, and select information suitable for the LLMs. To enable effective multi-agent coordination, de- velop tree-structured rollout approach for re- ward credit assignment in reinforcement learning. Extensive experiments in both in-domain and out- of-distribution scenarios demonstrate that C-3PO significantly enhances RAG performance while maintaining plug-and-pn sagg-nnd-pay flexibility and superior generalization capabilities. Code available https://githn: cww/ Hom/Chen-Gie GA----XX--PPPPP 3PO.
with LLM preferences, (2) optimizing LLMs adapt to retriever behavior, and (3) introducing intermediate mod- ules bridge the gap between them (Ma et al., 2023; Shi et al., 2024; Asai et al., 2024; Wei et al., 2025; Yu et al., 2024a:b). Despite progress, these methods face notable challenges: fine-tuning retrievers often requires carefully curated data and may not be feasible for commercial search engines (Schmidt, 2014; Nakano et al., 2021), while opti- mizing LLMs is resource-intensive and risks compromising their original capabilities (Zhou al., 2024). Approaches that introduce intermediate modules to avoid modifying ei- ther the retriever or the LLM primarily focus on optimizing individual tasks, such as query rewriting or document rerank- ing (Ma et al., 2023; Wang et al., 2023; Tan et al., 2024). However, optimizing a single task in isolation often leads to suboptimal results, as the effectiveness of RAG systems relies the cohesive interaction and collaboration among multiple components throughout the entire pipeline (Fan et al., 2024; Zhou et al., 2024).
1. Introduction
Recent advances in retrieval-augmented generation (RAG) for large language models (LLMs) have demonstrated re- markable capabilities in various tasks (Anthropic, 2024:
"Corresponding Authors. 'Gaoling School Artificial Intelli- gence. Reamin University China Tongyi Lab 'Soochow Univer- sity *University of Oxford 'Tsinghua University. Correspondence to: Guoxin Chen <gx.chew www.eceeechnnn @ @gmail.com>> Minpeng Liao <minpeng. hap@ammpp@@aaaaaa aleeabbbbaaiiin com>, Wayne Xin Zhao <batmann fly@ com Kai Fan <kk aalibaba-inc.com
In human's 's search behavior, the process typically involves an iterative back-and-foo process of proposing search queries and reviewing documents until the correct response is either found in the retrieved documents or emerges the person's 's mind. Similarly, LLMs emulate this process
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Here, alignment refers the functional coordination between retrievers and LLMs, rather than human preference alignment.

---

C-3PO: Compact Plug-and-Pla Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
by taking on multiple roles within search pipeline: propos- ing search queries, reviewing documents, deciding when to terminate retrieval, and generating the final response, among other tasks. However, assigning all these tasks to LLMSs results in numerous calls, leading to high computa- tional costs, especially for complex questions. To address this, it is desirable to design a compact proxy capable of handling most tasks, while reserving the most challenging ones LLMs such as planning the overall roadmap and generating the final response.
and LLMs, validating its effectiveness plug-and-play solution for RAG systems.
2. Related Work
Eetrieval-Augmenter Generation. Retrieval-augmented generation (RAG) has emerged as crucial technique for enhancing L.MMs' capabilities by incorporating external knowledge sources (Gao et al., 2023; Yu et al., 2024b). Re- cent studies have highlighted the importance of aligning the retriever with the LLM to achieve superior performance (Fan et al., 2024; Chan et al., 2024). This alignment can be ap- proached through three main strategies: retriever fine-tuning methods (Shi et al., 2024), LLM fine-tuning methods (Asai et al., 2024; Wei et al., 2025; Yu et al., 2024a), and inter- mediate modules methods (Ma et al., 2023; Wang et al., 2023; Tan et al., 2024). However, these methods often face practical limitations, such as focusing on local optimization, the inability to align with commercial search engines, and the substantial computational costs of LLM optimization. Different from previous work, we introduce lightweight, proxy-centric alignment framework that implements align- ment without modifying either the retriever or LLM while optimizing the entire RAG pipeline holistically.
Therefore, we propose C-3PO, a proxy-centric alignment framework that employs a lightweight yet effective proxy to facilitate seamless communication between retrievers and L.LMs without modifying them or compromising their original capabilities. As illustrated in Figure I, C-3PO integrates a lightweight multi-agen collaborative system within single proxy model, where multiple agents work in human-like manner to assist the entire RAG pipeline. To optimize this proxy, we employ multi-agent reinforce- ment learning (MARL) for end-to-end training, treating the retrievers and I.LMs as part of the environment. To address the key challenge of optimizing multiple agents with distinet tasks, introduce tree-structured rollout mechanism and Monte Carlo credit assignment to improve reward distribution among different agents. In this way, our C-3PO redistributes the sampling efforts from the question level to the action level, enabling more efficient credit as- signment in multi-agent systems through expectation-be based reward distribution. Experimental results demonstrate that the RI.-trained proxy achieves robust performance on both in-domain and out-o--distribution datasets, even with un- seen retrievers and LLMs, highlighting its plug-and-play modularity and superior generalization capability. Our con- tributions are summarized as follows:
Multi-agent Systems. Multi-agent systems have recently garnered increasing attention, especially in the context of complex task-solving and decision making (Guo et al., 2024; Zhang et al., 2024; Chen et al., 2025). A ma- jor challenge in multi-agent frameworks is credit assign- cout-determiningg each agent's is contribution to the overall system performance which becomes particularly crucial in multi-agent reinforcement learning (Yuan al., 2023a; Zhu et al., 2024). In our work, we propose Monte Carlo credit assignment mechanism to distribute system-level rewards to each agent in the form of expectations, enabling effective coordination within agents.
We propose C-3PO, novel proxy penticcentric alignment framework that bridges the gap between retrievers and LLMs through a lightweight multi-agent system, en- abling seamless integration without modifying existing RAG components.
3. Preliminaries
Before introducing C-3PO, we first review the preliminaries of cooperative multi-agent reinforcement learning (MARL), where multiple agents work collaboratively to accomplish given tasks. A cooperative MARL problem is generally formalized as cooperative stochastic game, represented by a tuple (N, (Sthen, {A') Jex. T, R), where:
We design an efficient multi-agent collaborative system within a single proxy model that emulates human-like search behavior, where specialized agents handle dif- ferent aspects of the RAG pipeline while maintaining computational efficiency.
We develop an innovative training approach combining MARL with a tree-structured rollout mechanism and Monte Carlo credit assignment, effectively addressing the challenge of optimizing multiple agents towards the system-level performance in an end-to-end manner.
N = 2,., n} is the set of agents in the system.
S is the state space of agent i, where each agent main- tains its agent-specific state information.
A' is the action space of agent i, defining the joint action space A == A'
X A".
Extensive experiments demonstrate that C-3PO achieves superior performance across diverse datasets and exhibits strong generalization capability with unseen retrievers
T: {S* heN x A + her is the deterministic state transition function, specifying how the states of agents
2

---

C-3PO: Compact Plug-and-Plas Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Misalignment
Proxy-centric Alignment
Retriever
LLMs
Retriever
LLMs
Multiple Agents
Human-guided Alignment
Reasoning Router
Information Filter
Decision Maker
No Retrieval
Humon Cognitive Capabilities
Question
Simple Retrieval
Information Filter
Does this question need external knowledge?
Is this information relevant the question?
Do we have sufficient information to answer?
Decuments
Reasoning
Complex Retrieval
Decision Maker
Answer
Can this answered with single search?
Does this information provide new insights?
What information should we retrieve next?
Roadmap
Collaborative Strategy
Iterative
Accomulated
Information
R
Reasoning Router
R
R
Information Filter
R=0
Decision Maker
Teredit
R
{sf at Teredit (8 diee
Deterministic
Stochastic
Teredit (s at)
E [R(T) s aj]
Node
Tree-structured Rollout
Monte Carlo Credit Assignment
Multi-agent Optimization
Regroup
Figure Overall framework of C-3PO, (Upper left) Essential cognitive capabilities required effective RAG system interaction Homan-guides alignment. (Upper right) Our proxy-centric alignment simulates these human-like interaction through lightweight multi-agent system with collaborative strategies. (Bottom) The end-to-end optimization pipeline our multi-agent system.
update given their joint action A.
nication between the retriever and the LLM, as illustrated in Figure 1. These agents operate as distinct roles within a single lightweight proxy using targeted instructions, collabo- ratively managing various aspects of the RAG pipeline. This unified design ensures efficient coordination while maintain- ing simplicity for edge deployment. Formally, we define each agent follows:
R (SE x x N is the system-level re- ward that measures the overall task completion, where indicates success and 0 otherwise.
In this cooperative setting, all agents share the same system- level reward R and work together to accomplish the task. Each agent follows its policy -' S + A'' to se- lect actions based on its local observations. A trajectory (fsb hev, au { hev, ......) represents the sequence of agent states and joint actions, where of E St is the state of agent time step t, and a, {at A is the joint action at time step t under the joint policy (ar ...., ,...,,.,,,, "").
This proxy plays the role of Reasoning Router agent to determine the optimal reasoning strategy for given ques- tion from a high-level perspective. Given the current state (the question), it selects actions using a maximum two-step operation:
1. Decide Retrieval Necessity: the agent outputs [No Retrieval], question directly processed by the LLM, leveraging its internalized knowledge.
4. Cooperative Multi-agent System
In this section, we elaborate on the role of each agent (Sec-
2. Determine Question Complexity: If the agent outputs [Retrieval], the agent also evaluates whether the ques- tion requires complex reasoning.
tion 4.1) with their collaborative strategies (Section 4.2).
4.1. Multiple Agents
For simple questions, the agent continues to generate sin- gle <query content> and interacts with the retriever to obtain documents. The retrieved documents are then pro- cessed by the Information Filter agent to extract relevant,
Inspired by human behavior mentioned in the Introduction, we design three specialized agents- Reasoning Router, In- formation Filter, and Decision Maker-to to facilitate commu-
3

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
LLM-friendly content. Finally, the LLM uses the filtered documents generate answer to the question.
high-level guidance for the proxy.
2. Iterative Retrieval-filter Loop: Guided by the roadmap. the Decision Maker evaluates the current progress, de- termines the next objective, and generates subqueries for the retrieval-filet loop. This process is carried out in coordination with the Information Filter and continues iteratively until the Decision Maker determines that the accumulated documents contain sufficient information to address all subgoals.
For complex questions, the agent outputs [Planning], which will trigger a multi-step reasoning strategy that re- quires coordination with multiple agents. Further details on this strategy will be introduced later.
This proxy plays the role of Information Filter agent to pro- cess and filter retrieved information for identifying content suitable for LLMs. Its state space includes the question, the retrieved documents, and the current reasoning objective (if operating in [P]anning] mode). Based on the given state, the agent selects an action to analyze and filter relevant documents using the following structured format:
3. Final Answer: All accumulated information is passed to LLM for answer generation.
Notably, generating the roadmap is the only role that is not played by the proxy in our communication pipeline; however, the LLM is invoked only once in the [Planning] strategy, minimizing computational overhead. Additionally, it is important to note that the number of retrieval-filtes loops may not directly correspond to the number of subgoals, as a single retrieval might address multiple subgoals or require multiple attempts for single subgoal.
Thought: <Analysis of each documents>> Action: [<Selected document IDs>]
This proxy plays the role of Decision Maker agent to de- termine the optimal action within the [Planning] strategy based on the current state. Its state space includes the ques- tion, the I.I. -egeerrte roadmap, and the accumulated documents from the reasoning history. Given the current state, the agent selects action evaluate progress and decide the next operation, using the following structured format:
Through these three strategies, our multi-agent system adap- tively handles questions of varying complexity. The Rea- soning Router automatically selects the most appropriate strategy based on the characteristics of each question: the Direct Answering Strategy provides immediate responses for general knowledge, the Single-pass Strategy efficiently retrieves information for fact-based questions, and the Multi- Step Strategy addresses complex questions through guided iterative reasoning. This hierarchical approach ensures op- timal resource utilization by aligning computational effort with question complexity while potentially maintaining high response quality. Next, the key focus is to optimize the proxy to learn the knowledge boundaries of the LLM and master the specific capabilities of each agent.
Thought: <Analysis of current progress and objective Action: [[eetrieva]] <subquery content> (Con- tinue with retrieval-file fiiltell-fiterr loop), or [LLM] (Pass to LLM for answering)}
4.2. Collaborative Strategy
With the three specialized agents defined, we now describe how they collaborate to efficiently handle different types of questions. Their coordination follows a structured workflow, enabling adaptive and multi-granular information process- ing through various strategies, detailed below.
5. Multi-Agent Proxy Optimization
Since the final answer generated by the LLM is straightfor- ward to evaluate as the system-level reward, it is intuitive to employ reinforcement learning to optimize the proxy. However, each agent within the proxy serves as an interme- diate module, responsible for only a partial trajectory of the RAG pipeline. This makes difficult define agent-level rewards (Guinet et al., 2024). For example, a high-qualit, generated query might still result in a low sestem-level re- ward due to poor subsequent document filtering. To tackle this challenge, we propose tree-structured rollout approach for robust on-policy learning, utilizing deterministic rollout in the early stages and stochastic rollout in later stages.
The Direct Answering Strategy and Single-pan pass Strategy have already been introduced in the definition of the Rea- soning Router agent, corresponding to the agent outputs [No Retrieval] and [Retrieval] <query content>, respectively.
Multi-Step Reasoning Strategy corresponds the [Planning] output by Reasoning Router agent. Designed to address complex questions requiring high-level roadmap from LLM and multiple retrieval-filter loops, this strat- egy enables iterative information gathering and reasoning through the following three phases:
5.1. Credit Assignment
1. Generate Roadmap: The LLM decomposes the com- plex question into structured set of subgoals, providing
To avoid the sparse reward in traditional single trajectory rollout, we propose the tree-structured rollout for credit
4

---

C-3PO: Compact Plug-and-Plag Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
assignment, which distributes system-level rewards across agents to mitigate the high variance of local rewards for each agent. Through this way, C-3PO effectively redis- tributes the sampling effort from the question level to the action level while maintaining a similar computational bud- get. The core idea is to evaluate each agent's ' contribution by forcing the Reasoning Router to explore all possible rea- soning strategies during the rollout for each question, and enable information sharing across different reasoning paths through action-level exploration.
Monte Carlo Credit Assignment. Instead of constructing single trajectory rollout for each question, we create tree-structured rollout containing multiple trajectories. This structure enables us to trace how individual decisions impact the system-level outcome. For each agent-generated node (sf, aj), we compute its credit reward based on the expected system-level reward:
EiELCH Ra [E(s, at)l
Feredii (s), o/) E, [R( 2
(5)
Deterministic Rollout. Given a question 9. we begin by forcing the Reasoning Router to explore both [No Retrieval] and [Retrieval]. For the [Retrieval] branch, we further force the agent to explore simple reason- ing by directly generating <query content> and complex reasoning through [Planning]. As shown in Figure I, we deterministically construct a decision tree during the first stage of the rollout. Current tree, with depth of 2, enables simultaneous exploration of multiple reasoning paths, pro- viding a comprehensive understanding of how each decision impacts the final outcome.
where Lis), a,) denotes the set of leaf nodes reachable from (sf, af), and Rí is the final reward leaf node I.
Our proposed credit assignment mechanism offers several key advantages over single trajectory rollout: (1) Our rollout thoroughly explores all possible strategies for each question, generating numerous intermediate training exam- ples for each agent. (2) Most importantly, while directly redistributing a single system-level reward to agent nodes in a single trajectory is challenging, our approach accurately estimates the reward of each agent node in probabilistic expectation using the tree-structured rollout.
Stochastic Rollout. Once the overall reasoning strategy is confirmed, the subsequent rollout employs sampling to expand the decision tree. For each non non-terminal node, we randomly sample K (tt) candidate actions from the proxy x for the i-th agent at depth 1.2 Specifically, the tree is expanded using the following children (actions):
5.2. Training Method
For each sampled tree, we disassemble it into individual nodes and group them by their corresponding agents, as illustrated in Figure 1. The token sequence within each node, along with its corresponding reward computed via Eq (5), is added to the replay buffer for Proximal Policy Optimization (PPO) (Schulman et al., 2017). The overall training objective follows the standard PPO framework but incorporates multi-agent aggregation.
K(()
TT-1s instruction.)
(1)
2,
ift 4
K(1)
1, if 4
where instruction; refers to the task-specific instruction for the i-th agent, and K(I) represents the dynamic branching factor at depth I, balancing exploration and computational efficiency. Each sampled action of triggers a state transi- tion governed by:
LCSPO
Exime SEN'
(6)
where the loss LpPo can represent either value loss or policy loss. Details of the loss functions are provided in the Ap- pendix A.I. Following the PPO recipe in Ouyang al. (2022), the PPO for LLMs typically requires an initial- ization from the supervised fine-tuning model. Therefore, we employ our tree-structured rollout with rejection sam- pling (Yuan et al., 2023b) to collect seed data and use cross- entropy loss for the supervised warm-up phase.
++1 = Tsi .
(3)
where j E / denotes the next agent in the predefined strat- egy (Section 4.2).² By recursively applying this sampling process until leaf nodes are reached, we construct a decision tree containing multiple trajectories T:
T= [ ai, $141,))
6. Experiments
where each leaf node includes the final system-level reward (R = for success and R for failure).
6.1. Experimental Setup
Datasets. To comprehensively evaluate our C-3PO, we experiment on both single-hop datasets including Natural Questions (NQ) (Kwiatkowski et al., 2019), PopQA (Mallen et al., 2023), and TriviaQA (TQA) (Joshi et al., 2017), as well as multi-hop datasets including 2WikiMultiHoOAA
"The time step reused as the depth of the tree. Since nodes are expanded layer by layer in our implementation, the depth corresponds to the time step.
'The transition function T is deterministic and involves state updates. See Appendix B.2 for more details.
5

---

C-3PO: Compact Plug-and-P%a Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Table I. Main results. Training Testing with Wiki Retriever.
LLM
Tuned Params
Multi-hop
Single-hop
Methods
Proxy
Average
Server
2Wiki
HQA
Musique
NQ
PopQA
TQA
Direct
Qwen2-72B
X
41.6
45.9
20.5
53.3
24.3
76.3
43.65
Standard
Qwen2-72B
X
34.4
55.7
41.1
60.8
38.0
78.1
51.35
Retriever Fine-tuning
REPLUG
Qwen2-72B
X
109M
33.5
50.4
31.9
55.6
39.3
77.6
48.05
LLM Fine-tuning
Self-RAG
Qwen2-7B
X
7B
50.4
38.5
66.2
51.70
InstructRAG
Qwen2-7B
X
7B
35,8
48.1
39.5
66.6
47.50
Auto-RAG
Qwen2-7B
X
7B
41.8
37.8
53.4
36.8
63.3
46.62
Self-RAG
Qwen2-72B
X
72B
56.6
40.3
78.2
58.33
InstructRAG
Qwen2-72B
X
72B
49.7
57.8
40.1
77.9
56.37
Auto-RAG
Qwen2-72B
X
72B
47.7
45.7
55.3
40.9
72.1
52.34
Intermediate Module
Reranker
Qwen2-72B
Qwen2-7B
7B
32.8
46.4
22.6
57.2
20.9
76.3
42,70
QueryRewrite
Qwen2-72B
Qwen2-1.5B
1.5B
36.2
55.8
40.2
62.3
36.7
78.9
51.68
SKR-KNN
Qwen2-72B
X
40.6
55.6
41.3
61.9
38.8
78.1
52.71
SlimPLM
Qwen2-72B
Qwen2-7B
3x7B
24.4
62.4
76.2
54.33
C-3PO (Ours)
Qwen2-72B
Qwen2-0.5B
0.5B
66.1
66.8
49.4
63.7
46.1
80.4
62.08
t 16.4
1 11.0
8.1
1.3
5.2
1.5
C-3PO (Ours)
Qwen2-72B
Qwen2-1.55
1.5B
65.2
69.0
54.2
65.9
44.8
82.1
63.53
t 15.5
f 13.2
t 12.9
3.5
3.9
3.2
(2Wiki) (Ho et al.. 2020), Musique (Trivedi et al., 2022), and HotpotQA (HQA) (Yang et al., 2018). For each dataset, we only use 6000 randomly sampled questions instead of the full training set.
6.2. Main Results
We report the performance both single-hop and multi-hop datasets in Table 1. First, our C-3PO consistently outper- forms various baselines across different datasets, achieving superior average performance 62.08% and 63.53% with lightweight proxies only 0.5B and 1.5B parameters, re- spectively. This demonstrates the effectiveness of our proxy- centric alignment approach in bridging the gap between the retriever and the LLM. Second, compared to single-hop datasets, our method yields particularly notable gains in challenging multi-hop reasoning tasks. Specifically, C-3PO achieves significant improvements on multi-hop datasets (2Wiki +15.5%, HQA +13.2%, Musique +1 +12.9%), while maintaining strong performance on single-hop tasks (NQ +3.5%, PopQA +3.9%, TQA +3.2%). This significant per- formance gain suggests that, without training original RAG system, our C-3PO effectively enhances the coordina- tion between the retriever and LLM, which is particularly crucial for addressing complex multi-hop tasks. Third, although retriever fine-tuning method (Shi et al., 2024) re- quires fewer tuned parameters, it does not overcome the limitations of standard RAG systems in handling complex cognitive and multi-hop reasoning tasks. Both L.LM fine- tuning and intermediate module methods show promising results, but are constrained by either large tuning parame- ters (7B/7BB) or inconsistent performance across different reasoning datasets. contrast, our C-3PO achieves consis- tent improvements across all datasets with only 0.5B/1.5B additional parameters, demonstrating both efficiency and effectiveness in enhancing RAG systems.
Baseline. We compare our method against a diverse set of baselines, including (1) Direct: Directly answer ques- tions without retrieval. (2) Standard RAG: The standard retrieval-augmentte method retrieves documents based on the question. (3) Retriever Fine-tuning Method: RE- PLUG (Shi et al., 2024). (4) LLM Fine-tuning Method: Self-RAG (Asai al., 2024), InstructRAG (Wei et al., 2025),
and Auto-RAG (Yu et al., 2024a). (5) Intermediate mod- ule Method: Reranker (Li et al., 2023), QueryRewrite (Ma al., 2023), SKR (Wang et al., 2023), and SlimPLM (Tan et al., 2024).
Implementation Details. Following Asai et al. (2024), we construct our retrieval system using the 2018 Wikipedia dump (Yang et al., 2018) as the knowledge source and use centriever-msmarre (Izacard al., 2022) as our dense re- triever. We utilize Qwen2-72B -IIntrutttt (Yang et al., 2024a) fixed LLM server, while Qwen2-0.5B Qwen2-1.55 is trained candidate lightweight proxy for efficient edge de- ployment. In the warm-up phase, we collect 4 solutions for each question with Qwen2-72B-Innttne rate 4e-5, with 3 epochs and a batch size of 512. For the RL phase, we set learning rate of 5e-7 policy model and 5e-6 for value model with batch size of 1024 and maximal depth of 13. More details please refer to Appendix A.
We use a learning
6

---

C-3PO: Compact Plug-and-Plo Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Table 2. Out-of-Distriuttion results. Testing with Google Search Engine.
OOD Datasets
OOD Datasets + Retrieval LLM Server
+ Retrieval
Dataset
FQA
M-RAG
FQA M-RAG
FQA
M-RAG Llama3.3-70B
FQA M-RAG GPT-4o-mini
Average
LLM Servers
Qwen2-72B
Qwen2-7B
Direct Standard
58.2
42.2
40.6
34.4
60.8
43.8
58.8
51.7
48.81
66.4
46.3
57.4
39.2
65.6
41.7
71.6
52.3
55.06
LLM Fine-tuning
Self-RAG InstructRAG
49.4
39.2
44.30
51.2
40.7
45.95
Auto-RAG
48.6
41.6
45.10
Intermediate Module
Reranker
58.2
42.3
36.0
30.7
63.4
41.1
52.0
48.8
46.56
QueryRewrite
67.2
45.9
56.8
38.5
67.4
39.2
72.0
51.8
54.85
SKR-KNN
65.6
44.1
57.8
37.8
66.6
41.7
71.2
52.5
54.66
SlimPLM
60.8
44.7
47.2
35.2
54.8
40.0
68.6
53.7
50.63
C-3PO (Ours)
72.8
50.0
61.0
41.6
71.6
47.2
74.6
55.4
59.28
5.6
3.7
3.2
2.4
4.2
3.4
1 2.6
1.7
1 4.22
6.3. Analysis of Plug-and-Play Proxy
00
wio Tree-strutured Rollout
In this section, conduct comprehensive investigation of performance across three out-of-distribution (OOD) dimen- sions, including OOD datasets, retrieval systems, and LLM servers. This analysis aims to demonstrate that our proxy is a plug-and-play module with superior generalization capabili- ties. In Table 2, we assess modularity and generalization by introducing two recent and challenging OOD datasets: FreshQA (FQA) (Vu al., 2024) and MultiHop-RAAl AA (M- RAG) (Tang& & Yang, 2024). Additionally, replace the retriever with the Google search engine (Schmidt, 2014) and
SOTA Baseline
60
ZWiKI
HQA
Musique
NQ
PopOA
TQA
Average
Figure 2. Ablation Study.
70B, and GPT-4o-mini), with consistent improvements rang- ing from 1.7% to 5.6%. This platform-agne agnostic performance demonstrates the slug-and-play capability of our method, enabling seamless integration with various retrievers and LLM servers without requiring any modifications.
experiment with different LLM servers.
First, LLM fine-tuning approaches exhibit notably inferior performance compared to the standard RAG. This signifi- cant degradation suggests that directly fine-tuning LLMs, while potentially effective for specific tasks, may compro- mise the inherent generalization capabilities and lead to sub- OOD performance. Second, while piermediate-modiiee methods maintain competitive performance, their focus on optimizing individual tasks may compromise their robust- ness. In contrast, our C-3PO holistically optimizes com- munication tasks of the entire RAG pipeline through multi- agent collaboration, effectively aligning the retriever and LLM while preserving their inherent generalization capabil- ities. This enables C-3PO to achieve superior generalization across all OOD settings, consistently outperforming existing approaches with a large margin, 4.22% over the best per- forming bascline on average. Finally, even all three dimen- sions are OOD, C-3PO exhibits robust performance across different LLM servers (Qwen2-72B, Qwen2-7B, Llama3.3-
6.4. Ablation Study
Ablation Training Paradigm. To thoroughly evaluate the effectiveness of different components in our training process, we conduct comprehensive ablation studies across six in-domain datasets. Specifically, we examine the follow- ing variants: (1) "w/o Tree-structure Rollout": variant without the tree-structured rollout and Monte Carlo credit assignment, meaning that we directly optimize each agent using the system-level reward (a single trajectory). (2) "w/o RI.h: The performance in the supervised warm-up phase. (3) "SOTA Baseline": The strongest baseline of each dataset.
The experimental results reveal several key findings. First, removing the tree-structured rollout (and Monte Carlo credit

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
70
Table 3. Ahlation Study on Collaborative Strategy.
In-domain Performancs
O0D Performance
11
Average Inference Time per Questoo
2Wiki
PopQA
FQA
M-RAG
Average Perforaace (%)
Avg Inference Time
10
C-3PO
65.2
44.8
72.8
50.0
[No Retrieval]
41.6
24.3
58.2
42.2
[Retrieval] 64.7
44.6
68.8
48.3
[Planning] 65.5
46.9
74.8
50.4
LAVI
assignment) leads to unstable performance during the RL phase, occasionally degrading below the supervised warm- up model. This degradation can be attributed to the direct use of system-level rewards as supervised signals for all agents, which fails accurately assess individual agent contributions and may mask detrimental actions within suc- cessful trajectories. In contrast, our Monte Carlo credit assignment mechanism enables reward allocation in proba- bilistic expectation through tree-structured exploration, en- suring that each agent receives appropriate feedback for its specific actions. Second, comparing with the supervised warm-up model ("w/o RL"), our C-3PO achieves substan- tial improvements across all datasets. This performance boost demonstrates that end-to-end RL optimization effec- tively aligns the behaviors of multiple agents towards the system-level objectives, going beyond the limitations of supervised learning that only optimizes for local agents. The improvement is particularly significant on challenging datasets like 2Wiki (+1.6%), Musique (+4.5%), PopQA (+1.7%), and TQA (+2.0%), where sophisticated coordina- tion among agents is crucial for task success.
40
Standard
RAG QueryRewrite SlimPLM C:3P0(Ours)
Figure 3. Performance and Efficiency Comparison.
64
Average Accuracy (%)
62
81-
60
C-3PO-1.BB-SFT
58-
C-3PO-0.5B-SFT
C-3PO-1.5B-RL
C-3PO-0.5B-RL
57
150
200
300
Steps
Figure 4, Average Accuracy of C-3PO during RL training.
method (3.6s), C-3PO yields significant performance gains across both in-domain and out-of-generation evaluations. Ablation on Collaborative Strategy. To better under- stand the effectiveness of our collaborative strategy, we conduct ablation studies on OOD datasets by forcing C- 3PO to consistently use fixed strategy for questions, as shown in Table Notably, [No Retrieval] only uti- lizes the inherent knowledge of LLMs exhibit the lowest performance, which is suited for addressing straightforward problems. We observe that the [Planning] strategy consis- tently achieves the best performance among other strategies, which is reasonable given its more sophisticated reasoning process and higher inference cost. Moreover, even our sim- ple [Retrieval] strategy significantly outperforms other baselines shown in Table 2. demonstrating the effectiveness of the retrieval-file capability in our C-3PO.
Furthermore, C-3PO outperforms most methods, such as AutoRAG (Yu et al., 2024a) and SlimPLM (Tan et al., 2024), in both efficiency and effectiveness. This demonstrates that C-3PO achieves an optimal balance between performance and computational efficiency.
Training Dynamics in RL. Figure 4 depicts the average performance trajectory of our C-3PO across six in-domain benchmarks throughout the RL training process. The results demonstrate consistent and stable improvement in accuracy for both model (C-3PO-1.5B and C-3PO-0.5B) during RL training. The final accuracy of C-3PO-1.5B (63.53% sur- passes that of C-3PO-0.5B (62.08%), suggesting that model capacity plays role in the ultimate performance ceiling, Both models exhibit rapid improvement during the initial training phase and eventually outperform the SFT model. This significant improvement highlights the effectiveness of our tree-structured multi-agent optimization framework in optimizing multi-agent collaborative systems over time.
6.5. Detailed Analysis
Inference Efficiency Analysis. To investigate the inference efficiency of C-3PO, compare both the performance and inference cost across different methods, as illustrated in Figure 3. Our approach achieves superior performance (49.2% for in-domain and +4.2% for OOD scenarios) while maintaining a reasonable inference time of 4.8s per ques- tion. Although slightly slower than the Standard RAG
6.6. Performance on HLE
We evaluate our C-3PO on Humanity's Last Exam (HLE) (Phan et al., 2025), a recently released challenging benchmark. We use the OOD Retrieval (Google Search) and L.M Server (Qwen2.5-72B, 2-----7--B-Intruue (Yang al., 2024b))
8

---

C-3PO: Compact Plug-and-Pla, Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Table 4. Main Results Humanity's 's Last Exam (text-only questions, evaluated with official prompt (Phan al., 2025)).
Humanity's 's Last Exam
Method
Bio/Med
Chem.
CS/AI
Engineering Humanities
Math
Physies
Other
Avg.
Proprietary Models (For Reference)
OpenAl Deep Research
26.60
Deepseek RI
8.54
ol
7.75
GPT-4o
2.32
Open-Source Models
Qwen2.5-7B
5.42
3.00
1.76
3.22
4.66
3.58
1.98
4.00
3.52
Qwen2.5-72B
11.31
6.00
1.76
1.61
7.25
3.07
3.96
2.28
4.27
C-3PO (Ours)
9.95
7.00
4.86
9.67
4.66
5.43
3.46
5.71
5.79
Table 5. Comparative Analysis between C-3PO-ICL and C-3PO-RL.
Method
Proxy
2Wiki
HQA
Musique
NQ
PopQA
TQA
Average
Efficiency
C-3PO-ICL
Qwen2-72B
54.1
62.5
45.5
63.4
45.7
82.9
59.01
10.7s
C-3PO-RI.
Qwen2-1.5B
65.2
69.0
54.2
65.9
44.8
82.1
63.53
4.8s
as our system components. As shown in Table 4, our model achieves an average score of 5.79%, demonstrating a sig- nificant improvement over its base model Qwen2.5-72B- Instruct (4.27%). Notably, proprictary models like GPT-4o (2.32%) and approaches the performance of ol (7.75%), highlighting the effectiveness of our method in narrowing the gap between open-source and proprictary models.
achieve optimal performance on more challenging tasks (e.g., 2Wiki, HotpotQA, and Musique). Through reinforce- ment learning. we further optimize individual agent capabil- ities, leading to substantial improvements on the complex tasks. In conclusion, our proxy-centric alignment frame- work demonstrates strong performance across both variants. While C-3PO-ICL showcases the framework's effectiveness through few-shot learning, C-3PO-RL offers practical ad- vantages through reduced computational requirements and enhanced performance on challenging tasks. Our C-3PO-RL successfully aligns the retriever and L.M without modify- ing either component, while facilitating edge deployment and robust performance across diverse scenarios.
model surpasses several
6.7. More Analysis of C-3PO-ICL and C-3PO-RL
We further conduct a comprehensive comparison between C-3PO-ICL (Qwen2-72B-Innn Instructt and C-3PO-RL (Qwen2- 1.5B), where C-3PO-ICL is used to generate seed data through rejection sampling in our supervised warm-up phase. Our experimental results, presented in Table 5, reveal several important findings. First, C-3PO-ICL demon- strates remarkable performance, surpassing all baseline methods across different datasets (as shown in Table and Table 2). This result validates the effectiveness of our frame- work, where collaboration among multiple agents enables effective alignment of the LLM and the retriever. However, this approach faces practical limitations due to substantial inference overhead from multiple LLM queries, making it less suitable for efficient responses and edge deployment.
7. Conclusion
In this paper, we presented C-3PO, a proxy-centric align- ment framework that bridges retrievers and LLMs through lightweight multi-agent system. By leveraging MARL with our proposed tree-structured rollout and Monte Carlo credit assignment, our framework effectively optimizes multiple specialized agents toward the system-level performance without modifying existing RAG components. Extensive experiments demonstrate C-3PO's superior performance and strong generalization capability across different out-of- distribution datasets, retrievers, and LLMs, establishing it practical plug-and-pil and-play solution for RAG systems.
To address these limitations, we introduce a compact proxy that significantly reduces computational requirements while maintaining framework effectiveness. Our analysis reveals that while C-3PO-ICL performs well overall, it may not

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Acknowledgements
value preference optimization for mathematical reason- ing. In Al-Onaizan, Y., Bansal, M., and Chen, Y.- N. (eds.), Findings of the Association for Computa- tional Linguisties: EMNLP 2024, pp. 7889-7903, Mi- ami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 10.53/44////224 findings-emmlp.463. URL attps://aclanthologgy org/2024, www.iffsiniinniiininn 463/.
This work was done during an internship at Tongyi Lab and was supported by Alibaba Research Intern Program. This work was partially supported by National Natural Sci- ence Foundation of China under Grant No. 92470205 and 62222215.
Impact Statement
Chen, G., Zhang, Z., Cong, X., Guo, F. Wu, Y., Lin, Y., Feng, W., and Wang. Y. Learning evolving tools for large language models. In The Thirteenth International Conference Learning Representations, 2025, URL https://openrestt wttp:riwww.neeforuueeeeee net/foormmid==wtrDD...
The advancements in Retrieval- Augmmette Generation (RAG) systems, such those proposed in this work, hold significant potential for diverse applications. By enhancing modularity, generalization, and plug-and-plas capabiliiiis, these systems can empower applications in edge deploy- ment at relatively low computational cost. However, the deployment of RAG systems still poses risks that need to be carefully considered, despite our efforts to mitigate them. These systems heavily rely on external retrieval sources, which may include biased or unreliable data, leading to the propagation of misinformation. In high-stakes applications like medical or legal advice, incorrect or incomplete re- trieval could have severe consequences. Although our proxy is not directly responsible for the content retrieved by the retriever, future research can focus on improving alignment in fairness, robustness, and transparency during information filtering.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A.,
Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong. C., Ferrer, C. C., Nikolaidis, C.,, Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choud- hary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hup- kes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Di- nan, E., Smith, E. M., Radenovic, F., Zhang. F., Synnaeve, G., Lee, G., Anderson, G. L.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Bil- lock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton, J.. Spisak, J., Park, J., Rocca,
References
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. October 2024.
Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self- rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / //operreiie www.eeeeffeeoe wttp://oorrrddddd//ieeeeeeeeeeee
J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The llama 3 herd of models. CoRR, abs/2407.2178.. 2024. doi: 1044555/ARXIV.240..1773 URL https: //doi org/ 10. 10855////xiiv. 2407. 21783.
Chan, C., Xu, C., Yuan, R., Luo, H., Xue, W., Guo, Y., and Fu, J. RQ-RAG: learning to refine queries for retrieval augmented generation. CoRR, abs/2404.00611 2024. doi: 10.4850/ARXXV..40400066 org/10 10. 48444405555a.. www.exiii.........
Fan, W., Ding. Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T., and Li, Q. A survey on RAG meeting 1lms: Towards large language models. In Baeza- Yates, R. and Bonchi, F. (eds.), Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona. Spain, August 25- 29, 2024, pp. 6491-6501 ACM, 2024. doi: 10.1145/ 6637528.3671470. URL https::/doot org/100 1145/ 3637528 3671470.
URL https://doi.
Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: Process supervision without process. In Advances in Neural Information Processing Systems, volume 37, pp. 27789427722. Curran As- sociates, Inc., 2024a. URL https::t///// https://proceeddngss neurips.cc http://pppp..tteeee///////////// 30fte///ccceessfffffaccccoooo000 Reteeepeeeeeeeeeeeeeeeeeeeeeeaae pdf.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Guo, Q. Wang, M., and Wang, H. detrieval-augmentte generation for large language mod- els: A survey. CoRR, abs/2312. 109977 2023. doi: 10. 48550/ARXIV.2312.00997 URL 10. 00.48555/arrce 2092.
Chen, G., Liao, M., Li, C., and Fan, Step-level
10

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Guinet, G., Onidvar-Tehrani, B., Deoras, A., and Callot, L. Automated evaluation of retrieval-augee -agmmentdd language models with task-specific exam generation. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: http:://openrevier www/teeunnnteeee 4jq0V6N1Uz.
2024, doi: 10488500AARXXX2220002276 URL https: //6o1..org/110444550/////111 attp:///0//ivvvv/v 2410.21276.
Izacard, G., Caron, M., Hosscini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL https://openrevies. net/forunnid=-jKN111
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 0058-8057. ijcai.org, www.aa.t.g..ppp...eeeeep org/proceedings..
Joshi, M., Choi, E.,, Weld, D., and Zettlemooyr L. TriviaQA:
A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/1//11-1111 org/P17-1147/.
2024. URL htw.s:tt////ttt 2024/890.
Ho, X., Duong Nguyen, A.-.., Sugawara, S., and Aizawa, A. Constructing a multi-hop QA dataset for compre- hensive evaluation of reasoning steps. In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th Inter- national Conference on Computational Linguistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. /ol///66////222oooooooigoii 580. URL https: //aclanthologg.orgg
K Kwiatkowski, T., Palomaki, J., Redfield, O., Collins. M., Parikh, A. P., Alberti, C., Epstein, D., Polosukhin, I. Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M., Dai, A. M., Uszkoreit, J., Le, Q. and Petrov, S. Natural questions: benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466, 2019. doi: 10.1 A)) _0026.. URL https://doi.org 10. 1al2/ a_072.
Hu, J., Wu, x., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and nigh-performance rlhf framework. arXiv preprint 2rXiv:2405.11143, 2024.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A. T., Kir- illov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kondrich, A., Tul- loch, A., Mishchenko, A., Baek, A., Jiang, A., Pelisse, A. Woodford, A., Gosalia, A., Dhar, A., Pantuliano, A., Nayak, A., Oliver, A., Zoph, B., Ghorbani, B., Le- imberger, B., Rossen, B., Sokolowsky, B., Wang. B., Zweig, B., Hoover, B., Samic, B., McGrew, B., Spero, B. Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn, B. Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lugaresi, C., Wainwright, C. L., Bassin, C., Hudson, C. Chu, C., Nelson, C., Li, C., Sherm J. Conger, C., Barette C., Voss, C., Ding, C., Lu, C., Zhang, C., Beau- mont, C., Hallacy, C., Koch, C., Gibson, C., Kim, C., Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C., Czarnecki, C., Jarvis, C., Wei, C., Koumouzelis, C., and Sherburn, D. Gpt-4o system card. CoRR, abx/2400.21276
Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2308.0322 2023. doi: 10. 10.485550/AXXX.2200.. 00t.:22330000081.... URL https: //doi org/ 10. 48550/arXiv 2308 03281.
Radford, A., Madry, A., Baker-
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019, URL https: //openreview. //opemrerier..tt//foruur/un/aaae hntt//oorrrr///ddBggd81
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mor- datch, I. Multi-agent actor-critic for mixed cooperative- competitive environments. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural In- formation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December
4-9, 2017, Long Beach, CA, USA, pp. 6379-63999 77999639,.90 2017.
Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. Query rewriting in retrieval- augmented large language mod-

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
els. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 5303-5315. As- sociation for Computational Linguistics, 2023. doi: 1/066333V122223MMMLPPPP //doi. //go/ioor/10. 181.11..111111112022 emnlp-main.322
C. B. C.,, Shaaban, M., Ling, J., Shi, S., et al. Humanity's last exam. arXiv preprint arXiv:2501. 14249, 2025.
Qiao, Z., Ye, W., Jiang, Y., Mo, T., Xie, P., Li, W., Huang, F., and Zhang, S. Supportiveness-base knowledge rewrit- ing for eetrieval-augmentee language modeling. CoRR, abs/2406.0B 16, 2024. doi: 10. 10.48350/ARXIV.24066
URL https:
08116. URL https://doi.org// 48550/ 485500/rr1i... arxiv. 08116.
Mallen, A., Asai, A., Zhong, V. Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language mod- els: Investigating effectiveness of parametric and non- parametric memories. In Rogers, A., Boyd-Graber. J., and Okazaki, N. (eds.), Proceedings of the 6lst Anmual Meer- ing of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pp. 9802-9822 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1/116653/1122223a2lllloooom aclanthology .rr.eogg.....ooo.....o 2223g acl-long.5ed 546/.
Schmidt, How google works. Hachette UK, 2014.
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. nigh-dimensional continuous control using generalized advantage estimation. In Bengio, Y. and Le- Cun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: / /arxiv. org/oogg///.. abs/ 11066 02438.
URL https://
Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J.. Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L.,, Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Gr- ishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., and et al. Gemma: Open models based on gemini research and technology. CORR, abs/2403.0822) 2024. doi: 10.45550/ARXXII.2445..0222 //doi. org/ 10/g///1...0 48550 wttp:///XXvv...4000 08295.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1 007000007.7..77 2017. URL http://arxix: org/abs hbg/abs//o 1tt./777..00000 .63477.
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W. REPLUG: retrieval- augmented black-box language models. In Duh, K., Gómez-Adorno H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 8371-8384 Association for Computational Lin- guisties, 2024. doi: 1tt.://0////10..11///////00/000 463. URL naacl 1ong.. 463.
URL https:
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Floundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, Webgpt: Browser-assisted question-answering with hu- man feedback. CoRR, abs/2112.09332,, 2021. URL
Sun, H.-L., Sun, Z., Peng, H., and Ye, H.-J. Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL, 2025a.
http://axxxv...o/aboee 09332.
Sun, H.-L., Zhou, D.-W., Li, Y., Lu, S., Yi, C., Chen, Q.-.., Xu, Z., Luo. W., Zhang, K., Zhan, D.-C., et al. Parrot: Multilingual visual instruction tuning. In ICML. 2025b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J.. Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,
Tan, J., Dou, Z., Zhu, Y., Guo, P., Fang, K., and Wen, J. Small models, big insights: Leveraging slim proxy mod- els to decide when and what to retrieve for llms. In Ku, L.. Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 1 11-16, 2024, pp. 4420- 4436. Association for Computational Linguistics, 2024. URL https: //ooa/..orr......6//////////.aa0 242.
LA, USA, November 28- December 9, 2022, 2022.
doi: 10..B33W.//222AACLLLLLOCC //doi.org/10/ http:///1..11053//////222222
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang,
12

---

C-3PO:
Compact Plug-and-P%, Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Tang, Y. and Yang, Y. Malt Benchmarking aetrieval-uggmeeted generation for multi-hop queries. CoRR, abs/2401.15399 15391,, 2024. doi: 10.48550/ARXIV.
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhut- dinov, R., and Manning, C. D. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsu- jii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18- --21- URL https::: org/D18-1259/
2401.15391. URL https://doi.org 48550/ arXiv. arXiv.2401. 1531.
Trivedi, H., Balasubramanian, N., Khot, T., and Sabhar- wal, A. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguisties, -5:4,9-55,, 2022. doi: 10.1162/ 00475. URL https: httpps://aclantholoogyy org/2022.tac111.333 tac1-1.31/.
Yu. T., Zhang. S., and Feng, Y. Auto-rag: Au- tonomous etrieval-augmenter generation for large lan- guage models. CoRR, abs/2411. 1.19443, 2024a. doi: 10. 48350/AAXXIV.411 19444.. URL https:///oi.onn 10. A48550/ariiv....////
Vu, T., lyyer, M., Wang, X., Constant, N.. Wei, J. W., Wei, J. Tar, C., Sung. Y., Zhou, D., Le, Q. V., and Luong, T. Freshllms: Refreshing large language models with search engine augmentation. In Ku, L.,, Martins, A., and Srikumar, V. (eds.), Findings of the Association for Com- putational Linguistics, ACL. 2024, Bangkok, Thailand and virtual meeting, August 1 1-16, 2024, pp. 13697-13720. Association for Computational Linguistics, 2024. doi: 10. 9663//1/222..ENNDNNS-AALLLLLLLL
Yu, Y., Ping, W. Liu, Z., Wang, B., You, J.. Zhang, C., Shoeybi, M., and Catanzaro, B. Rankrag: Unifying con- text ranking with retrieval-augmmente generation in llms. In Globerson, A., Mackey, I.., Belgrave, D.,, Fan, A., Pa- quet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pP. 121156-121184, Curran Associates, Inc., 2024b.
URL https: :// doi org/10. 1ww.3/11661122222 f findings--ccll.... 8133.
Wang, Y., Li, P., Sun, M., and Liu, Y. Self-knowledge guided retrieval augmentation for large language mod- els. In Bouamor, H., Pino, J., and Bali, K. (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp- 10303-1031 10315. Association for Computational Linguis- EMMLP. 11tp//110.1113/////22033
Yuan, L., Zhang, Z., Li, L., Guan, C., and Yu, Y. A sur- vey of progress on cooperative multi-agent reinforcement learning in open environment. CoRR, 10to//221.0108. 2023a. doi: 10.48550/ARXXI223122000 12.0155 //doi. 10. 48550/arKiv 2312.01058.
URL https:
tics, 2023. doi:
Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. Scaling relationship on learning mathematical reason- ing with large language models. CoRR, abs/2308.01822 2023b. doi: 10.4855/ARRXI.22300001 //doi org/10 10. 1000.48550/aaaa
691. URL https://doli findings-emnie -er.png--emllp...... 691.
Wei, Z., Chen, W.-L., and Meng, Y. InstructRAG: Instruct- ing retrieval-augmeene generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview seenreview.ee/fforee www.neeetoororioo (tm
URL https: 01825.
Zhang, B., Mao, H., Li, L., Xu, 2., Li, D., Zhao, R., and Fan, G. Sequential asynchronous action coordination in multi-agent systems: A stackelberg decision transformer approach. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 59559-59575. PMLR, 21-27 Jul 2024. URL pttpss//proccedinng mlr. htmll
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C.,, Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H. Tang, J., Wang, J., Yang, J. Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang. P., Peng R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan,
Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.. Li. Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt- mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans. LA, USA, December 10 16, 2023, 2023.
Y., Chu. Y., Liu, Y., Cui, Z., Zhang, Z., Guo. Z., and Fan, Z. Qwen2 technical report. CORR, 04//2407.106771 2024a. doi: 10445550/ARXXX.247.10666 10671. URL https: //doi.org//11 10. 48550/arXiv... 2tt.5///2XXxv..27.11106.. 240.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C. Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024b.
13

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmente Generation
Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z.,, Feng, Z., and Ma, Y. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- 13372.
tics. URL http:// http:///aaa.v....aaa..aas/ess
Zhou, Y., Liu, Y. Li, X., Jin, J., Qian, H., Liu, Z., Li, C., Dou, Z., Ho, T., and Yu, P. S. Trustworthiness in aurieval-augmentee generation systems: survey, CoRR, abs/2409. 101022 2024. doi: 10-48500/RXIV.24009. 10102. URL https://doi.org org/10. 10. 48550/arXiv. 2409.10102.
Zhu, C., Dastani, M., and Wang, survey of multi-
agent deep reinforcement learning with communication. Auton. Agents Multi Agent Syst. 38(1):4, 2024. doi: 10. 0007/S104488-023-99633--- URL attpss//dof org/ 10. 10.1007/810458--03--66333
14

---

C-3PO: Compact Plug-and-Pla, Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
A. More Implementation Details
A.1. RL Training Process
Having obtained the credit rewards that reflect each agent's contribution, we develop an optimization framework to guide end-to-end training across all agents. The key idea to use these credit signals for optimizing the collaborative behavior of the entire system. The optimization objective for multi-agent system can be formulated maximizing the expected credit rewards:
[0) ETTTT
ΣΣ Teredi (s at)
Since each agent's action sequence of tokens, we decompose this optimization using Proximal Policy Optimization (PPO) (Schulman et al., 2017; Yuan al., 2023a; Zhu et al., 2024) follows:
C.300 Σ (p.
(8)
EN
Specifically, for each agent i, we define:
(0) = BrrNe ΣΣ min (0)A clip(ri,m (0), -,, e)A{mm)
(9)
Ta(a [ )m where ri... (0) Rew (ai, m))
is the probability ratio, si.. represents the concatenation of current state and the first m
tokens the action sequence for agent at time step and af.m denotes its m-th token. We compute the advantage estimate ( where M is the token length of the action sequence. To estimate state values across the multi-agent system, employ centralized state-value function Vo that takes each agent's 's state smm as input. The value function is optimized to minimize the mean squared error (Lowe et al., 2017):
using GAE (Schulman al., 2016): Åi.m
(Ø) Eysnt ΣΣ (Vo(s) (Voss G m²
(10)
where Gi Al, + Vo( is the empirical return. The final optimization objective combines the policy and value losses:
Eiro (0, Φ) ()) + (e)
(11)
where e, controls the weight of the value loss. This joint objective enables end-to-end training of both policy and value networks across all agents.
A.2. Implementation Details
Supervised Warm-up Phase: We utilize Llama-Factory (Zheng et al., 2024) as our training framework for the initial supervised fine-tuning phase. The detailed pyper-parameters for this phase are presented in Table
Reinforcement Learning Phase: For the RI. training phase, we adopt OpenRLHF (Hu et al., 2024) as our primary training framework, coupled with VLLM (Kwon et 2023) inference engine. The complete set of RL training hyper-parameters is detailed in Table 7. To initialize both the policy and value models, we leverage the model obtained after one epoch of supervised fine-tuning, with the language model head replaced by a value head for the value model.
Inference Phase: For the deployment of our system, we establish comprehensive infrastructure that integrates multiple components:
Retriever Server: We construct our retrieval server using the 2018 Wikipedia dump (Yang et al., 2018) as the primary knowledge source. We employ contriever msmarco document retrieval. Our inference code also supports Google search engine (Schmidt, 2014) as the retriever server.
comtriever-mmmaroo (Izacard et al., 2022) as our dense retriever for efficient and effective
15

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Table 6. Key hyperparameters the supervised warm-up phase.
Hyperparameter
Value
Learning Rate
4c-5
Batch size
512
#Epochs
Optimizer type
AdamW (Loshchilov & Hutter, 2019)
Chat template
Qwen (Yang al., 2024a)
Base model
Qwen2-1.5B or Qwen2-0.5B (Yang et al., 2024a)
Cutoff length
4096
Warmup ratio
0.03
LR scheduler type
Cosine
Table 7. Key hyperparameters in the RL phase.
Hyperparameter
Value
Learning Rate of Policy model
5e-7
Learning Rate of Value model
5e-6
Batch size
1024
KL Coefficient
0.005
Optimizer type
Adam
Prompt max len
4096
Generate max len
2048
Maximal depth
13
LR scheduler type
Cosine
LLM Service:
We integrate SGLang* as our LLM server, which provides compatibility with various state-of-the-arr language models, including 1ecen--22-Instruuu (Yang al., 2024a) and 1amma33.-70B-Inttttte (Dubey et al., 2024). Moreover, also support GPT series models
Inference Optimization: Our implementation supports two high-performance inference engines: SCLang and VLLM,
allowing users to optimize for different deployment scenarios and hardware configurations.
This modular architecture ensures both flexibility in model selection and efficiency in deployment, while maintaining robust performance across different configurations.
A.3. Dataset Details
Table 8. Training Dataset Statisties.
Multi-Hop
Single-Hop
Data Name
Total
2WikiMultiHopQA
HotpotQA
Musique
Natural Questions
PopQA
TriviaQA
Raw Data Size
167,454
90.447
19,938
79,169
12,868
78,785
448,661
Our Train Data Size
6,000
6,000
6,000
6,000
6.000
6,000
36.000
Sampling ratio
3.5%
6.6%
30.1%
7.5%
46.6%
7.6%
8.02%
In-domain Datasets. As shown in Table conduct extensive in-domain experiments three single-hop and three multi-hop datasets. For each dataset, we randomly sampled 6,000 instances as the training set, with sampling ratios detailed in Table 8. Overall, we utilize only 8% of the original data as the training set. For the in-domain test sets, we randomly sampled 1,000 instances as the test set.
"ntcps://docs.eg http://.occ..ssggg....... 'We use -0n---44-mini2224---2 in our outtaf--generalization experiments.
16

---

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Table 9. Out-of-generalizzttin Dataset Statistics.
FreshQA
Multihop-RAG
Data Size
500
2556
Out-of-gen gen----generaiiatiion Datasets. To comprehensively evaluate the plug-and-play capability of our C-3PO in out-of- distribution generalization scenarios, we introduce two recent challenging datasets: FreshQA (Vu et al., 2024) and Multihop- RAG (Tang & Yang, 2024). The statistics of the OOD datasets are summarized in Table 9.
A.4. Overall Algorithm
In this section, we present the inference process of C-3PO for reference, as shown in the Algorithm 1.
Algorithm 1 Inference Process of our C-3PO
input question q, the retrieval server (Retriever), the LLM server (LLM), the proxy model in our C-3PO æ, instruction for different agent (Reasoning Router, Information Filter, and Decision Maker).
output The Answer.
1: a' (g. instruction:) [Reasoning Router agent)
2: if Ta'==(B [No Retrieval] then
3: Answer + (LM g) [Direct Answering Strategy, if g does not require retrieval]
4: else if a== [Retrieval] <query content> then
5: docs + ( <query content>)
6: selected (a)s(a?) f- ((g does, instructiong) (Information Filter agent)
Answer - LLM((, selected docs) (Single-pass Strategy, if q requires retrieval and is simple question]
8: else
9: Accumulated_does <- ()
10: Roadmap < (g) (al == [Planning]] }
11: a² < (g, Roadmap, Accumulated docs, instructiona) (Decision Maker agent)
12: while a² * [LLM] do
docs <- (sarieval(<subuurry content> a²)
selected docs(a") (q, does, instructiong) (Information Filter agent)
15: Accumulated does <- (Accumulated docs) U (selected does)
16: a² f- (q, Roadmap, Accumuatede.um instructions) (Decision Maker agent)
17:
end while
18:
Answer +- LLM((g Accumulated.docs) (Multi-step Reasoning Strategy, if requires retrieval and complex]
19:
end if
B. Instructions and State Transition Function
B.1. Instructions for Each Agent
In this section, we details the state space and action space fo each agent our C-3PO.
Reasoning Router. The Reasoning Router agent operates with state space S {q). where q represents the input question. This agent is responsible for determining whether retrieval necessary for the given question and assessing the question complexity when retrieval is needed. For question that does not require retrieval, this agent outputs [No Retrieval] If the retrieval is needed, the agent outputs one of the following actions based on the complexity of the question q: for simple questions requiring retrieval: [Retri eval] <query content>, initiating single-pass retrieval-fille loop, where <query content> defines the space of possible queries; for complex questions: [Planning], triggering the multi-step reasoning strategy. The specific examples are as follows:
17

---

C-3PO: Compact Plug-and-PPl Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Reasoning Router
Instruction Reasoning Router
You are an intelligent assistant tasked with evaluating whether given question requires further information through retrieval or needs planning arrive at an accurate answer. You will have access to large language model (LLM) for planning or answering the question and retrieval system provide relevant information about the query.
Instructions:
1. EEvaluate the Question**: Assess whether a precise answer can be provided based on the existing knowledge of LLM.
Consider the specificity, complexity, and clarity of the question.
**Decision Categories:
the question is complex requires a planning phase before retrieval, your response should be:
[Planning]
If the question requests specific information that you believe the LLM does not possess pertains to recent events or niche topies outside LLM's knowledge scope, format your response follows:
[Retrieval] 'YOUR QUERY HERE
If you think the I.I.M can answer question without additional information, respond with: [No Retrieval]
3. **Focus Assessment**; Avoid providing direct answers to the questions. Concentrate solely on determining the necessity for retrieval or planning-
State of Reasoning Router Now, process the following question:
Question: {question}
Output (All possible Actions) of Reasoning Router
For No Retrieval
[No Retrieval]
% For Retrieval
[Retrieval] <query content> (for simple questions)
[Planning] (for complex questions)
Information Filter. The state space of Information Filter consists of the question q, the retrieved documents, and the current objective (if in [Planning] mode), i.e., S {q, retrieved documents] for single-pass strategy (Retrieval <query content), or 2 {4, retrieved documents, current objective} for multi-step reasoning strategy ([Planning]).
Information Filter
Instruction for Information Filter You intelligent assistant tasked with analyzing the retrieved documents based given question and the current step's objectives. Your role is to determine the relevance of each document in relation the question and the specified objectives.
Instructions:
1. ee Anallyee Relevance**: Evaluate each document whether it aligns with the objectives of the current retrieval step and contains direct answer the question.
**Thought Process**, Provide brief analysis for each document, considering both the content and the retrieval objectives.
3. **Filter Documents** After your thought process, generate list of document indices indicating which documents to retain.
State of Information Filter
Now, process following question:
Current step's objectives: (objective) (only for [Planning] mode)
Question: {question)
Documents: [documents]
Output of Information Filter
Thought: <Analysis of each 'ocuments>>
18

---

C-3PO: Compact Plug-and-Pla, Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
Action: |<Selected document [D>>]
Decision Maker. The Decision Maker agent operates with state space S = {q. Accumulated Documents, Roadmap }.
Based the current state, this agent outputs one of two possible actions: [Retrieval] <subquery content> (requesting
additional retrieval-filtering loop through the sub-query) [LLM] (passing all accumulated documents LLM for generating the final answer).
Decision Maker
Instruction for Decision Maker
You intelligent assistant tasked with determining the next appropriate action based the provided existing documents, plan, and question. You access to large language model (LLM) for answering question retrieval system gathering additional documents. Your objective is to decide whether to write query for retrieving relevant documents or to generate a comprehensive answer using the LLM based on the existing documents and plan.
Instructions:
***aalltte Existing Documents**; Assess existing documents determine sufficient to answer the question.
2. *FFoolo the Plan**; Understand the next steps outlined in the plan.
3. **Decision Categories:**
If the existing documents is insufficient and requires additional retrieval, respond with:
[Retrieval] 'YOUR QUERY HERE'
If the existing documents is adequate to answer the question, respond with:
[LLM]
4. Foocus * Do answer the question directly; concentrate identifying the next appropriate action based on the existing documents, plan, and question.
State of Decision Maker
Now, process the following question:
Existing Documents: {accumulated documents}
Roadmap: {roadmap}
Question: {question}
Output Decision Maker
Thought: (Your analysis for current situation (need retrieval for additional informations LLM to answer))
Action: Your decision based the analysis ([Retrieval] <subquery content> or [LLM])]
B.2. State Transition Function
Given state sf and an action of in each agent E N, the transition function T in our framework is deterministic. Based on the three collaborative strategies introduced in Section 4.2, the state transitions are defined as follows:
Direct Answering Strategy ([No Retrieval]): this strategy, the LLM directly generates the answer without retrieval, resulting in state transitions between agents.
Single-pass Strategy ([Retrieval] <query content>): This strategy involves state transition between the Reasoning Router and Information Filter agents:
T = [g} x A [[Retrieval] Kquery content>} retrieval S
{g. retrieved documents}
(12)
where S represents the initial state with the question q, and S represents the state for the Information Filter agent after retrieval. The Information Filter is responsible for filtering the helpful documents based on S
Multi-Step Reasoning Strategy ([Planning]): This strategy involves multiple state transitions in a cyclic manner:
Reasoning Router - Decision Maker:
T: S = {q) A [[Planning] [plancing]
S =({ (g, Accumulated Documents, Roadmap), (13)
19

---

C-3PO: Compact Plug-and-Pae Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
where the roadmap is generated by the LLM and the accumulated documents is empty for initial step.
Decision Maker + Information Filter:
T: : = {q. Accumulated Documents, Roadmap) x A = [Retrieval] <subquery content>,
current objective]
retrieval
8 {q. retrieved documents, current objective),
(14)
where the current objective is generated by the Decision Maker agent S
Information Filter Decision Maker:
T : S² = {q, retrieved documents, current objective) x A = {Selected Documents}
Gitter S new
{g. Updated Accumulated Documents, Roadmap).
(15)
This retrieval-filter loop between the Decision Maker agent and the Information Filter agent continues until the Decision Maker outputting [LLM] or a termination condition is met. The state transitions in our C-3PO are deterministic and well-defined, ensuring consistent behavior across the multi-agent system.
C. Additional Experimental Results
C.1. More Analysis in RL
70
Strategy Ratio (%)
60
50
40
30
Direct Answering
20
Single-pass
10
Multi-Step Reasoning
0
50
100
150
200
250
300
Step
Figure 5. Strategy Ratio in RI. training process.
Strategy Ratio during RL Training Process. As introduced in the Section 4.2, our C-3PO incorporates three distinct strategies: Direct Answering Strategy ( (DNo Retrieval)), Single-pass Strategy ([Retrieval] <query content>), and Multi-Step Reasoning Strategy ( ([Planning]), ) each designed for different question complexities. Figure 5 reveals how C-3PO dynamically adapts its strategy selection during the RL training process.
The evolution of strategy ratios shows clear trend: the Multi-Step Reasoning Strategy gradually dominates the decision space, stabilizing at approximately 60-70%, while the Single-pass Strategy decreases to around 30%. The Direct Answering Strategy maintains a consistent but low ratio of about 5%. This distribution pattern offers several insights into our framework's learning behavior: First, the limited use of Direct Answering Strategy aligns with our experimental findings in Table confirming that solely relying on the model's inherent knowledge is insufficient for complex question-answering tasks. Second, the substantial proportion of Single-pas. pass Strategy usage demonstrates scenarios where simple external information retrieval suffices. Most notably, the increasing preference for Multi-Step Reasoning Strategy indicates that our C-3PO recognizes the importance of multi-step reasoning in handling complex queries effectively. These learned ratios demonstrate that our framework effectively develops balanced strategy selection mechanism. By dynamically choosing appropriate strategies based on question complexity, our C-3PO achieves a balance between computational efficiency and reasoning capability, making it well-suited for real- world applications.
C-3PO's ability to identify
20

---

C-3PO: Compact Plug-and-PP Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
ZWikiMultiHoOOA
0.25
HotpotQA
0.6
Musique
0.25
0.5
0.20
0.4
Densky
0.15
Denoity
0.15
Density
0.10
0.10
0.2
0.05
0.05
0.00
0.00
0.0
2
8
10
12
13
18
12 13
14
15
5
10
12
13
Depth
Depth
Depth
2.0
NaturalQuestions
PopQA
TriviaQA
1.50
1.5
1.25
Density
Denalty
Denaity
1.0
1.0
0.75
0.50
0.00
2
6
7
8
9
10
11
12
13
1
2
5
6
8
10
11
12
14
15
16
2
5
6
8
10
11
12
13
14
15
18
Depth
Depth
Depth
Figure 6. Depth Distribution in Test set.
Depth Distribution. Figure 6 presents the depth distribution of reasoning processes different datasets, revealing distinct patterns that align with the inherent complexity of each task. We observe three clear categories of reasoning depth requirements: (1) Simple Complexity (Depth 3-5): Datasets like NaturalQuestions PopQA, and TriviaQA show concentrated distributions around depths 3-5, indicating that most questions in these datasets can be effectively addressed with the Direct Answering Strategy (NNo Retrieval]) and Single-pass Strategy ([Retrieval] <query content>). This aligns with the nature of these datasets, which primarily contain straightforward factual questions. (2) Mixed Complexity: HotpotQA and 2WikiMultiHopAA exhibit multiple peaks in the depth distribution, with notable concentrations around depths 3-4 and depths 9-15, indicating diverse range of question complexity. This bimodal distribution suggests that while some questions require simple reasoning steps, others need more complex reasoning chains. (3) Complex Complexity: Musique displaies broader distributions with significant density at higher depths (9-13), particularly pronounced in their rightward skew. Musique's distribution is notably spread across higher depths, consistent with its design for multi-step reasoning questions.
These distributions validate our framework's adaptive capability in handling queries of varying complexity. The frame- work naturally adjusts its reasoning depth based on task requirements, demonstrating efficient resource utilization while maintaining the ability perform deep reasoning when necessary.
D. Additional Prompts
In this section, we supplement additional prompts based on Appendix B.
D.I. Roadmap
In the multi-step reasoning strategy, we introduce an LLM -LM-generated roadmap as high-level guidance for our proxy. The specific prompt and example are as follows:
21

---

C-3PO: Compact Plug-and-Pla Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
An example of Roadmap
Prompt Roadmap
You are am expert assistant tasked with analyzing the following question and formulating a detailed plan. You will utilize a retrieval system to gather relevant information in your planning. Your goal is to analysis the question and provide structured sequence of actions to address it effectively.
Instructions:
1. *Question Analysis**; Identifying the core components of the question. Determine what key information we currently know and what additional information is necded through retrieval. **Step By Step Planning**: answers.
Develop detailed plan step by step. Focus the planning process rather than providing direct
**Focus on P Keep your response clear and structured, concentrating solely on the analysis and planning aspects.
Now, process the following question:
Question: {question)
Example of generated roadmap
(Take What nationality is the director of film The Caper The Golden Bulls? as an example) To answer the question, we need to find information about director the film "The Caper of Golden Bulls." " Then we should determine which nationality is the director born using the retrieval.
Step 1: Retrieve the relevant documents that mention the film "The Caper of the Golden Bulls."
Step 2: Identify the director of the film from the retrieved documents.
Step 3: Retrieve the relevant information about 'Which nationality is the director born..
Step 4: Provide the answer based on the retrieved information.
D.2. Evaluation
In our experiments, we found that traditional evaluation metrics such as Exact Match (EM) are often inaccurate, as they strictly require identical generated answers. To address this issue, following previous work (Zheng et al., 2023; Vu et al., 2024), leverage LLM to assess answer correctness by comparing the predicted answer with the ground truth. The specific example is as follows:
Prompt of Evaluation
You are a precise answer validator. Your task is to compare the predicted answer with set of acceptable correct answers and determine if the prediction matches any of them.
Input format:
Question: [The question text]
Correct Answers: [Array or list of acceptable correct answers]
Predicted Answer: [The answer to be evaluated]
Rules:
Consider semantic equivalence, not just exact string matching
2. Ignore minor differences in formatting, spacing, or capitalization
3. For numerical answers, consider acceptable margin of error if applicable
4. For text answers, focus the meaning rather than exact wording
The predicted answer is considered correct if matches ANY ONE of the provided correct answers The matching can be exact or semantically equivalent to any the correct answers
7. Return only "True" if the predicted answer is correct, or "False" if it is incorrect.
Now, process the following question:
Question: {question}
Correct Answer: [tru_answer]
Predicted Answer:
22


Reviews:
[{"review_id": "FlS4JCVaKe", "reviewer": "Reviewer_B7TK", "summary": "The paper proposes C-3PO,  which introduces a multi-agent system that optimizes retrieval, query generation, and information filtering. It uses multi-agent reinforcement learning (MARL) with tree-structured rollout and Monte Carlo credit assignment. Experiments show that C-3PO significantly enhances RAG performance across in-domain and out-of-distribution datasets, demonstrating its plug-and-play flexibility and strong generalization capabilities​", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer B7TK\n\nWe sincerely appreciate your thorough review. We have carefully addressed each of your concerns below:\n> Issue 1 & W3\n\nWe appreciate the mentioned related works, and would like to clarify several important points:\n1. First, we acknowledge the importance of these works. We will **cite these related works and incorporate detailed discussions in our revised version**.\n2. Regarding the timeline and reproducibility of mentioned related works:\n- [1] is published on **Jan 25, 2025**, 2 days later than ICML abstract submission deadline **Jan 23, 2025**.\n- [3] is published on Dec 17, 2024, which can be considered **concurrent work**.\n- For [2] and [3], despite their relevance, **the absence of publicly available implementation** makes faithful reproduction challenging.\n3. Our evaluation have covered 3 major baseline categories (retriever/LLM fine-tuning and intermediate approaches) across 6 in-domain and 2 OOD datasets, demonstrating thorough effectiveness and generalization.\n\n> Issue 2\n\nWe apologize for any confusion, while the experimental setup of each row in Table 3 was presented in `Line 418-426`, we would like to provide further clarification:\n- [No Retrieval]: Relies solely on LLM's inherent knowledge\n- [Retrieval]: Employs single retrieval-filter loop\n- [Planning]: Utilizes multi-step reasoning\n\nThe full C-3PO system's ability to adaptively select strategies leads to robust performance across different datasets.\n\n> Issue 3\n\nWe appreciate the reviewer's careful examination of our tree-structured rollout. We would like to provide additional clarification:\n1. Regarding the performance gains:\n- We observe substantial gains on challenging tasks such as Musique, HQA, and PopQA.\n- While our method **enhances agent decision-making instead of directly answering the question**, the performance ceiling **ultimately** depends on the LLM (remains frozen in C-3PO). Our approach still outperforms many recent methods that fine-tune LLMs (AutoRAG) as shown in Tables 1/2.\n\n2. On complexity concerns:\n- We would like to emphasize that our tree-structured rollout **does not introduce additional computational overhead** compared to standard RL. \n- Traditional RL methods typically require **multiple independent sampling trajectories per question in parallel**, and our approach may reuse partial trajectories during rollout. In SGLang inference system, reused context can be efficiently cached. Algorithmically, our approach **redistributes these sampling efforts from the question level to the action level**, enabling more efficient credit assignment in multi-agent systems through expectation-based reward distribution. \n- We can also control the breadth/depth of the tree to balance exploration and cost (Eq. 4), making it flexible for various computational budgets.\n\nWe believe the clarifications show that our approach offers meaningful improvements and maintains computational efficiency.\n\n> Eval Criteria\n\nWe appreciate the reviewer's suggestion regarding evaluation metrics. We would like to clarify our choice of metrics and provide additional results:\n\n1. **Limitations of EM metrics**:\n- Through our preliminary studies, we observed that rule based metrics, such as EM, can be unreliable, especially when using frozen LLMs that may express correct answers in varied formats.\n- Inaccurate rewards from strict rule based matching could potentially harm the RL training.\n\n2. **The choice of LLM-based evaluation**:\n- Recent benchmarks, such as FreshQA, HLE (Humanity's Last Exam), have increasingly adopted LLM-based evaluation due to its ability to capture semantic correctness beyond EM.\n- In our human verification process, we found that Qwen2-72B-instruct demonstrates high accuracy in assessment, making it as a more reliable source for both evaluation and RL rewards.\n\n3. **Additional EM Results**:\n- To address this concern, we provide some EM results (due to limited chars, a full table can refer to the response of W4 for Reviewer TgQU).\n\n|Methods|2Wiki|HQA|Musique|NQ|PopQA|TQA|AVG|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Standard|26.1|41.3|31|52.1|38.1|73.8|43.73|\n|Auto-RAG|44.7| 41.3| - |43.8|39.2|72.1|48.22|\n|C-3PO-0.5B|60.5|61.1|50.1|65.9|52.7|80.3|61.76|\n|C-3PO-1.5B|63.7|63|54.8|67.7|53.8|82|64.16|\n\n> W1\n\nRegarding inference efficiency, we already presented a detailed analysis in `Section 6.5 and Figure 3`. It shows that C-3PO achieves the best performance-efficiency trade-off. \n\n> W2\n\nWe appreciate the suggestion and would like to clarify that our evaluation **already covers a diverse range of LLMs across different scales and types**, such as `Qwen2-7B`, `Qwen2-72B`, `Llama3.3-70B`, and `GPT4o-mini` (commercial API), as shown in Tables 1/2.\nWhile we acknowledge that testing on other commercial APIs like Claude and o1 would be interesting, the significant costs make such extensive evaluation prohibitively expensive.\n\nWe sincerely thank you for your detailed comments and hope our responses have adequately addressed your concerns.", "claims_and_evidence": "Most of the claims in this paper are supported by the evidence. \n\nSome issues:\n- The paper does not compare against certain related baselines that also use tree-based rollout [3] or multi-agent training for RAG [1, 2], making it unclear how C-3PO improves over existing methods.\n- There is no detailed analysis of the role of each agent in the system. While Table 3 may provide some insights, the experimental setup is unclear, and it is not explicitly explained what each row represents.\n- The performance gain of tree-structured rollout over standard reinforcement learning appears marginal in Figure 2, raising concerns that the proposed approach may be overly complex without substantial benefits.\n\n\nReferences:\n\n[1] Chen, Yiqun, et al. \"Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning.\" arXiv preprint arXiv:2501.15228 (2025).\n\n[2] Shao, Zhihong, et al. \"Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.\" arXiv preprint arXiv:2305.15294 (2023).\n\n[3] Jiang, Jinhao, et al. \"RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement.\" arXiv preprint arXiv:2412.12881 (2024).", "methods_and_evaluation_criteria": "The choice of datasets is reasonable.\n\nHowever, it is unclear why EM/F1/Accuracy scores were not used as the final performance metrics, given that they are widely adopted in prior work (numerous references support this). It is recommended to at least provide some numbers on one/more of these metrics.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "See above sections for details.", "relation_to_broader_scientific_literature": "This paper proposes an online RL training method that is reasonable and has some novelty. However, the added complexity raises concerns about whether the performance gains justify the additional computational cost.", "essential_references_not_discussed": "RAG with multi-agent systems:\n\nChen, Yiqun, et al. \"Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning.\" arXiv preprint arXiv:2501.15228 (2025).\n\nShao, Zhihong, et al. \"Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.\" arXiv preprint arXiv:2305.15294 (2023).\n\nZhu, Junda, et al. \"ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator.\" arXiv preprint arXiv:2405.18111 (2024).", "other_strengths_and_weaknesses": "Strengths:\n\n- Clear Modular Design for Multi-Agent Collaboration\n\n- Strong Performance on RAG Tasks\n\n- Detailed prompt format, simple,mentation details provided\n\nWeaknesses:\n\n- Additional studies using alternative metrics (e.g., EM/F1) and inference efficiency analysis would strengthen the empirical results.\n\n- The method should be tested on a wider range of LLM APIs and local models to assess its generalizability across different deployment settings.\n\n- Including additional baselines that use tree-based rollout or multi-agent training would provide a more comprehensive comparison.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnznpgL1p6", "reviewer": "Reviewer_TgQU", "summary": "The paper proposes C-3PO, a plug-and-play multi-agent system used to enhance the alignment of retrievers and LLMs in RAG systems. Specifically, C-3PO consists of three LLM agents: a reasoning router designed to determine the reasoning strategy for a specific question, an information filter agent used to identify relevant documents from retrieved ones, and a decision maker agent designed to determine the optimal action based on the current state. To optimise these agents, the paper uses reinforcement learning to train these agents and proposes a simple tree-structured rollout approach for robust on-policy learning. For the tree-structured rollout, it computes reward by enumerating all possible reasoning strategies for each question. Experimental results on both in-domain and out-of-domain datasets validate the effectiveness of the proposed  C-3PO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TgQU,\n\nThank you for your thorough and constructive review of our paper. We would like to address each of your concerns in detail:\n\n> W1\n\nWe appreciate your concern about computational efficiency. We would like to clarify that our tree-structured rollout **does not introduce additional computational overhead** compared to standard RL:\n\n- Traditional RL methods typically require **sampling multiple independent trajectories per question in parallel**, and our approach may reuse partial trajectories during rollout. In SGLang inference system, the reused context can be efficiently cached for further inference. Algorithmically, our approach just **redistributes this sampling effort from the question level to the action level**, maintaining a similar computational budget.\n\n- The tree structure actually provides **several advantages**:\n  - It enables more systematic exploration of the action space\n  - It allows for expectation-based credit assignment\n  - It reduces the variance in training compared to random sampling\n\n- We can also control the breadth and depth of the tree to balance between exploration and computational cost (Eq. 4), making it flexible for different computational budgets.\n\nTherefore, while our approach may appear computationally intensive at first glance, it actually offers a more structured and efficient way to explore the action space within the same computational constraints as traditional RL methods.\n\n> W2\n\nWe apologize for any confusion regarding the warm-up phase. We would like to clarify several key points:\n- As mentioned in `Section 5.2 and Appendix A.2`, we collect seed data through rejection sampling from Qwen2-72b-instruct, specifically gathering 2 correct solutions for each question.\n\n- The detailed training hyper-parameters are provided in `Table 4`.\n\n- To further validate the effectiveness, we conducted comparative experiments between C-3PO-RL and C-3PO-ICL in `Table 8`. These results demonstrate the feasibility of our warm-up strategy.\n\nWe hope these clarifications address your concerns about the warm-up phase implementation.\n\n> W3\n\nWe agree that the evaluation metrics deserve more prominence in the main text. We will move a concise version of the evaluation metrics `from Appendix D.2 to Section 6.1`, making these important details more accessible while maintaining the paper's flow and readability.\n\n> W4\n\nThank you for raising this important point about evaluation methodology. We would like to address this concern from multiple aspects:\n\n1. **Additional EM Results**: \n\nWe have conducted additional experiments using the EM metric. The results show that on the EM metric, C-3PO still achieves significant improvements over all baselines:\n\n| Methods|2Wiki|HQA|Musique|NQ|PopQA|TQA|AVG|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Direct|36.4|36.8|17.5|44.1|25.1|73.4|38.88|\n|Standard|26.1|41.3|31|52.1|38.1|73.8|43.73|\n|REPLUG|25.2|39.8|24|43.2|37.7|74.3|40.7|\n|Self-RAG| - | - | - |41.7|40.5|74.9|52.36|\n|InstructRAG|45.9| - | - |51.6|40.9|75.6|53.5|\n|Auto-RAG|44.7|41.3| - |43.8|39.2|72.1|48.22|\n|ReRanker|29.8|37.6|19.4|47.6|20.7|73.3|38.06|\n|QueryRewrite|42.9|47.3|44.5|60.6|40.3|79.1|52.45|\n|SKR-KNN|38.6|54.8|37.7|56.2|38.6|73.5|49.9|\n|SlimPLM| - | - |19.8|57.6| - |76.4|51.26|\n|C-3PO-0.5B|60.5|61.1|50.1|65.9|52.7|80.3|61.76|\n|C-3PO-1.5B|63.7|63|54.8|67.7|53.8|82|64.16|\n\n2. **Limitations of Traditional Rule Based Metrics**:\n- Through our preliminary studies, we found that rule based metrics, such as EM, can be unreliable, especially when working with frozen LLMs that may express correct answers in unpredictable formats.\n- These inaccurate rewards from strict rule based matching could potentially harm the training of reinforcement learning.\n\n3. **Adoption of LLM-based Evaluation:**\n- Recent famous benchmarks like FreshQA and Humanity Last Exam (HLE) increasingly adopt LLM-based evaluation to capture semantic correctness beyond exact matching.\n- This trend reflects the community's recognition of the limitations of traditional metrics for complex QA tasks.\n\n4. **Reliability of Our Evaluation**: \n\nWe conducted rigorous human verification of `Qwen2-72B-instruct`'s evaluation capabilities.\n\nWe understand the importance of using standardized metrics. However, we believe that combining both traditional and LLM-based evaluation provides a more comprehensive assessment of model performance. We appreciate this feedback and have enhanced our evaluation section accordingly.\n\n> References Not Discussed\n\nWe sincerely thank you for suggesting these valuable references. We will incorporate these citations and related discussions in our revised manuscript to better position our work in RAG.\n\nWe appreciate again for your time and effort in reviewing our paper. We believe that addressing these concerns has helped strengthen our work, and we hope our responses have satisfactorily addressed your questions. We look forward to your further feedback.", "claims_and_evidence": "The claims are well-supported by the experimental results.", "methods_and_evaluation_criteria": "The proposed method is sold. However, the paper relies on an LLM (see Appendix D.2) to evaluate the generated answers, which raises concerns about potential biases and reliability. It is unclear which LLM is used for evaluation and how different LLMs would affect the results.  The paper should also provide results on existing QA evaluation metrics, such as Exact Match (EM) and F1-score, to offer a more standardized and quantitative assessment of the answers.", "theoretical_claims": "There is no theoretical analysis in the paper.", "experimental_designs_or_analyses": "The experimental design appears reasonable and well-structured.", "relation_to_broader_scientific_literature": "Existing works on leveraging intermediate component to bridge the gap between retrievers and LLMs focus on optimising a single task in isolation, which may lead to suboptimal performance. The paper proposes C-3PO to faciliate seamless communication between retrievers and LLMs.", "essential_references_not_discussed": "The following iterative/adaptive RAG models are missing from the paper: \n1. Trivedi, Harsh, et al. \"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.\" ACL 2023. \n2. Jiang, Zhengbao, et al. \"Active retrieval augmented generation.\" EMNLP 2023. \n3. Su, Weihang, et al. \"DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Models.\" ACL 2024. \n4. Jeong, Soyeong, et al. \"Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity.\" NAACL 2024.", "other_strengths_and_weaknesses": "Strengths: \n1. The paper is well-written and easy to follow. \n2. The proposed C-3PO seems novel. Experimental results on six in-domain datasets and two out-of-the domain datasets validate the effectiveness of the proposed C-3PO. \n3. Ablation studies are conducted to verify the effectiveness of each component. \n\nWeaknesses: \n1. The proposed tree-structured rollout method incurs high computational cost, as it requires exploring all possible reasoning trajectories for each question. This exhaustive search significantly increases the training overhead, limiting its practicality. \n\n2. The paper states that it employs a warm-up phase to train the multi-agent system. Despite some descriptions of the supervised warm-up phase, the details remain unclear. Although the appendix provides some additional information, it does not fully explain the specifics of the training process, including the training data and training methodology. \n\n3. The introduction of the evaluation metrics should be moved from Appendix to the main paper. \n\n4. A major concern is the use of LLM for evaluation, raising questions about bias and reliability. It is unclear why conventional QA metrics such as Exact Match and F1 are not reported.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please see the questions in above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgPAhb2sop", "reviewer": "Reviewer_Tndf", "summary": "This paper proposes a proxy-centric framework that enhances communication between retrievers and Large Language Models (LLMs) through a lightweight multi-agent system named C-3PO. Unlike the vanilla RAG framework, the proposed framework incorporates multiple specialized LLM agents to manage different stages of the pipeline:\n1. Reasoning Router Agent: Evaluates the complexity of the query to determine whether retrieval and reasoning are required. For simple queries, the process proceeds directly to the Information Filter Agent. For complex queries, the system enters a planning mode, engaging all agents collaboratively.\n2. Information Filter Agent: Processes and extracts relevant information from the retrieved data.\n3. Decision Maker Agent: Identifies the optimal action during the planning mode.\n\nTo train the framework, the authors propose a tree-structured rollout mechanism for credit assignment, addressing the issue of sparse rewards, and utilize a PPO training objective. Experiments conducted on multiple QA datasets across various RAG systems, including those with retriever tuning or LLM tuning, demonstrate significant improvements in performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Tndf,\n\nThank you for your thorough and constructive review. We would like to address each of your concerns in detail:\n\n> W1\n\nThank you for raising this important issue about the framework's generalizability. We would like to clarify several aspects:\n1. Design Philosophy:\n- Our proxy-centric alignment is inspired by human interaction patterns in knowledge-intensive tasks, where information gathering and reasoning are fundamental operations.\n- The core design focuses on how proxy can align retrievers and LLMs through planning and reasoning to collect information, rather than being strictly QA-specific.\n\n2. Preservation of LLM Capabilities:\n- Importantly, our framework does not fine-tune the LLM, preserving its general capabilities (e.g., writing, summarization).\n- The agents serve as information gathering and coordination proxies, which are inherently applicable to various knowledge-intensive tasks beyond QA.\n\nWe will include this as an important direction for future work, while maintaining that the current design principles are fundamentally task-agnostic.\n\n> W2\n\nThank you for raising this important issue. We would like to address your concerns from multiple aspects:\n1. **Training Efficiency:**\n- Our approach **does not introduce significant additional training overhead** compared with standard PPO.\n- Traditional RL methods typically require **sampling multiple independent trajectories per question in parallel**, and our approach may reuse partial trajectories during rollout. In SGLang inference system, the reused context can be efficiently cached for further inference. Algorithmically, our approach just **redistributes this sampling effort from the question level to the action level**, maintaining a similar computational budget.\n\n2. **Inference Efficiency:**\n- `Figure 3` shows our method does not introduce substantial inference latency.\n- This efficiency is achieved through our Decision Maker, which dynamically allocates optimal strategies to balance computation and performance.\n- `Figure 5` shows how different strategies evolve during RL iterations, providing transparency into our method's adaptation.\n\n3. **Regarding Baseline Comparisons:**\n- While works [1,2] are not yet open-sourced for faithful reproduction, we have included another reranking work in `Tables 2/3`.\n- We chose QueryRewriting as our efficiency baseline due to its parameter efficiency (1.5B) and consistent stability across scenarios.\n\nWe appreciate these suggestions and will incorporate the related works [1,2] to better position our work.\n\n> Q1\n\nThank you for this detailed question about computational efficiency. Let us break down the number of forward passes required for each strategy:\n1. **Empirical Evidence:**\n- `Figure 6` provides `detailed distributions of inference depths` across different datasets\n- `Figure 3` shows the inference latency of C-3PO compared to baselines\n\n2. **Forward Passes by Strategy:**\n- [No Retrieval]: 1 proxy pass + 1 LLM pass\n- [Retrieval]: 2 proxy passes + 1 LLM pass\n- [Planning]: 2 LLM passes + variable proxy passes (distribution shown in Figure 6)\n- Note that Proxy are lightweight (0.5/1.5B) and LLM can be 7/72B\n\nThis strategic allocation of computational resources allows us to maintain efficiency and achieve superior performance. The actual number of forward passes is optimized for each specific query rather than using a fixed number for all cases.\n\n> Q2\n\nThank you for this insightful observation. The [Planning] strategy, while powerful, involves collecting additional information that may introduce more noise and potentially mislead the LLM. Meanwhile, for many RAG datasets, a well-crafted query combined with effective filtering ([Retrieval] in our C-3PO) might suffice, especially when search engines can retrieve the necessary information in a single pass. This suggests that the optimal strategy depends on the alignment between dataset and proxy capabilities rather than following a one-size-fits-all approach.\n\n> Q3\n\nThank you for this thoughtful question about extending our framework beyond QA tasks. While our C-3PO framework is specifically designed for **knowledge-intensive tasks** where proxy-centric alignment between retrieval and LLM components is crucial, the tasks in [3] primarily focus on **logical reasoning** that may not heavily rely on external knowledge. For such pure logical reasoning tasks, our retrieval-oriented multi-agent system might offer limited benefits in its current form.\n\nHowever, we believe our framework could be adapted for logical reasoning tasks by:\n- Combining a reasoning verification agent\n- Integrating training approaches similar to Deepseek-R1 for pure reasoning tasks\n\nWe appreciate this suggestion as it opens up interesting directions for future research.\n\nWe sincerely appreciate your detailed review and thoughtful questions. We believe addressing these points has helped strengthen our paper. We look forward to your further feedback.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Here arises a minor concern regarding the generalization capability beyond QA tasks. The current agent functions and pipeline appear to be QA-oriented, and the evaluation datasets are exclusively focused on QA tasks. It would be beneficial to either explicitly position this work as specific to QA or extend the evaluation to include a broader range of tasks to demonstrate the framework's versatility and applicability beyond question answering.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "For the RAG baseline involving LLM fine-tuning, the use of Qwen2 to control variables raises concerns about reproducibility. To ensure fairness and simplicity, I think a more straightforward baseline could simply employ retrirever with an instruction-tuned Qwen2-7B server. Instruction tuning is a standard and widely accessible approach compared to the custom fine-tuning proposed in this work, making it a more practical and reproducible baseline for evaluation.", "relation_to_broader_scientific_literature": "This work proposes a multi-agent cooperative framework and training method, which extends beyond QA tasks. Its modular design and tree-structured rollout approach offer potential for broader applications with customizable agents and pipelines.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n1. The use of multi-agent systems to handle complex tasks is a highly sought-after approach, and this paper presents a well-designed framework with significant performance improvements.\n2. The paper is well-written and easy to follow, making it accessible to a broad audience.\n\n### Weaknesses\n1. The designed agent functionality and pipeline appear to be overly specific to QA tasks, limiting the framework's generalizability to other applications.\n2. The reported improvements come at a significant cost, including the computational and resource overhead of training these customized agents and the increased complexity during inference. \n\nDespite the inclusion of an Inference Efficiency Analysis to highlight performance trade-offs, the comparison baseline is somewhat outdated and relies on costly methods (e.g., query rewriting). Recent works have focused on more efficient single-dimension improvements for RAG (e.g., reranking [1], drafting [2]), which were omitted in the main experiments and the Efficiency Analysis. \n\n[1] RankRAG: Unifying context ranking with retrieval-augmented generation in llms. NIPS 2024.\n\n[2] Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting. ICLR 2025.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. To better assess the efficiency and the cost of the framework, could you elaborate on the average number of 8B LLM forward passes (from the additional agents) required for each task? \n\n2. In Table 3, the [Planning] module shows limited improvement on 2Wiki, PopQA, and M-RAG, while demonstrating significant improvement on FQA compared to the [Retrieval] module. Could you provide insights into this discrepancy?\n\n3. Do you think the designed multi-agent framework could be applied to broader tasks beyond QA? For example, tasks in [3]. If not, what adjustments would be necessary to made?\n\n[3] Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models. NIPS 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HsEfPYktmt", "reviewer": "Reviewer_stM2", "summary": "This paper presents a novel proxy-centric framework for addressing the alignment challenge in RAG systems. The key innovation lies in its introduction of a lightweight multi-agent system that mediates between retrievers and LLMs without requiring modifications to either component. The framework is inspired by human search behavior and implements three specialized agents that work collaboratively to optimize the RAG pipeline. The key technical contributions of C-3PO include a proxy-centric alignment architecture that maintains plug-and-play flexibility, an efficient multi-agent system design, and a tree-structured rollout approach for multi-agent reinforcement learning that enables effective reward credit assignment. Through extensive experimentation in both in-domain and out-of-distribution scenarios, the authors demonstrate that their approach significantly enhances RAG performance while maintaining generalization capabilities across different retrievers and LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer stM2,\n\nThank you for your thoughtful review and constructive suggestions. We particularly appreciate your recommendations about extending our evaluation to more challenging benchmarks and exploring alternative training strategies. These insights will help strengthen our work. We would like to address your suggestions in detail:\n\n> W1\n\nThank you for this valuable suggestion about testing on more challenging benchmarks. We agree that evaluation on complex reasoning tasks is crucial for validating our framework's capabilities.\n\nWe have conducted additional experiments on Humanity's Last Exam (HLE) text-only questions using Google as the retriever (an out-of-domain search engine for C-3PO):\n\n| LLM                  | Method         | n docs | HLE (text) |\n|----------------------|----------------|:------:|:----------:|\n| Deepseek-R1          | -              | -      |     8.6    |\n| o3-mini (high)       | -              | -      |     14     |\n| Qwen2.5-72B-Instruct | Vanilla LLM    | -      |    4.85    |\n| Qwen2.5-72B-Instruct | Vanilla RAG    |   10   |    5.35    |\n| Qwen2.5-72B-Instruct | C-3PO          |   10   |    6.46    |\n| Qwen2.5-72B-Instruct | C-3PO-Planning |   10   |    6.84    |\n\nThe results show that:\n- C-3PO improves performance by 1.11 % compared to vanilla RAG\n- C-3PO-Planning further boosts performance by 1.49%\n- These improvements demonstrate our framework's effectiveness even on highly challenging reasoning tasks with out-of-domain retrieval\n\nWe will include these results in our revised manuscript to provide a more comprehensive evaluation of our framework's capabilities.\n\n> W2\n\nThank you for this insightful suggestion about exploring from-scratch RL training. We agree that this direction, similar to Deepseek-R1's approach, is very interesting and could potentially lead to more general training strategies for multi-agent systems.\n\nWhile our current warm-up approach helps ensure stable and smooth training in the multi-agent setting, we believe exploring from-scratch training could:\n- Reduce dependency on seed data collection\n- Potentially discover novel agent interaction patterns\n- Lead to more generalizable training strategies\n\nWe will include this as an important direction for future research. The challenge of balancing exploration and stability in from-scratch multi-agent RL training presents an exciting opportunity for advancing the field.\n\nWe sincerely appreciate your valuable suggestions that have helped us identify important directions for both immediate improvements and future research. Your feedback about evaluation on challenging benchmarks has already led to meaningful additional results. We will incorporate these improvements in our revised manuscript.", "claims_and_evidence": "The authors make several key claims that are well-supported by the presented evidence:\n* Claim 1: The proposed proxy-centric framework (C-3PO) effectively bridges retrievers and LLMs while maintaining plug-and-play flexibility.\n\nThe authors detailed technical descriptions that clearly distinguish their approach from existing methods to support this claim in Section 1 and 2. The authors demonstrate the plug-and-play flexibility through extensive experiments in both in-domain and out-of-distribution scenarios (Table 1 and Table 2). Evidence appears convincing as they test with unseen retrievers and LLMs to validate generalization capabilities.\n\n* Claim 2: The tree-structured rollout mechanism and Monte Carlo credit assignment effectively optimize multi-agent coordination.\n\nThe authors provide a theoretical foundation in Section 5 with detailed mathematical formulation. The effectiveness of this approach is empirically validated through comprehensive ablation studies in Section 6.4, which quantitatively demonstrate its advantages over alternatives. The design are well-motivated and the consistent performance improvements across different experimental settings further strengthen this claim.\n\n* Claim 3: The human-inspired multi-agent collaborative system enhances RAG performance.\n\nThe evidence for this claim is particularly strong in Section 6.6 and Appendix C, where the authors demonstrate the effectiveness of their approach through in-context learning experiments. Notably, C-3PO-ICL shows impressive performance even without any training, outperforming many baselines from Tables 1 and 2. The detailed case studies and comprehensive analysis across different tasks and scenarios provide convincing support for the benefits of the multi-agent collaborative approach.", "methods_and_evaluation_criteria": "* Methods:\nThe proposed proxy-centric framework makes sense as it addresses the key challenge of aligning retrievers and LLMs without modification. The multi-agent design mimicking human search behavior is intuitive and well-motivated. The use of MARL with the proposed tree-structured rollout is appropriate for optimizing multiple agents towards the system-level objectives. The lightweight design ensures practical applicability while maintaining effectiveness.\n\n* Evaluation criteria:\nThe evaluation is comprehensive and well-structured. The authors conduct extensive experiments across a diverse range of datasets, including three single-hop datasets (NQ, PopQA, TriviaQA) and three multi-hop datasets (HotpotQA, 2WikiMultihopQA, MuSiQue). The inclusion of FreshQA and MultiHop-RAG as out-of-distribution test sets further demonstrates the model's robustness and adaptability. Furthermore, the authors evaluate C-3PO's plug-and-play and generalization capabilities by testing with previously unseen retrievers and LLMs. This comprehensive evaluation protocol provides strong evidence for the framework's versatility and practical applicability in real-world settings.", "theoretical_claims": "This paper does not make formal theoretical claims requiring rigorous proofs.", "experimental_designs_or_analyses": "The experimental design and analyses in this paper are thorough and well-executed. The authors conduct comprehensive experiments across a diverse range of datasets, including both single-hop and multi-hop benchmarks , which effectively validates the model's capability in handling varying complexity levels of tasks.\n\nParticularly noteworthy is their extensive evaluation of out-of-distribution (OOD) generalization across three dimensions: OOD datasets (FreshQA and MultiHop-RAG), different retrieval systems (from Contriever to Google Search), and various LLM servers (from Qwen to GPT-4). This comprehensive OOD evaluation protocol strongly supports their claims about the framework's plug-and-play capability and generalization ability.\n\nThe ablation studies are systematic and well-designed. The authors thoroughly examine both the training paradigm and collaborative strategies, providing clear insights into each component's contribution. The comparison of different fixed strategies particularly helps understand the model's behavior. Furthermore, the efficiency analysis comparing both performance and inference cost across different methods demonstrates practical considerations for real-world deployment.", "relation_to_broader_scientific_literature": "This work makes meaningful connections to several important research directions in the broader scientific literature:\n\nFirst, the work builds upon and extends retrieval-augmented generation (RAG) research. While previous works mainly focus on modifying either retrievers (e.g., REPLUG) or LLMs (e.g., Self-RAG, Auto-RAG), this paper proposes a novel perspective of using a lightweight proxy for alignment, which provides a more practical and efficient solution.\n\nSecond, the tree-structured rollout mechanism for multi-agent reinforcement learning builds upon classic MARL literature. This work presents a solution by introducing Monte Carlo credit assignment with tree-structured exploration, advancing the field of multi-agent coordination.", "essential_references_not_discussed": "After a thorough review of the paper's citations and related work section, I did not identify any essential references that are missing from the discussion. The citation coverage appears complete and up-to-date, providing adequate context for understanding the paper's contributions and positioning in the broader research landscape.", "other_strengths_and_weaknesses": "Strength\n1. The proxy-centric alignment framework is innovative, offering a practical solution that enhances RAG systems without modifying existing components. This approach significantly reduces deployment barriers while maintaining strong performance.\n2. The multi-agent collaborative system design is elegant and well-motivated, effectively mimicking human search behavior through specialized agents. The lightweight implementation (0.5B/1.5B parameters) demonstrates impressive efficiency.\n3. The training methodology combining MARL with tree-structured rollout and Monte Carlo credit assignment is technically sound and novel, effectively addressing the complex challenge of multi-agent optimization.\n4. The empirical validation is remarkably comprehensive, demonstrating strong performance across both in-domain scenarios and out-of-distribution settings (datasets, retrievers, and LLMs), convincingly validating the framework's effectiveness and generalization capability.\n\nWeakness\n1. While the current evaluation is comprehensive across in-domain and out-of-distribution settings, testing on more challenging benchmarks like Humanity's Last Exam (HLE) would further validate the model's capabilities on highly complex reasoning tasks.\n2. The training paradigm currently relies on seed data collection. While this is a practical approach, exploring the possibility of from-scratch RL training (similar to recent advances in RL (Deepseek-R1)) could provide interesting insights into more general training strategies, though this is beyond the scope of the current work.", "other_comments_or_suggestions": "The paper is well-written and clearly structured. The authors have done a thorough job in presenting their ideas and experimental results. The figures and tables are informative and well-organized. I would encourage the authors to explore the framework's capabilities on more challenging tasks (such as HLE) and investigating its potential for broader applications.", "questions_for_authors": "The key points have been covered in the previous sections. I have no additional questions that would substantially impact my evaluation of this work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes C-3PO, a proxy-centric framework that facilitates communication between retrievers and LLMs through a lightweight multi-agent system. The reviewers found the method to be interesting and reasonable, yielding strong empirical performance. Some reviewers raised concerns including discussion with similar (and concurrent) works, evaluation metrics, and LLM-based evaluation, which were addressed by the authors quite comprehensively in the rebuttal. 

Minor note: Some references used the arXiv version instead of the conference proceeding version (e.g., RankRAG is a NeurIPS'24 paper and InstructRAG is an ICLR'25 paper), which are better to be updated.
Decision: Accept (poster)