Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
Authors: Wei Yang
Defu Cao
Jiacheng Pang
Muyan Weng
Yan Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Agent
Adaptive Collaboration
Policy Optimization
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: While scaling individual Large Language Models (LLMs) has delivered remarkable progress, the next frontier lies in scaling collaboration through multi-agent systems (MAS). However, purely autonomous MAS remain ``closed-world'' systems, constrained by the static knowledge horizon of pre-trained models. This limitation makes them brittle on tasks requiring knowledge beyond training data, often leading to collective failure under novel challenges. To address this, we propose the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework, a principled paradigm for human--agent collaboration. HILA trains agents to learn a metacognitive policy that governs when to solve problems autonomously and when to defer to a human expert. To operationalize this policy, we introduce Dual-Loop Policy Optimization, which disentangles immediate decision-making from long-term capability growth. The inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning, transforming expert feedback into high-quality supervised signals that strengthen the agent's reasoning ability. Experiments on challenging mathematical and problem-solving benchmarks show that HILA, equipped with Dual-Loop Policy Optimization, consistently outperforms advanced MAS, establishing a principled foundation for collaborative and continually improving agentic systems.

Published as a conference paper at ICLR 2026
ADAPTIVE COLLABORATION WITH HUMANS:
METACOGNITIVE POLICY OPTIMIZATION FOR MULTI- AGENT LLMS WITH CONTINUAL LEARNING
Wei Yang; Defu Cao; Jiacheng Pang, Muyan Weng, Yan Liu University of Southern California (wyang930, defucao, pangi, muyanwen, yanliun ca
ABSTRACT
While scaling individual Large Language Models (LLMs) has delivered remark- able progress, the next frontier lies scaling collaboration through multi-agent systems (MAS). However, purely autonomous MAS remain "closed-world" sys- tems, constrained by the static knowledge horizon of pre-trained models. This limitation makes them brittle on tasks requiring knowledge beyond training data, often leading to collective failure under novel challenges. To address this, we propose the Human-In-the-Loop Multi-Agent Collaboration (HILA) frame- work, a principled paradigm for human-agen collaboration. HILA trains agents to learn a metacognitive policy that governs when to solve problems autonomously and when to defer to a human expert. To operationalize this policy, we introduce Dual-Loop Policy Optimization, which disentangles immediate decision-maker from long-term capability growth. The inner loop applies Group Relative Policy Optimization (GRPO) with cost-aware reward to optimize deferral decisions. while the outer loop implements continual learning, transforming expert feed- back into high-quality supervised signals that strengthen the agent's 's reasoning ability. Experiments on challenging mathematical and problem-solving bench- marks show that HILA, equipped with Dual-Loop Policy Optimization, consis- tently outperforms advanced MAS, establishing principled foundation for col- laborative and continually improving agentic systems. The code available at com - Mar.eeaadyyHLLLL..aa HAIAAAIAACC. git
INTRODUCTION
While scaling individual Large Language Models (LLMs) has produced remarkable progress, the next frontier lies in scaling collaboration through multi-agent systems (MAS) (Hong et al. 2023
Chen et al. 2023b; Jiang et al.] 2023; Ning et al. 2023 Han et al. 2025 Wang et al. 2025a
Chen et al. 2025b). By coordinating multiple agents to tackle problems beyond the reach of any single model, this paradigm has inspired wave of innovations from structured debates dynamic workflow optimization (Zhang et al. 2024a Qiao et al. 2024 Han et al.) 2025). Yet these systems face an inherent ceiling: no matter how sophisticated their interaction protocols, purely autonomous agents remain fundamentally cloled-world, Their knowledge horizon is bounded by pre-training corpora (Wang al.) 2023b (Srivatsa al., 2024: Du et al., 2023 Liu al. 2024).. While they can recombine existing in information, they cannot generate new knowledge adapt to unseen contexts. This creates vulnerabilities when tasks demand real-time information, domain-specific expertise, or reasoning patterns absent from training (Zhang et al., 2024d, Chen et al.) 2025a). In such cases, internal collaboration alone cannot bridge the gap, often leading to collective failure. To break this ceiling and enable open-ended intelligence, a paradigm is needed. We argue that the most principled path is to integrate external human expertise, transforming closed systems into adaptive frameworks capable of continual learning and growth (Sun et al. 2025 Zou al.).
Within this clrled-world paradigm, research has followed two main directions. sizes optimizing autonomous collaboration through increasingly sophisticated interaction proto-
The first empha-
"Equal contribution.

---

Published as a conference paper at ICLR 2026
cols. Frameworks based on structured debate (Chan al. 2023 Liu al. 2024 topology con- trol (Ong et al. 2024; Chen et al., 2024b), and workflow graph optimization (Zhang et al. 2024b Li et all 2025a) have demonstrated notable improvements in refining and recombining agents' in- ternal knowledge. However, these methods largely engage in collective introspection (Zhang et al. 20244] Chen et al.] 2024a), maximizing the of existing information without extending beyond the aggregate knowledge boundary. They act as powerful integrators, but not true learners capable of acquiring genuinely new capabilities. Recognizing this intrinsic limitation, second line of work has sought to incorporate human expertise (Takerngsaksiri et al. 2025 Mozannar et al.) 2025) Many
systems (Liu et al., [023] Pandya al. 2024) treat humans primarily as
passive oracles or supervisors for sub-tasks. This leaves two critical questions unresolved: when to
defer to the expert, often reduced to heuristics such as low-confidence thresholds rather than learned policies (Kenton et al.) 2024 Li et al., 2024b); and how to learn from human input, which is typi- cally applied as one- time fix rather than as a catalyst for long-term capability growth (Mu et al.) 2024 Wang et al. 2025b). Importantl, human intervention holds the potential to operate at multiple levels, offering both localized corrections to specific reasoning errors and broader adjustments that reshape the overall collaborative process (Triem & Ding. 2024 Grondin et al., 2025).
This analysis highlights that the key challenge not whether agents can interact with humans, but whether they can do intelligently and strategically. Addressing this requires metacognitive pol- icy, a high-level strategy for reasoning about both self-competence and peer competence to guide collaboration. Such a policy must solve two intertwined problems: when to ask, which demands moving beyond heuristics to model uncertainty and balance the risk of failure against the cost of intervention; and how to grow, which requires mechanisms that turn expert feedback into lasting capability improvements rather than one-time fixes. A paradigm that unifies these elements is es- sential for building open and continually evolving agentic systems.
To address these challenges, we propose Human-In-the-Loap Multi-Agent Collaboration (HILA), a principled framework for adaptive human collaboration. The central idea of HII.A is not simply to place a human in the loop, but to equip agents with metacognitive policy that decides when external expertise is needed and how it should be used. HILA instantiates this idea through three coordinated components: (i) Autonomous Operation, where agents first attempt to solve problems using their current capabilities; (ii) Metacognitive Assessment, where they assess their own confidence and the difficulty of the task to recognize potential knowledge boundaries; and (iit) Strategic Deferral, where external expertise is invoked as a targeted intervention rather than passive fallback. To optimize this metacognitive behavior, we further introduce Dual-Loop Pol- icy Optimization (DLPO), a training framework that separates short-term intervention decisions from long-term capability improvement. Its inner loop applies Group Relative Policy Optimization (GRPO) with cost-aware reward to refine deferral behavior online, while its outer loop turns ex- pert feedback from deferral events into supervised training signals that continually strengthen the model's 's underlying reasoning ability.
In summary, the main contributions of this paper are as follows:
We propose the Human-in-the-Loop Multi-Agent Collaboration (HILA) framework, paradigm for human-agent collaboration that equips agents with a metacognitive policy to decide when to strategically defer human expertise.
We introduce Dual-Loop Policy Optimization (DLPO), a training methodology that sep- arates short-term deferral decisions from long-term capability growth. The inner loop em- ploys GRPO with a cost-aware reward, while the outer loop leverages expert feedback as supervised signals for continual learning.
Extensive experiments on mathematical reasoning and general problem-solving bench- marks demonstrate that HILA with DLPO outperforms both autonomous multi-agent sys- tems, establishing robust foundation for continually improving agentic collaboration.
2 RELATED WORK
Large language models (LLMs) acting alone are limited by context length, sequential generation, and restricted skill coverage, which constrains their ability to solve complex reasoning tasks (Gabriel
et al. 2024 Ping al. 2025b: Li al. 2025b Ye al. 2024 Xia et al. 2025a). To mitigate these
2

---

Published as a conference paper at ICLR 2026
issues, multi-agent systems (MAS) have been widely explored, where multiple LLMs are organized into collaborative structures for collective problem solving (Hong et al. 2023 Yang et al. 2025a
Jiang et al.] 2023 Qiao et al. 2024) Yang & Thomason 2025). Early efforts relied prompt braset-based based paradigms that assign predefined roles or workflows, enabling debate, critique, or corporate-style pipelines (Du et al. 2023; Chan et al., [2023] Yang et al. 2025b; Chang et al. 2025). While effective, these designs lack adaptability since their interaction protocols are fixed and cannot evolve through experience. More recent work moves toward structured coordination and adaptive communication. Predefined schemes employ debate or peer-review across chains, trees, or graphs (Liu et al.) 2024
Qian al.] (024), while adaptive methods restructure interactions dynamically via routing, pruning,
or workflow search (Yang et al. 2026; Chen et al. 2026 Yue et al. 2025 Ping et al. 2025a).
complementary line of research introduces human-in-the-loop collaboration. Humans have been positioned as supervisors, oracles, or evaluators, providing corrections or domain knowledge to
strengthen agent performance (Takerngsaksiri et al. 2025| Mozannar et al., [025]] Liu et al., [023] Pandya et al. 2024). Closely related, Siedler & Gemp (2025) study LLM-mediated guidance in
MARL, where an LLM serves as a natural-language controller that interprets and delivers interven- tions to shape agents' learning trajectories and accelerate training. However, such systems often rely on heuristics (e.g., confidence thresholds) to trigger deferral (Kenton et al., 2024] Li et al.) 2024b), and feedback is usually treated one-time fix rather than signal for sustained capability growth (Mu et al. 2024; Wang et al. (0255)). Recent discussions highlight that human involvement can occur at multiple levels, from correcting local reasoning errors reshaping global collaborative
dynamics (Triem & Ding 2024 Grondin et al. 2025). Together, these directions have advanced the field of MAS, yet challenges remain in moving beyond closed-word world recombination toward open and adaptive collaboration. A detailed review of related work is provided in Appendix A
Human-in-the-Lee Multi-Agent Collaboration (HILA)
Dual-Loop Pollcy Optimization (DLPO)
Inner-Loap
AL for Meda Policy
Eaput
Defer for Human
Decision Reasoning
Action
Execution
w
Reasoning
Quantdore
Agent
Quantions
Rellout
Out of Boundary
Agent
Answer
Proactive Guidance
Multi- Mruti--Aeent RL
Reward
(R R...al
Commeniicaton
Ing. ORPI
Neremnan Cemeut
Cogaltiee Mgri
Stratogla Action Delode Craleting Pnevlegee
Outer-Loop
Defar
CL for Improving
Borld
Set
Grede
Decisionn Ressoning
Human Feedback
moriag
Datar grit
Offline 6
1OB
Data
Meta-Cognittve
Stratogic Action
Knowledgee Boundary
Appntt
Agent
fine-tuning
feading
Overview of the proposed HILA framework and its Dual-Loop Policy Optimization (DLPO) training paradigm. Left: HILA coordinates multi-agent collaboration with both proactive human guidance and reactive expert feedback via metacognitive states and strategic actions (EVAL, CREATE, DEFER). Right: DLPO optimizes the meta-policy in an inner RL loop with cost-aware re- wards, and expands the model's knowledge boundary in an outer continual-learmin loop by storing DEFER-triggered human feedback as offline supervision.
Figure 1:
3 METHODOLOGY
Our methodology builds multi-agent system designed for adaptive collaboration with human ex- pert. Figure [ provides an overview of HILA and its Dual-Loop Policy Optimization, highlighting how multi-agent collaboration is coupled with human. At its core metacognitive policy that enables agents to reason about both their own competence and that of their peers, thereby deciding when to act autonomously and when to defer to external expertise. We formalize this collaborative process as a Metacognitive Markov Decision Process, which provides the foundation for our frame- work. The framework is then specified through a structured cognitive state space and a functional action space. Finally, introduce Dual-Loop Policy Optimization algorithm that combines inforcement learning to refine the metacognitive policy with continual learning to integrate expert feedback into lasting capability growth.
3

---

Published as a conference paper at ICLR 2026
3.1 PRELIMINARIES: THE METACOGNITIVE MARKOV DECISION PROCESS
We model human-agent collaboration as Metacognitive Markov Decision Process (Meta-MDD) which formalizes decision-making over high-level cognitive strategies such as autonomous problem- solving or deferral to human expertise. This abstraction provides a principled foundation for defining states, actions, transitions, and rewards in our collaborative framework. The full formalization and detailed design are provided in Appendix B
3.2 A FRAMEWORK FOR HUMAN-AGENT COLLABORATION
Building on the Meta-MDP, we introduce a framework that operationalizes human-agent interac- tion through three components: (i) a structured cognitive state space that encodes problem context and metacognitive assessments, (ii) a functional action space representing high-level collaboration strategies, and (iii) interaction protocol that specifies coordination across rounds. These elements allow agents to reason about tasks while regulating their decision boundaries in a principled way.
3.2. STRUCTURED COGNITIVE STATE SPACE
metacognitive policy should not make decisions based solely on narrow local response, since effective coordination depends on a broader view of the current reasoning context. In our setting, the policy should consider three main sources of information. First, it needs access to the task context, including the original question and relevant interaction history, which defines the decision environ- ment and the current objective. Second, it should observe the agent's own self context, namely its latest solution and local reasoning status, which reflect its current belief and confidence. Third, should incorporate peer context from other agents, since their responses provide complementary evidence about agreement, disagreement, and alternative solution paths. Together, these sources pro- vide richer basis for deciding whether to preserve the current trajectory, generate a new solution, or seek stronger external assistance.
To further support meta-policy decisions beyond the raw task/self/peer context, we optionally aug- ment 8, with a compact set of structured cognitive signals that instantiate monitoring-conttr view of metacognition together with standard docial-informationn cues. Under uncertainty, the con- troller benefits from three complementary evidence sources: social evidence about the collective state (Cannon-oowen Bowers et al. (1993); Jost et al. (2013), self-monitoring evidence about the reliability of the current solution Nelson & Narens ( 1994); Dunlosky & Metcalfe (2008), and cognitive con- trol evidence about whether further internal deliberation is likely to help whether escalation is warranted Risko Gilbert (2016). We operationalize these sources via lightweight summaries over conffict), metacognitive
observable interaction traces: social consensus cues z;"" (convergence monitoring cues zi!n (local reliability). and cognitive control cues z (progress vs. escalation). All cues are computed with simple parsing- and rule-hased heuristics, yielding explicit decision- oriented abstraction without introducing additional learned modules.
Formally, represent the policy state
concat( x
X; self
2t unc
mon
ctrl
(1)
where xHisk, x and xeer denote the task, self, and peer context, respectively, and 21" 2mm0 and 2ith are optional structured cognitive cues capturing social consensus, metacognitive monitoring. and metacognitive control evidence. In practice, these cues computed via lightweight parsing- and rule-based heuristics over the observable interaction traces, rather than introducing additional learned modules external supervision. This design provides the meta-policy with explicit, decision-oriented abstraction of the current collaborative state while keeping the structured compo- nent lightweight and auxiliary.
3.2.2
THE STRATEGIC ACTION SPACE
The action space A in our Meta-MDP is not defined by low-level text generation, but by a discrete set of high-level cognitive strategies. These actions empower the agent to manage its problem-solvingg process, balancing the exploitation of existing collective knowledge against the exploration of new solutions and the strategic deferral to external expertise. Formally, the action space is defined as A eva create deteer
4

---

Published as a conference paper at ICLR 2026
Evaluate (a Exploiting Collective Knowledge. This action represents the cognitive stance of convergence and synthesis. When selecting d the agent commits to exploiting the existing knowledge within the multi-agent group. Operationally, it must select and endorse of the lutions already proposed by its peers in the current round. This action allows the agent to leverage collective intelligence and reinforce high-quality, consensual solutions.
Create (a Creative Exploration and Hypothesis Generation. This action embodies the cognitive stance of divergence and exploration. By choosing a arreale, the agent posits that the current solution pool is insufficient and commits to generating novel solution sequence ('Choice'. 'Rea- son') from scratch. This action is crucial for breaking cognitive fixation, correcting shared errors within the group, and introducing new, potentially superior reasoning paths.
Defer (a Risk Mitigation and Knowledge Augmentation. This action represents the high- est level of metacognitive awareness- the ability to recognize the limits of the system's own ca- pabilities. Selecting adeler signals that the agent assesses the problem's uncertainty or difficulty to be beyond the collective's current ability to solve reliably. Operationally, this triggers a call to the external human expert, whose high-quality demonstration is then used the round's output. This action serves as both a mechanism for ensuring task success in critical situations and as a conduit for introducing new knowledge into the system via continual learning.
3.2.3 COLLABORATIVE INTERACTION MODEL
A defining feature of our framework the integration human expertise into collective reasoning through structured, multi-round protocol. At each round f, all N agents receive the shared cognitive state 8t. Each agent i independently samples a metacognitive action d₁ Ta(a)):) and executes it in parallel. The output Vi.t depends on the chosen action. When acting autonomously, the agent applies its internal generation process go(s)). When deferring, adopts the authoritative solution Yhuman, provided by the expert:
[o Uhummn.t
if as,t E aa create },
yi.t
if Og,t doferr
The collection {y1.t..... 3N..) } then forms the next state S1+1, ensuring that updates reflect the most reliable signals, whether from autonomous synthesis or human demonstration.
From a learning perspective, the Defer action plays dual role. It serves as risk mitigation, ensuring progress under uncertainty by overriding flawed solutions, and as knowledge augmentation, inject- ing expert demonstrations as high-quality samples for continual learning (Section (33). Thus, the human collaborator functions not merely as fallback oracle but as driver of system improvement.
3.3 ADAPTIVE POLICY OPTIMIZATION WITH CONTINUAL LEARNING
Mastering the metacognitive challenges described above requires an optimization strategy that bal- ances two competing paths: the high-risk but potentially high-reward route of autonomous problem- solving, and the low-risk but constrained option of deferring an expert. This trade-off naturally lends itself to reinforcement learning (RL), where the goal to learn policy Ty that maximizes the expected utility of the collaborative process. To address this, propose Dual-Loop Policy Optimization (DLPO) framework that integrates a reinforcement learning objective for strategic policy optimization with supervised objective for continual knowledge acquisition.
3.3.1 INNER LOOP: REINFORCEMENT LEARNING FOR METACOGNITIVE POLICY
The inner loop optimizes the agent's high-level policy (a(a)s)) over strategic actions. The key challenge is to provide a learning signal that reflects the trade-off between autonomous success, potential failure, and the cost of expert intervention. This problem is well-suited Group Relative Policy Optimization (GRPO), which contrasts the relative advantages of actions in each state.
Reward Formulation. We use a reward function R(st, o,) that combines task correctness with lightweight action-dependent costs. Let j(;;) denote the output obtained after executing action a;
5

---

Published as a conference paper at ICLR 2026
at state St, and let (g) denote the task reward induced by the resulting answer (e.g., binary correctness signal). We define
(ū(a))) Rig (ū(az)) Cereates
Gr EVAL,
R(st 01)
Gt CREATE,
Rog (Dhaman (ax)) Cdefer,
4, DEFER,
where Cereate and Caeeer are small tunable penalties reflecting the additional cost of internal re- generation and external expert intervention, respectively, with Cdefer > Crreate 2 0. this way, correctness remains the primary training signal, while the cost terms encourage the policy prefer lower-cost actions when multiple choices lead to similarly good outcomes.
GRPO Objective. Given the reward vector R, = [R(s,, a1)s,, R(s,, ax),. advantages are computed by centering rewards:
K
A(sr. 0x) R(s,, ax)
Σ az). j=l
(4)
K
The policy gradient objective is:
LPG = - Ex R [A(s,, a+) log no [o([)[=))]
(5)
Two regularizers ensure stability: a KL-penalty constrains deviation from the reference policy Tea and entropy bonus promotes exploration. The final inner-loop loss is:
Linner LPG BarCII - -ennn
(6)
3.3.2 OUTER LOOP: CONTINUAL LEARNING FROM EXPERT FEEDBACK
While the inner loop optimizes how the agent uses its current abilities, a truly adaptive system must also expand them. Reinforcement learning alone cannot overcome the knowledge ceiling of the base LLM, as improves decision policies without introducing fundamentally new skills. To break this ceiling, we introduce an outer optimization loop for Continual Learning from Expert Demonstrations.
This loop is activated by the 'Defer' action, which indicates that the agent has identified a knowledge gap. When deferring. the agent receives a high-quality demonstration Vhuman = (ti....,ta tt) from the expert, which converted into supervised fine-tuning (SFT) sample. The training objective is to maximize the likelihood of this sequence, conditioned on the state st, by minimizing the cross- entropy loss:
(0) = Σ log (1(4 82, b1)
In this design, the inner RL loop determines when to defer, while the outer loop teaches what to learn from expert input. Together, they establish an apprentice-mentor dynamic: the agent strategically invokes human guidance and systematically assimilates it into lasting capability growth.
3.3.3 THE FINAL DUAL-LLOP POLICY OPTIMIZATION OBJECTIVE
The inner and outer loops are optimized jointly to train a single agent that is both strategically adept and continually improving. The final training objective is principled combination of the reinforcement learning signal from the inner loop and the conditional supervised signal from the outer loop. The total loss, Lisat, is computed over batch of experiences:
Lioual (0) B(sc,,)) [Limeer (0) + Asn T(a = deterr LaFr(())
where Linner (8) is the full GRPO objective, An is hyperparameter balancing the two learning signals, and II(-) is the indicator function that ensures the SET loss is only applied when the 'Defer" action is taken.

---

Published as a conference paper at ICLR 2026
Model
Type
GSM8K
AMC
AIME
HvaanEval
MMLU
Vanilla
SA
72.76 (+0.00)
8.03 (+0.00)
2.96 (+0.00)
47.56 (+0,00)
57.99 (+0.00)
CoT
SA
74.22 (+1.46)
11.65 (+3.62)
3.70 (+0.74)
51.42 (+3.86)
61.57 (+3.58)
SC
SA
80.79 (+8.03)
12.45 (+4.42)
4.07 (+1.11)
57.52 (+9.96)
68.30 (+10.31)
PHP
MA
80.01 (+7.25)
15.66 (+7.63)
(44..44448)
56.50 (+8.94)
68.46 (+10.47)
Debate
MA
83.52 (+10.76)
19.28 (+11.25)
5.56 (+2.60)
57.72 (+10.16)
67.59 (+9.60)
G- G-Debate
MA
83.98 (+11.22)
20.48 (+12,45)
5.19 (+2.23)
57.93 (+10.37)
69.89 (+11.90)
DyLAN
MA
82.03 (+9.27)
19.68 (+11.65)
3.70 (+0.74)
61.59 (+14.03)
66.85 (+8.86)
G-Swarm
MA
84.89 (+12.13)
15.66 (47.63)
5.78 (+2.82)
59.55 (+11.99)
69.67 (+11.68)
A-Prune
MA
84.38
(+11.62)
16.47 (48.44)
4.81 (+1.85)
57.11 (+9.55)
69.09 (+11.10)
AFlow
MA
83.75 (+10.99)
12.05 (+4.02)
4.44 (+1.48)
62.20 (+14.64)
69.31 (+11.32)
HILA
MA
89.86 (+17.10)
35.83 (+24.47)
9.37 (+6.41)
72.15 (+24.59)
73.62 (+15.63)
Table I: Comparison of baseline and proposed methods using the LLaMA3-8B backbone. All values are percentages (the percent sign is omitted in the table). Values in parentheses denote absolute dif- ferences relative to the Vanilla baseline (first row). Underlined numbers indicate the best-peri rest-performing baseline on each benchmark. "SA" denotes single-agent, and "MA" denotes multi-agent settings.
4 EXPERIMENTS
Experimental Setup. We evaluate our method on broad suite of benchmarks, including general language understanding (MMLU), program synthesis (HumanEval), and quantitative mathematics (GSM8K, AIME, AMC). Following related works (Liu et al. 2023 Pandya al. [024], we employ GPT-4o-mini as a proxy human expert, leveraging its strong reasoning capability to simulate human interventions. Detailed experimental and training settings are provided in Appendix C.1
Overall Performance. Table I shows that our HILA framework with DLPO achieves the best overall performance among all compared methods, consistently surpassing strong autonomous multi-agent baselines across all reported benchmarks. These baselines, including debate-style (e.g., I.L.-Debate), appology-based (e.g., DyL.AN), and graph-optimization (e.g., GPTSSarm, AFL.WW) methods, remain confined to "closed-world" collaboration, where performance is capped by the agents' internal knowledge and often degrades on problems requiring non-obvious reasoning paths. In contrast, HILA introduces an "open-world" dynamic by enabling agents to strategically access external expertise, directly addressing this knowledge ceiling. On the Llama3-8B -BB backbone, HILA yields substantial gains over the strongest autonomous baseline on every benchmark, with abso- lute improvements ranging from 3.7 to 15.4 points. The gains are particularly pronounced on competition-style math benchmarks such as AMC and AIME, where cascade failures from flawed premises are common. By learning metacognitive policy to defer under high uncertainty, HILA avoids these pitfalls and effectively leverages superior guidance. These results suggest that the per- formance gains arise not merely from more complex interaction, but from the principled integration of external knowledge and the agent's learned ability decide when to invoke
Cross-Backbong Generalization. We further evaluate HILA across four heterogeneous LLM backbones, spanning both the Qwen and LLaMA families and covering multiple model seales. As shown in Table HILA consistently achieves the best performance on GSM8K across all back- bones, outperforming both single-agent prompting baselines and strong autonomous multi-agent methods. This result shows that the effectiveness of HILA is not tied to specific model family or parameter scale, but transfers robustly across substantially different pretrained backbones. More- over, the gains especially pronounced on smaller or weaker models, suggesting that HILA can more effectively compensate for limited intrinsic reasoning capacity when the base model is less ca- pable. Taken together, these results demonstrate that the proposed framework is both general across backbone choices and scalable across model strengths, while maintaining consistent benefits from adaptive collaboration and strategic human intervention.
Beyond Better Intervention Decisions. To further analyze the source of HILA''s performance, we separate the contributions of policy learning and capability growth. Table three vari- ants of HILAA under progressively stronger training regimes. Moving from the unoptimized policy to
7

---

Published as a conference paper at ICLR 2026
Model
Qwen2.5-7B
Qwen2.5-3B
LLaMA3-8B
LLaMA3-3B
Vanilla
90.71 (+0.00)
83.25 (+0.00)
72.76 (+0.00)
45.26 (+0.00)
CoT
91.13 (+0.42)
84.36 (+1.11)
74.22 (+1.46)
52.49 (+7.23)
SC
91.79 (+1.08)
87.54 (+4.29)
80.79 (+8.03)
58.68 (+13.42)
PHP
91.35 (+0.64)
86.83 (+3.58)
80.01 (+7.25)
66.21 (+20.95)
LLM- Debate
92.47 (+1.76)
87.41 (+4,16)
83,52 (+10.76)
76.32 (+31.06)
DyLAN
92.64 (+1.93)
87.85 (+4.60)
82.03 (+9.27)
76.17 (+30.91)
GPT
92.83 (+2.12)
86.45 (+3.20)
84.89 (+12.13)
71.85 (+26.59)
AgentPrune
91.76 (+1.05)
86.20 (+2.95)
84.38 (+11.62)
68.71 (+23.45)
AFlow
92.37 (+1.66)
86.78 (+3.53)
83.75 (+10.99)
70.89 (+25.63)
HILA
94.72 (+4.01)
91.17 (+7.92)
89.86 (+17.10)
83.85 (+38.59)
Table 2: Performance of baselines across four LLM backbones on GSM8K. All values are percent- ages (percent sign omitted). Parentheses show absolute differences (percentage points) relative to the Vanilla row for each backbone. HILA refers to the proposed method with DLPO
Model
GSM8K
AMC
MMLU
Vanilla (Base)
72.76
8.03
57.99
Model
GSM8K
AMC
MMLU
Vanilla (DLPO)
82.11
10.84
61.58
HILA (Init Policy)
88.15
33.33
68.30
DyLAN (Base)
82.03
19.68
66.85
HILA GRPO
88.38
32.50
70.47
DyLAN (DLPO)
88.32
25.30
68.72
HILAA DLPO
89.86
35.83
73.62
Debate (Base)
83.52
19.28
67.59
Table 3: Performance of HILA under progres- sively stronger training regimes. Inil Policy de- notes the unoptimized system, GRPO applies only the inner-loop policy optimization, and DLPO adds the outer-loop update from expert demonstrations. The results show that the full dual-loop objective delivers the strongest over- all performance.
Debate (DLPO)
88.93
21.69
69.57
Table Transferability of the backbone up- dated by DLPO. Base denotes the original I.LaMA3-8B backbone, while DLPO denotes the same backbone after DLPO training. For each inference framework, we keep the test-time protocol unchanged and only replace the back- bone. All values are percentages.
GRPO yields only modest changes overall, with the main benefit appearing in more reliable strategic control rather than uniformly higher task accuracy. In contrast, introducing the full DLPO objective leads clear second-stage improvement across benchmarks, indicating that the gains cannot be explained by better action selection alone. This pattern suggests that reinforcement learning primar- ily improves when the system chooses to defer, while the outer supervised loop is responsible for transforming those deferral events into lasting gains in reasoning ability.
Table further verifies this interpretation by evaluating standard inference pipelines with the back- bone updated after DLPO training- Across single-agent prompting and autonomous multi-agent baselines, replacing the original backbone with the DI.PO-updated one consistently improves per- formance, showing that the benefit transfers beyond the HILA interaction protocol itself. Notably, these gains appear even when the downstream method does not use strategic deferral at inference time, which indicates that the expert demonstrations collected during training HILA make better local decisions, but are also improving the underlying L.LM's general reasoning competence. Taken together, these results show that HILA'' gains arise from two complementary sources: a better metacognitive policy for intervention, and continual learning that strengthens the backbone model itself.
merely helping
Policy Distribution Across Training Stages. To better understand how training changes HILA''s collaboration behavior, we examine the distribution of the three strategic actions (EVAL, CREATE, and DEFER) across different training stages. Table 5 shows clear and interpretable trend. Starting from the unoptimized policy, HILA assigns a nontrivial fraction of decisions to DEFER, indicat- ing substantial reliance external intervention before the system learns calibrated metacognitive strategy. After applying GRPO, the share of DEFER decreases consistently across datasets, while EVAL and CREATE become slightly more frequent. This trend suggests that inner-loop policy opti-
8

---

Published as a conference paper at ICLR 2026
GSM8K CREATE
AMC CREATE
MMLU CREATE
Model
EVAL
DEFER
EVAL
DEFER
EVAL
DEFER
HILA (Init Policy)
36%
35%
29%
36%
40%
24%
47%
34%
19%
IIILA GRPO
38%
36%
26%
40%
40%
20%
48%
37%
15%
HIIAA DLPO
55%
28%
17%
64%
24%
12%
74%
21%
Table 5: Distribution of strategic actions across training stages. As training progresses, the propor- tion of DEFER consistently decreases, while EVAL becomes more dominant, indicating improved metacognitive control and reduced reliance on external intervention.
100
CSMBK
80
MMILLI
Accuracy (%)
20
GPT:3.5-turbo
GPT-4o-minl
Figure 2: Effect of human proxy capability on HILA. Using stronger language models the external expert consistently improves performance on GSM8K, AMC, and MMLU.
mization learns cost-aware intervention strategy. Because deferral carries explicit penalty, the agent becomes more selective about invoking external expertise and makes better use of au- tonomous alternatives.
The effect becomes even more pronounced after full DLPO training. Compared with both the ini- tialization stage and GRPO alone, DEFER drops substantially
all benchmarks, accompanied by marked increase in EVAL. This shift suggests that the agent is not merely learning to avoid costly de- ferral heuristically, but is becoming Since DI.OO combines inner-loop policy optimization with continual learning from expert demon- strations, the reduced deferral rate further suggests that these demonstrations improve the model's underlying reasoning ability. As the backbone becomes stronger, the system relies less on external help and more often exploits or refines internally available solutions. These results show that GRPO improves when to ask, while DLPO further improves whether asking is needed at all.
capable of resolving tasks within the multi-agent system.
Effeet of Human Proxy Capability. We further study how the strength of the external expert shapes HILA's effectiveness by instantiating the human proxy with three language models of differ- ent capabilities, namely gpt 3. 5-turbo, qpt-4o-mint and gpt-4o. As shown in Figure and 6 stronger proxies consistently lead to better overall performance across all benchmarks. This ordering is stable across GSM8K, AMC, and MMLU: gpt-3. 5-turbo yields the weakest results, gpt- 40--mi-mm provides a clear improvement, and got-gppttooooo -o achieves the best performance. This trend is consistent with the role of DEFER in HILAA, which designed to bring in higher-quality reasoning precisely when the autonomous multi-agent system reaches its limit. The results therefore show that HILA depends not only on learning when to defer, but also on who the system defers Strategic intervention and expert quality are complementary: stronger metacognitive policy can identify when outside help is needed, while the final gain still depends on the informativeness and reliability of the returned guidance.
The Value of Collective Exploration. To examine how the size of the collective affects both ef- fectiveness and cost, we evaluate HILA with different numbers of autonomous agents on AMC and MMLU, shown in Figure aa), with full results provided in Appendix C.2 Table and Table g Across both benchmarks, increasing the number agents generally improves accuracy, with the most noticeable gains appearing when moving from very small collective moderate-sized

---

Published as a conference paper at ICLR 2026
0.70
Accuracy (AMC)
0.09
Accurr (MML)
Accuracy (AMC)
0.32
0.68
0.30
0.67
0.66
AMC
AMC
HNLU
5
Number of Agents
Number of Rounds
(a) Accuracy function of the number of agents.
(b) Accuracy function of the number of rounds.
Effect of scaling collaborative configurations. (a) shows how increasing the number of agents impacts accuracy AMC and GSM8K, while (b) analyzes how varying the number of interaction rounds influences performance. Together, the results highlight the trade-off's between broader exploration through more agents and deeper refinement through additional rounds.
Figure 3:
one. This trend supports central intuition of HILA: a larger agent pool enables richer collective exploration, allowing the system to generate a broader set of candidate reasoning trajectories and increasing the chance that strong solution are not linear. As the collective continues to grow, improvements become smaller and less stable, suggesting diminishing returns trast, computational cost rises sharply with the number of agents, as both input and output tokens increase substantially with broader participation. These results highlight practical trade-off: larger collectives can strengthen reasoning through broader exploration, but this benefit must be balanced against rising inference cost, making the number of agents an important balance choice in HILA.
be identified and propagated. However, the gains
the solution space is already sufficiently diversified. By con-
Optimal Depth of Iterative Collaboration. To assess the role of iterative refinement, vary the number collaborative rounds and evaluate HIILA on AMC and MMLU, as shown in Figure 7t), with full results provided in Appendix C.2, Table and Table [0] Across both benchmarks, the effect follows a clear non-monotonic pattern. Increasing the number of rounds from a shallow interaction consistently improves accuracy, indicating that multi-round collaboration depth to moderate intermediate solutions, incorporate peer feedback, and correct earlier mis- helps the system refine takes. This supports the intuition that iterative interaction strengthens reasoning by enabling agents to progressively revise and consolidate candidate solutions rather than relying on single-pass ex- change. However, the benefit does continue indefinitely. Performance peaks an intermediate depth and then declines slightly as the interaction is extended further, even though token usage con- tinues to rise. This suggests that additional rounds initially create useful opportunities for correction, but excessive interaction leads to diminishing returns and can begin to reinforce suboptimal trajec- tories. These results show that iterative collaboration is valuable but not unbounded, and the number of rounds should be chosen to balance refinement gains against rising computational cost.
5 CONCLUSION
In this paper, we presented HILA, framework that equips multi-agent systems with metacogni- tive policy for deciding when to act autonomously and when to defer to external expertise. Through Dual-Loop Policy Optimization, which combines GRPO-based intervention control with continual learning from expert demonstrations, HILA enables both adaptive crriision-making and long-term term capability improvement. Extensive experiments across diverse and challenging reasoning bench- marks show that HILA consistently outperforms strong autonomous and multi-agent baselines. In future work, plan to study dynamic collaboration mechanisms and further strengthen the evolutionary capabilities of multi-agent systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported in part by the Department of Defense under Cooperative Agreement Num- ber W91 INF-24-2-0133. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Cooperation, competition, and maliciousness: LIm-stakeholders interactive negotiation. Advances in Neural Information Processing Systems, 37:83548-83599 2024.
Shayan Meshkat Alsadat and Zhe Xu. Multi-agent reinforcement learning in non-cooperative stochastic games using large language models. IEEE Control Systems Letters, 2024.
Janis A Gannon-Bowers Eduardo Salas, and Sharolyn Converse. Shared mental models in expert team decision making. 1993.
Defu Cao, Michael Gee, Jinbo Liu, Hengxuan Wang, Wei Yang, Rui Wang, and Yan Liu. Con- versational time series foundation models: Towards explainable and effective forecasting, arXiv preprint arXiv:2512. /6022, 2025.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint 1xXxx:330.0020000 2023.
Ching Chang, Yidan Shi, Defu Cao, Wei Yang, Jechyun Hwang, Haixin Wang, Jiacheng Pang, Wei Wang, Yan Liu, Wen-Chit Peng, et al. A survey of reasoning and agentic systems in time series with large language models. arXiv preprint 1rXxv:2509./157. 2025.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309. 17288, 2023a.
Jajju Chen, Yuxuan Lu, Xiaojie Wang. Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao. and Dakuo Wang. Multi-agent-as-juee Aligning llm-agent-basedd automated evaluation with multi-dimensional human evaluation. arXiv preprint urXiv:2507.21033 2025a.
Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei- Rei Tu, Zhaofeng He, and Lijie Wen. Llmarena: Assessing capabilities of large language models in dynamic multi-agent environments. arXiv preprint ar 2002::2402.1 /6499, 2024a.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. Routerde: Query-based router by dual contrastive learning for assembling large language models. Advances in Neural Information Processing Systems, 37:66305-666 66328, 2024b.
Weize Chen, Yusheng Su. Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2(4):6, 2023b.
Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Xinyu Ma, Wei Yang. Daiting Shi, Jiaxin Mao, and Dawei Yin. Tourrank: Utilizing large language models for documents ranking with a tournament- inspired strategy. In Proceedings of the ACM on Web Conference 2025, pp. 1638-1652, 2025b.
Yiqun Chen, Jinyuan Feng, Wei Yang. Meizhi Zhong, Zhengliang Shi, Rui Li, Xiaochi Wei, Yan Gao, Yi Wu, Yao Hu, et al. Self-compression of chain-of- thouugt via multi-agent reinforcement learning. arXiv preprint arXiv:2601.21919, 2026.

---

Published as a conference paper at ICLR 2026
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference Machine Learning, 2023.
John Dunlosky and Janet Metcalfe. Metacognition. Sage Publications, 2008
Xueyang Feng, Zhi- Yuan Chen, Yujia Qin, Yankai Lin, Xu Chen, Zhiyuan Liu, and Ji-Rong Wen. Large language model-based human-agent collaboration for complex task solving. arXiv preprint arXiv:2402.12914 2024.
Adrian Garret Gabriel, Alaa Alameer Ahmad, and Shankar Kumar Jeyakumar. Advancing agentic systems: Dynamic task decomposition, tool integration and evaluation using novel metrics and dataset. arXiv preprint arXiv:2410.22457. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Suzie Grondin, Arthur Charpentier, and Philipp Ratz. Beyond human intervention: Algorithmic collusion through multi-agent learning strategies. arXiv preprint arXiv:5501.16935, 2025.
Ai Han, Junxing Hu, Pu Wei, Zhiqian Zhang, Yuhang Guo, Jiawei Lu, and Zicheng Zhang- Joyagents-rl: Joint evolution dynamics for versatile multi-llm agents with reinforcement learning. arXiv preprint arXiv:2506. 19846, 2025.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2023.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint 24ti::2008.044333 2024.
Ziqi Jia, Junjie Li, Xiaoyang Qu, and Jianzong Wang. Enhancing multi-agent systems via reinforce- ment learning with lim-based planner and graph-based policy. arXiv preprint arXiv:2503. 10049, 2025.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0256/, 2023.
Zhouyang Jiang, Bin Zhang, Airong Wei, and Zhiwei Xu. Qlim: Do we really need a mixing network for credit assignment in multi-agent reinforcement learning? arXiv preprint arXiv:2504.12961, 2025.
John T Jost, Arie W Kruglanski, and Thomas O Nelson. Social metacognition: An expansionist review. In Metacognition, pp. 137-154. Psychology Press, 2013.
Zachary Kenton, Noah Siegel, János Kramár, Jonah Brown-Cohen. Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, al. On scalable oversight with weak lms judging strong llms. Advances in Neural Information Processing Systems, 37: 75276, 2024.
Boyi Li, Zhonghan Zhao, Der-Horng Lee, and Gaoang Wang. Adaptive graph pruning for multi- agent communication. arXiv preprint arXiv:2506.02955) 2025a.
Dapeng Li, Hang Dong, Lu Wang, Bo Qiao, Si Qin, Qingwei Lin, Dongmei Zhang, Qi Zhang, Zhiwei Xu, Bin Zhang, et al. Verco: Learning coordinated verbal communication for multi-agent reinforcement learning. arXiv preprint axXiv:2404.17780. 2024a.
Huao Li, Hossein Nourkhiz Mahjoub, Behdad Chalaki, Vaishnav Tadiparthi, Kwonjoon Lee, Ehsan Moradi Pari, Charles Lewis, and Katia Sycara. Language grounded multi-agent reinforcement learning with human- interppretteeebeee communication. Advances in Neural Information Processing Systems, 37:87908-87955 2024b.
12

---

Published as a conference paper at ICLR 2026
Jiate Li, Defu Cao, Li, Wei Yang, Yuehan Qin, Chenxiao Yu, Tiannuo Yang, Ryan A Rossi, Yan Liu, Xiyang Hu, al. someone hid it": Query-agnostic black-bor box attacks on lim-based retrieval. arXiv preprint arXhv:2602.00364, 2026.
Shixuan Li, Wei Yang, Peiyu Zhang, Xiongye Xiao, Defu Cao, Yuehan Qin, Xiaole Zhang, Yue Zhao, and Paul Bogdan. Climatellm: Efficient weather forecasting via frequency-aware large language models. arXiv preprint urXiv:2502. 1059,, 2025b.
Yuxi Li. Reinforcement learning applications. arXiv preprint arXiv: 1rXiv:1908.06990 2019.
Tian Liang, Zhiwei He. Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking large language models through multi- agent debate. arXiv preprint arXiv:2305. 2023.
Muhan Lin, Shuyang Shi, Yue Guo, Vaishnav Tadiparthi, Behdad Chalaki, Ehsan Moradi Pari, Si- mon Stepputtis, Woojun Kim, Joseph Campbell, and Katia Sycara. Speaking the language of teamwork: LIm-guided credit assignment in multi-agent reinforcement learning. arXiv preprint arXiv:2502.03723, 2025.
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. LIm-powered hi- erarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:231..15224, 2023.
Tongxuan Liu, Xingyu Wang, Weizhe Huang. Wenjiang Xu, Yuting Zeng, Lei Jiang, Hailong Yang, and Jing Li. Groupdebate: Enhancing the efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409. 14051, 2024.
Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. Self: Language-driven self-evolution for large language model. 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-4552 2023.
Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Ger- rits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde McLauhhin et al. Magentic-ui: Towards human-in-the-loop agentic systems. arXiv preprint arXiv:2507. 22355 2025.
Chunjiang Mu, Hao Guo, Yang Chen, Chen Shen, Die Hu, Shuyue Hu, and Zhen Wang. Multi- agent, human-agent and beyond: survey on cooperation in social dilemmas. Neurocomputing, 610: 610:128514, 2024.
Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. arXiv preprint arXiv:2310.08901, 2023.
Thomas O Nelson and Louis Narens. Why investigate metacognition. Metacognition: Knowing about knowing, 13:1 25, 1994.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of- thought: Prompting Iiss for efficient parallel generation. arXiv preprint arXhv:2307.15337. 2023.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and lon Stoica. Routellm: Learning to route mlms with preference data. arXiv preprint Ni..:2224....66.66. 2024.
Bo Pan, Jiaying Lu, Ke Wang, Li Zheng, Zhen Wen, Yingchaojie Feng, Minfeng Zhu, and Wei Chen. Agentcoord: Visually exploring coordination strategy for Ilm-based multi-agent collaboration. arXiv preprint arXiv:2404.119453 2024.
Ravi Pandya, Michelle Zhao, Changliu Liu, Reid Simmons, and Henny Admoni. Multi-agent strategy explanations for human-robot collaboration. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 17351-17357. IEFF, 2024.
13

---

Published as a conference paper at ICLR 2026
Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo- Moo-Kyung Kim. for collaborative large language models with reinforcement
Maporl: Multi-agent learning. arXiv preprint arXiv:2502.18439, 2025.
Feng Peiyuan, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of Ilm agents. Advances in Neural Infor- mation Processing Systems, 37:5244-555 -225244,, 2024.
Heng Ping, Arijit Bhattacharjee, Peiyu Zhang, Shixuan Li, Wei Yang, Anzhe Cheng, Xiaole Zhang, Jesse Thomason, Ali Jannesari, Nesreen Ahmed, al. Verimoa: mixture-offagentt framework for spec-to-hdl generation. arXiv preprint arXiv:2510.27617 2025a.
Heng Ping. Shixuan Li, Peiyu Zhang, Anzhe Cheng, Shukai Duan, Nikos Kanakaris, Xiongye Xiao, Wei Yang, Shahin Nazarian, Andrei Irimia, et al. Hdlcore: A training-free framework for mitigat- ing hallucinations in Ilm-generated hdl. arXiv preprint arXiv:2503. 16528, 2025b.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al. Scaling large language model-base multi-agent collaboration. arXiv preprint 24Xv:24066007155, 2024.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch for qa via self- planning. arXiv preprint MXNi::2001.02666 2024.
Evan Risko and Sam Gilbert. Cognitive offloading. Trends in cognitive sciences, 20(9):676-633 2016.
Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic Ilm agent search in modular design space. arXiv preprint arXiv:2410.06153, 2024.
Philipp D Siedler and Ian Gemp. Llm-mediated guidance of marl systems. arXiv preprint WrXi:2250311355. 2025.
KV Srivatsa, Kaushal Kumar Maurya, and Ekaterina Kochmar. Harnessing the power of multiple minds: Lessons learned from Ilm routing. arXiv preprint arXiv:2405.0046) 2024.
Chuanneng Sun, Songjun Huang, and Dario Pompili. Llm-based multi-agent decision-making: Challenges and future directions. IEEE Robotics and Automation Letters, 2025.
Mirac Suzgun and Adam Tauman Kalai. Meta-prompting: Enhancing language models with task- agnostic scaffolding. arXiv preprint 4rXiv:2401.12954, 2024.
Wannita Takerngsaksiri, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, and Ming Wu. Human-in -ntt----ot---o software development agents. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 342-352. IEEE, 2025.
Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Hamessinn the power of intelligent I1m agents. arXiv preprint arXiv:2306.03314 2023.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Haley Triem and Ying Ding. "tipping the balance": Human intervention in large language model multi-agent debate. Proceedings of the Association for Information Science and Technology, 61 (1):361-373, 2024.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, al. Rema: Learning meta-think for ilms with multi-agent reinforcement learning. arXiv preprint
(al::550300900) 2025.
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's 's game of thoughts: Battle against decep- tion through recursive contemplation. arXiv preprint 1r.0v:2310.013320 2023a.
14

---

Published as a conference paper at ICLR 2026
Xuejiao Wang, Guoqing Zhi, Zhihao Tang, Hao Jin, Qianyue Zhang, and Nan Li. Self-aware in- telligent medical rescue unmanned team via large language model and multi-agent reinforeement learning. In Proceedings of the 2024 International Symposium on Al and Cybersecurity, pp- 119-124, 2024.
Zhenhailong Wang, Shaoguang Mao. Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.053300 2023b.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprini arXiv:2504.200 20073, 2025a.
Ziyan Wang, Zhicheng Zhang, Fei Fang, and Yali Du M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality, arXiv preprint arXiv:2503.02077 2025b.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0 01600 2021.
Yuan Wei, Xiaohan Shan, and Jianmin Li. Lero: Lim-driven evolutionary framework with hy- brid rewards and enhanced observation for multi-agent reinforcement learning. arXiv preprint arXiv:2503.21 2025.
Muyan Weng, Defu Cao, Wei Yang, Yashaswi Sharma, and Yan Liu. Temporalhench: A benchmark for evaluating llm-based agents on contextual and event-informed time series tasks. arXiv preprint 26Xiv:2602.1327. 2026.
Yu Xia, Rui Zhong, Hao Gu, Wei Yang, Chi Lu, Peng Jiang, and Kun Gai. Hierarchical tree search- based user lifelong behavior modeling on large language model. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1758-1767, 2025a.
Yu Xia, Rui Zhong, Zeyu Song, Wei Yang, Junchen Wan, Qingpeng Cai, Chi Lu, and Peng Jiang. Trackrec: lterative alternating feedback with chain-of-thought via preference alignment for rec- ommendation. arXiv preprint arXiv:2508. 15388, 2025b.
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large lan- guage models collaboration: An in-depth analysis via debate. arXiv preprint arXiv:2305.1153 2023.
Yanggang Xu, Weijie Hong, Jirong Zha, Geng Chen, Jianfeng Zheng, Chen-Chun Hsia, and Xin- lei Chen. Scalable uav multi-hop networking via multi-agent reinforcement learning with large language models. arXiv preprint arXiv:2505.08844 2025.
Wei Yang and Jesse Thomason. Learning to deliberate: Mcta-policy collaboration for agentic 1lms with multi-agent reinforcement learning. arXiv preprint 1XXKv:2509003/111 2025.
Wei Yang, Jiacheng Pang, Shixuan Li, Paul Bogdan, Stephen Tu, and Jesse Thomason. Maestro: Learning to collaborate via conditional listwise policy optimization for multi-agent 1lms. arXiv preprint arXiv:25/ 1.06/44, 2025a.
Wei Yang, Muyan Weng, Jiacheng Pang. Defu Cao, Heng Ping, Peiyu Zhang, Shixuan Li, Yue Zhao, Qiang Yang, Mengdi Wang, et al. Toward evolutionary intelligence: Llm-based agentic systems with multi-agent reinforcement learning. Available at SSRN 58/9/82, 2025h.
Wei Yang, Shixuan Li, Heng Ping, Peiyu Zhang, Paul Bogdan, and Jesse Thomason. Auditing multi-agent I1m reasoning trees outperforms majority vote and alm-as-judge. arXiv preprint arXiv:2602.0934/ 2026.
Wen Ye. Wei Yang. Defu Cao, Yizhou Zhang, Lumingyuan Tang, Jie Cai, and Yan Liu. Domain- oriented time series inference agents for reasoning and automated analysis. arXiv preprint 14.4474 2024.
15

---

Published as a conference paper at ICLR 2026
Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. Exchange-of-thougghe Enhancing large language model capabilities through cross-mode cross-model communication. arXiv preprint arXiv:2312.01823, 2023.
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route ilms for multi-agent systems. arXiv preprint arXiv:2502. 1113, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 35:15476-15488, 2022.
Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, et al. Proagent: building proactive cooperative agents with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17591-17599, 2024a.
Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024b.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. arXiv preprint arXiv:2410. 10762, 2024c.
Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. Exploring collab- oration mechanisms for Ilm agents: A social psychology view. arXiv preprint arXiv:2310.02124,
2023.
Kaiqing Zhang. Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321- 384. 2021.
Xuan Zhang, Yang Deng, Zifeng Ren, See- KionK Ng, and Tat-Seng Chua. Ask-before-plee plan: Proac- tive language agents for real-world planning. arXiv preprint arXiv:2406. 22.224406...,9, 2024d.
Yang Zhang, Shixin Yang, Chenjia Bai, Fei Wu, Xiu Li, Zhen Wang, and Xuelong Li. Towards efficient Ilm grounding for embodied multi-agent collaboration. arXiv preprint arXiv:2405.145: 14314, 2024e.
Heng Zhou, Hejia Geng, Xiangyuan Xue, Li Kang, Yiran Qin, Zhiyong Wang, Zhenfei Yin, and Lei Bai. Reso: A reward-driven self-organizing Ilm-based multi-agent system for reasoning tasks. arXiv preprint arXiv:2503.023390 2025.
Guobin Zhu, Rui Zhou, Wenkang Ji, and Shiyu Zhao. Lamarl: Llm-aided multi-agent reinforcement learning for cooperative policy generation. IEEE Robotics and Automation Letters, 2025.
Yuan Zhuang, Yi Shen, Zhili Zhang, Yuxiao Chen, and Fei Miao. Yolo-marl: You only I1m once for multi-agent reinforcement learning, arXiv preprint ar arXiv:2410.0399) 2024.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen
Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024.
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Yankai Chen, Chunyu Miao, Hoang Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Langzhou He, et al. Llm-based human-agent collaboration and interaction systems: A survey.
16

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
A.1 COLLABORATION PARADIGMS IN MULTI-AGENT LLM SYSTEMS.
Early research has shown that single LLM agents face inherent limitations in context length, quential generation, and breadth of skills, which restrict their ability to solve complex tasks re- quiring diverse perspectives parallel reasoning (Gabriel et al., 2024 Liang al. 2023 Xiong iet al. 2023 Yin et al. 2023 Zhang et al.) 2023). To overcome these bottlenecks, recent work has explored multi-agent systems (MAS), where multiple LLMs are orchestrated realize collective
intelligence across domains such as software engineering, planning, and problem solving (Hong
ct al.
2023 Chen et al.. 2023b Xia et al.
2025b; Ning et 2023 et al. 2026 Pan al. 2024
Suzgun & Kalai 2024
Chen et al. 2023a Weng et al. 2026
Most existing frameworks rely on prompt-based paradigms that predefine roles, communication pro- tocols, or workflow structures. These designs enable debate, critigue, and corporate-styl pipelines, achieving notable gains in coordination efficiency (Du et al., 2023] Chan et al.] [023]] Cao et al.]
2025 Mukobi et al., [203 Wang et al.) 2023a, Abdelnabi et al. 2024] Han et al., 2025 However, because they are hand-crafted and do not adapt through experience, such systems remain fundamen-
tally closed-world they can only recombine existing knowledge rather than acquire genuinely new
capabilities (Wang et al. 2023b Liu et al. 2024 Chen et al. 2024b).
Beyond fixed prompts, two further directions have emerged. Prestructured coordination employs fixed debate or peer-review topologies, such as chains, trees, or graphs, to refine reasoning (Du 2024 Qian et al. 2024). In contrast, self-organizing approaches adapt the interaction graph dynamically ly through search, pruning, routing, or evolutionary mechanisms (Hu
et al.
2023 Liu et al.
al. 2024 Shang et al. 2024 Zhang et al. [024b] [Zhuge al. 2024 Zhang al.) [024c] Yue et al. 2025).
These advances highlight the importance of who communicates and when, yet they pri-
marily optimize internal coordination and leave unaddressed the problem of learning from external expertise.
In contrast, our work introduces a centralized and iterative collaboration framework that explicitly incorporates human expertise as an open-world resource. Rather than treating human feedback as a passive oracle or one-time correction (Takemgsaksiri et al. 2025 Mozannar al., 2025 Liu et al., 2023 Pandya et al. (024)), we propose to endow agents with a metacognitive policy that governs both the timing of deferral and the assimilation of human guidance into lasting improve- ments. This approach differs fundamentally from prior debate, routing, and workflow-search sys- tems, which often lack principled credit assignment between reasoning and decision outcomes (Chan) et al. [2023] Talebirad & Nadiri 2023] /Wii et al.) 2025). By integrating external knowledge with learned metacognitive adaptation, our framework moves beyond static collaboration to establish a pathway toward adaptive and continually improving multi-agent intelligence.
A.2 MUL.TI-GGENT REINFORCEMEN LEARNING FOR L.MS.
A growing line of work seeks to move beyond static prompt engineering and endow multi-agent L.MM systems with adaptive learning capabilities. Early efforts rely on supervised fine-tuning (SFT) to inject collaborative patterns by imitating expert demonstrations or curated trajectories (Lu et al,
2023 Madaan al. 2023 Zelikman et al. 2022 Wei et al. 202I). While effective for seeding cooperative behaviors, SFT remains limited by its offline nature and cannot adapt to novel contexts. Reinforcement learning (RL) has thus emerged as a natural complement, enabling agents to refine policies through tria--and-error interaction and reward-
adaptation (Zhu et al., 2025 Zhuang al. 2024). In practice, SFT often serves as initialization, with RL providing fine -eeeggraned policy
improvement under feedback (Zhu et al. 2025 Li 2019 Zhang et al. (021).
Recent research highlights three major directions. First, compiling language into structured con- trollers-such as graphs, code, or Mllnswallows RL to optimize execution policies over symbolic abstractions rather than raw text (Zhuang et al., 2024 Jia et al. 2025] Zhu et al., 2025). Second, online collaboration is adapted through RL-based task decomposition, communication routing, and role assignment, which allow dynamic coordination beyond static protocols (Zhou et al. 2025 Wang
et al. 2024 Xu et al. 2025 Li et al. 2024a). Third, several studies explore learning reasoning poli-
cies directly in language space using GRPO or PPO-style updates, often integrating tools or human
input when beneficial (Wan al. 2025
Park at. 2025 Han et al. 2025
Peiyuan al. 2024 Feng
17

---

Published as a conference paper at ICLR 2026
et al.
These approaches underscore the importance of credit assignment and reward shaping for a iigingg emergent behaviors in high-dimensional language action spaces (Wei et al. 2025 Jiang
2024).
et al. 2025 Alsadat & Xu 2024 Lin al. 2025
Our work is closely aligned with this trajectory but emphasizes a key gap: most existing RL ap- proaches focus on optimizing intra-agent inter-agent coordination while leaving the system's knowledge boundary fixed. contrast, we introduce a dual-loop perspective where RL is respon- sible for learning metacognitive deferral policy, and expert demonstrations triggered by deferral events fuel continual learning. This integration addresses both immediate pecision-mmking and long- term capability growth, providing principled path toward genuinely adaptive multi-agent systems.
B METHODOLOGY
B.1 PRELIMINARIES: THE METACOGNITIVE MARKOV DECISION PROCESS
We formalize the dynamics of numan-agent collaboration as a Meta-Cognitive Markov Decision Process (Meta-MDP), defined by the tuple M (8, P, Rt 7). A Meta-MDP provides prin- cipled framework for sequential decision-making where actions correspond to high-level cognitive strategies. Formally, a Meta-MDP defined by the tuple (S, P. R. 7). each round of the multi-agent collaboration, the process unfolds as follows: The state S₁ E S is structured cognitive state representation, which encapsulates not only the external problem context but also the agent's internal assessment of its own and its peers' current understanding, as we will detail in Section 3.2 Based on this rich state, the agent selects a metacognitive action a from a discrete action space A, which includes functional strategies such as solving the problem autonomously the expert. The system then transitions to a new state 8:41 according to the transition function P(s (8, 0z). reward R(s,, ax) is issued, designed to incentivize both task success and the ef- ficient utilization of the expert resource. The overarching goal is to learn an optimal metacognitive policy x² (a, (c)]))) that maximizes the expected cumulative reward, thereby training an agent that can rationally balance autonomous problem-solving with strategic reliance on human guidance.
deferring to
C EXPERIMENT
C.I EXPERIMENTAL SETTINGS
Benchmarks and Evaluation. To evaluate framework, conduct experiments broad collection of benchmarks that test complementary aspects of reasoning ability. These tasks span three domains: general knowledge and analytical reasoning, program synthesis, and mathematical problem solving.
For general knowledge, use the MMLU benchmark, which includes 57 subject areas in multiple-choice format. Performance is measured by Accuracy. To control evaluation cost, we sample 10 examples from each subject category. For program synthesis, we adopt HvmanEval, where models generate code solutions from natural-language specifications.
For quantitative reasoning, we consider four math-focused datasets with concise numerical answers: GSM8K (grade-school arithmetic word problems), AIME (short-form olympiad-sty/e tasks), and AMC (contest problems). Performance on these datasets is reported as Solve Rate, defined by exact match against each dataset's normalized reference solution.
All evaluations are conducted on the official datasets with standard prompting protocols. We exclude external tools and retrieval, ensuring that improvements stem from our collaboration framework rather than auxiliary resources. single-agent settings, inference run deterministically. For multi-agent experiments involving stochastic sampling, we set up random seeds, repeat runs, and report averaged results. This setup isolates the contribution of our proposed method and allows for fair comparison against existing approaches.
To ensure fair and comprehensive comparison, evaluate our framework against three broad families baseline methods that represent the dominant paradigms in collaborative reasoning with LLMs:
Baselines.
18

---

Published as a conference paper at ICLR 2026
1. Single-Agent Solvers. These methods rely single model instance without peer in- teraction. They capture the performance limits of prompting alone. Examples include direct decoding under standard prompt (Vanilla), reasoning traces generated by Chain-of- Thought (Co't) prompting, and multi-sample aggregation methods such as Celfssoons (SC). Self-reflection strategies (e.g., Reflection, RASC) are also included, where the model internally revises its outputs without external assistance.
2. Interactive Multi-Agent Deliberation. This class introduces explicit communication among multiple agents. Agents generate, critique, and refine one another's proposals. Ap- proaches such as LLM-Debate implement structured resuedd cycles, while pairwise or pooled critique frameworks (e.g. PHP) simulate peer-review processes. These baselines assess whether systematic interaction alone, without external expertise, can reduce errors and improve reasoning robustness.
3. System-Level Coordination Frameworks. Some approaches treat collaboration as an optimization problem over computational graphs. Adaptive topology and routing meth- ods (e.g.. DyLAN, MasRouter) dynamically determine communication patterns, while workflow- and hearch-basede systems (e.g., GPTSwarm, AFLOW) orchestrate reusable rea- soning modules. Communication prunnnn strategies such as AgentPrune improve scalabil- ity by filtering redundant interactions. These baselines highlight efficiency and coordina- tion at scale.
For all baselines, we control the backbone model. prompting setup, and generation budget (number of agents, rounds, and outputs). When multiple candidates are produced, we apply the baseline's s external tools are
canonical reduction method (e.g., majority vote). No retrieval augmentation used. This categorization clarifies whether improvements arise from stronger single-agent reason- ing, richer peer verification, or more effective coordination, providing a clear context for evaluating our method.
Implementation Details. Unless otherwise specified, our main experiments are conducted with three-agent collaborative setup running for three interaction rounds. All autonomous agents instantiated from the instruction-tuned open-source backbones @wen2.5--BBInnstrun 3B-Instruct (Team) 2024),
Qwen2.5- and 1lama-3.2-3B-Instruu (Grattafiori al.) 2024]. To enable efficient post-training while controlling memory overhead, we adopt parameter- efficient fine-tuming mne-tnnnng via LoRA, with a default rank of 16, scaling factor a 32, and dropout rate 0.05. In practice, we implement the system using the HuggingFace Transformers framework, to- gether with 8-bit quantization and key-value caching to reduce memory consumption and accelerate multi-round inference. For autonomous agent generation, decoding follows nucleus sampling strat- egy with top-p = 0.95, temperature 0.7, and a maximum generation length of 1024 tokens. For the external GPT-based human proxy, we use a lower temperature of 0.3 to encourage more stable and reliable responses during intervention. To reduce randomness evaluation, most main experiments repeated with three independent random seeds and the reported results are averaged across runs.
For optimization, we use Adam with cosine learning- rate decay. In our hyperparameter search, the learning rate is selected from (2 10-5, 10 1 10-1, 1, 2 x 10-4 5 x 10-1), and the number of training epochs is varied within the range 1-3. We further control the effective batch size through gradient accumulation, with batch-size settings explored in {8, 16, 32). For the GRPO-style inner-loop optimization, we use a clipping coefficient e 0.2, a KL regularization weight BaI = 0.02, reward scaling fixed to 1.0, and advantage normalization enabled by default. These settings are chosen to maintain stable offline policy optimization while preserving sufficient sensitivity relative reward differences across candidate actions.
The training data consist of both real and synthetic task instances. For GSM8K, we directly use the original training split. For the remaining benchmarks, in order better simulate realistic help- seeking scenarios in which agents may need to request external assistance, we construct synthetic training data with GPT-5, generating task-aligned samples that mimic the types of situations en- countered during collaborative reasoning. In total, we use approximately 10K training instances across all tasks. Data generation and offline trajectory construction are performed under the HILA framework. Specifically, for each agent state, enumerate all valid strategic actions and roll out the corresponding consequences to form grouped candidates, which are then converted into con- trastive comparisons for offline GRPO training. This exhaustive enumeration allows build
19

---

Published as a conference paper at ICLR 2026
action groups with explicit relative preferences, rather than relying only on the single action sam- pled during online execution.
In addition to the policy-training data, also construct supervised data for the outer-loop continual learning stage. Whenever an agent chooses to seek external help, we record the corresponding problem together with the expert-provided correct reasoning trace and final answer, and treat this pair as a ground-truth supervision instance for SFT. After preprocessing. the resulting offline dataset contains roughly SK grouped samples for GRPO training and about 2K expert-labeled samples for SFT. This split reflects the intended role of the dual-loop design: the inner loop is trained primarily from action-level relative comparisons, while the outer loop driven by a smaller but higher-quality set of expert demonstrations that directly expand the model's ' reasoning capabilityy
For reward shaping, we fix the overall reward scale to 1.0 and tune the penalties associated with different strategic actions. In particular, the penalty for CREATE is searched over {0.05, 0.1, 0.2), while the penalty for DEFER varied from 0.1 0.5 with step size 0.1. These coefficients control the trade-off between autonomous exploration, reliance peer solutions, and external intervention. We tune them on held-out validation performance to balance reasoning accuracy against the cost of invoking additional computation expert assistance. Across all experiments, we use the same implementation pipeline and tuning protocol unless a specific ablation explicitly studies one of these hyperparameters.
Definition of Human Expert. In principle, the human collaborator in our framework refers to real person who can provide external knowledge and corrective interventions. However, following prior studies that approximate human input with advanced language models (Liu et al. 2023 Pandya iet al. 2024), we also adopt strong LLMs as practical substitutes. For LLaMA- agents, whose reasoning ability is relatively weaker, we use GPT-4o-mini the proxy expert in most experiments to maintain favorable balance between cost and effectiveness. In contrast, for Qwen-based agents, whose reasoning ability is substantially stronger, more powerful expert is necessary provide sufficiently informative guidance; therefore, we use GPT-4o as the proxy expert in those settings.
Prompt Templates. Our framework relies on small set of task-specific and interaction-speett prompt templates to coordinate reasoning, collaboration, and metacognitive control. Concretely, use base prompts to initialize task solving under different task types (e.g., mathematical reason- ing, multiple-choice question answering, and code generation), debate prompt support iterative multi-agent refinement by exposing each agent to its own prior response and the responses of other agents, and meta-policy prompt govern high-level action selection among EVAL, CREATE TE, and DEFER. For clarity, we present the canonical forms of these prompts below, with placeholder variables indicating the task content and the dynamically constructed interaction history.
Base Prompt (Multiple-Choice)
Answer the following multiple-choice question. Choose the single best option. Solve the problem:
([question])
Think step by step, show your reasoning, and end your response with a single line that clearly states the final answer.
Base Prompt (Math: Numerie Symbolic)
Solve the following problem:
{[quest i onn]]
Think step by step, show your reasoning, and be careful with arithmetic. End your response with a single line that clearly states the final answer. If the answer is number, output only the number on that final line.
20

---

Published as a conference paper at ICLR 2026
Base Prompt (Math with Forced Boxed Answer)
Solve the following problem: [qqusstio]]
Think step by step, show your reasoning, and be careful with arithmetic. Must give the final answer in the form \boxed(...):
Base Prompt (Code Generation with Unit Tests)
You are given Python programming task. Write correct and efficient solution that passes all unit tests. Rules:
Output ONLY Python code.
Do NOT include explanations, comments outside the given prompt, or additional text.
Keep the original function signature exactly given.
Do not write any test code. Do not use nnput () or print ()
You may use the Python standard library.
[qqeesiion}]
Base Prompt (Generic Fallback)
Solve the following problem:
{fquestinn ion]] Think step by step. End your response with single line that clearly states the final answer. the answer is number, output only the number on that final line.
Multi-agent Collaboration Prompt
You are in multi-agent collaboration.
=== Original Prompt ===
{base_prropt]]
Your Previous Responses ===
{[self history block))
=== Other Agents' Responses ==
Now compare the solutions, resolve disagreements, and provide an UPDATED final answer.
Keep reasoning concise but correct. Finish with clear final answer line.
Meta-Policy Prompt
You are a meta-policy controller for a multi-agent system. Choose ONE action and output ONLY the action line.
Valid actions (no extra text):
DEFER
(ask a human expert)
EVAL <idx>
(copy Agent idx; idx in 0. NN11) (write new solution yourself)
CREATE
[
Problem
{[base pusse-prompt] }
=
Your Latest Solution ===
[[self wwwesflllttttteee |ollutiooo)
21

---

Published as a conference paper at ICLR 2026
Accuracy on GSMBK (%)
90.5
90.0
DLPO
89.5
89.0
88.5
GRPO
Init
83.0
87.5
16
18
20
22
24
26
28
30
(%)
Figure 4: Accuracy vs. DEFER rate on GSM8K across training stages (Init - GRPO - DLPO) DLPO achieves a joint improvement, increasing accuracy while reducing reliance external inter- vention.
=== Other Agents' Latest Solutions ==
{otherss. (lothers.lateste www.lat.ssllleee iuutiion solutions]] Now output ONLY one action line.
C.2 EXPERIMENTAL RESULTS
Cecurac-Deeferen Coupling Across Training Stages. To further probe whether the reduced DEFER rate reflects genuine capability gains (rather than merely cost-avoidance), we jointly analyze task accuracy and deferral frequency across training stages. Figures 4 reveal a consistent and in- terpretable pattern: moving from the initialization policy to GRPO and then to full DLPO generally shifts the system toward the upper-left region, i.e., higher accuracy with lower reliance external intervention. This coupling is especially informative because it distinguishes two qualitatively dif- ferent regimes: (i) frugal policy that reduces deferral but sacrifices correctness, and (ii) stronger policy that reduces deferral while improving accuracy. Across benchmarks, GRPO tends to yield a modest leftward shift (lower DEFER), consistent with learning cost-aware intervention policy. In contrast, DLPO produces clearer Pareto-style improvement on all three datasets, simultane- ously decreasing deferral and increasing accuracy, which difficult to explain by cost-awareness alone. Instead, it is consistent with the dual-loop design: the inner loop improves when to ask, while the outer loop assimilates expert demonstrations so that the system increasingly does not need to ask. Notably, the magnitude of this coupled improvement varies by task: GSM8K, the accuracy gains are relatively modest while deferral decreases substantially. suggesting that DLPO primarily reduces unnecessary escalation trade-off, whereas DLPO recovers and improves accuracy while further reducing deferral, indicat- ing that continual learning helps avoid "over-pruning" intervention; and on MMLU, the monotonic improvement of both metrics provides strong evidence that the learned policy becomes both more selective and more competent. Overall, these coupled trajectories provide an additional behavioral validation that HILA's gains arise from both calibrated intervention and capability growth.
easier arithmetic items; AMC,, GRPO alone exhibits mild
Effect of Human Proxy Capability. We further examine how the capability of the external expert affects the overall performance of HILAA by replacing the human proxy with language models of different strengths. As shown in Figure. 2 and Table. 6] The results show clear and consistent pattern: stronger human proxies lead to better final performance across all evaluated benchmarks. When the proxy is weaker, the gains from intervention proxy produces the best overall results. This trend is consistent with the design of HILA, where DE- FER intended provide access higher-quality reasoning trajectories whenever the autonomous collective reaches its limits.
noticeably reduced, whereas stronger
These results suggest that the effectiveness of HILA depends not only learning when to invoke external help, but also on the quality the guidance received once intervention occurs. In other words, strategic deferral and expert quality are complementary: well-trained metacognitive policy
22

---

Published as a conference paper at ICLR 2026
42.5
Accuracy on AMC (%)
40.0
37.5
DLPO
Init
32.5
27.5
10
12
14
16
18
20
22
24
26
DEFER Rate (%)
Figure 5: Accuracy vs. DEFER rate on AMC across training stages. GRPO reduces deferral with small accuracy fluctuation, while DLPO simultaneously improves accuracy and further lowers deferral, suggesting that continual learning mitigates over-conservative intervention.
Accuracy on MMLU (96)
DLPO
73
71
GRPO
69
5.0
7.5
10.0
12.5
15.0
17.5
20.0
DEFER Rate (%)
Figure 6: Accuracy vs. DEFER rate on MMLU across training stages. Accuracy increases mono- tonically as deferral decreases, indicating that the learned policy becomes both more selective and more capable.
can identify situations where outside assistance is valuable, but the ultimate benefit still depends on how informative and reliable that assistance is. Overall, this experiment shows that HILA can effectively benefit from stronger external guidance when available, while also highlighting the im- portance of the chosen human proxy determining final system performance.
Deferral Cost as a Control Knob for Collaboration. We further examine how the deferral cost Cdefer modulates the balance between autonomous reasoning and external intervention in HILA. Figure (a shows the accuracy trends on AMC and MMLU under different values Cdefers while Figure (b) illustrates how the action distribution changes AMC when comparing low cost 0.1 with a high cost 0.5. A clear pattern is that increasing Cde pushes the policy away from DEFER and toward autonomous actions, especially EVAL. This confirms that the learned policy responds directly to the intended intervention penalty and adjusts its behavior accordingly. At the same time, the accuracy curves reveal nontrivial trade-off. moderate cost can reduce unnecessary reliance on external help while preserving competitive performance, whereas a larger cost suppresses deferral more aggressively but may also hurt final accuracy. This effect is particularly visible on the more difficult benchmark, where overly discouraging intervention can prevent the system from accessing useful external guidance when
still needed. These results support the view that C' serves as an effective control knob for collaboration, regulating how often HILA relies on outside expertise while exposing the performance- cost trade-off that must be tuned in practice.
The Value of Collective Exploration. To examine how the size of the collective influences both effectiveness and cost, evaluate HILA with varying numbers of autonomous agents AMC and MMLU, as shown in Table. and Table. g The results show a broadly consistent trend across both benchmarks: increasing the number of agents generally improves accuracy, especially when moving
23

---

Published as a conference paper at ICLR 2026
Model
GSM8K
AMC
MMLU
HILLA with gpt-4o-mini
88.15
33.33
68.30
HILA with gpt-3.5-turbo
83.33
17.50
66.84
HILA with gpt-4o
89.76
36.67
72.28
Table 6: Effect of human proxy capability on HILAA Using stronger language models as the external expert consistently improves performance on GSM8K, AMC, and MMLU, showing that the quality of external guidance an important factor in the effectiveness of strategic deferral.
0.36
AMC
0.34
MMUU
0.74
Accuracy (AMC)
Accuracy (MMLU
Caste - 0.1
Casla - 0.5
0.72
0.30
DEFER
0.28
0.70
27%
CREATE
EVAL
0.24
Codfor
CREATE
(a) Accuracy on AMC and MMLU under different deferral costs Carfer.
(b) Action distribution on AMC for low and high de- ferral costs (0.1 vs. 0.5).
Figure 7: Deferral cost as control knob for collaboration. (a) shows how changing the deferral cost C' affects task accuracy AMC and MMLU. (b) shows how increasing cost shifts the learned policy on AMC away from DEFER and toward autonomous actions, especially EVAL. Together, the results illustrate the trade-off between reducing external intervention and preserving final performance.
from very small collective to moderate-sized our framework, namely that larger agent pools enable richer collective exploration. With more agents generating and refining candidate solutions parallel, the system can access broader set of reasoning trajectories, increasing the chance that at least one high-quality solution emerges and can be selected or propagated through the metacognitive policy.
This pattern supports central premise of
At the same time, the gains are not linear. Accuracy improves most clearly in the low-agent regime, but becomes less stable as the collective grows larger, with smaller incremental gains and occasional fluctuations across neighboring settings. This suggests diminishing returns from simply scaling number of agents: once the solution space has been sufficiently diversified, additional agents contribute progressively less new information. In contrast, computational cost increases sharply with the agent count. Both input and output tokens grow rapidly as more agents participate, leading to substantial increase in total token usage when accuracy improves only marginally. Taken together, these results highlight a practical trade-off: larger collectives can strengthen reasoning through broader exploration, but the benefit must be balanced against the rising inference cost. This makes the number of agents an important Pifcieecy-performmnce design choice in deploying HILA.
Optimal Depth of Iterative Collaboration. To assess the role of iterative refinement, we vary the number collaborative rounds and evaluate HILA on AMC and MMLU, shown Table and Table I0 The results show a clear non-monotonic pattern across both benchmarks. Increasing the number of rounds from shallow interaction depth to a moderate one consistently improves accuracy, indicating that multi-round collaboration helps the system refine intermediate solutions, incorporate peer feedback, and recover from errors made in earlier stages. This behavior supports the intuition that iterative interaction strengthen reasoning by allowing agents to progressively revise and consolidate candidate solutions rather than relying on a single-pass exchange.
However, the benefit of additional rounds does not continue indefinitely. Performance peaks at an intermediate depth and then declines slightly when the interaction is extended further, even as token usage continues to increase. This suggests that while additional rounds initially provide useful opportunities for correction and refinement, excessive interaction can introduce diminishing returns and may begin reinforce suboptimal trajectories. One plausible explanation is that repeated
24

---

Published as a conference paper at ICLR 2026
Agents
AMC
Avg. Input Tokens
Avg. Output Tokens
Avg. Total Tokens
26.67
3146
1299
4445
28.33
10184
2564
12748
32.50
23285
4463
27748
31.67
42540
6349
48889
33.33
64097
7585
71682
32.50
92135
9501
101636
34.17
162142
12704
174846
10
35.83
238278
14928
253206
Table 7: Effect of the number of agents on performance and token cost on AMC. Increasing the collective size generally improves accuracy, but also leads to a steep rise input, output, and total token usage, revealing clear performance- efficiency trade-off.
# Agents
MMLU
Avg. Input Tokens
Avg. Output Tokens
Avg. Total Tokens
66.32
2054
508
2562
2
67.54
5575
972
6547
3
68.49
11049
1525
12574
4
68.95
18302
2077
20379
5
68.60
27133
2609
29742
6
69.47
38317
3223
41540
8
69.12
65295
4333
69628
10
69.65
94827
5612
100439
Table Effect of the number of agents on performance and token cost on MMLU. larger collec- tive typically yields better accuracy, especially in the low-agent regime, while substantially increas- ing token consumption as the system scales.
exchanges increase the chance propagating noisy or flawed intermediate reasoning, making the collective more vulnerable to error accumulation rather than correction. Taken together, these results indicate that iterative collaboration valuable but not unbounded: the number of rounds should be chosen to balance the gains from refinement against the rising computational cost and the risk of over-iteration.
C.2.1 REAL HUMAN-IN-THE-L-ON
Motivation and Evaluation Scope. Our main experiments strong GPT-based models prox- ies for external experts, which provides controlled and scalable approximation of human interven- tion. However, the central claim of HILA is fundamentally about human-agent collaboration rather than model-model interaction alone. To validate that the framework remains effective when the external expert replaced by real human participants, we conduct additional human-in-the-Be study with real PhD-level experts. Because real human annotations are substantially more expensive than API-based proxy supervision, focus three representative benchmarks that jointly cover arithmetic reasoming, competition-ssyle mathematical problem solving, and general knowledge rea- soning: GSM8K, AMC, and MMLU. This selection provides diversity in task format and difficulty while keeping the study practically feasible. Accordingly, we instantiate real human involvement two operational forms: proactive guidance provided initialization and reactive assistance, as summarized in Figure 8
Balanced Subset Construction for Human Evaluation. To ensure that the human-evaluation subset faithfully reflects the range of collaborative states encountered by HILA, we do not simply sample instances uniformly at random. Instead, for GSM8K and MMLU, subset based on the -re-intervention agreement pattern among agents, so that the selected instances cover qualitatively different collective reasoning regimes. Concretely, each item is assigned one of the following categories: (i) all agents agree and are correct, (ii) the majority agrees but is incorrect, (iii) the majority agrees and correct, and (iv) all agents disagree. We then approximately balance
construct stratified
25

---

Published as a conference paper at ICLR 2026
Rounds
AMC
Avg. Input Tokens
Avg. Output Tokens
Avg. Total Tokens
1
18.33
455
2195
2650
2
22.50
13406
3480
16886
3
31.67
24129
4580
28709
40.83
33287
5203
38490
5
35.00
43984
6627
50611
Table 9: Effect of the number of collaborative rounds on performance and token cost on AMC. Accuracy improves substantially as the interaction depth increases from one to four rounds, but declines when an additional round added, indicating diminishing returns from over-iteration.
# Rounds
MMLU
Avg. Input Tokens
Avg. Output Tokens
Avg. Total Tokens
0.6404
5352
933
6285
0.6491
6385
1251
7636
0.6849
11049
1525
12574
0.6930
15318
1798
17116
0.6895
19689
2144
21833
Table 10: Effect of the number of collaborative rounds on performance and token cost on MMLU. A moderate number of rounds improves accuracy, while further increasing the interaction depth yields only limited benefit and slightly higher cost.
these categories in the final evaluation subset. This stratification is important because it exposes human experts to both easy and hard collaborative contexts, including cases of consensus failure and severe disagreement, rather than over-representing only trivial instances. For AMC, since the benchmark itself is comparatively small, we evaluate on the full set instead of a stratified subset.
Human Expert Pool and Participation Protocol. As shown in Figure we recruit 20 PhD students from a broad range of technical disciplines, including Computer Science, Computer En- gineering, Electrical Engineering, Mathematics, Industrial and Systems Engincering, Biomedical Engineering, Earth Sciences, and Economics, and treat them as domain-genersa expert annotators. To make real human participation compatible with repeated and reproducible evaluation, we adopt an offline expert collection protocol: each human expert answers the selected questions in advance, rather than being queried interactively during live inference. For every problem, each expert pro- vides two fields: human-idea, which contains concise problem-solving intuition high-level plan, and human.reasoning... which contains a more complete reasoning trace together with the final answer. This design allows to evaluate both lightweight human hints and full expert lutions under the same system. Formally, if x denotes the input problem and denotes a human expert, then the collected annotation can be written
(x)
idea (x), ch (pa)on(x))
where cidea (x) and rrasson (x) denote the levels of expert guidance.
e)
Two Modes of Human Participation. We study two distinct modes of real human involvement under HIIAA In the first mode, the human serves as a reactive expert, i.e., the human is consulted only when an agent selects the DEFER action. In this case, the human response functions exactly as the external expert signal in the standard HILAA loop. In the second mode, the human serves as proactive expert, i.e., expert information is injected before the agents begin their initial reasoning. This proactive information can be either the concise numan_ide Importantly, these two roles are not mutually exclusive: even when proactive human information is provided at initialization, the system may still choose to defer later to GPT-based external expert during downstream collaboration. This lets us separate two complementary effects of human in- volvement: initial guidance (better starting trajectories) and on on-demand correction (targeted rescue under uncertainty). In notation, the proactive setting augments the original problem input into
the full human_reason ng.
x' = (x, ginit
(i:) (1:)),
where Cinit (1x) may be either (2)( )
e geason (3:).
26

---

Published as a conference paper at ICLR 2026
Proactive
Reactive
Human Expert
Human Knowledge
Human
Al Agent
Expert
Human
Feedback
Figure 8: Real human-in-the-loop collaboration in HILA with bidirectional interaction. Real hu- man experts can act proactively by injecting prior knowledge at initialization (e.g., high-level human_idea or full human collaboration begins. They can also act reactively by providing corrective feedback when the meta- policy selects DEFER. This unified view highlights that human expertise in HILA is not limited to post-hoc supervision, but can be integrated both as upfront guidance and as on-demand intervention during multi-round collaboration.
shaping the agents' starting state before autonomous
Human Ideas
Help Request
>>
>>
Human Reasoning
OE
E
Experts
Offline
Real Human Beinge
Figure 9: Real human feedback pipeline in HILA. Upon help request (DEFER), the system routes the query to a pool human experts, who provide either high-level human ideas or full human reasoning. The collected feedback is then stored structured supervision for subsequent training and continual improvement.
Expert Quality in the Single-Expert Regime. Before evaluating the full HILA framework, we first compare the standalone problem--olving accuracy of candidate experts, including the base LLAMA3-8B model, several GPT models, and the real human expert pool. This serves direct estimate of the quality ceiling available the system once external intervention occurs. shown in Table real human experts achieve the strongest overall performance across the three benchmarks, with especially large gains on AMC, where they substantially outperform model-based experts. This result is important because it shows that the human expert channel is not merely concep- tual replacement for GPT proxies, but can provide genuinely stronger guidance in domains where structured reasoning and careful mathematical judgment especially important. At the same time, the margin on MMLJU is smaller, suggesting that the relative advantage of human experts depends on task type. Overall, this comparison establishes that real humans constitute high-quality and meaningful source of external expertise for HILA.
Reactive Human Experts Under HILA. We next replace the GPT-based expert in HILA with different reactive experts while keeping the autonomous backbone fixed to LLAMA3-8B. The re- sults in Table 12 show that stronger experts consistently lead to better downstream performance, and real human experts provide the best overall results. This trend mirrors our earlier proxy-based observations. but now with actual human participants: once the system learns when to defer, the final gain still depends critically on the quality of the external assistance. The improvement especially striking AMC, where replacing GPT experts with real humans yields much larger jump than on the other two datasets. This suggests that the value of human intervention particularly strong in settings where the collective may otherwise converge plausible but incorrect line of reason-
27

---

settings under the same Published as a conference paper at ICLR 2026
Expert
GSM8K
AMC
MMLU
LLaMA3-8B
58.67
16.67
57.33
GPr-3.5-Turbo
66.00
17.50
58.00
GPT-4o-mini
85.67
39.17
86.67
GPT-4o
88.33
46.67
87.33
Human Experts
95.00
97.50
88.00
Table 11: Standalone performance of candidate external experts on the human-evaluation bench- marks. Human Experts denote the aggregated real-expert pool collected from 20 PhD) participants. All values are reported as percentages.
Reactive Expert in HILA
GSM8K
AMC
MMLU
GPT-3,5-Turbo
61.67
20.83
59.67
GPT-4o-mini
74.33
29.17
64.33
GPT-40
77.33
33.33
67.33
Human Experts
78.67
61.67
75.33
Table 12: Performance of HILA with different reactive experts, i.c., experts consulted only when the system selects DEFER. The autonomous backbone is fixed LLaMA3-8B. All values are per- centages.
ing. In other words, human experts are not only more accurate in isolation; when used as reactive intervention targets, they also provide more effective corrective signals for HILA's decision process.
Proactive Human Guidance at Initialization. Finally, we examine whether humans can con- tribute not only reactive experts but also as proaetive sources of guidance before the multi-agent reasoning process begins. Table 13 compares several numan-participae
LLAMA3-88 8B backbone. When humans provide only a high-level idea at initialization, the benefit mixed: the system shows some improvement on MMLU,, U, but the gains are limited or inconsistent on the more mathematical datasets. This suggests that a coarse human hint may improve the initial search direction, but is often insufficient reliably steer the full collaborative process toward the correct solution. In contrast, when humans provide a full reasoning trace at initialization, perfor- mance improves substantially across all three benchmarks, yielding the strongest results among all compared human- particiiaaiiiitioonnn settings.
Reactive vs. Proactive Human Roles. The contrast between human_idea and full saman-reasonia reveals a key mechanism of real human value in HILA. Lightweight proactive hints primarily act as weak priors: they may bias the initial reasoning trajectory, but the downstream agents must still reconstruct the full solution path on their own. Full proactive reasoning, by con- trast, provides much stronger structured signal that can reshape the collaborative search process from the outset. Notably, this proactive human information can still be combined with subsequent GPT-based reactive intervention, creating a hybrid setting in which human knowledge improves ini- tialization while automated expert proxies remain available for later rescue. Taken together, these results suggest that real humans can improve HILA in two distinct but complementary ways: as re- active experts that correct failures when the system explicitly requests help, and proactive experts that improve the quality of the initial reasoning state before collaboration unfolds. This provides stronger empirical evidence that HILA is not merely compatible with human involvement in princi- ple, but can meaningfully leverage real human expertise under multiple operational modes.
Action Allocation Under Real Human Participation. To better understand how real human in- volvement changes the behavior of HILA beyond final task accuracy, we further analyze the distri- bution of strategic actions under different human-partichi particppatiiiaatioon modes. Specifically, we report the proportion of EVAL, CREATE, and DEFER decisions across GSMSK, AMC, and MMLU when humans act as reactive experts, proactive idea providers, or proactive full-reasoning providers. This analysis is important because the performance gain arise from qualitatively different col- laboration regimes: one system may improve by becoming more self-reliant, while another may improve by more aggressively invoking external help. By comparing action distributions, can
28

---

Published as a conference paper at ICLR 2026
Human Participation Setting
GSM8K
AMC
MMLU
GPT-4o reactive expert only Human Experts reactive expert only
77.33
33.33
67.33
78.67
61.67
75.33
Human proactive idea + GPT-4o reactive expert
78.33
34.17
79.33
Human proactive full reasoning + GPT-4o reactive expert
90.67
65.83
86.67
Comparison of different real human participation modes under HIILA. In the proactive setting, human information is injected before the initial agent reasoning, while GPT-4o remains available as the reactive expert for later DEFER decisions. All values are percentages.
Table 13:
GSMBK
AMC
MMLU
DEFER
EVAL
DEFER
EVAL
DEFER
37.0%
26.5%
26.7%
EVAL
36.9%
60.0%
35.7%
CREATE
CREATE
CREATE
Figure 10: Action distribution under HILA when real humans reactive experts, meaning that feedback is provided only after the system chooses DEFER. Across datasets, CREATE and DEFER dominate on GSM8K and AMC, while MMLU shows noticeably larger proportion of EVAL.
characterize how the source and form of human input reshape the policy's internal decision dynam- ics.
Reactive Human Experts Encourage Direct Help-Seeking. When real humans as reactive experts, i... they are only consulted after the policy selects DEFER, the resulting action distribution shows a substantial allocation deferral across all three datasets (Figure (O). This particularly notable on GSM8K, where DEFER becomes the most frequent action. The pattern is consistent with the strong standalone quality of human experts observed earlier: once the policy learns that real humans provide highly reliable corrective signals, it becomes more willing to invoke exter- nal intervention in uncertain settings. At the same time, the policy still retains nontrivial mass on CREATE, suggesting that HILLA does not collapse into pure expert reliance even when high-quality human help is available. Instead, it maintains a mixed strategy in which autonomous exploration and expert consultation coexist.
Proactive Human Ideas Shift the Policy Toward Greater Deferral. A different pattern emerges when humans provide only a concise idea at initialization, while GPT-4o remains available as the reactive expert for later DEFER decisions (Figure [T]. In this setting, the proportion of DEF ER in- creases further across three benchmarks relative to the hueeiieehummn-only condition. This sug- gests that lightweight human idea, although useful an initial prior, is often insufficient to fully resolve downstream uncertainty, As a result, the policy appears to rely more heavily on later reac- tive intervention, using the initial hint guide the search while still frequently requesting stronger expert assistance when difficult reasoning steps remain unresolved. this sense, high-level human ideas may improve early orientation, but they do not necessarily reduce the model's dependence external rescue during the full reasoning trajectory.
Full Proactive Human Reasoning Rebalances the Collaboration Strategy. When the proac- tive human input is upgraded from a high-level idea to full reasoning trace, the action distribu- tion changes again (Figure 12 Compared with the proactive-idea condition, DEFER decreases on GSM8K and AMC, while CREATE generally becomes more prominent. This pattern suggests that richer upfront human guidance can better stabilize the initial reasoning state, allowing the agents to continue refining or extending stronger partial solution without needing invoke reactive help as often. In other words, full human reasoning trace appears to function less like weak prior and more like structured scaffold for downstream collaboration. Although DEFER remains
29

---

Published as a conference paper at ICLR 2026
GSMBK
AMC
MMLU
DEFER
EVAL
21.5%
$1.25
25.7%
EVAL
DEFER
45 15.0%
33.9%
35.5%
CREATE
CREATE
CREATE
Figure 11: Action distribution under HILA when humans provide proactive idea at initialization, while GPT-4o remains available as the reactive expert for later DEF ER decisions. Results are shown for GSM8K, AMC, and MMLU.
GSMBK
AMC
MMLU
EVAL
DEFER
DEFER
EVAL
DEFER
39.0%
32.3%
35.8%
35.2%
CREATE
CREATE
Figure 12:
Action distribution under HILA when humans provide full proactive reasoning at initial- ization, while GPT-4o remains available as the reactive expert for later DEFER decisions. Results are shown for GSM8K, AMC, and MMLU.
negligible, the reduction relative the proactive-ides setting indicates that stronger proactive human input can partially substitute for later intervention by lowering uncertainty earlier in the process.
Human Guidance Changes Policy Regimes, Not Just Accuracy. Taken together, these distribu- tions reveal that different forms of human participation induce meaningfully different policy regimes HILA. Reactive human experts primarily increase the attractiveness of DEFER by making external intervention highly valuable when explicitly requested. Proactive human ideas, while beneficial as initial hints, can still leave enough residual uncertainty that the system defers even more frequently downstream. By contrast, proactive full reasoning provides stronger structural guidance upfront, which reduces the need for subsequent intervention on some tasks and shifts more probability mass toward autonomous continuation through CREATE and EVAL. These observations complement the accuracy results by showing that real human involvement affects not only how well the system per- forms, but also how chooses collaborate.
D DECLARATION ON THE USE OF LARGE LANGUAGE MODELS
In preparing this work, made of several large language models for different purposes. First, GPT-4o-mini and GPT-4o were integrated directly into our experimental framework, where they served proxies for human experts in the human-in-the-loor setting. This design choice follows prior research and allowed us to evaluate the framework under controlled and repeatable conditions while balancing cost and effectiveness. GPT-5 was used generate synthetic training data for model optimization, enabling scalable and controlled construction of training instances. Second, GPT-5 was employed assist with improving the clarity, organization, and readability of the manuscript. The model helped refine phrasing and grammar, but all conceptual contributions, methodological de- sign, and experimental analysis were developed by the authors. All content was carefully reviewed, edited, and validated by the authors, who take full responsibility for the accuracy and integrity of the final publication.
30


Reviews:
[{"review_id": "8dWxcHtuMf", "reviewer": "Reviewer_3GsK", "summary": "The paper proposes LIMA, a metacognitive framework where LLM agents can decide whether to exploit current knowledge or to defer to a human expert for help. This decision policy is trained with Dual-Loop Policy Optimization (DLPO), where the inner loop is GRPO with a cost-aware reward for deferring, and the outer loop is continual learning where expert demonstrations are turned into lasting capabilities. Extensive experiments across multiple datasets all confirm the effectiveness of the proposed method.", "questions": "1. How should the deferring cost C be set in practice?\n\n2. The goal of GRPO is to maximize the cumulative reward for each episode. Why can't the agents first learn to always defer to experts to learn the capabilities, and then choose to always not defer to maximize the return?\n\n3. Do you have any mechanism to prevent forgetting the knowledge given by the expert?\n\n4. It will also be interesting to see how well this method perform in single-agent domains as I think this method is perfectly applicable to single-agent as well.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Paper is well-written, technically sound, and easy to follow.\n\n* The method proposed is novel and principled. The option to defer to expert is interesting for a multi-agent system. And the dual-loop training process combined into a single DLPO loss is principled and easy to implement.\n\n* The empirical evaluations are extensive across multiple benchmarks, as well as ablation and scaling studies, which all demonstrate the superiority of the method proposed.", "weaknesses": "* Cost model is constant $C$, which seems to be a strong assumption. There are many scenarios where querying the expert with different levels of question would incur different costs. It is also unclear how sensitive the outcomes are to C.\n\n* \"Human\" is proxied by gpt-4o-mini for the main experiments. It is interesting to see how this would scale with more capable models like gpt-4o.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNLWpMgP8y", "reviewer": "Reviewer_QNPF", "summary": "This paper explores how to learn \"when to seek help from humans and how to continually learn from human feedback\" in human–agent collaboration. The core idea is to enable multi-agent systems to proactively trigger \"deferral actions\" (defer action) in situations of uncertainty (confidence) or beyond their capabilities. Then leverage feedback for continuous optimization. The inner loop is used for immediate decision optimization, while the outer loop is used for long-term capability growth. The experiments cover various reasoning tasks (GSM8K, MATH, AIME, AMC, HumanEval and MMLU), demonstrate the framework's effectiveness.", "questions": "- Corresponding to weakness1: How human expert were selected? who these participants are?\n- Corresponding to weakness2: How much of the performance gain comes from external high-level feedback? Could the presence of such advanced feedback (from human experts) lead to unfair comparisons?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Novel Framework: The idea that an agent can adaptively seek help from humans based on its own capabilities while continually improving itself through higher-level feedback is novel.\n- Method's Effectiveness: From paper's table 1, LIMA and LIMA (w/ DLPO) acquire a significant gains compared to baseline models across six representative benchmarks.", "weaknesses": "- For human expert: The paper mentions “real human experts” but does not clarify who these participants are, how they were selected, or whether the framework accounts for human cognitive cost or fatigue.\n- Performance gains: The relationship between this performance gain and external feedback requires careful analysis.\n- Presentation: Table 2 column 1 Model.\n- Although the proposed method achieves improvements across the six benchmarks in Table 1, its performance remains below the sota （broad perspective) results on these benchmarks, which weakens the overall significance of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YpjqJz787R", "reviewer": "Reviewer_TKNM", "summary": "The Learning to Intervene via Metacognitive Adaptation (LIMA) framework introduces a novel approach to human–agent collaboration, enabling agents to metacognitively determine when to defer to human expertise. To train these agents, the authors propose Dual-Loop Policy Optimisation (DLPO), which separates short-term deferral decisions from long-term skill development. DLPO’s inner loop uses GRPO with a cost-aware reward, while its outer loop incorporates expert feedback for continual learning. Experiments on mathematical reasoning and general problem-solving tasks show that LIMA with DLPO outperforms existing autonomous multi-agent systems, offering a strong foundation for adaptive, continually improving collaboration between humans and AI, especially investigating when and how to consider human intervention.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty of the idea is good", "weaknesses": "- Could include further related work such as \"LLM-Mediated Guidance of MARL Systems\" (Siedler et al) to \"human-in-the-loop\" paragraph\n- Figure 1 has not been incorporated in the text (sorry if i missed this)\n- A flow/process diagram supporting the methodology would elevate comprehension drastically\n- I would not consider MMLU as a \"general knowledge reasoning\" benchmark - its language understanding\n- I think there should have been either more experiments for models such as \"GPT-4o-mini as a proxy human expert\" to verify the \"human expert\" llm, or real human (at least partly) in the loop to verify the llm based \"human expert\".\n- For the \"Human involvement under two modes\" Table 3, why are there only reports on GSM8K and AMC? This feels cherry-picked\n- For the results in \"Figure 3: Effect of scaling collaborative configurations.\", I think you should also consider additional inference costs per performance increase\n- \"identify their knowledge boundaries\" - Unfortunately, there are no results supporting this claim\n- \"assess its own certainty\", \"its relation to the collective\", \"intrinsic quality of its current reasoning path\" - Unfortuantely also here, no results supporting this claim at all", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y4PCt80Sdd", "reviewer": "Reviewer_mLVp", "summary": "This paper tackles the \"closed-world\" limitation of autonomous multi-agent LLM systems (constrained by pre-trained knowledge, prone to collective failure) by proposing the LIMA framework for human-agent collaboration and DLPO dual-loop optimization. LIMA gives agents a metacognitive policy to choose autonomous problem-solving (evaluate/generate solutions) or human deferral, using a structured cognitive state space. DLPO’s inner loop (GRPO with cost-aware rewards) refines deferral decisions, while the outer loop turns expert feedback into supervised signals for continual learning. Experiments on reasoning benchmarks (MMLU, HumanEval, GSM8K, etc.) show LIMA outperforms state-of-the-art autonomous MAS across LLM backbones.", "questions": "Please see weaknesses.  I will adjust the paper’s score accordingly in my rebuttal to reflect the strengths identified.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper addresses multi-agent LLM systems’ \"closed-world\" flaw with the LIMA framework, integrating metacognitive assessment (via structured cognitive states) to let agents strategically choose autonomy or human help, surpassing passive \"human-in-the-loop\" designs. The DLPO dual-loop optimization creatively unites short-term decision refinement (GRPO) and long-term capability growth (expert feedback-driven learning), solving prior disconnections between decision and knowledge enhancement. Formal mathematical formulations boost reproducibility.\n\n2. The experimental design is comprehensive. It tests across three key domains (general knowledge, program synthesis, math reasoning) with diverse benchmarks, ensuring generality. A broad baseline set (single-agent, interactive, system-level) enables fair SOTA comparisons. Cross-model tests (Llama3, Qwen2.5) show adaptability, even improving small models. Ablation studies and configuration analyses strengthen conclusions, while AI proxy and real human expert tests enhance real-world relevance.\n\n3. The paper is clear, well-structured, and accessible.", "weaknesses": "1. While the paper identifies \"4 agents\" as the optimal configuration, it provides no analysis of why performance plateaus beyond this number, nor tests scalability to larger agent populations (e.g., 8+ agents) , a key gap for deploying the framework in complex tasks (e.g., multi-step scientific research). The current LIMA design relies on full inter-agent consistency checks to inform metacognitive decisions, which would incur prohibitive communication and computation costs as agent count grows (e.g., O(n²) consistency comparisons for n agents)\n\n2. The paper frames \"long-term capability growth\" as a core goal of DLPO’s outer loop, but its experiments only measure performance improvements over fixed benchmarks (e.g., MMLU, GSM8K) rather than tracking sustained learning across sequential, novel tasks. For example, it does not test whether the framework retains previously learned skills when fine-tuned on new task feedback (e.g., first learning to solve GSM8K, then AMC, then retaining GSM8K performance). This omission hides potential catastrophic forgetting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)