ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning
Authors: Ziyu Wan
Yunxiang LI
Xiaoyu Wen
Yan Song
Hanjing Wang
Linyi Yang
Mark Schmidt
Jun Wang
Weinan Zhang
Shuyue Hu
Ying Wen
Venue: NeurIPS 2025 poster ()
Keywords: Reasoning
Large Language Model
Multi-Agent
Reinforcement Learning
Primary area: deep_learning
TL;DR: Training a new reasoning paradigm of LLMs explicitly contains meta-thinking in a multi-agent and multi-turn setting with RL

Abstract: Recent research on Reasoning of Large Language Models (LLMs) has sought to further enhance their performance by integrating meta-thinking—enabling models to monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving.
However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting in low efficacy.
To address this challenge, we introduce Reinforced Meta-thinking Agents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors, encouraging LLMs to think about thinking.
ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions.
Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading to improved generalization and robustness.
Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks.
Additionally, we further extend ReMA to multi-turn interaction settings, leveraging turn-level ratio and parameter sharing to improve efficiency.
Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta-thinking reasoning process enhances the reasoning capabilities of LLMs.

ReMA: Learning to Meta-think for LLMs with Multi-agent Reinforcement Learning
Ziyu Wan 12 Yunxiang Li"; Xiaoyu We Yan Song', Hanjing Wang', Linyi Yang', Yan Mark Schmidt Jun Wang Weinan Zhang', Shuyue Hu?! Ying Wen!!
Shanghai Jiao Tong University
Shanghai Artificial Intelligence Laboratory
3 University of British Columbia
University College London
Canada CIFAR Al Chair (Amii)
Abstract
Recent research on Reasoning of Large Language Models (LLMs) has sought further enhance their performance by integrating meeta-hinninggennbbaee monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting low efficacy. To address this chal- lenge, we introduce Reinforced Meta-thinking Agents (ReMA), a novel frame- work that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta- thinking behaviors, encouraging LLMs to think about thinking. ReMA decouples the reasoning process into two hierarchical agents: high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions. Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading improved generalization and robustness. Empirical results from single-turn exper- iments demonstrate that ReMA outperforms single-agent RL baselines plex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks. Additionally, we further extend ReMA multi- turn interaction settings, leveraging turn-level ratio and parameter sharing to im- prove efficiency. Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta- thinking reasoning process enhances the reasoning capabilities of LLMs. Our code can be found in https: //githubb com/ wwt.w/www......i..e. ziyu -MeMM-eMMM--icccccc
models
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities in knowledge under- standing and complex reasoning tasks [Chowdhery et al. 2023 Achiam al. 2023 Anil et al.)
2023 Dubey et al. 2024). The paradigm in developing LLM-based reasoning models is shifting from scaling training-ting computation towards scaling test-time computation (Snell et al.) 2024 Recent advancements, such as OpenAl-ol [OpenAT] 2024), Deepseek RI [DeepSeek-A) et al.
2025 and Gemini 2.0 Flash Thinking [DeepMind] 2025 have demonstrated that allowing [.L.ss to
think before generating answers can significantly enhance performance and lead to the emergence of
"Equal contribution.
Work done during internship at Shanghai Artificial Intelligence Laboratory Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question: T 9.5. Illog =²
log, log, x" is identity all comguic value
Construction based method
RI-like method
ReMA: Reinforced Meta-thinking Agents
Question
Queition
Lansbica dune piddica
Undvrrand 00
Meds-thinking
Poutrained
LLM
Ressoing Trmplates
Template
Powerful
Ressonine
Question
High-leool agraan
Meda Thinking
levelleellllee sgcat
Slafe
MAIL
Ressoning Steps
Sepercited
Rejection Stappling
Reinfereennnt Learsing
Outcame Reawed
Iaga" 2
bgg
Ingar
Fineruning
lag'
kag'
() Brokk dune if y stile Mil
Clayy sp rood to Mines lufs brows fire
Mitaa Thinking
Romeening sigps
[OOT] Ere, The equation glar
1. Lot merlie each lemnuding the nonural speritin Ds Stort eg with the hond sicer (Dog o
Sigirt) spick)..
The withie
[VERIF)] Vorly snsent
Maill foold Los of oes
fiis equation
[oom Tharcken, krdf is the right arawes
Tharskon, kndt the right anaoes
Ga0m di Ai- 9-
step
miwaakss.. (PMM
Phas Annoer
Mampkend
Resnoriig
Thanknn
raa Armanerie
27
Rassonin
Thansing
27
The Frent Ro
Figure 1: Left: A construction-basee method that fine-tunes LLMs using rejection sampling, search- ing among combinations of pre-defined templates. Middle: R1-like method learns mix meta- thinking and detailed reasoning steps during training. Right: Our method ReMA separates the meta-thinking and reasoning steps in multi-agent system and updated by reinforcement learning.
human- like reasoning patterns. These patterns like "Wait, hold on." or "Let's break this down.". indicate that LLMs can develop a form of meta-thinking abilities that can generalize well to out-of- distribution (OOD) tasks [Xiang et al. 2025]. Meta-thinking, also known as metacognitive skills [Flavell] 1979], is an ability traditionally considered uniquely human [Didolkar et al.) [024]].
To cultivate meta-thinking patterns from LLMs themselves, recent construction-basse supervised approaches leverage supervised finetuning on structured reasoning trajectories. Specifically, these methods sampling reasoning trajectories from predefined meta-thinking templates and then use
pervised finetuming (SFT) or direct preference optimization (DPO) [Rafailoy et al. [223] to teach
LLMs imitate these patterns eQi al. 2024 Yue et al. 2024 Xi al. 2024 Yang et al. 2025 Muen- nighoff et al. 2025 Ye et al.) 2025c However, such methods lack sulficient flcxibility for LLMs to explore suitable meta-thinking patterns. Thus, they often fail to generalize out-of-distribution
(OOD) problems, leading to unstable performance on unseen data [Kirk et al.) Chu et al. 2025).
Besides construction-based methods, Ri-like single-agent reinforcement learning (SARL) has also been adopted for meta-thinking in reasoning (DeepSeek-Al et al., 2025 Xie et al. 2023 How-
ever, these SARL attempts typically rely on strong foundational models for easier exploration or extensive task-specific fine-tuning for stable training (Xu et al. 2025 Gandhi et al. 2025). Further- more, SARL needs to learn meta-thinking and reasoning within a single forward pass, seeking to capture complex reasoning structures purely in autoregressive manner (Xie et al. [025). This can potentially lead to issues such as incfficient exploration as well as reduced readability and carly convergence to local optima [DeepSeck-Al et al. 2025 Xiang al. 2025].
To address these limitations, we introduce Reinforeed Meta-thinking Agents (ReMA), a novel framework that leverages multi-agent reinforcement learning (MARL) to encourage LLMs to think about thinking. Our approach employs a multi aggett system (MAS) composed of high-level meta- thinking agent, responsible for strategic oversight and instruction generation, and low-level rea- soning agent tasked with detailed executing processes based on provided guidance. We compare the inference process among the construction- Since MAS distributes the exploration space of SARL into multiple agents, it enables each agent to explore more structurally and efficiently during training. Then we apply reinforcement learning on each agent with aligned reward functions. In this way, ReMA effectively balances the trade-off between generalization capability and exploration efficiency. As result, they can learn to play the best of their role (either to meta-think think or to follow instructions). at the present of the other agent.
method, R1-like method, and ReMA in Fig. [
To our knowledge, we are the first to formally define and optimize multi-agent meta-thinking rea- soning process (MAMRP) through multi-agent reinforcement learning. Our extensive experiments span both math reasoning and L.L tasks, where ReMA consistently achieves the high- est average performance across three backbone pretrained models. We further extend ReMA to multi-turn interaction settings on math reasoning tasks, implementing turn-level ratio to optimize

---

trajectory returns and stabilize training. Through comprehensive ablation studies, we illustrate the evolving dynamics between agents, revealing unexpected interaction patterns such as role reversals under different reward settings. These findings provide valuable insights into how meta-thinking processes enhance the reasoning capabilities of L.I.ss.
Preliminaries
In this section, outline the formulation of the vanilla reasoning process (Sec. 2.1 and the repre- sentative training methods (Sec. 2.2) along with the notation used throughout the paper.
2.1 Vanilla Reasoning Process (VRP)
The probability of generating a response equals the product of its stepwise probabilities. Given model To and a prompt x (xi,...; ØN) ). the vanilla reasoning process (VRP) autoregressively produces response y = (y), - Vr.) with
To ( II To(y 11, 22 NN U1.... ....... Vt 1)
II No (o(y)xx,<<))
The response usually contains intermediate reasoning steps before arriving the final answer, this process is also known as chain-of-thought (CoT) (Wei al. 2022], which can be represented as:
reasoning stops
a,
where a is the extracted final answer, which is included in the answer y.
2.2
Training VRP via Reinforcement Learning
RL frames VRP decoding process as a deterministic, token-level Markov Decision process (MDP) Wang et al. 2024a]. Its objective is
((6) = Ex )>D.y (Ryyy")))
where R(::)) represents a reward function comparing generated answer y with the golden answer y for any question sampled from dataset 'D.
To compute the gradient V T(),,, computationally efficient algorithms GRPO (Shao et al. 2024 and REiNFORCE++ (Hu 2025) are widely adopted. Take GRPO as an example, given a question- answer pair x, y and a group of G' generated responses y.. denote yi, as the j-th token of the i-th response, it optimizes the following loken-level objective:
[()) ( JwD, (yr I (:x))
(1 Aeg, (1)(),s(()) 1 - 1+ Acs) - B DRa. (Toll )}
(2)
where the token-level ratio Tı. (0) and the group normalieed advantage As, are defined as:
(()) ne (yij x, Yi,ci)
A R Ra-mean( (
Teaia (Yij X, Vi,<))
[ [
However, RL on base LLMs that haven't 't been well-aligned may suffer from issues like poor read- ability and language mixing, preventing researchers from verifying, understanding, and further de- veloping their LLMs. And huge searching space makes efficient learning of veta-thinking daunting.
3 Method
In this section, we present Reinforced Meta-thinking Agents (ReMA), RI. method integrating meta-thinking into the reasoning process L.LM under multi-agent settings (Sec. 3.1 then describe the learning process enabled by MARL of single- and multi-turn LLM setting (Secs. 3.2. and 3.2.2

---

3.1 Deploying Meta-Thinking Reasoning Process for LLMs
Beyond VRP (Sec. 2.1 recent studies [Muennighoff' et al. 2025 Ye et al. 2025c have shown that integrating meta- seeta-thinking behaviors in reasoning process can largely improve the accuracy of the fi- nal answers. By integrating Meta-thinking, ReMA decomposes problem solving into two sequential phases: a meta-thinking phase that plans, monitors, or revises strategy, followed by reasoning phase that produces the detailed solution. We analyse Meta-thinking Reasoning Process along two orthogonal axes sinele- vs. multi-agent and single- vs. multi-turn.
In single-agent setting, such process calls LLM once and generates meta-thinking and the follow- ing reasoning autoregressively. We formulate the meta-thinking reasoning process (MRP) below:
Ty x, m) T (m x),
where and y are the output of meta- meta-thinking and reasoning respectively. We present the procedure as shown below:
X
meta meta-thinking
reasoning steps
a.
(4)
Exploring MRP reasoning through a single-agent approach is often inefficient, as it requires the lan- guage model to simultancously master both meta-thinkiin and detailed problem-solving within one call. Prior research has demonstrated that activating different model capabilities through specialized agents significantly improves MRP exploration efficiency. To leverage this insight, we decouple meta-thinking and reasoning into two separate LLM agents: a high-level agent dedicated to gener- ating meta-thinking, and low-level agent focused on executing reasoning steps.
During a conversation, the high-level and low-level agents (i.e., Ty and my) act in an interleaving manner. The high-level agent generates and summarizes meta-thoughts from the prompt and inter- action history, while the low-level agent executes detailed problem-solving under those instructions. We formulate the multi-agent meta-thinking reasoning process (MAMRP) as follows:
y m x, m) (m x).
(5)
While the single-turn MAMRP offers a straightforward approach, it lacks the ability perform immediate and fine-graine cognitive switching during the reasoning process, which limits its effec- tiveness on complex, long-horiza planning tasks. Therefore, we extend Eq. (5) and formulate the multi-turn MAMRP follows:
T
Yr II (y: x, (m,y)<e m.) my (m, x, (m,y)<u
(6)
where T is the number of turns. Similarly, present the process with directed graph:
meta-dhinking
reasoning
meta-thinking
reasoning
meta-thinking
reasoning
Y1
m2
Y2
yT
Wh
Wh
As a complex reasoning system, MAMRP provides various optimization opportunities in scaling inference- time computation. We leave further discussion of these aspects in Appendix C.I
3.2 Training MAMRP: A Multi-Agent RL Method
Multi-agent RL, unlike single-agent RL in deterministic MDP, must contend with stochastic, non- stationary dynamics and rewards, making optimization more challenging. We start by considering easier case, the optimization of single-turn MAMRP.
3.2.1 Optimizing Single-turn MAMRP
To train the system from Sec. 3e. we embed it as a Markov Game between the two agents. Suppose the two LLM agents parameterized by 02 and 6,, respectively. Define a joint hierarchical policy over sequential decisions m and y:
y ()) x) == ( x, m) Te, (m x))
(8)
Let R(y.y*) denote the final reward serves as the objective function [[th, 01) for the joint policy:
I [O 01) = Ex.x- R By o)) R(y,

---

RL for VRP & MRP
ReMA with Separate Parameters
ReMA with Shared Parameters
Quertina
Questien
Querdoa
Purveaaio High-leve agent
Trileatle
Trainalle
Lonskerel agrnt Not (8)
Single Reasaning Agent
High-lenel agrnd
Ciilg level ageat
MAMIRP Bolleer
Mera Thinking
Meta Thinking
Meta Thinking
chinklbi
Reasoning
Trainatio
Srain agont
mL
Yi
IB
y2
Solution
Rensoning
Reasening
Ratio clip & ago Ene v, E yul Zayet
Faick vk
Selution
Peelbork
Solution
mistry diplrul 1 saAu
Figure 2: Comparison of training pipelines. Left: agent updated either with mixed (VRP) or explicit (MRP) meta-thinking. Middle: ReMA with separate parameters for the high-level (meta-thinking) and low-level (reasoning) agents; training alternates between freezing one agent and updating the other. Right: ReMA with shared parameters and multi-turn interactions: both agents share the same parameters and are distinguished by their system prompts. Training employs a turn-level ratio for stable multi-turn reinforcement learning and efficient updates, ensuring each turn's 's contribution is controlled to prevent instability.
RL training of VRP and MRP, where single LM
During optimization procedure, the high-level policy Te, and low-level policy ne, aim to maximize their respective rewards independently. The optimization goals for agents are:
0% = arg max L (x.y*) >n mnmoo yw [Ra (m, y,), 0,
(10)
(0%) arg max LE (x.y*) ) monto y s (m, y, y.y"
(11)
0:
where R2 and R2 are policies' individual reward functions, including It and regularization according to tasks and models, e.g., different format rewards (refer to Appendix C22 for details). The detailed algorithm is the Algorithm [
We illustrate the MAMRP inference procedure and the proposed training method in Fig. 2 We also provide an analysis of different loss functions in Appendix C.5
3.2.2 Scaling up Multi-turn MAMRP
To scale up to multi-turn MAMRP, we can still adopt the iterative training strategy in Sec. 3.2.1 However, we make some changes to improve the efficiency of rollout and training.
First, we implement parameter-sharing strategy where both high-level and low-level agents utilize identical model weights 0, distinguished only by role-specific system prompts S, and S.. Formally, we define ne ( Sz. -) and m, TT(- -). sharing the same underlying parameters rather than maintaining separate model instances. This approach eliminates the need for frequent model swapping on GPU during rollout, avoiding inefficient wait times, while enabling larger batch sizes during training to simultancously optimize policies for both meta-thinking and reasoning roles.
Second, we propose a multi-turn GRPO with turn-level ratio to address the challenges of multi- turn MAMRP. The trajectory- averaged objective with turn-level ratio of is defined as (The objective of T. is the similar but with different system prompt):
J(0)= = (xxx DVD, ((m))))) (-1x)
T
[yael
Σ
m0)n(r,,(0)AA:s. (lip(r..(0), - -c,1,,,))))...... - -3DKI (Fo)(
1=
(12)
5

---

where ys.e,j is the j-th token at turn & of the reasoning agent of the i-th trajectory. And the turn-level ratio for clipping is defined as:
Yo.el
(0) exp
log (i
(13)
Viiel
1
yr,el
To (y
x, {m, Ni, Vi.Jee (i,,5,
exp
log
(14)
Yi.e
Moaia (iin x, {mi.. Vi, Jes mi, Vi,eki)
The introduction of the turn-level ratio serves two key purposes. First, using a token-level ratio (Eq. (2) in the objective introduces bias for multi-turn training, as averages over tokens trajectory. This means that tokens within longer turns (those containing more tokens) can dispropor- tionately influence the overall loss, and averaging at the token level may encourage excessively long single-turn responses. Second, clipping each token independently risks instability during training.
In contrast, the turn-level ratio aligns more closely with the underlying MDP formulation by treating all tokens within a turn as single action and applying clipping at the turn level. Intuitively, this approach stabilizes training by preventing the LLM from making unstable updates that could result in extreme outputs, such as overly long repetitions or incoherent text. We conduct experimental verification in subsequent empirical results (Sec. 4.3)
Experiments
To evaluate the effectiveness and efficiency of ReMA, we conduct experiments on challenging benchmarks for two types of tasks: mathematical reasoning and LLM-as-a-Judee with three dif- ferent LLMs. Then, we investigate the models' performance in both single- multi-turn settings. Finally, we provide ablation studies and qualitative analyses of our method.
4.1 Experiment Settings
We first analyze the single-turn case of ReMA, i.e., = 1. The high-level agent generates a com- plete meta-thinking trace in one shot, and the low-level agent follows the instructions and outputs the final results. Single-turn ReMA reduces stochasticity and training cost while our experiments show that still provides meaningful performance gains.
Benchmarks We conduct experiments on two types of tasks: mathematical reasoning and LLM- as-a-Judge. For mathematical reasoning experiments, we train models 7.5k training samples in
MATH (Hendrycks et al., 2021] and use MATH500 [ightman et al. 2023 as the in-distribution test
dataset. Additionally,
test the optimized models out-of-distrrbution datasets: GSM8K (Cobbe
et al.
2021 A1ME24 AMC2:1 GaoKao2023En [Zhang et al. 2023 Minerva Math Lewkowycz
al.
2022 and Olympiad Bench (He al. 2024
For LLM-as-a-Judge benchmarks, we train models on RewardBench [ambert et al. 2024]. Specifi- cally, we convert the original data into pair-ranking format and split it into training set of 5k items and test set of 970 items, denoted as RewardBench9770. The models are also tested on JudgeBench
Tan et al. 2024] to assess distrribitii performance. We refer to Appendix D.1.2 for detailed comparisons and results.
Baselines, Models, Training Settings We compare pass@1 performance across the following
methods: (1) VRP (CoT, step-by-step prompting, Sec. 3.I): (2) VRPRL (RL under VRP); (3)
MRPRL. RL. (RL under MRP with high-level task analysis. Eq. (4)). and (4) ReMA (ours, RL under
MAMRP, Eq. (
We train and test 1amma-3-8B-Instruuee 7B-Instruct [Team] 2024 on mathematical reasoning benchmarks. For M-aa--a-judgee bench- marks, train and test 1--ma-.1--BB-Inttaee and -wenn.5-7B-Insstteeee We instruct-tuned
(Dubey al. 2024]. and Qwen2.5-
'ntrps://hugginngfac.coo -ttp://g..n.attaoo//aaaaaataa/// http://iiiigacc.c//aaataaaaaeeee wtt.////oatt////eeeeeo
aaation-ame aatta//aaaaaaaaaaaaaaaaaaaaeeeea

---

MATH500 (Medium)
GSM8K (Easy)
AIME2024 (Hard)
0.92
0.200
Testing Accurac)
0.60
0.175
0.159
0.74
0.36
0.025
50
100
150
200
100
150
200
250
50
100
150
200
250
Training Steps
Training Steps
Training Steps
Meta-Thinking
Figure 3: An RL experiment with 3 training schemes. While RL from SFT excels on easier prob- lems, RL under Meta-thinking shows superior generalization harder problems like AIME24.
LLMs to prompt them perform VRP, MRP, and MAMRP directly during training. Unless speci- fied, we use two separate copies of the same model for high- and low-level agents in ReMA. We use the base reward setting Appendix C.2 by default. And for the underlying RL algorithm, we use REINFORCE+ Hu 2025
We refer to Appendix for detailed training settings.
4.2 Results of Single-turn ReMA
Question Does single-turn ReMA outperforms baselines on both in-distribution and out-of-
distribution test sets?
Table D compares the greedy decoding performance of ReMA against various RL baselines across mathematical benchmarks (Table Tab and LLM-as-a-Judge benchmarks (Table lb.. Results across different LLMs indicate that, on average, ReMA outperforms all baselines. achieving a maximum improvement of 6.68% mathematical benchmarks and 8.49% on LLM-as-a-Judge benchmarks.
Notably, ReMA achieves the highest performance most benchmarks, particularly out-of- distribution datasets, with a maximum improvement of 20% on AMC23 for Llama3-8BBInstruuct 13.33% AIME24 for Qwwen2.5-7B-Instrueet, 14.23% RewardBench970 for Llama3. 1-8B- Instruct. These results demonstrate the superior out-of-distribttion generalization ability conferred by the meta-thinking mechanism in ReMA. However, we observe that the accuracy gains from RI. training on instruction-turge LMs are smaller than from base models (Sec. [.2.1]) This may be due to the higher initial performance and the relatively fixed output distribution of instruction-tuned models, which limits the improvement and peak performance in RL.
We also compare ReMA with other inference-time meta-thinking and multi-agent baselines, e.g. Self-Refine Madaan et al.] (203) and Multi-Agent Debate (MAD)(Du et al.] 2023], we refer to Appendix E. for detailed results.
4.2.1 Meta-thoughts boost low-level generalization
Question 2. Can Reasoning benefit from Meta-thinking?
Here provide tiny but motivating example of how ReMA gives better learning dynamics. We use Qwen2.5-Math-7B (Yang et al. 2024] as the starting base model, MATH (level 3-5, about 5.5K number of instances) as the training dataset, and we compare three reinforcement learning training schemes, in particular:
(1) RL from Base: train the base model directly on MATH with binary outcome reward; (2) RL from SFT: SFT the base model with GPT-4o's CoT answers; then RL on train dataset with binary outcome reward; (3) RL under Meta-thinking: SFT the base model with GPT-4o's 's meta- weeta-thinking plans; then RL train dataset with binary outcome reward.
The models are evaluated on benchmarks (Fig- 3h. SFT brings the best initial accuracy on distribution and easier sets, but fails to improve harder ones. RI from Base yields limited gains. In contrast, RT. under Meta-thinking achieves the best learning dynamics and generalizes better to challenging problems (AIME24). See Appendix F.I for case studies.
7

---

Table I: Performance in-distribution test sets and out-of-distribution test sets. We also report the improvemenn/deeraaattion w.r.. basic CoT performance( (VRP). On average, ReMA outperforms all baselines. Particularly out-of-distribution datasets, ReMA achieves the highest performance on most of the benchmarks.
(a) Performance on math benchmarks
VRP
ReMA
Model
Benchmark
VRPAL
MRPRL
(CoT)
(Ours)
MATH500
30,80
33.40 (+2.60)
32.80 (+2.00)
33,80 (+3.00)
GSM8K
67.48
81.80 (+14.32)
79.68 (+12.20)
79.38 (+11.90)
AIME24
0.00
0.00 (+0.00)
3.33 (+3.33)
0.00 (+0.00)
Llama3
AMC23
2.50
10.00 (+7.50)
12.50 (+10.00)
22.50 (+20.00)
-SB
Gaokao2023en
22.34
27.53 (45.19)
23,38 (+1.04)
28.57 (+6.23)
IInttucct
Minerva Math
8.82
16.54 (+7.72)
18.01 (+9.19)
13.97 (+5.15)
Olympiad Bench
8.44
8.89 (+0.45)
9.33 (+0.89)
8.89 (+0.45)
Average
20,05
25.45 (+5.40)
25,58 (+5.53)
26.73 (+6.68)
MATH500
50.80
50.20
48.60 (-2.20)
53.20 (+2.40)
GSM8K
86.05
84.53 (-1.52)
85.37 (-0.68)
87.26 (+1.21)
AIME24
10.00
3.33 (-6.67)
6.67 (-3.33)
13.33 (+3.33)
Llama3.1 -8B
AMC23
27.50
12.50 (-15.00)
30,00 (+2.50)
20.00 (-7.50)
Gaokao2023en
38.96
36.10 (-2.86)
37.14 (-1.82)
37.14 (-1.82)
IInstruct
Minerva Math
22.79
26.84 (+4.05)
25.37 (+2.58)
28.31 (+5.52)
Olympiad Bench
15.11
19.70 (+4.59)
15.70 (+0.59)
19.56 (44.45)
Average
35.89
33.32 (-2.57)
35.55 (-0.34)
36.97 (+1.08)
MATH500
75.00
77.20 (+2.20)
76.40 (+1.40)
74.40 (-0.60)
GSM8K
92.04
91.36 (-0.68)
91.81 (-0.23)
90.60 (-1.44)
AIME24
6.67
6.67 (+0.00)
10.00 (+3.33)
20.00 (+13.33)
Qwen2.5 -7B -Instruct
AMC23
47.50
50.00 (42.50)
52.50 (+5.00)
57.50 (+10.00)
Gaokao2023en
56.62
54.81 (-1.81)
55.06 (-1.56)
57.92 (+1.30)
Minerva Math
35.66
34.93 (-0.73)
32.35 (-3.31)
34.93 (-0.73)
Olympiad Bench
38.22
38.37 (+0.15)
37.78 (-0.44)
36.30 (-1.92)
Average
50.24
50.48 (+0.24)
50.84 (+0.60)
53.09 (+2.85)
(b) Performance LLM--asa-Judge benchmarks
VRP
ReMA
Model
Benchmark
VRPRI.
MRPRL. PR.
(CoT)
(Ours)
Llama3.1
RewardBench970
69.48
82.89
(+13.41)
81.13 (+11.65)
83.71 (+14.23)
-8B -Instruct
JudgeBench
51.29
51.94 (+0.65)
52.90 (+1.61)
52.90 (+1.61)
Average
60.39
67.41 (+7.02)
67.02 (+6.63)
68.31 (47.92)
Qwen2.5
RewardBench970
78.56
85,36 (+6.80)
86.49 (+7.93)
83,51 (+4.95)
-7B Instruct
JudgeBench
58.39
56.94 (-1.45)
58.39 (+0.00)
56,94 (-1.45)
Average
68.47
71.15 (+2.68)
72.44 (+3.97)
70.22 (+1.75)
4.2.2 Diverse thia-thg characteristics of LLMSs
Question 3. How well L.LMs learn meta-think?
To further analyze meta- veeta-thinking behaviors, we train models with structured JSON-format actions inspired by Yue et al. [024]. The meta-thinking agent generate two entries in one LM call, first selects from three actions: DECOMPOSE (breaking into subproblems), REWRITE (simplifying the problem), EMPTY (direct solving), then generates the corresponding text. We compare Llama- 3.1-8B-Instruc. and Llama-3.2- 1-t.a...------stteeeee I-Istrutt to study scale effects (two 1B models vs two 8B models) on meta-thinking agent's training. We use vLLM guided JSON decoding (Dong al. 2024 for valid formatting and base reward (reasoning agent's 's solution accuracy with format constraints).
We observe that smaller LMs produce simpler outputs, likely due to limited capacity to maintain valid ISON formatting while exploring diverse reasoning strategies. As Fig. 4 shows, smaller LMs like 3lama-3.2-1 3.2--B-Innn Instrrcct quickly converge to the simplest EMPTY action avoid format-

---

ting penalties, while larger LMs like 1aama-31-8B-Instruu can adapt meta- veeta-thinking strategies for detailed case studies.
based on problem difficulty. See Appendix F3
Average Problem Difficutyy by Action Type Over Steps (h111)
Average Problem Difiiclty by Action Type Over Steps (hai8)
Average Problem Dificcltty
DECOMPOSE
Average Problem Difficultt
2.3
REANTE EMPTY
4.5
2.0
DECOMPOSE
2.5
HEWRITE EMPTY
2.0
2.5
20
55
50
200
20
93
60
300
122
Training Steps
Training Stops
Figure 4: Average problem difficulty by action type during training. Left: 1B LM collapses to the
EMPTY action. Right: 8B LM adapts to more complex meta-thinking strategy for harder problems.
4.3 Extending ReMA to Multi-turn MAMRP
We further extend ReMA to multi-turn MAMRP settings, enabling multiple rounds of interaction between the meta-thinking agent and the reasoning agent as defined in Eq. 7
Unlike the inherent VRP capabilities of most L.Mss, multi-turn ReMA requires initial bootstrapping. Thus, we constructed a supervised fine-tuning dataset (about 0.8K samples) from LIMO (Ye et al.) 2025c] using GPT-4o to establish the starting point for multi-turn interaction capabilities. Then finetune Qwen2.5-7B before RL training.
As described in Sec 3.2.2 we deploy the proposed GRPO with turn-level ratio clipping and trajectory-lever averaging loss during training. And we remove the KL-divergence term to allow more flexible exploration. By default, the agents share the same parameters and are simultaneously updated using their trajectories. We refer to details in Appendix D.2
4.3.1 Results and Ablations
Question 4. Can ReMA be scaled to multi-turn settings?
There are two key points revealed by our multi-turn RcMA experiments, as shown in Fig. 5 On one hand, the algorithm can demonstrate effective convergence the training set, with accuracy steadily increasing from approximately 55% to 70% during training. It also achieves an average per- formance gain of about 5% across all seven test benchmarks, indicating stable improvements out- of-distribution data. (Experiment with the rollout config of turn30 token512, see Appendix D.2.2 and Fig. for more details.)
On the other hand, we observe that the performance of multi-turn ReMA is highly sensitive to hyper- parameters such the maximum response length per turn and the maximum number of turns. For certain configurations, the model either collapses into producing massive repetitions within a single turn or generates empty responses after only a few turns. Similar phenomena have been reported in concurrent works such as RAGEN [Wang et al. 2025) and nondeterminism between inference and training engines in current RL infra [He and Lab 2025 and improper token processing that deviates from the As result, multi-turn RL becomes susceptible to long-horizon credit assignment challenges and state drift, often leading to reduced exploration diversity-a phenomenon Trap"" To address this challenge, it is essential comprehensively explore the training recipe w.r.t. model, data, algorithm and, most importantly, the RL infra.
principle (e.g., performing risky Excode--ncode operations on tokens).
phenommmnoo referred to as the "Echo
Question 5. How does parameter sharing and turn-level ratio affect multi-turn ReMA?
As shown in Fig. 6 compare different configurations on a smaller dataset consisting of 133 samples-1 19 from each of the MATH problem types to evaluate sample efficiency and conver- gence speed. We run each experiment with 3 different random seeds and report the average perfor- mance. First, all configurations eventually achieve nearly 100% accuracy on the training dataset. Notably, the trajectory-level loss with turn-level ratio (Turn-Ratio, Eq. (14) demonstrates sub- stantially better sample efficiency than its token-level variants (Eq. 2 reaching higher training rewards with fewer steps. We also present the training curve of separate weight setting, the empirical results show that shared parameters with simultaneous updates converge noticeably faster.

---

Main Results MATH-LV3-5
Ablations trained on Subsampled MATH Data
0.70
0.9
0.9
0.85
Reward
Train Rewarg
0.55
0.7
0.50
per
0.6
0.5
0.40
Teat
Tam-Rate Parama
Soparate Pammr
0.35
200
300
400
500
200
100
200
300
Training Steps
Training Steps
Training Steps
Figure 5: Training results of multi-turn Figure Ablations of multi-turn ReMA on a tiny sub- ReMA on MATI--LeveI3---KK under differ- set of MATH, we only show here the training curves ent rollout configurations.
of different training and rollout configurations.
5 Conclusion
In this paper, we introduced ReMA, a novel framework that leverages multi-agent reinforcement learning elicit meta-thinking large language models. By explicitly separating meta-thinking and reasoning processes into distinct agents, our approach enhances both exploration during train- ing and the interpretability of model outputs. We tailored RL algorithms and reward functions to ensure reliable performance. Through comprehensive experiments on mathematical reasoning and LLM-as--aJudge benchmarks. ReMA consistently achieved superior results. particularly on out-of- distribution datasets. We further extend ReMA to multi-tur turti-ttummm settings, enabling the framework to handle more complex reasoning scenarios that require more communication between agents. Our ablations demonstrate how effective coordination between agents evolves, highlighting the promise of reinforcement learning and structured agents' collaboration for advancing the capabilities of lan- guage models in complex reasoning tasks.
6 Acknowledgment
This work was supported by the Shanghai Municipal Science and Technology Major Project. The Shanghai Jiao Tong University team is supported by National Key R&D Program of China 2024YFC3505402. Doubao Large Model Fund. The Shanghai Jiao Tong University team is also partially supported by Shanghai Municipal Science and Technology Major Project (202ISHZDZX0102) and National Natural Science Foundation of China (62322603). This work was partially supported by the Canada CIFAR AI Chair Program, the Natural Sciences and Engi- neering Research Council of Canada (NSERC) Discovery Grants RGPIN-2022-03669 and enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).
10

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv: 08774, 2023.
Elif Akata, Lion Schulz, Julian Coda-Forno, Scong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXiv:2305. 2023,
Cem Anil, Guodong Zhang, Yuhuai Wu, and Roger B. Grosse. Learning to give checkable answers with prover-verifiet games. CoRR, abs/2108. 12099, 2021. URL/ttpss//arxiv.com
21 08. 12099
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -aan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1 2020.
Jiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: Self-organizing agents in open-ended environments. arXiv preprint arXiv:2402. 04578, 2024a.
Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. Unlocking the capabil- ities of thought: A reasoming boundary framework to quantify and optimize chain-of- chain-of-thought. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL
https:: hptp::///ppeeeiiieeeee net forum?
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. Routerde: Query- based router by dual contrastive learning for assembling large language models. arXiv preprint arXiv:2409. 1109/99880 19886, 2024c.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representa- tions, 2023.
Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. Au- totamp: Autoregressive task and motion planning with 1lms as translators and checkers. In 2024 IEEE International conference robotics and automation (ICRA), pages 6695-6702. IEEE, 2024d.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240); 1-113, 2023.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1110.1411.. 14168, 2021.
Google DeepMind. Gemini flash thinking. 2025. technol ogi / www.///minniiim fassh th i nk ing/
URL hitps: httpts::/ddeemiiid.ggmm google Accessed: 2025-01-29.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao,
11

---

Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Juniong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025. https://:
org/ 2501 12948
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabillitess of Ilms: An exploration in mathematical problem solving. arXiv preprint arXiv:2405. 2024.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14611 2024.
Kefan Dong and Tengyu Ma. Stp: Self-play IIm theorem provers with iterative conjecturing and proving, 2025. URL hhttts:://aartti/ org/ abs /2502. 00212
Yixin Dong, Charlie F Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen. Xgrammar: Flexible and efficient structured generation engine for large language models. arXiv preprint arXiv:2411. 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Andrew Estornell, -ran-Francois Ton, Yuanshun Yao, and Yang Liu. Acc-debate: An actor-critic approach multi-agent debate, 2024. URL https::/ /tttpp:://axxi.../ wttp://///aa//.....///0
John H Flavell. Metacognition and cognitive monitoring: A new area of
inquiry. American psychologist, 34(10):906, 1979.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cogni- tive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. 2025.
URL
01307
Peizhong Gao, Ao Xie, Shaoguang Mao, Wenshan Wu, Yan Xia, Haipeng Mi, and Furu Wei. Meta reasoning for large language models. arXiv preprint arXiv:2406. 1698, 2024.
12

---

Alex Graves. Sequence transduction with
recurrent neural networks.
arXiv preprint
arXiv:1211.3711, 2012.
Fatemeh Haji, Mazal Bethany. Maryam Tabar, Jason Chiang, Anthony Rios, and Peyman Najafirad. Improving I1m reasoning with multi-agent tree-of-thought validator agent, 2024. URL https:
11527
Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. Al Open, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm infer- ence. Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.222500
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv 00384774 2021.
Sirui Hong, Xiawu Zheng. Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308.0033 00352, 3(4):6, 2023.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11145; 2024a.
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403. 12031, 2024b.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pseudo feedback. arXiv preprint 1oxXi:24111163444 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training lims to reason and leverage search engines with reinforcement learning, 2025.
Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of I1m outputs. arXiv preprint arXiv:2407.13692 2024.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. In The Twelfth International Conference on Learning Representations.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. /29/7, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, IJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403: 13787, 2024.
13

---

Pat Langley, Kirstin Cummings, and Daniel Shapiro. Hierarchical skills and cognitive architectures. In Proceedings of the anmual meeting of the cognitive science society, volume 26, 2004.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Ming Li. Jiuhai Chen, Lichang Chen, and Tianyi Zhou. Can Ilms speak for diverse peo- plc? tuning Ilms via debate to generate controllable controversial statements. arXiv preprint arXiv:2402. 10614 2024.
Tian Liang Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking large language models through multi- agent debate. arXiv preprint arXiv:2305. 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2008. 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. CoRR, 20782503.20783, 2025. doi: 10.48550ARRXV..503...2. 20783. URL httpt::///oooiiii. http://66...gg111..44455//..x...
20783
Chengdong Ma, Ziran Yang. Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, and Yaodong Yang. Red teaming game: A game-theoretic framework for red teaming language models. arXiv preprint arXiv:2310.00322. 10.00222, 2023.
Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, and Min Chen. Co- evolving with the other you: Fine- tine-tuning LLM with sequential cooperative multi-agent rein- forcement learning. CoRR, abs/2410.06101, 2024. doi: 10-8550/ARXV.241006000.. URL https://dof htt.//oo..oggg.... www.rrg........... 48550/ arxiv. 24110.01000
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedhack. Advances Neural Information Processing Systems, 36:46534 46594, 2023.
Dakota Mahan, Duy Van Phung, Rafacl Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. /2832, 2024.
Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Markian Rybchuk, Philip H. S. Torr, Ivan Laptev, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Wit.. Malt: Improv- ing reasoning with multi-agent Ilm training, 2024. TRLLhttps:/// http:://wwwxiv.oo/ab//////a 01928
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
OpenAl. Openai ol system card, 2024. URL httts:://opeaaa..oom httppppnai.cooooo////
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances neural information processing systems, 35: 2022.
14

---

Chanwoo Park, Seungju Han, Xingzhi Guo, Asuman Ozdaglar, Kaiqing Zhang, and Joo- Moo-Kyung& Kim. Maporl: Multi-agent post-co-training for collaborative large language models with reinforcement learning. 2025. http::/tt::////t /ttp//aavvvvv.v www.vvo.rrg/bb///...// 18439.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.032880 2022.
Israel Puerta-Merino, Carlos Núñiz-Molina, Pablo Mesejo, and Juan Fernández-Olivares. A roadmap to guide the integration of Ilms in hierarchical planning. arXiv preprint
arXiv:2501.0806 2025.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reason- ing makes smaller Iims stronger problem-solvers. arXiv preprint arXiv:2408.06199, 2024.
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 01 replication journey; A strategic progress report part 1, 2024. URL http::tts:///et http::://aaxiv.ooo org/ htt.b////411.19999
Lv Qingsong, Yangning Li, Zihua Lan, Zishan Xu, Jiwei Tang, Yinghui Li, Wenhao Jiang, Hai-Tao Zheng, and Philip S. Yu. Raise: Reinforenced adaptive instruction selection for large language models, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, (--53728-537411 2023.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task plan- ning. arXiv preprint wrXiv:2307.06111 2023.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to arXiv preprint arXi::2501.18099,
plan & reason for evaluation with 2025.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, vol- ume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org, 2015. www.nngs....nnn.mme
URL
scchu man15. 1mman15.htmm hhtml
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs/1707.063444 2017. URL
1707.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0338 03300, 2024.
Maohao Shen, Guangtao Zeng. Zhenting Qi, Zhang-We Hong. Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain- tf-action-thought enhances I1m reasoning via autoregressive search, 2025. URL https:// arxiv.org/ abs /20.2/2522..m 02508
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033 2024.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEFF/CVF international conference on computer vision, pages 2998-3009, 2023.
15

---

Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao. Salman Avestimehr, and Chaoyang He. Tensoropera router: A multi-model router for effi cient Ilm inference. arXiv preprint arXiv:2408. 12320, 2024.
Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains, 2025. URL
https: fitps://arxiv: www. abs 2501 05707
Chuanneng Sun, Songjun Huang, and Dario Pompili. Retrieval-augmented hierarchical in-context reinforcement learning and hindsight modular reflections for task planning with llms. arXiv preprint arXiv:2408.06525 2024.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog). 13(1):38, 2019.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating Ilm-based judges. arXiv preprint arXiv:2410.12784 2024.
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint urXiv:2311.10553 2023.
Qwen Team. Qwen2.5; A party of foundation models, September 2024. WtLhttts:://// //wen.m.. github. gwwpo/////eee..... 57|
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671 2024a.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Faral-Zarandi,, Jason Weston, and Xian Li. Self-taught eval- uators. arXiv preprint MrXiv:2408.02660 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint arxiv:2203. 2022.
Yuqing Wang and Yun Zhao. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05344 0533400 2023.
Zhenhailong Wang, Shaoguang Mao. Wenshan Wu. Tao Gc, Furu Wei, and Heng Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300) 2023.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Il agents via multi-turn reinforcement learning, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-248311 2022.
Sean Welleck, Ximing Lu, Peter West, Facze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning self-correct. arXiv preprint arXiv:2211.00053 2022.
Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, and Ying Wen. Reinforcing language agents via policy optimization with action decomposition. CoRR, abs/2405. 2024. doi: 10.48550/ 200.55000xxiv..45.... 15821 2482.
ARXIV.2405.1588 15821. http:///a...aaaa//.ooo../0/////0
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing I1m reasoning via critique models with test-time and training-time supervision. arXiv preprint 2rXiv:2411.165777 2024.
16

---

Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, al. Towards system reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04000 2025.
Yihang Xiao, Jinyi Liu, Yan Zheng, Xiaohan Xie, Jianye Hao, Mingzhi Li, Ruitao Wang, Fei Ni, Yuxiao Li, Jintian Luo, et al. Cellagent: An l1m-driven multi-agent framework for automated single-cell data analysis. BioRxiv, pages 2024-05, 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing I1m reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
Fengli Xu, Qianyue Hao, Zefang Zong. Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprini arXiv:2501.09688 2025.
Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.03617 2024.
Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, and Jun Wang. Ask more, know better: Reinforce-learned prompt questions for decision making with large language models. arXiv preprint (1XXx:23310.81177 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical ex- pert model via self iaff-mppooveeet.. arXiv preprint arXiv:2409.1220 12121 2024.
Ling Yang. Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical I1m reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772. 2025.
Guanghao Ye, Khiem Due Pham, Xinzhi Zhang. Sivakanth Gopi, Baolin Peng. Beibin Li. Janardhan Kulkarni, and Huseyin A Inan. On the emergence of thinking in 1lms Searching for the right intuition. arXiv preprint 2x17:2500.06673, 2025a.
Peijun Ye, Tao Wang, and Yue Wang. A survey of cognitive architectures in the past 20 years. IEEE transactions on cybernetics, 48(12):3280-37)) 3290, 2018
Yaowen Ye, Cassidy Laidlaw, and Jacob Steinhardt. Iterative label refinement matters more than preference optimization under weak supervision. arXiv preprint arXiv:2501.0755 2025b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, arXiv preprint 07X.v.2502.03887 2025c.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Ab- sorbing abilities from homologous models as free lunch. In Forty-first International Conference on Machine Learning, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin. Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025.
Murong Yue, Wenlin Yao, Haitao Mi, Dian Yu, Ziyu Yao, and Dong Yu. Dots: Learning to reason dynamically in lims via optimal reasoning trajectories search. arXiv preprint 11X0v:2410.03864, 2024.
Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang. Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route lms for multi-agent systems, arXiv preprint arXiv:2502. / 113, 2025a.
17

---

Yu Yue, Yufeng Yuan, Qiying Yu. Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025b.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yugiang Li, Wanli Ouyang, and Dongzhan Zhou. Llama-berry: Pair- wise optimization for ol-like olympiad-level mathematical reasoning, 2024a. URL https:
wlaxxivoorg/aat abs///b////// 24100 02884
Hangfan Zhang, Zhiyao Cui, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, and Shuyue Hu. If multi-agent debate is the answer, what is the question? arXiv preprint 25K::25002.8788 2025a.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. arXiv preprint (rXiv:2410.117666 2024b.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models gaokao benchmark, arXiv preprint arXiv:2305. /24744
2023.
Yiqun Zhang, Peng Ye, Xiaocui Yang. Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue Hu. Nature-inspired population-based evolution of large language models. arXiv preprint arXiv:2503.01 155, 2025b.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: post-training amplifies behaviors learned pretraining, 2025.
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang. Weihua Luo, and Kaifu Zhang. Marco-ol: Towards open reasoning models for open-ended solutions, 2024. TRRLttpss:
wttee/i..rr//bb//2a1/00 14405
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024.
Mingehen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-bars societies of mind. arXiv preprint arXiv:2305. Nnn:23355.7066 17066, 2023.
18

---

Appendix Table of Contents
A Related work
1
A.
Single LLM Reasoning
9
2
Multiple LLM Reasoning
20
Hierarchical Reasoning
20
RL in LLM
Limitation and Future Work
Supplementary Materials for Method in Section
Inference-time Scaling For ReMA.
Detailed reward design
Pseudocode of ReMA
Brief convergence analysis
Learning to reason from the perspective of Leader Follower Game
D Training Details
26
D.1
Single-turn ReMA
26
D.1.1
Supervised fine-tuning data collection
27
D.I.
Dataset Curation of RewardBench970
27
D. 1.1
Training MATH
28
*
D.1.4
Training on Reward Bench
28
D.2
Multi-turn ReMA
D.2.1
SFT data collection of multi-turn MAMRP
D.2.2
Training MATH
Other Experiments
29
Experiments of comparing to more baselines
Reward functions shape cross- agent behaviors
29
Detailed Training Curves Different Datasets of Multi-turn ReMA
30
Qualitative results
High-level policy finds better plans
Case study for Experiments of Different Reward Functions in Appendix
E.2
Case study for Adaptive Meta- Meta-thinkinn in Single-Tune Turle ReMA in Section 4.2.2
Prompts
A
Related work
Drawing from the bitter lesson [Sutton 2019).
methods that appear scale effectively
searching and learning, aligning with current trends in large language models [Xu et al. (025). At present, researchers are leveraging these methods to maximize the capabilities of individual trans- formers, while other efforts are exploring architectures that involve multiple interacting entities. In this paper, we examine this divergence within context LLM reasoning, capability that lows large language models to solve problems through logical reasoning, step-by-step analysis, and inference (Wang al. 2024a].
A.1 Single LLM Reasoning
Main research works in reasoning involving single LLM utilize search-basee based and post-training methods. The fundamental elements of searching methods are text generation and evaluation. Gen- eration schemes include In-Context Learning [Brown et al.) [2020] Beam Search [Graves] 2012], and various tree-based searching [Snell al., 2024]; Evaluation approaches often use outcome ac- curacy, self-consistency [Wang | 2022]. process reward signal [Lightman et al. [023 as the criteria to select high-quality responses from the generated texts. Post-training method another research line in opposition to pre-training. Popular training pipelines often involve specific data
19

---

construction followed by Supervised Fine-tuning [Qin et al. 2024 Ouyang et al., 2022 Hui et al.)
2024
Liu et al. 2024]. or reinforcement learning to interactively explore learning patterns [aan
et al.
2024a hang et al. 2024a DeepSeek-AI et al. 2025 Xu et al. 2025
A.2 Multiple LLM Reasoning
Integrating multiple entities can potentially surpass the intelligence of the individual model [Chen jt al., 2023]. With the rapid emergence of large language models showing a varying level of abili-
ties, some studies have explored facilitating discussions among multiple off-the-shelf LLMs (Zhang
ct al. 2025a CCenn et al. |024a| Wang et al. |023] Du et al. 2023 Zhuge et al. 2023 Tang al.)
2023 Hao et al. 2025 Akata et al. 2023 Hong et al., 2023] Zhang al. 2024bb, taking the form
of free discussion [Du et al. 2023 Liang et al. 2023 or structured role assignments [Hong et al.
2023
Zhang et al. [024bb] Some have applied routing mechanisms to assign tasks to the most
suitable expert models (Hu et al. 2024b Stripelis et al. 2024 Ding et al. 2024| Yue et al. 2025a
Chen et al. 2024c] or merging mechanisms to develop more versatile models (adav et al. [224
Yu et al. 2024 Zhang et al.] 2025b]. Beyond aggregating static knowledge from multiple agents, multi-agent LLM training can also enhance reasoning capabilities. For example, multi-agent de-
bates can generate diverse synthetic data, which can subsequently be used for supervised fine-tuning
[Estomell et al. 2024 Li et al.; 2024 Motwani et al. 2024 Dong and Ma 2025 Perez al.) 2022
Ye et al. [025a Subramaniam et al.)
2025]. Reinforcement learning (RL) methods have also been
adopted to improve LLM reasoning in areas such alignment (Perez et al. 2022 Ma et al. 2023
and legibility (Kirchner et al. 20244. Motwani et al. (2024) utilize a three-agent system for genera- tion and fine-tune the models using Direct Preference Optimization (DPO). Reinforcement Learning with Generative Reward Models (GenRM) (Mahan et al.. 2024 Ye et al. 2025b] Jiao et al. 2024
Wang et al. 2024b] represents another common approach of multi-agent training, where the reward signal is derived from the token probabilities of another LLM, coupled with the reasoning process. While our work aligns with these efforts. it diverges by using an additional tunable LLM to pro- vide metacognitive instructions, guiding the low-level LLM during learning, rather than relying on static GenRM. The most closely related works to ours are MAPORL. [Park et al., [025] and COPRY (Ma et al.] 2024]. . MAPoRL is a multi-agent debating framework that uses multi-agent reinforce- ment learning (MARL) with a learned verifier to fine-tune each LLM agent. COPRY duplicates an LLM into two agents, training them simultaneously in the roles of pioneer and observer using www--off-----ootooooooo
RL. et al. 2025) trained with a novel beds meta-action tokens for self-re reffereflction and exploration into autoregressive search process. However, unlike our approach, which explicitly separates metacognition from plan execution, these methods do not decompose the reasoning process but instead focus on improving direct chain-of- thought generation. Furthermore, our experiments are conducted on a larger scale and include more challenging problems.
(COAT) framework that em-
A.3 Hierarchical Reasoning
Partitioning reasoning into hierarchical processes has been explored in prior research to make bio- logical sense YYe et al. 2018 Langley et al. 2004). In the context of language models, a hierarchical
structure has been used to facilitate diverse reasoning patterns, including planning (Puerta-
ett al.
2025 [Sun et al. 2024 Song et al.] [023] Rana et al., 2023, Chen et al., [024d]] Yan et al.
2023
Xiao et al.] 2024 validation (Haji al.] 2024 Xi et al. [024] and self-refinement Madaan
jet al.
2023 Kumar et al. 2024] Welleck et al. 2022] For instance, EvalPlanner (Saha et al. 2025
framework that conducts reasoning through plan generation and execution. DOTS Yuue al.
2024) extends decomposition by integrating a tree-based searching method with Analysis, Solution, and Verification layers. Marco-ol [Zhao al., 2024] focuses on open-ended problem-solving and abstract thinking, dynamically adjusting reasoning granularity and incorporating reflection mecha- nisms to enhance reasoning performance. Beyond these approaches, metacognition [Flavell 1979) has been identified as another critical component of reasoning, referring to the intuitive understand- ing of one's 's own cognitive and reasoning processes [Gao et al., 2024 Wang and Zhao 2023). Wang and Zhao (2023 proposed metacognitive prompting strategy to improve large language model (LLM) capabilities. [Diookkar et al. (2024) further developed a prompt-guided method that enables models to label math problems with the required skills and subsequently use these labels to solve new problems. Gao et al. (2024) introduce meta-reasoner which use contextual multi-arm bandit
to learn high-level "advisor" over low-level reasoning process. Xiang et al. (0225 provides a
20

---

Meta-CoT framework to think about its own thinking. They use construction based methods as well as reinforcement learning develop meta-cognitive skills. Qingsoon et al. [2025] introduces RL framework for dynamic instruction selection during fine-tuning- In our work, we also value reflect- ing on reasoning processes, and we enhance metacognitive abilities through two-agent interaction and reinforcement learning at both end.
A.4
RL in LLM
Recent advancements in applying RL to LLMs have enhanced their reasoning and decision-making capabilities. [iu et al. 2025 examines token-level optimization biases by introducing Dr. GRPO to stabilize policy gradients. VAPO (Yue et al. 2025b] enhances PPO with value-aware perturba- tions and adaptive reward shaping to improve robustness in sparse-reward reasoning tasks. DAPO (Yu et al., 2025 provides a scalable, modular RL framework that integrates distributed rollout col- lection and dynamic replay buffers for reproducible training at scale. SimpleRL-Zoo (Zeng al. 2025] conducts zero-shot RL experiments across open-base LLMs to uncover emergent cogmitive behaviors under minimal reward signals. Echo Chamber (Zhao et al. 2025 investigates how RL fine-tuning algorithms can amplify pretrained model biases and proposes regularization to mitigate over-amplification. [Wen et al. (2024) decomposes high-level language actions into token-level op- erations to achieve finer-grained credit assignment. Some works push RL training for single-turn multi-turn. Search-R1 (Jin al., 2025) trains LLMs orchestrate multi-turn search strategies with RL-optimized decision policies to improve question-answert
accuracy. ArCHer [Zhou et al.) 2024) employs hierarchical, multi-tume RL architecture with manager and worker policies to effi ciently handle long-horizon dialogue tasks. RAGEN (Wang et al. (2025) introduces trajectory filter- ing and critic modules within multi-turn RL framework stabilize learning and reduce shallow policy behaviors.
B Limitation and Future Work
In this work, we only test ReMA on math and JLM-as-a-Judge benchmarks. Though the results shows the effectiveness of ReMA, adopting ReMA tasks where naturally needs multi-turn teraction between several interleaved agents has great potential. Moreover, a comprehensive un- derstanding of the learning dynamics of multi-turn RL and multi-turn MARL for LLMs needed. Finally, there's still sufficient space to further improve the procedure of multi-turn multi-agent roll- out through modern LLM speed up techniques, e.g. prefill-decode disaggregation and asynchronous rollout.
C Supplementary Materials for Method in Section 3
C.1 Inference-time Scaling of ReMA
In this section, we discuss how to enhance the inference-tingg computation of our hierarchical sys- tem, specifically focusing on the interaction between the high-level and low-level agents. The total number of model samples required for inference is determined by the product of the sampling budget allocated each agent.
For instance, in a simple single-turn setting, if the high-level agent samples k responses and each of these responses leads to foz samples from the low-level agent, the total number of model calls required is:
Total samples kn x k2-
Given fixed computational budget, an important question arises: how should the sampling bud- get be distributed between the high-level and low-level agents to maximize performance? Al- locating more samples to the high-level agent may increase diversity in reasoning strategies while allocating more to the low-level agent may yield more refined solutions for given metacognitive plan.
Another crucial consideration is how perform reranking on the final outputs. Two potential strategies include:
21

---

Hierarchical reranking: First, for each high-level response, rank and aggregate the low- level responses under it. Then, rank the aggregated results across different high-level re- sponses.
Flat reranking: Directly rank all sampled responses together, regardless of the hierarchy of high-level reasoning steps.
Balancing sampling allocation and designing effective reranking strategy are key challenges in efficiently scaling our multi-agent reasoning system. the next section, we explore empirical results comparing different allocation strategies and ranking methods.
C.2 Detailed reward design
As described in Sec. 3.2 we update both high-level and low-level agents by assigning rewards based on the low-level policy output. Below, we outline several potential reward designs:
Correctness reward: For tasks with explicit ground truth, assign rewards based the correctness of the low-level agent's 's output.
Format reward: For tasks that require a specific output format, we enforce adherence to the prescribed structure by providing format reward.
To encourage the high-level agent to generate informative and unambiguous meta mneta-thnniing, and to stabilize the low-level outputs, we reward the high-level agent when the low-level agent produces consistent responses. Specifically, the consistency reward is defined as
max occurrence of answer
total number of responses
To examine multi-agent netacognition-inttgrall reasoning with different reward designs, we ex- periment with different reward function designs to encourage effective collaboration and structured reasoning. Below, we introduce and justify several reward schemes.
1. Correctness and Format-Aware Reward (Base Setting) In primary reward setting, the system's 's overall correctness is used as the primary reward signal, supplemented by format-baser rewards for both the high-level and low-level agents. Using mathematical problem-solving an example:
Low-level agent (To,): Receives reward of +1.0 for a correct answer. If the answer is incorrect, the agent is further penalized based on format compliance. Specifically:
the output contains the designated answer-indicating format (e.g., boxed La- TeX), it receives -0.5.
Otherwise, receives -1.0, as a missing format often suggests an incomplete or un- structured response.
High-level agent (Te,): Receives the average correctness of the low-level agent's 's sampled responses its reward. Additionally, to prevent the high-level agent from directly gener- ating explicit answers instead of guiding reasoning, strong penalty of -1.0 is applied if includes an explicit answer format (e.g., boxed).
Consistency- Reward Instead using correctness as the high-level reward signal, this approach rewards the high-level agent for promoting consistent responses from the low-level agent, regardless of actual correctness. The consistency reward is defined as the proportion of the most frequently occurring answer among all sampled responses:
max occurrence of an answer R
(15)
total number of responses
the majority of responses do not contain a definitive answer, the reward is set to zero. We also add the format penalty to the high-level agent if its output contains the designated answer-indicating format. This incentivizes the high-level agent to guide the low-level agent toward more stable, detailed, reproducible outputs rather than erratic reasoning paths.
22

---

Pronutt low-level LLM
Question
T&
Prompt for high-level
n
T
Our method can be viewed as combination of practical TRPO and block coordinate ascent, with the high and low-level models treated as distinct components within a larger neural network. Note that the figure does not represent the exact gradient back pocckppopagttion flow but rather highlights the key idea that we separate the high- and low-level models. This separation allows for the independent computation of gradients and the independent training of cach model.
Figure 7:
These different reward formulations allow us investigate various dimensions of metacognitive rea- soning: correctness, consistency, etc. We empirically compare their effects on learned metacognitive reasoning patterns in Sec. F.2
C.3
Pseudocode of ReMA
The pseudocode is shown Algorithm
Algorithm Single turn MAMRP
Require:
High-level policy Th, Low-level policy my Dataset D, Optimizers for Th and mi Emin: Emax filter training dataset
1: Initialize тr and m
2: while not converged do
3: build training dataset D/ with Thi Emins Emax
4: for Sample (x, m. y") Di do
5: Generate y ~ (x, m)
6:
Compute low-level reward Ry(y
7: Update using Va, B(R])
8:
end for
9:
build training dataset Dr, with Th, Ti, Emin: Emax for Sample (x, y)) ~ Dr, do
10:
Generate 11:
Ta (x) and mx m) Compute high-level reward R2 (m, y.y") Update Th using Vo, [[[]
12:
13:
14:
end for
15:
end while
C.4 Brief convergence analysis
reuse the notations from Sec. 3.2 where x is task prompt, y is generated answer, y" is ground- truth, m is metacognition on task solving, To, and ma, are high- and low-level agents with parame- ters 0n and 0, We consider the joint hierarchical policy defined in Eq. (8) and update the objective in Eq. 9
23

---

To leverage existing RL and optimization convergence analysis methods, we treat the two models as components of a larger model, as illustrated in Fig. 7 When updating one model, we treat the other model as part of a stationary environment. The gradients with respect to 0n and O1 are:
V Vo [O,, B1) Ex.y*
Σ
Ve, Th (m
x; 0h)
[E (xriiit).. R(y,
Vo 0t) Ex,y'
Σ
Vorily x, m: ())
(9
We can compute the gradients with log trick and estimate (ylx.) R(y, yx) with Monte Carlo method.
Equipped with the objective function and gradient computation, we update the models iteratively. Without loss of generality, we analyze the case where the high-level policy is updated first:
g(t+1) arg max J(h, ())),
0 (++1) arg max ((0
(L+11 01)
Regarding the different regularizations R2 and R2 Eqs. (T)) and TT for the different policies, instead of directly integrating them into the loss function, we treat them as constraints, as done in Trust Region Policy Optimization (TRPO) [Schulman et al., 2015). Note that when one policy is fixed, the other policy operates in a stationary decision process.
Based on the defined objective and update method, apply TRPO and block coordinate ascent. First, recall that when updating single policy, TRPO guarantees monotonic improvement by opti- mizing lower bound. Specifically, let oold and represent the old and current policies, respectively. We define a surrogate objective as:
π (aass)
La (T) = E Egnnaat Boal
Told A TT (s, (s.
As shown by Schulman et al. (215), the true objective of is lower-bounded by:
(T) 2 Lea (T) C max KL[[Tala 8), m {)]:
for some constant C. By optimizing the right-hand side of the above inequality, we are guaranteed improve the performance of æ. Therefore, for policies n² and =++1 obtained from iterations and t ++ using the TRPO method, we have: ( 2 (TT)).
Now, returning to our updating method, we treat the high- and low-level policies as two blocks of single agent. The iterative update process can thus be viewed as a cyclic block coordinate ascent, where the two policies are updated in a fixed order. By updating each block using the TRPO method, and improving the surrogate objective within the KL constraint, each block update does not decrease J:
J(Ott1 ,0,) 2 J(A,, 8)), J( (1/,,,,0+1) 2 ((, 00))
Thus J(O)+1 (1(++++++++)))))+ 2 J(H 8f). This repeated coordinate maximization converges to fixed point, where no single coordinate update can further improve J(O,, 0:).
Given the theoretical monotonic improvement with TRPO and block coordinate ascent, we adopt practical version of TRPO in experiments, specifically Proximal Policy Optimization (PPO)
[Schulman et al. 2017) or GRPO [Shao et al. 2024].
C.5 Learning to reason from the perspective Leader Follower Game
Besides the loss function in the main part, we also propose to frame the problem leader-follower game. By analyzing the equilibria the leader-follower game, we demonstrate that our framework inherently identifies the optimal sub-tasks aligned with the capabilities of the low-level model. This ensures that the high-level decisions are guided by the low-level model's strengths, leading to more efficient and targeted task decomposition.
24

---

C.5.1 Leader-follower game
The seader-followrr game, also known as the Stackelberg game, models interaction between two agents with parametrized strategies 0 (01, 02) and differentiable objective functions (L1, L2) - IR. In this framework, the leader announces its strategy first, and the follower observes this de- cision to respond optimally. This sequential structure enables the leader to anticipate the follower's reaction and adjust its strategy accordingly. A Stackelberg equilibrium occurs when neither agent can unilaterally improve its objective. Denoting 01 as the leader's 's strategy and 02 as the follower's, the loss functions Lı and L2 are optimized with the following bi-level structure:
0 = argmin 0. (0, (01)).
w (0:) = argming C2 (,00,,02) 02)
Anil et al. 2021 apply the leader-follower game to ensure checkable answers in a prover-verifier game (PVG). The objective is verifier that is both complete (accepts all correct proofs from verifier) and sound (rejects all incorrect proofs from a verifier). They analyze different scenarios where the verifier
as the leader, the prover as the follower, and both announce strategies simul- taneously, forming a Nash equilibrium. The study concludes that in verifier-led PVG, a Stackelberg equilibrium is both necessary and sufficient for achieving sound and complete verifier, whereas in other configurations, Stackelberg equilibrium is not necessary or sufficient for this outcome.
C.5.2 Efficacy of LLM
Because the high-level policy possesses strong generalization capabilities, it is impractical for it to exhaustively explore every potential sub-task for each question. Instead, it naturally focuses on tasks within a feasible range of difficulty, leveraging only limited set of coarse planning actions. Rather than pinpointing perfectly tailored sub-tasks, the policy searches for general tasks of particular com- putational complexity, i.e., difficulty, that it can handle reliably. Motivated by this perspective, we incorporate the concept of a reasoning boundary for large language models (LLMs) [Chen et al. 2024b]. Intuitively, the reasoming boundary circumscribes the maximum difficulty of problems model can solve at desired accuracy level. Formally, for a model 6, a task f. and a predefined threshold A, the reasoning boundary of 0 represents the maximum problem difficulty d that satis- fies:
Bec== (t|0) sup(d]Aco(tld,0) = A).
d
where denotes the problem difficulty. By quantifying the difficulty level a model can reliably handle, the reasoning boundary provides systematic way to align the high-level policy's focus with the model's actual capabilities, gauge the efficacy of the low-level policy, and determine the optimal strategy for solving the question.
C.5.3 Leader-follower Game for LLM Reasoning
Our goal to find the high-level policy that searches for the sub-task sequence based on the efficacy of the low-level policy to solve the question. We design the loss functions as follows:
Ln E (( [- log [I(KK xr, f1.k, V1:K -1)]
C. Ex-pp.tia de [-r(yk 0s t1.ks 11:k-1]],
where r(y:: Sh | Ir, 11 31:k-1) represents the step reward for the correctness 9N derived from the question the sub-task sequence f1s from the high policy and prior intermediate answer V1-k-1- The loss functions can be interpreted as follows: the high-level policy incentivized find sub- tasks that lead to the correct answer based on the capabilities of the low-level policy, while the low-level policy is incentivized enhance its citructtoo-follwinng ability.
How to minimize the loss functions and whether such minimization leads to the desired results remain questions. To explore this, we consider simplified case of our method, where the high- level policy plans the complete sub-task sequence at the beginning and the low-level executes the instruction in a single interaction. The corresponding parameterized policies are defined as TA ((t,,..., tx) x) and (i@(,, .k)) x, (f1,...,txx t))) The corresponding loss functions are:
Ln E( ((..u)~pp.ke |- log (yK x, te)k,,)),
(16)
C, = E Excpo te [-r 0s xr, tikk Vi.k V1-k-1)]
(17)
25

---

In this step, the high-level policy generates the entire sub-task sequence without relying on interme- diate answers, while the low-level policy follows the sequence to produce answers for the sub-tasks. The low-level policy can still leverage prior intermediate answers to sequentially refine its responses.
To analyze the result agents by minimizing the loss functions, adopt the completeness and sound- ness properties from the PVG framework for LLM reasoning. Specifically, if the high-level policy generates a sub-task sequence that executable within the low-level policy's ' capabilities, the prob- lem must be solved (completeness). Conversely, if the sub-task sequence is incorrect or beyond the low-level policy's 's capacity, the problem cannot be solved (soundness). To achieve this, we utilize the conclusion from Anil et al. [2021), which positions the low-level policy as the leader and the high-level policy as the follower, equilibria guarantee the complete and sound low-level policy.
When the high-level policy takes the lead, the low-level policy is forced to adapt to the specific strategy defined by the high-level policy, which can result in neither complete nor sound low-level policy. For example, if the high-level policy dictates that it will only generate sub-tasks involving addition and subtraction, the low-level policy is constrained optimize only for these tasks. While they may reach an equilibrium, the low-level policy remains incomplete, and this limitation impacts Nash equilibrium is
both policies. In the case of the simultaneous PVG game, convergence possible, but it is not sufficient for completeness and soundness. For instance, the low-level policy might disregard the high-level policy entirely (e.g., if the high-level provides incorrect instructions, but the low-level still performs correctly). This approach, however, is challenging to implement due to the significantly larger search space involved.
Furthermore, the loss functions we design ensure that, at a Stackelberg equilibrium, the high-level policy identifies sub-task sequences that the low-level policy can execute solve the problem with the highest probability. With the low-level policy acting as the leader, it establishes its reasoning boundary for tasks. Based on the reasoning boundary, let 0n and 0: represent the policy parameters for the high-level and low-level policies, respectively. The probability that the low-level policy correctly solves the question is defined as:
K
TI(KK 2. t..k)
II Acc(tx: 22, 61).
k=1
where we can compute the difficulty de from tę and xr where the difficulty de can be derived from fy and x. The loss function in Eq. (T6) ensures that the selected sub-tasks are optimal for the low-level policy. Here we provide a theoretical condition under which the most efficient solution strategy can be identified, according to the efficacy of the LLM.
This approach can be viewed as game between high-level "prover" and low-level "verifier". The verifier, representing the low-level policy, adheres the high-level policy's 's instructions to validate its reasoning. Unlike the classic PVG setting, where the prover has ground-truth labels, the label of our high-level policy depends on the tunable low-level policy. This distinction, where the low- level policy (leader) is inherently more complex, contrasts with traditional PVG setups and adds complexity due to the interdependence between the high- and low-level policies.
By framing the problem-solving process leader-follower game, with the low-level policy des- ignated as the leader, we can construct a bi-level optimization problem to identify an equilibrium. Following the formulation in Sec. C.5. the problem is expressed as:
0 arg min L, (8) 01) (B:) arg min Ln (9h. 0t)
Then we can apply bi-level optimization techniques.
D Training Details
D.I Single-turn ReMA
We refer to Appendix G for prompts we use during training. We implement the training pipeline with OpenRLHF (Hu et al. 2024a] which is a highly efficient codebase and is easy to scale up. We select REINFORCE+ to save resources and for efficient training. All experiments are conducted in a node of 8 NVIDIA 1000 GPUs. We use bf16, Zero2, Flash-Attention and gradient checkpointing to run our experiments.
26

---

During rollout, we set temperature=1.0, top p=1.0, top k=-1, and use vLLM for inference accelera- tion. We set the max generation length be 2048 and. the rollout batch size to be 1000. The number of samples per prompt is 4. During training, we use Adam Optimizer with learning rate of 5e-7. We set the mini-batch size to be 500, and the clip ratio to be 0.2. Other hyperparameters, such as KI. coefficients and the number of training episodes, were carefully tuned based on validation set performance to ensure robust and reliable results. To align with the hyperparameter in OpenRLHF, use #Training Episode as the number of reinforcement learning epoch on the entire dataset.
In ReMA, during prompt filtering of the high-level model, the high-level agent first samples 10 candidates for each question with t=1.0, and for each output the low-level agents sample solution with t=0.0, then we select questions of success rate between [émin: Emax]. And for the low-level agent's s prompt filtering, the high-level agent first samples candidate for each question with t=0.0,
and for each output the low-level agents sample 10 solutions with t=1 1.0, then we select questions of
success rate between [Emin; Emax] and use the high-level agent to sample meta-thoughts with t=1.0 as the input.
D.1.1 Supervised fine-tuning data collection
For experiments in Sec. 4.2.1 we collect expert data to enhance the reasoning pattern, i.e. RL from SFT. Specifically, we collect demonstration data from GPT-4o Mini on MATH training dataset (7.5k problems) Hendryckk et al. [2021] and use it to fine-tune the LLMs. The data generation follows these steps: First, we prompt GPT-40 Mini to produce metacognitive reasoning for high-level model training. Specifically, we use different prompts to instruct it to rewrite and decompose a given ques- tion without providing a final answer. We collect metacognitive reasoning using two predefined ac- tions, "rewrite" and "decompose", which align with human approaches complex wroblem-solving while preserving answer diversity. Next, we use the generated instructions to prompt GPT-4o Mini to follow the metacognitive steps and solve the question, obtaining SFT data for low-level policy training. Below, we present the prompts used for both high-level and low-level models. Prompts can be found in Appendix G.1.I
D.1.2 Dataset Curation of RewardBennhh70
Table 2:
Performance on LLM-as-a-Judge benchmarks, trained on dataset under the loose setting, The two-agent workflow in ReMA
VRP
ReMA (Ours)
Model
Benchmark
VRPRL RL
MRPRL.
(CoT)
1lama3.1 -8B
RewardBench970
71.24
81.86 (+10.62)
80.41 (+9.17)
86.29 (+15.05)
JudgeBench
51.77
51.45 (-0.32)
50.65 (-1.12)
53.71 (+1.94)
-Instruet
Average
61.51
66.65 (+5.14)
65.53 (+4.02)
70.00 (+8.49)
Qwen2.5
RewardBench9900
86.49
87.22 (+0.73)
80.31 (-6.18)
90.72 (+4.23)
-7B -Instruct
JudgeBench
58.39
54.84 (-3.55)
55.81 (-2.58)
58.71 (+0.32)
Average 72.44
71.03 (-1.41)
68.06 (-4.38)
74.72 (+2.28)
We process the original dataset in RewardBench by splitting it into training set containing 5,000 tuples of (instruction, response A, response B) and a test set with the remaining 970 tuples.
To ensure meaningful dataset split, we validate two separation strategies:
Loose setting: We only ensure that there is no direct overlap of tuples between the training and test sets.
Strict setting: We further enforce that no instruction appears in both the training and test sets. The results for this setting are presented in the main results (Table Tb
Additionally, since the original RewardBench data originates from different subsets, we ensure that all original subsets evenly represented in both the training and test sets.
Table reports the learning performance of various methods under the loose dataset split setting. Compared to the results in Table [b] ReMA significantly outperforms other RL tuning basclines
27

---

across all models, particularly on out-of- distribbutioo (OOD) benchmarks. The consistent improve- ments on OOD datasets of these two settings suggest that ReMA enhances meta-thinking abil- ity, resulting in better generalization across diverse task distributions.
D.1.3 Training on MATH
VRP For Llama3-8B Ba-aaa-----Intuutt.. Llama3. 1-mma3..----Isstiii Ins-tustut,, and Owen2.5-7B-Instruuc we all use KL coefficient of le-2, and for #Training Episode, we use 12,6,6 for these 3 models respectively. For Llama3-8B-Instuu we set the learning rate of 2e-7 for stable training.
all KI. coefficient of le-2, and for #Training Episode, we use 10,6,6 for these 3 models respectively.
MRP For Llama3-8B-Instnuu Llama3.1 1-....1-1111 Inst-utt and Siven..--7B-Insteene
MAMRP We use Emin 0.2, Emax 0.8 for prompt filtering. We use the same #Training Episode=4 for all models, and for #Update Iteration, we use 3 for Llama3-88 1are3-8B-Imse I-B-Instuct and 1lama3.1- 1- 8B-Instruct, 10 for Qwen2.5-7B- In-stuutt... And we set the KL coefficient to be 1e-2 for all the models.
D.1.4 Training on Reward Bench
VRP For Llama3. --struutt and Bren2..-7B-Instrreee we all use a KL coefficient le-2, and for #Training Episode, use 4,6 for these 2 models respectively.
MRP For Llama3. 1lamaa.--8B-Instruce Instructt,, and Qwen2.5-78 -een2..5-BBInnttruet,, we all use KL coefficient le-2, and for #Training Episode, we 4,6 for these 2 models respectively.
MAMRP We set #Update Iteration=1 for all models. We set the KL coefficient to be le-2 for and le-2 for Qwen2.5-7B-Instruct all models. For Llama3. 1-B-aa.---------tt- use Smin = 0.2, E max = 0.8 for prompt filtering and we use #Training Episode of 2 during training. For Llama3. 1-8B-IInttuuee we use min 0.1, max 0.9 for prompt filtering and we use #Training Episode of I during training.
we
D.2 Multi-turn ReMA
We refer to Appendix for prompts we use during training. We implement multi-turn ReMA training pipeline with VeRL [Sheng et al.] 2024) since it's easier to implement complex training pipeline with a single centralized controller. Similar to OpenRLHE, VeRL also a highly efficient and scalable codebase for further development.
For the multi-turn ReMA rollout,
parameter sharing and simultaneous update by de- fault. In details, we maintain two message lists with the system prompt of meta-thinking agent and reasoning agent respectively. During rollout, each agent act as 'assistant" in its own mes- sage list and the other agent act as 'user'. We use three hyperparameters to control the roll- out length: (I) 'max mnax-num.turns" mhaxrresponse-eengthy
the maximum number of turns for each trajectory. (2) the maximum number of tokens for each turn's response. (3)
max.prompp..eengee : the maximum number of tokens for each trajectory.
During training, we apply the collected message list to Qwen2.5-7B's chat template and build loss masks in order compute the loss for all turns of one trajectory (message list).
Morever, for multi turt ReMA rollout, unlike single agent single turn rollout, we need to carefully design the termination logic. Basically, we let the meta-thinking agent automatically decides when to finish the solving procedure, we use a special tag [FINISH] to indicate the end of the solving procedure. After we detect this tag, we will terminate trajectory after the reasoning agent generates its output.
We also design other termination conditions to ensure the quality of the generated trajectories. If the last agent's response is too long, we will terminate the whole trajectory and setting the reward to O. We also introduce different version of format reward: we give a reward of 1.0 only if the reasoning agent's last turn response is correct and the meta-thinking agent's last turn response include [FINISH]* We use math-verify the default verifier.
28

---

D.2.1 SFT data collection of multi-turn MAMRP
We use GPT-4o to translate 817 samples in LIMO Ye al. 2025c by prompting it to wrap each sentence with meta-thinking and reasoning tags. We use a temperature of O. After filtering, we get 800 conversations for training. The prompt can be found in Appendix [.2.1] For supervised fine- tuning, we use LlamaFactory as the codebase and train the model for 3 epochs with learning rate of le-5, consine learning rate scheduler, and batch size of Use DeepSpeed Zero2 for distributed training.
D.2.2 Training MATH
For training of multi-tura turn ReMA on MATH, we use GRPO (hao et al. 2024] as the default learning algorithm. We refer to Appendix G.2.2 for prompts. For experiment in Sec 4.3 128 prompts, each with 16 trajectories. During training, we drop the KI. loss term to improve the numerical stability. We use learning rate of 1e-6, bíloat16 precision, FSDP backend for distributed training. We split the rollout data into 4 bati-b for update. For the sake of numerical stability, do pre-clip before computing the exponential of log-prob for upperbound of 3.0.
we use sample
For the main result in Fig 5 we test different rollout configurations with wwa.pprmpp..enee of 4096, training for 500 steps. We use 32 NVIDIA A800 GPUs, the longest training cost about 40 hours due to large scale validation per 10 steps.
For the ablation results in Fig 6 we use tiny subset of MATH Level 3-5, training for 300 steps. Specifically, we sample 19 questions for every single type (133 instances total). We use NVIDIA A800 GPUs, the training cost about 30 hours
We test different rollout configurations:
(I) max_num.turns 30, max max.ressoonee..enghh225555 max.prompt_.enghh-4009 (2) max_num_turns-33 max.response__eegtt>--O00 pwppp
And for the experiment of separate parameter in multi-turn ReMA, we iteratively train each agent with the same configuration above, but with switch interval of 10 steps, starting from the meta- thinking agent.
E
Other Experiments
E.1
Experiments of comparing to more baselines
To further comparing ReMA to other meta-thinking and multi-agent baselines, we focused on adding two representative baselines: Multi-Agent Debate (MAD) (Du et al. 2023] and Self-refine (Madaan
ct al. 2023].
For both two experiments, we use the prompts and implementation in Du et al. 2023 Note that MAD and self-refine are inference-tine methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Relf-Refine, we let the LLM first propose an answer and then prompt critique and refine the answer only once. We use temperature of 0 for most generations except the 1st round of MAD to maintain diversity. We summarize the additional baseline results Table 3
Self-Refine and MAD do not exhibit strong performance, which we attribute to insufficient scaling of test-time compute. However, under similar compute, single-turn ReMA enhances the system's problem-solving capabilities through RL training.
E.2 Reward functions shape cross-agent behaviors
We also investigate the impact of different reward function designs on ReMA's behavior. In addi- tion to the base reward setting described in Appendix C.2, we evaluate a consistency- function using Swen2.5-7BB-Instrue agent to generate more detailed guidance. Indeed, we observe that the high-level agent trained in this manner produces more detailed solution steps compared the one trained with the basic cor- rectness format reward. However, we also find that this approach often leads to jailbreak behavior,
reward This reward function designed to encourage the high-level
29

---

Table 3: Additional baselines reordered as CoT, Self-Refine, MAD, ReMA (Ours). CoT comes from prior VRP results.
Model
Benchmark CoT
Self-Refine
MAD
ReMA (Ours)
MATH500
30.80
24.60
22.40
33.80
GSM8K
67.48
65.58
66.76
79.38
AIME24
0.00
0.00
0.00
0.00
Llama3 -8B
AMC23
2.50
2.50
11.25
22.50
Gaokao2023en
22.34
17.40
20.00
28.57
Inntrrcct
Minerva Math
8,82
12.13
11.76
13.97
Olympiad Bench
8,44
4.59
4.67
8.89
Average
20.05
18.12
19.55
26.73
MATH500
50.80
26.60
39.20
53.20
GSM8K
86.05
56.41
78.43
87.26
A1ME24
10.00
0.00
5.00
13.33
Llama3.1
AMC23
27.50
12.50
16.25
20.00
Gaokao2023en
38.96
27.53
32.08
37.14
-Instruct
Minerva Math
22.79
13.24
18.93
28.31
Olympiad Bench
15.11
6.52
10.37
19.56
Average
35.89
20.40
28.61
36.97
MATH500
75.00
76.40
72.60
74.40
GSM8K
92.04
91.51
90.00
90.60
AIME24
6.67
6.67
11.67
20.00
Qwen2.5 -7B -Instruct
AMC23
47.50
45.00
45.00
57.50
Gaokao2023en
56.62
62.34
61.82
$7.92
Minerva Math
35.66
34.93
36.03
34.93
Olympiad Bench
38.22
38.96
39.41
36.30
Average
50.24
50.83
50.93
53.09
where the high-level agent tends to include the final answer within its output, compromising the intended hierarchical reasoning process.
Furthermore, we discover an interesting evolution of a pattern during training: although our exper- imental setup is designed for the high-level agent provide solution plan while the lower-level agent executes it, we find that under the consistency-bart increases its attempt of verification rather than straightforward execution. We observed certain sen- tence commonly appearing in the low-level agent's 's responses: "Ler's go through the solution step by step to ensure clarity and correctness."
reward, the lower-level agent significantly
quantify this effect, we track the frequency of We analyze this pattern across all mathematical test sets, sampling eight completions per question at temperature of 0.7. Our empirical results have identified 30x increase of such self-verifying pat- terns in the model trained with the consistency-based reward compared to the one trained with the base reward. Moreover, we also observe additional variations of this pattern, c.g. "Let's 's''s carefully re-evaluate the problem and solution to ensure accuracy and clarity." These phrases indicate that the low-level agent is actively exploring to verify the detailed response provided by the high-level agent.
This suggests that (1) meta-thinking can not only emerge and be reinforced the high-level agent but also in the low-level agent. During reinforcement learning (RL) training, the two agents develop a novel problem-solving pattern characterized by a role reversal. (2) Consistency-based rewards promote a more self-corrective approach at the lower level, potentially disrupting the intended separation of roles between planning and execution. For detailed case study, refer Appendix
E.3 Detailed Training Curves Different Datasets of Multi-turn ReMA
We show the detailed training curves of the multi-turn ReMA on different datasets in Fig.
30

---

Train Reward
MATH500
alme24
0.75
0.20
0.15
0.65
Rewerd
0.70
0.60
0.55
0.10
0.05
0.05
Training Steps
Training Steps
Training Steps q5mBk
Training Steps
amc23
@aokao2023en
minerva math
0.6
0.350
0.5
0.60
0.90
Reward
0.4
0.55
0.300
0.3-
0.250
0.2
0.45
0.75
0.200
200
400
200
400
Training Steps
Training Steps
Training Steps
Training Steps
tumá0 token225
turnoo taken256
tum30 tosen512
Figure Detailed Training Curves on Different Datasets of Multi-turn ReMA
F
Qualitative Results
High-level policy finds better plans
Here is an example of how high-level policy alters the solving method of LLM, increasing the likelihood of providing correct answers. As we can see from the following example, without high-level policy, the LLM counts all integer coordinates, including those on the boundary, and then subtracts the boundary coordinates. In contrast, the high-level policy identifies a better approach, directly instructing the LLM to count only the coordinates strictly inside the boundary.
F.2
Case study for Experiments in Section
H2
Fig.
10 and Fig. show an case study experiments Sec.
E.2
Although both agents are prompted with the same instructions as in our main results, the consistency reward of the high-level agent significantly alters the learning dynamics. As illustrated in Fig. [T] the high-level agent generates detailed solution attempts rather than strategic plan. Consequently, the low-level agent evolves to verify the high-level agent's solutions. This suggests that, during reinforcement learning (RL) training, the two agents develop a novel problem-solving pattern characterized by role reversal, where one agent generates an answer while the other verifies
F.3
Case study for Experiments in Section 4.2.2
Fig. 12 shows the detailed outputs of the two groups LLMS trained JSON action. The outputs finally converge relative easier and shorter sentences, compared main experiments. This may due to insufficient capacity of a small LM to produce a valid JSON-format response while simultaneously exploring diverse reasoning strategies.
31

---

Consider the rectangular regiinn the fullowing points as verticess (5,,), -)) -4) How many points with integer courdinates will be strictly in the interior of this rectangular region?
Output without hightlevei planning salves question less optimal way, leading to wrong answer:
with high-level planaing identiifes optimal sub-tasks solve, incrcasing chance amswering the question correctly
To determine how many points with integer cooedinates will be strictly the
High-level policy plansing: identify boundaries the rectangular region Then, determine the integer cocrdinates that lie strictly within thes boundarics. Finally, count the number of such points.
interior of the roctangular region with vertices (5, 4), (-5,4), (-5, (66,)43).00 and
will fdlow
stepe:
Identiff the dimensions the restangle:
The witth of the rectingle can be calesiited as the distance between the x- coundinates the vertices. The coordinates and
Lon-Aevel policy reasening following the high-policy plan: To solve the prublem, follow the steps outlined instrucions.
(-5)
Step I: Identify the boundaries of the rectangular region. The vertices of the rechingular
are: (5,4) (-B. 4)- 4)- (G, 4jf'rom these vertaess we can determine boundaries- The boundary
height the rectungle can be caleniated as distance between the coordinates of the verlices. The y-cooriinates are and -4.
-5.-
The right boundary is x 5. The bottom boundary y 4. The tup poundary y
Height 4 (-4)
2. Calculate the total number of integer
The total number of integer calculated
Step Determine integer conndinttes strictly within houndaries, For point (a.u) to be strictly inside the recungle, it must satisfy- -5 &
poines roccangle rincluding the boundary)
considering the range of and y values: The x valuee range from -5 to
giving 11 integer values (-5, -4,-, 22, -1,0 0, 2, values
'The
values from
d, giving
Step 3: Count the integer coordinates that satisfy thes cunditions. Now, we will find the inseger values for and that fall within these boundanes. For
4, -3, -2, 11,0, 3,,2,-1,0 2,3, d). The
of points rectingle
inclnding the boundary) is.
values
gives us possible integer For yo- The integer values that satisfy -4 y mrc -3, -,, 2, This
-B, 1, 2,
Total points - 11 x 99
gives 7 possible intoger values.
3. Calculate the number of points on the houndary: The boundary consises of
rectungle.
botcems
and y
Step Calculate ronal mamber points. Now, mulliply mumber of possible integer values for a by the number of possible integer values for
each have 11 intger points (from -5 to 5). The left and right sides (x
und x :5) each have integer points (from - to 4). Therefore, the total
mumber puints boundary
Total points
7
Boundary points
Finally, number points with cooedinates inserioe rectangular region is [6a]
the
number points strictly in the interior, we subunet the points on the boundary from the total number points:
Calculate the mumber strictly interioe points
points
points Boundary points 99
Thus, of points with integer coordinates will strictly the
intztior of this rectangular region 30
Figure 9: Case Study comparing with and without high-level metacognition results.
G
Prompts
G.1 Single-turn ReMA prompts
G.I.1 Prompts for JSON data collection
Prompt for metacognition reasoning to rewrite:
System prompt:
You are a math expert trying to solve mathematical problems.
Before answering question, your task is to rewrite the original question to make it clearer.
Provide your rewritten content in JSON format:
{[
"action": "REWRITE", "output"; "((clearer question content)]"
}}
Respond only with valid JSON. Do not write introduction summary.
User prompt:
Here is the question:
[problem.text]
Prompt for metacognition reasoning to decompose:
32

---

HightLeeol Agrat' Ouigpr
Levilggerr Ohepue
Lateprikongg cdditee slep dorty camuuiats
Cadermed
Vedentaad gremouates
The of (122-11*
Th lue halua apalii aeviiiaar
Nodes But upuiliii roscilan
Coeater nete
Crender
Lis the proucty euepii uni repeeate
propert uompl mosapate
auptor, and lnv the reptred uaee
madaka
Th
and ofe
sencal rflacco
We fid ville
Llig ampory cap- thi
firs
PPP
P27
PiP
Fi?
saac cutuuta turhus mu squatio aff 130e 17th the ragil allo jud pairs
Saoc nurs nuts
oiecciioion "The 11% magela bny
indo the do the lo laju and regataee of 1d Thardher
The mugrecedd of
creider respitat of eftir cperiic
1)% maps
f4 and de mogiaaal righraldit [Da Therther
Ief pdde 1
H 106
Golli Reiua Les
alea Thas
-- [1he-1
Sihilg socol
Selvng be commooue
Mrunce de and canoonappaaat shal.
Saas Inl Scillss aav
Sloce Py) d fialli vokue
(
Thanhee
Thodlin 8o camile
144 106 3
1A4 344 1A4 TALL 354 161
The rolucce
700
The sali
Figure 10: Case Study for consistency reward of high-level agent
Question The opecice af (tiee 11 = dless 10 Mooogglee rons to Pas Ta For Pau where hur denizs deneoos oxgagation Fied of ole
Apent's
Carppr t (PPing
Aguot's Dorpat (port
Undertand the chis oqaatin: The the
Thaing do 10t you ribos
Montt The
Porn Purr Puri stmerdie
per The oo plich ardtt
G. A
rercondie do menerr di prodeet T- lo?.
witt 1) x
3000
We sodd ear onee pain offeen:
Simgith coprodeec 1o euchto
Thus dhe of uquadii Aoc
ti the +he
Appb property comglas
[I- Laae G- Daall - LOT (I- 200000 Die susty 1ik
Sax de wogpeaaatc stus de dy "Ty :2 por "Iy"
I-- 28A! t 188 1330 4 Kory 3TE -1 bow (100e) (1) 120 20wm (1
ferelte remoonaal medil and
as de Bortar de serv
(Ir Mone (122--22
Fiaalas Ba eitaiiii Conderatii
nnd New
and aigie Be day.
Figure 11: Case Study for base correctness reward of high-level agent
33

---

Question: The product of 36" + and 4e"
is 124 26r 8r²
What is
1-mma.1188-Iustui training Output:
Ui--ttt---U-lssttruuet training Output:
"action": "DECOMPOSE",
"action": "EMPTY". "oulpu"":
"oulpul": "Decompose the problem into finding the cooffiiient of the solving for
product of polynomials
Step find the produci of the two polynomials
ge 64 and 4f le
Step The problem asks us find the values of and i the quadratic expressions at and 4t respectively, that their product is 12r + 204"
multiply
firs polynomial
the
second pelynomial and then combine like temss
10c
Stap The product the polynomials be written
Step
we multiply quadratic expressions equaie the resalting expression the given product
(3t2 + a)(at" + bt 2).
Step 3: Using the distributive property, we multiply each temn the first
12c4 201 6/
polynomial by
second polynomiall:
(de") (Br) + (a=*)-2)
(50(()))) (50)(M) (55) (a](1(") (a)(8)) (m)(2)
Step We start mulliplying the quadratic expressions: (3xf o)(4s
Step Simplilying the products, we get
2).
12c 304* 66* 4 20r + 5&4 100 dat udi
Step gives 12r (38 5je" (4a 58 6)F
Step Combining
12r + (35 20je* (56 6 Aaae' + (ab 10je 2a
(
26
10r
20je
We
that the product the polynomials
12r 28r sc 16t
Step equal to the coctTisients of the given product 12c.
resulting expression
the cuollicients of the corresoonding puwers of in the expressions.
We
200"
8t²
162
Step Equating the coefficients of gives us the equation 36 + 26.
Seep 7: Equating the coefficients of t², we get 36 20 26. Equating coefficients of
which simplifies to 21 and thus -
get
Step 10: Solving the equation -2 we find that -33 we ger
Step 9: Equating constani terms, we gel -2a
Step Equating the constant terms gives us 20 6, which simplitties
Step Subatirating
equation 6
58 + 4[ 3) 8 which simplifies to 56 18 -8.
Step Solving
that 56
Step Now that have the values of and find the
Step 13: Now that have found the values of and à we van calculate their sum:
Step 9: A:Adiing and gives 10.
The final answer is:
The rinal answer 10
Figure 12: Case Study for interpretability experiments in Section 4.2.2
System prompt:
You are a math expert trying to solve mathematical problems. Before answering question, your task is decompose the original question to make it clearer.
Provide your rewritten content in JSON format:
[[
"action": "DECOMPOSE", "output"; "[fdecomposed question content]]"
}}
Respond only with valid JSON. Do not write an introduction summary.
User prompt:
Here is the question:
[problem.text]
Prompt for generating final answers using on the question and metacognition reasoning:
System prompt:
You are a math expert tasked with solving problems step by step. Follow the provided instructions precisely, showing all reasoning and intermediate steps.
Present the final answer within (boxed(()).
User prompt:
Here is the question and instructions:
Question
[problem.text]
Provided Instruction
[instruction.text]
34

---

G.1.2 Prompts for Math problems
VRP prompt:
System prompt:
You are a math expert tasked with solving problems ems step by step.
Present the final answer within \boxed().
User prompt:
Here is the question:
{Question]
MRP prompt:
System prompt:
You are a math expert tasked with solving problems. When solving problem, your first task is provide high-level solution plan as an instruction. Then you need to follow the provided instructions precisely, showing all reasoning and intermediate steps. Finally, you must present the final answer within |boxed[].
User prompt:
Here is the question: [Question]
MAMRP prompt:
high-level agent:
System prompt:
You are a math expert specialized in solving mathematical
problems, you need to teach weaker agent with minimal capability in math how to solve a problem step-by-step.
Your task is to provide high-level solution plan for the given problem, in order to guide a low-level math solving agent to solve the problem.
You can not directly answer the question. You'11 be punished if you include any answer in your response.
You need to first think deeply in mind and output your final instruction.
User prompt:
Here is the question: {Question]
low-level agent:
System prompt:
You are math expert tasked with solving problems step by step. Follow the provided instructions precisely, showing all reasoning and intermediate steps. Present the final answer within \boxed(). User prompt:
Here is the question and instructions:
[Question]
[Question}
[nd of Question]
[Provided Instruction]
(instruction]
[End of Instruction]
35

---

G.1.3
Prompts for LLM-as-a-Judge problems
We adopt the prompts from Saha et al. [2025]]
VRP prompt:
System prompt:
impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, zelevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two zesponses and provide a short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.
Please act
Do not allow the length of the responses to influence your evaltation.
Do not favor certain names of the assistants. Be as objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better.
User prompt:
[User Question]
instructoon)
[End of User Question]
[The Start of Assistant A's Answer] [response_A]
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer] (response_B)
[The End of Assistant B's Answer]
36

---

MRP prompt:
System prompt:
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's s instructions and answers the user's question better. First of your task is to build an evaluation plan that can then executed to assess the zesponse quality. Whenever appropriate, you can choose to also include step-by-step reference answer as part of the evaluation plan.
Enclose your evaluation plan between the tags "[Start of Evaluation Plan]" and "[End Evaluation Plan]".
After that, please act as an impartial judge and evaluate the quality of the responses provided by two assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better.
Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.
Do not allow the length of the responses to influence your evaltation.
Do not favor certain names of the assistants. Be as objective possible. After providing ng your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant is better.
User prompt:
[User Question]
(instruction] [End of User Question]
[The Start of Assistant A's Answer] (response A)
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
[response_B]
[The End of Assistant B's Answer]
MAMRP prompt: high-level agent:
System prompt:
We want to evaluate the quality of the responses provided by AI assistants to the user question displayed below.
For that, your task is to help us build an evaluation plan that can then be executed to assess the response quality. Whenever appropriate, you can choose to also include step-by-step reference answer as part of the evalvation plan.
User prompt:
[User Question]
(Question)
[End of User Question]
37

---

low-level agent:
System prompt:
Please act as an impartial judge and evaluate the quality of the responses provided by two assistants the user question displayed below. Your evaluation should be performed by following the provided evaluation plan step-by-step. Avoid copying the plan when doing the evaluation.
Please also only stick to the given plan and provide explanation of how the plan is executed to compare the two responses.
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.
not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible.
After providing your evaluation, output your final verdict by strictly following this format: "[[A]]" if assistant is better,
"[[B]]" if assistant B is better.
User prompt:
[User Question]
(instruction]
[End of User Question]
[The Start of Assistant A's Answer]
[response_A]
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer] (response_B)
[The End of Assistant B's Answer] [The Start of Evaluation Plan] (evaluation.plan)
[The End of Evaluation Plan]
38

---

G.2 Multi-turn ReMA prompts
G.2.1
SFT data collection of multi-turn MAMRP
System prompt:
You are classifying reasoning process data into two types of thinking. You will be given question question-answer answer pair from a reasoning dataset. Your task is to split all words into two parts. These words are crucial for analyzing reasoning patterns, do not skip any details.
+-Meta-Thinking Agent (MTA) :** Responsible for high-levei thought. processes. This includes planning, evaluating steps, expressing uncertainty, making observations, or setting goals. Avoid detailed calculations. The content should be enclosed in <<metathinnknng>> and ></metatthnnking""
**Raasninng Agent (RA) Responsible for detailed steps, such as calculations, logical deductions, or breaking down problem into subproblems. The content should be enclosed in >reesooning>' and ></reasoning>',
**Rules to follow:**
1. **Do not assign large chunks of text to single type of thinking.** The reasoning process consists small, nonlinear thinking steps, and Reasoning steps.
alternate appropriately between Meta-Thinking
2. **eeep the words from the original solution unmodified whenever possible..* Words like "Wait," "Hmm," " "But, " etc., typically indicate Meta-Thinkimg
and should be preserved. 3. **ehe finalizing the answer:**
The **MetaThinnking Agent (MTA) must explicitly confirm the answer before completion and output '[FINISH]'
The **Reasoning Agent (RA)** should then provide the final answer in the correct format.
4. **Do not skip any reasoning steps, redundant, incorrect or irrelevant**
if they seem
5. *DD not modify or remove any part of the original reasoning process**, even if it seems redundant or repetitive. The goal is **preserve the exact flow of thought**
it naturally occurs. 6. **Retain all expressions such as "Wait," "Hmm," "But wait," etc., exactly as they appear. These indicate important cognitive processes and should not be skipped or altered. Here are examples for you: [Examples]
User prompt:
[Begin of Question]
[question]
[End of Question]
[Begin of Solution]
{solution}
[End of Solution]
39

---

G.2.2 Prompt for math problems
Meta-Thinking Agent (MTA):
System prompt:
You are a meta-think agent that represents human high-level think process, when solving a question, you will have a discussion with human, each time you think about what to do next: e.g.
Exploring multiple angles and approaches
Breaking down the solution into clear steps Continuousil reflecting adapt your strategy as you progress
intermediate results honestly and
Backtracking when necessary
Requesting exploration of multiple solutions individually
Finally confi rm the answer with the tag [FINTSH]
User prompt:
{question}
Reasoning Agent (RA):
System prompt:
Please reason step by step follow the given instruction, when asked to finalize your answer, put your answer within (boxed(] User prompt:
{question] [instruction]
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe the hierarchical framework and the training method, experiments, ablation study, which align with the content in the paper (Sections 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a areech- speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
41

---

Answer: [Yes]
Justification:
We provide analysis of iterative training method Appendix C.4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides training dataset, out-of-distributiin evaluation dataset and the preprocessing procedure in Section 4.2, the base models and the training algorithms in Section 4. and Appendix C.3 supervised fine-tuning data collection in Appendix D
Justification:
and prompts in Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
42

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We put code in anonymous 4open www.esceeeeeeeeaaaee.ee...e
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nipp) ca/
pub mibblic/on /ttp/quuiesssseeee Codeeuumeeiiiiiio ssisnssssoosssssssss ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. http:ppppllccceeeee wwoegiideeeeeeeeuuuiinmi ssi
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
experimental results for the new
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The dataset splits are Section 4.1 and Appendix the training settings
including hyperpaprameters and training pipeline are in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
We report the confidence intervals and other statistical significance informa- tion for ablations. E.g. confidence intervals for experiment in Section 4.2.1 For experi- ment of Figure 6 in Section 4.3 we observe the same conclusion under multiple training settings. Due high computational burdens, the paper does include error bars for all of our experiments, but we provide every detail necessary for experiment reproduction.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
43

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detailed hardware information are in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://nuripppp
c/ cwwhiiiaaiieeeee nes ines?
Answer: [Yes]
Justification: We use the datasets that conforms to the NeurlPS code of Ethics and do not involve human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiciii..)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses exclusively on the technical aspects of L.LM reasoning and does not incorporate human value judgments.
44

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper proposes framework and training method for LLM reasoning and does not release any data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: credit the original owner of the code, data and the models. We mention the license and strictly follow the usage terms.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
45

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [aaprswittcoe com/ datasetts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coodemmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing experiments or research with hu- man subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The work does not involve human subjects, therefore IRB approval is not applicable.
Guidelines:
46

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our work focuses on the LLM reasoning task and explicitly details every use of the LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer
LLM policy (htpps: /tne:s:///rriise/ http:pps.cccccooocncc.e.e 2025/LLM for what should or should not be deseribed.
47


Reviews:
[{"review_id": "fp0fN6TUAW", "reviewer": "Reviewer_2Abh", "summary": "The paper treats meta thinking for LLMs as a multi-agent learning problem that can be trained alongside the standard generation process with RL and leads to strong results on benchmarks.", "questions": "1. Could the authors suggest a method of incorporating intermediate rewards/learnt values to fix the credit assignment issue and try out small scale experiments for this?\n\n2. How does ReMA compare to simply prompting a single model with meta-thinking instructions and then performing RL on the subsequent model to best use these steps?\n\n3. Could the authors try replacing the meta-thinking agent with some other baseline agent as an ablation? Is it really the meta-thinking that is helpful in increasing performance or spending more inference compute?\n\nI would be happy to increase my score once these questions are answered.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. The idea of focusing on a meta-thinking agent in a multi-agent system and training end-to-end is novel and interesting.\n\n2. The method achieves good results on a set of models and benchmarks.\n\n3. The turn-level ratio clipping seems like a useful solution to prevent degenerate behaviors/collapses by treating each conversational turn as one action (not necessarily novel but an important solution in the method).\n\nWeaknesses\n1. Credit assignment remains an issue. A good meta-thought but poor reasoning steps would lead to negative rewards for both. This severely limits training efficiency. The state drift because of this is an important concern. It would be interesting to see a credit assignment technique like (https://arxiv.org/abs/2410.08115 or https://arxiv.org/pdf/2412.01928) used.\n\n2. The improvements are not very high and not necessarily statistically significant. Standard RL baselines on Qwen models have often led to higher results on benchmarks than reported here.\n\n3. The method seems quite fragile to different configurations and hyperparameters.", "was_revised": true, "final_justification": "I have addressed the rebuttals in my comment below and would like to note that they are not sufficient. I recommend rejecting the paper due to the issues highlighted in my comments.", "rebuttal": "### **Weakness 1: Credit assignment remains an issue & Question 1: Suggestion of a method of incorporating intermediate rewards/values**\n\nThank you for your constructive review and highlighting the credit‐assignment challenge.\nHere we address both Weakness 1 credit assignment issue and Question 1 incorporating intermediate rewards/values.\n\nAs you point out, “poor reasoning steps would lead to negative rewards for both.” In our manuscript, we emphasize that we develop two agents collaboratively, where the meta‐thinking agent generates guidance for the reasoning agent. When the reward is negative, the agents search for a better collaborative strategy.\n\nFor the papers you recommand (https://arxiv.org/pdf/2410.08115 and https://arxiv.org/pdf/2412.01928), they estimate the value of each turn using a Monte Carlo method, which we agree is very promising and plan to explore in future work. For example, sampling multiple reasoning trajectories for each meta‑thinking decision and averaging their rewards to compute the advantage, yielding more accurate credit assignment. We will discuss these works in our manuscripts.\n\nWe also propose two additional methods for intermediate rewards/values. \n1. Baseline reuse: We can leverage our existing “no‑meta‑thinking” samples as a baseline to reduce reward variance and support credit assignment. \n2. Tabular Markov‑chain modeling: For tasks with a limited state space, we can treat the problem as a tabular Markov chain, akin to GiGPO[1] , and compute more accurate state values for advantage estimation. \n\n### **Weakness 2: Improvement is not significant. RL baselines on Qwen models have often have higher results.**\n\nWe thank the reviewer for their thoughtful comments. As shown in Section 4.2 Table 1, ReMA achieves substantial improvements on individual hard benchmarks, notably: AMC23 with Llama3-8B-Instruct: ReMA → +20.00% (2.50% → 22.50%), AIME24 with Qwen2.5-7B-Instruct: ReMA → +13.33% (6.67% → 20.00%). These results demonstrate large gains on challenging tasks. Our ablation study in Section 4.3.1 (Fig 6) further disentangles the contributions of turn‑level ratio clipping and the learned meta‑thinking policy, confirming that the observed improvement is directly attributable to the guidance provided by the meta‑thinking agent.\n\nWe acknowledge that Qwen models fine‑tuned with RL can achieve larger improvements. However, most recent work applies RL to base models, which start from a lower initial performance and are hard to sample MAMRP-style data for training. In contrast, our approach leverages the full potential of already capable models and further enhances their reasoning ability.\n\n### **Weakness 3: The method seems quite fragile to different configurations and hyperparameters.**\n\nThank you for highlighting these important points. We agree that multi‑turn ReMA can be sensitive to some hyper-parameters and training configurations. That’s why we introduced turn‑level ratio clipping combined with parameter sharing between agents, which noticeably reduces collapse and stabilizes learning (Section 4.3.1, Question 5 and Figure 6). We are actively developing more robust credit‑assignment mechanisms and training recipes to further enhance stability.\n\n### **Question 1: Suggestion of a method of incorporating intermediate rewards/values**\n\nThanks for your constructive comments. Please see above reply to Weakness 1 and Question 1.\n\n### **Question 2: How does ReMA compare to simply prompting a single model with meta-thinking instructions and then performing RL on the subsequent model to best use these steps?**\n\nWe appreciate the reviewer’s detailed critique. We view the question from two perspectives: (1) the high‑level meta‑thinking agent, specifically, the quality of the meta‑thinking instructions it generates; and (2) the low‑level reasoning agent, its ability to follow those instructions and the instructions’ suitability for that agent.\n\nIn our manuscript, we address these two aspects in separate sections respectively. In Section 4.2.1 and Figure 3, especially the comparison between the first and third settings, `RL from base` and `RL under Meta-thinking`. Figure 3 shows that this meta‑thinking guidance significantly boosts performance, demonstrating that the good instructions help with RL training of the reasoning agent. In Section 4.2.2 and Figure 4, we restrict the meta‑thinking instructions to “DECOMPOSE”, “REWRITE”, or “EMPTY”. Different models eventually converge to different meta-thinking instructions, the small model selects “EMPTY”, and the large model adapts to problem difficulties at the end, showing that we need different meta-thinking instructions for different models. We hope these experiments fully address your concerns.\n\nTo eliminate confusion, we will add at the end of Section 4.2.2: “Sections 4.2.1 and 4.2.2 together demonstrate that RL with high-quality meta-thinking instructions enhances performance, and that the model subsequently selects instructions appropriate to its own capability.”\n\n### **Question 3: Replacing Meta-thinking Agent with another baseline agent**\n\nWe sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: **Multi-Agent Debate (MAD)** and **Self-refine**, to cover key concerns raised by the reviewers.\n\nFor both two experiments, we use the prompts and implementation in [2]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.\n\nWe summarize the additional baseline results below:\n\n##### Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)\n\n| Benchmark       | Qwen2.5-7B-Instruct         |               |               | LLaMA3.1-8B-Instruct      |               |               | LLaMA3-8B-Instruct        |               |               |\n|-----------------|-----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|\n|                 | **ReMA**                    | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** |\n| MATH500         | 74.40%                       | 72.60%        | **76.40%**      | **53.20%**                  | 39.20%        | 26.60%          | **33.80%**                  | 22.40%        | 24.60%          |\n| GSM8K           | 90.60%                       | 90.00%        | **91.51%**      | **87.26%**                  | 78.43%        | 56.41%          | **79.38%**                  | 66.76%        | 65.58%          |\n| AIME24          | **20.00%**                   | 11.67%        | 6.67%           | **13.33%**                  | 5.00%         | 0.00%           | **0.00%**                   | **0.00%**     | **0.00%**       |\n| AMC23           | **57.50%**                   | 45.00%        | 45.00%          | **20.00%**                  | 16.25%        | 12.50%          | **22.50%**                  | 11.25%        | 2.50%           |\n| Gaokao2023en    | 57.92%                       | 61.82%        | **62.34%**      | **37.14%**                  | 32.08%        | 27.53%          | **28.57%**                  | 20.00%        | 17.40%          |\n| Minerva_math    | 34.93%                       | **36.03%**    | 34.93%          | **28.31%**                  | 18.93%        | 13.24%          | **13.97%**                  | 11.76%        | 12.13%          |\n| OlympiadBench   | 36.30%                       | **39.41%**    | 38.96%          | **19.56%**                  | 10.37%        | 6.52%           | **8.89%**                   | 4.67%         | 4.59%           |\n| **AVG**         | **53.09%**                   | 50.93%        | 50.83%          | **36.97%**                  | 28.61%        | 20.40%          | **26.73%**                  | 19.55%        | 18.12%          |\n\n\n\nCompared to these inference-time methods, our proposed method (**ReMA**) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.\n\nWhile MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, **ReMA** leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.\n\n**Reference**\n\n[1] Feng et al., \"Group-in-group policy optimization for llm agent training,\" arXiv:2505.10978, 2025.\n\n[2] Du et al., \"Improving factuality and reasoning in language models through multi-agent debate,\" ICML2023."}, {"review_id": "ag0CPQkBSa", "reviewer": "Reviewer_ydZo", "summary": "This paper introduces Reinforced Meta-thinking Agents (ReMA), a novel framework that incorporates Multi-Agent Reinforcement Learning to enhance the reasoning capabilities of LLMs by fostering meta-thinking—the ability to monitor, evaluate, and control reasoning processes. Specifically, ReMA separates the reasoning process into two specialized agents: a high-level meta-thinking agent that provides strategic oversight and plans, and a low-level reasoning agent that executes detailed problem-solving steps. \n\nBy aligning their objectives through reinforcement learning, ReMA outperforms single-agent baselines on complex reasoning tasks like mathematical benchmarks. The framework also extends to multi-turn interactions, leveraging parameter sharing and turn-level optimization for improved efficiency. Extensive experiments and ablation studies demonstrate the effectiveness of ReMA in boosting reasoning accuracy, adaptability, and collaboration between agents, providing new insights into structured multi-agent reasoning for LLMs.", "questions": "See above", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper introduces an innovative approach to training large language models (LLMs) by explicitly decoupling strategic oversight (meta-thinking) from problem-solving (reasoning).\n2. The proposed method achieves state-of-the-art performance on challenging reasoning benchmarks, including out-of-distribution (OOD) datasets such as GSM8K and AIME24.\n3. The experimental section provides comprehensive theoretical support and includes ablation studies to dissect the dynamics and contributions of the high-level (strategic) and low-level (problem-solving) agents.\n\n**Weakness:**\n\n1. Missing important baselines: The paper does not evaluate the proposed method using baselines where ReMA is applied while keeping the MAMRP rollout fixed and freezing the weights of one agent. This additional experiment could further validate the impact of Leader-Follower game interactions and strengthen empirical claims.", "was_revised": true, "final_justification": "My concerns were adequately addressed in the rebuttal and discussion.", "rebuttal": "### **Weakness 1: Missing important baselines**\n\nWe sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: **Multi-Agent Debate (MAD)** and **Self-refine**, to cover key concerns raised by the reviewers.\n\nFor both two experiments, we use the prompts and implementation in [1]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.\n\nWe summarize the additional baseline results below:\n\n##### Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)\n\n| Benchmark       | Qwen2.5-7B-Instruct         |               |               | LLaMA3.1-8B-Instruct      |               |               | LLaMA3-8B-Instruct        |               |               |\n|-----------------|-----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|\n|                 | **ReMA**                    | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** |\n| MATH500         | 74.40%                       | 72.60%        | **76.40%**      | **53.20%**                  | 39.20%        | 26.60%          | **33.80%**                  | 22.40%        | 24.60%          |\n| GSM8K           | 90.60%                       | 90.00%        | **91.51%**      | **87.26%**                  | 78.43%        | 56.41%          | **79.38%**                  | 66.76%        | 65.58%          |\n| AIME24          | **20.00%**                   | 11.67%        | 6.67%           | **13.33%**                  | 5.00%         | 0.00%           | **0.00%**                   | **0.00%**     | **0.00%**       |\n| AMC23           | **57.50%**                   | 45.00%        | 45.00%          | **20.00%**                  | 16.25%        | 12.50%          | **22.50%**                  | 11.25%        | 2.50%           |\n| Gaokao2023en    | 57.92%                       | 61.82%        | **62.34%**      | **37.14%**                  | 32.08%        | 27.53%          | **28.57%**                  | 20.00%        | 17.40%          |\n| Minerva_math    | 34.93%                       | **36.03%**    | 34.93%          | **28.31%**                  | 18.93%        | 13.24%          | **13.97%**                  | 11.76%        | 12.13%          |\n| OlympiadBench   | 36.30%                       | **39.41%**    | 38.96%          | **19.56%**                  | 10.37%        | 6.52%           | **8.89%**                   | 4.67%         | 4.59%           |\n| **AVG**         | **53.09%**                   | 50.93%        | 50.83%          | **36.97%**                  | 28.61%        | 20.40%          | **26.73%**                  | 19.55%        | 18.12%          |\n\n\n\nCompared to these inference-time methods, our proposed method (**ReMA**) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.\n\nWhile MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, **ReMA** leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.\n\n**Reference**\n\n[1] Du et al., \"Improving factuality and reasoning in language models through multiagent debate,\" ICML2023."}, {"review_id": "F2vgjQxZKe", "reviewer": "Reviewer_uS42", "summary": "This paper introduces the ReMA framework, which uses Multi-agent Reinforcement Learning (MARL) to encourage meta-thinking in Large Language Models (LLMs). Meta-thinking allows models to not only perform reasoning but also to evaluate and control their reasoning processes. The paper proposes a two-agent system where a high-level agent handles meta-thinking (strategic oversight) and a low-level agent handles detailed execution.", "questions": "See weakness", "limitations": "See weakness", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe concept of separating meta-thinking from reasoning and executing it through two distinct agents is a novel approach. This allows for more efficient exploration and training compared to conventional methods, where a single agent is responsible for both tasks.\n\nThe paper presents thorough experiments in both single-turn and multi-turn scenarios, achieving impressive results across various benchmarks. This strongly supports the effectiveness of the proposed approach.\n\nThe framework demonstrates potential for tackling more complex, long-term reasoning tasks, thanks to its capability to function in both single-turn and multi-turn interaction settings.\n\nWeaknesses：\n\nThe application of MARL in multi-agent environments can be computationally intensive, and scalability may become an issue as the number of agents or the complexity of tasks grows. This challenge needs to be addressed to ensure the approach remains effective in larger-scale scenarios.\n\nWhile the paper compares ReMA with single-agent reinforcement learning and other baseline methods, it would be beneficial to include comparisons with existing multi-agent meta-reasoning approaches to better highlight the unique advantages of ReMA.\n\nThe paper mentions that the multi-agent approach improves exploration; could the authors provide more details on how exploration is structured in the multi-agent setting compared to single-agent models?\n\nAre there any plans to extend ReMA for multi-agent cooperation in collaborative tasks, such as cooperative problem-solving or resource-sharing scenarios?", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1: Concern about computational efficiency**\n\nWe appreciate the reviewer’s insightful comment about computational efficiency and scalability. We fully acknowledge this limitation. This has always been one of the primary challenges in MARL, and yet mainstream training frameworks such as VeRL still offer limited native support for multi‑agent RL. \nHowever, there are already increasing efforts trying to solve this issue: e.g. \n(i) parameter sharing across agents (we tried in this work) \n(ii) PD disaggregation that reuses cached states, and \n(iii) an asynchronous actor–learner pipeline. \nThese design choices allow our method to keep near‑linear training speed as the number of agents or task complexity grows. In fact, the community is likewise beginning to focus on scalable solutions, such as Multiverse[1] and SGLang[2]. And we are willing to further iterate on our codebase moving forward. We hope these additions resolve the reviewer’s concern and are grateful for the opportunity to clarify.\n\n\n### **Weakness 2: Comparisons with existing multi-agent meta-reasoning approaches**\n\nWe sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: **Multi-Agent Debate (MAD)** and **Self-refine**, to cover key concerns raised by the reviewers.\n\nFor both two experiments, we use the prompts and implementation in [3]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.\n\nWe summarize the additional baseline results below:\n\n##### Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)\n\n| Benchmark       | Qwen2.5-7B-Instruct         |               |               | LLaMA3.1-8B-Instruct      |               |               | LLaMA3-8B-Instruct        |               |               |\n|-----------------|-----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|\n|                 | **ReMA**                    | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** |\n| MATH500         | 74.40%                       | 72.60%        | **76.40%**      | **53.20%**                  | 39.20%        | 26.60%          | **33.80%**                  | 22.40%        | 24.60%          |\n| GSM8K           | 90.60%                       | 90.00%        | **91.51%**      | **87.26%**                  | 78.43%        | 56.41%          | **79.38%**                  | 66.76%        | 65.58%          |\n| AIME24          | **20.00%**                   | 11.67%        | 6.67%           | **13.33%**                  | 5.00%         | 0.00%           | **0.00%**                   | **0.00%**     | **0.00%**       |\n| AMC23           | **57.50%**                   | 45.00%        | 45.00%          | **20.00%**                  | 16.25%        | 12.50%          | **22.50%**                  | 11.25%        | 2.50%           |\n| Gaokao2023en    | 57.92%                       | 61.82%        | **62.34%**      | **37.14%**                  | 32.08%        | 27.53%          | **28.57%**                  | 20.00%        | 17.40%          |\n| Minerva_math    | 34.93%                       | **36.03%**    | 34.93%          | **28.31%**                  | 18.93%        | 13.24%          | **13.97%**                  | 11.76%        | 12.13%          |\n| OlympiadBench   | 36.30%                       | **39.41%**    | 38.96%          | **19.56%**                  | 10.37%        | 6.52%           | **8.89%**                   | 4.67%         | 4.59%           |\n| **AVG**         | **53.09%**                   | 50.93%        | 50.83%          | **36.97%**                  | 28.61%        | 20.40%          | **26.73%**                  | 19.55%        | 18.12%          |\n\n\n\nCompared to these inference-time methods, our proposed method (**ReMA**) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.\n\nWhile MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, **ReMA** leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.\n\n\n\n### **Question 1: More details on how exploration is structured in the multi-agent setting compared to single-agent models?**\n\nThank you for your helpful and constructive suggestions! Below we outline the two main mechanisms through which ReMA structures exploration more efficiently. \n(i) Hierarchical decomposition shrinks the search space: single‑agent RL must discover high‑level planning, sub‑goal tracking, reflection and correction within one long autoregressive rollout. The agent therefore explores an undifferentiated action space that mixes meta decisions with low‑level reasoning tokens, leading to sparse rewards and early convergence to sub‑optimal patterns; ReMA decouples those roles. A high‑level meta‑thinking agent proposes a compact plan, while a low‑level reasoning agent executes it. Because each agent now solves a simpler sub‑problem, the effective search space factorises; this yields more diverse trajectories and faster policy improvement, as shown in Figure 3.\n(ii) Context‑engineered workflow accelerates computation & execution: In multi-agent systems, the agents can exchange succinct messages instead of the entire history. Consequently the context window processed per forward pass is shorter, which saves the computations. Moreover, multiple agents can solve sub-tasks in parallel, which accelerates the task execution process.\n\n### **Question 2: Are there any plans to extend ReMA for multi-agent cooperation in collaborative tasks, such as cooperative problem-solving or resource-sharing scenarios?**\n\nThanks for your spot-on insight. We are actively exploring broader applications of the ReMA framework, and a promising next step is an AI‑Scientist‑style expert ensemble, in which ReMA’s meta‑agent is promoted to a principal investigator who assigns sub‑problems to a team of domain‑expert agents with different tools working cooperatively.\n\n\n**Reference**\n\n[1] Yang et al., \"Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation,\" arXiv:2506.09991v2, 2025.\n\n[2] Zheng et al., \"SGLang: Efficient Execution of Structured Language Model Programs,\" arXiv:2312.07104v2, 2023.\n\n[3] Du et al., \"Improving factuality and reasoning in language models through multi-agent debate,\" ICML2023."}, {"review_id": "9srWTJoSE3", "reviewer": "Reviewer_5jA5", "summary": "The use ideas from MARL to induce improvements in meta-thinking for LLMs.  They show results indicating that this approach is better than standard meta-thinking approaches (e.g. CoT).", "questions": "My main concern is that the authors haven't compared their method to a spectrum of state-of-the-art meta-thinking methods in the literature.  I am not enough of an expert in the domain to know what the perfectly right comparisons are, but many such methods have been proposed.     Some potential candidates are:\n\n-- Tree of Thought\n-- self-consistency - you run the prompt a bunch of times, let the model make different chains, then majority-vote the answers. (wang et al., 2022, https://arxiv.org/abs/2203.11171)\n-- least-to-most prompting - break the hard question into sub-questions, solve them in order, add the pieces together at the end. (zhou et al., 2022, https://arxiv.org/abs/2205.10625)\n-- self-refine - model drafts an answer, critiques itself, rewrites, repeat. (madaan et al., 2023, https://arxiv.org/abs/2303.17651)\n-- reflexion - llm writes little “what i learned” memos after each run and reads them next time, improving without training. (shinn et al., 2023, https://arxiv.org/abs/2303.11366)\n\nI would like to either see the authors compare to some of those methods, or some others of their choosing, or explain very clearly why they don't have to.", "limitations": "Seems ok to me -- with the exception of the limitations of their method as compared to alternatives (mentioned above in my comments).", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-- The idea of the paper is clear and interesting\n-- the benchmarks are meaningful \n-- the ablations are pretty comprehensive\n-- the results are compelling in relation to the shown baselines. \n\nWeaknesses\n-- I believe there are tons of other variants of meta-rethinking in the literature besides CoT.   It seems to me (though I am not really a very very expert person in this domain) that it would be great to compare to some of these other methods.   To me this seems like a very important point.", "was_revised": true, "final_justification": "The authors addressed some of my concerns in the rebuttal/discussion process, so I am raising my score.", "rebuttal": "### **Weakness 1: Not compared to meta-thinking baselines**\n\nWe sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: **Multi-Agent Debate (MAD)** and **Self-refine**, one as a strong multi-agent meta-reasoning baseline and the other as a strong single-agent meta-thinking baseline.\n\nFor both two experiments, we use the prompts and implementation in [1]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.\n\nWe summarize the additional baseline results below:\n\n##### Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)\n\n| Benchmark       | Qwen2.5-7B-Instruct         |               |               | LLaMA3.1-8B-Instruct      |               |               | LLaMA3-8B-Instruct        |               |               |\n|-----------------|-----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|----------------------------|---------------|-----------------|\n|                 | **ReMA**                    | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** | **ReMA**                   | **MAD**       | **Self-refine** |\n| MATH500         | 74.40%                       | 72.60%        | **76.40%**      | **53.20%**                  | 39.20%        | 26.60%          | **33.80%**                  | 22.40%        | 24.60%          |\n| GSM8K           | 90.60%                       | 90.00%        | **91.51%**      | **87.26%**                  | 78.43%        | 56.41%          | **79.38%**                  | 66.76%        | 65.58%          |\n| AIME24          | **20.00%**                   | 11.67%        | 6.67%           | **13.33%**                  | 5.00%         | 0.00%           | **0.00%**                   | **0.00%**     | **0.00%**       |\n| AMC23           | **57.50%**                   | 45.00%        | 45.00%          | **20.00%**                  | 16.25%        | 12.50%          | **22.50%**                  | 11.25%        | 2.50%           |\n| Gaokao2023en    | 57.92%                       | 61.82%        | **62.34%**      | **37.14%**                  | 32.08%        | 27.53%          | **28.57%**                  | 20.00%        | 17.40%          |\n| Minerva_math    | 34.93%                       | **36.03%**    | 34.93%          | **28.31%**                  | 18.93%        | 13.24%          | **13.97%**                  | 11.76%        | 12.13%          |\n| OlympiadBench   | 36.30%                       | **39.41%**    | 38.96%          | **19.56%**                  | 10.37%        | 6.52%           | **8.89%**                   | 4.67%         | 4.59%           |\n| **AVG**         | **53.09%**                   | 50.93%        | 50.83%          | **36.97%**                  | 28.61%        | 20.40%          | **26.73%**                  | 19.55%        | 18.12%          |\n\n\n\nCompared to these inference-time methods, our proposed method (**ReMA**) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.\n\nWhile MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, **ReMA** leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.\n\n**Reference**\n\n[1] Du et al., \"Improving factuality and reasoning in language models through multi-agent debate,\" ICML2023."}]

Author rebuttal:

Meta-review: ReMA separates meta-thinking from reasoning using two agents trained together with multi-agent RL. A high-level agent handles strategic oversight while a low-level agent carries out reasoning steps. The two coordinate through RL. The authors show improvements on reasoning-heavy benchmarks like AMC23, AIME24, GSM8K, and MATH500, and provide ablation studies and new baselines like Multi-Agent Debate and Self-Refine to highlight what the meta-thinking layer contributes.

The reviews started off mixed. Some reviewers were positive, calling the separation of meta-thinking from execution novel and appreciating the experiments and ablations. Others were skeptical about fragility, credit assignment, and whether the gains are statistically meaningful or just variance across small test sets. After rebuttal, three reviewers were satisfied, noting the added baselines and clarifications, and raised their scores to accept. One reviewer remained unconvinced, stressing that stronger credit-assignment methods and more careful comparisons are still missing.

The paper asks how to make LLMs explicitly reason about their own reasoning, and makes a concrete step toward answering it. The method is creative, the experiments are fairly comprehensive, and most reviewers ended up leaning positive after discussion. At the same time, the remaining concerns about stability, variance, and ablation depth suggest the work is not yet definitive. To me the contribution feels solid and timely enough for acceptance as a poster.
Decision: Accept (poster)