Title: ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning

Large Language Model

Multi-Agent

Reinforcement Learning
Abstract: Recent research on Reasoning of Large Language Models (LLMs) has sought to further enhance their performance by integrating meta-thinking—enabling models to monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving.
However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting in low efficacy.
To address this challenge, we introduce Reinforced Meta-thinking Agents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors, encouraging LLMs to think about thinking.
ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions.
Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading to improved generalization and robustness.
Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks.
Additionally, we further extend ReMA to multi-turn interaction settings, leveraging turn-level ratio and parameter sharing to improve efficiency.
Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta-thinking reasoning process enhances the reasoning capabilities of LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe use ideas from MARL to induce improvements in meta-thinking for LLMs.  They show results indicating that this approach is better than standard meta-thinking approaches (e.g. CoT).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-- The idea of the paper is clear and interesting\n-- the benchmarks are meaningful \n-- the ablations are pretty comprehensive\n-- the results are compelling in relation to the shown baselines. \n\nWeaknesses\n-- I believe there are tons of other variants of meta-rethinking in the literature besides CoT.   It seems to me (though I am not really a very very expert person in this domain) that it would be great to compare to some of these other methods.   To me this seems like a very important point.\n\nQUESTIONS:\nMy main concern is that the authors haven't compared their method to a spectrum of state-of-the-art meta-thinking methods in the literature.  I am not enough of an expert in the domain to know what the perfectly right comparisons are, but many such methods have been proposed.     Some potential candidates are:\n\n-- Tree of Thought\n-- self-consistency - you run the prompt a bunch of times, let the model make different chains, then majority-vote the answers. (wang et al., 2022, https://arxiv.org/abs/2203.11171)\n-- least-to-most prompting - break the hard question into sub-questions, solve them in order, add the pieces together at the end. (zhou et al., 2022, https://arxiv.org/abs/2205.10625)\n-- self-refine - model drafts an answer, critiques itself, rewrites, repeat. (madaan et al., 2023, https://arxiv.org/abs/2303.17651)\n-- reflexion - llm writes little “what i learned” memos after each run and reads them next time, improving without training. (shinn et al., 2023, https://arxiv.org/abs/2303.11366)\n\nI would like to either see the authors compare to some of those methods, or some others of their choosing, or explain very clearly why they don't have to.\n\nLIMITATIONS:\nSeems ok to me -- with the exception of the limitations of their method as compared to alternatives (mentioned above in my comments).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the ReMA framework, which uses Multi-agent Reinforcement Learning (MARL) to encourage meta-thinking in Large Language Models (LLMs). Meta-thinking allows models to not only perform reasoning but also to evaluate and control their reasoning processes. The paper proposes a two-agent system where a high-level agent handles meta-thinking (strategic oversight) and a low-level agent handles detailed execution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe concept of separating meta-thinking from reasoning and executing it through two distinct agents is a novel approach. This allows for more efficient exploration and training compared to conventional methods, where a single agent is responsible for both tasks.\n\nThe paper presents thorough experiments in both single-turn and multi-turn scenarios, achieving impressive results across various benchmarks. This strongly supports the effectiveness of the proposed approach.\n\nThe framework demonstrates potential for tackling more complex, long-term reasoning tasks, thanks to its capability to function in both single-turn and multi-turn interaction settings.\n\nWeaknesses：\n\nThe application of MARL in multi-agent environments can be computationally intensive, and scalability may become an issue as the number of agents or the complexity of tasks grows. This challenge needs to be addressed to ensure the approach remains effective in larger-scale scenarios.\n\nWhile the paper compares ReMA with single-agent reinforcement learning and other baseline methods, it would be beneficial to include comparisons with existing multi-agent meta-reasoning approaches to better highlight the unique advantages of ReMA.\n\nThe paper mentions that the multi-agent approach improves exploration; could the authors provide more details on how exploration is structured in the multi-agent setting compared to single-agent models?\n\nAre there any plans to extend ReMA for multi-agent cooperation in collaborative tasks, such as cooperative problem-solving or resource-sharing scenarios?\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reinforced Meta-thinking Agents (ReMA), a novel framework that incorporates Multi-Agent Reinforcement Learning to enhance the reasoning capabilities of LLMs by fostering meta-thinking—the ability to monitor, evaluate, and control reasoning processes. Specifically, ReMA separates the reasoning process into two specialized agents: a high-level meta-thinking agent that provides strategic oversight and plans, and a low-level reasoning agent that executes detailed problem-solving steps. \n\nBy aligning their objectives through reinforcement learning, ReMA outperforms single-agent baselines on complex reasoning tasks like mathematical benchmarks. The framework also extends to multi-turn interactions, leveraging parameter sharing and turn-level optimization for improved efficiency. Extensive experiments and ablation studies demonstrate the effectiveness of ReMA in boosting reasoning accuracy, adaptability, and collaboration between agents, providing new insights into structured multi-agent reasoning for LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper introduces an innovative approach to training large language models (LLMs) by explicitly decoupling strategic oversight (meta-thinking) from problem-solving (reasoning).\n2. The proposed method achieves state-of-the-art performance on challenging reasoning benchmarks, including out-of-distribution (OOD) datasets such as GSM8K and AIME24.\n3. The experimental section provides comprehensive theoretical support and includes ablation studies to dissect the dynamics and contributions of the high-level (strategic) and low-level (problem-solving) agents.\n\n**Weakness:**\n\n1. Missing important baselines: The paper does not evaluate the proposed method using baselines where ReMA is applied while keeping the MAMRP rollout fixed and freezing the weights of one agent. This additional experiment could further validate the impact of Leader-Follower game interactions and strengthen empirical claims.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper treats meta thinking for LLMs as a multi-agent learning problem that can be trained alongside the standard generation process with RL and leads to strong results on benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The idea of focusing on a meta-thinking agent in a multi-agent system and training end-to-end is novel and interesting.\n\n2. The method achieves good results on a set of models and benchmarks.\n\n3. The turn-level ratio clipping seems like a useful solution to prevent degenerate behaviors/collapses by treating each conversational turn as one action (not necessarily novel but an important solution in the method).\n\nWeaknesses\n1. Credit assignment remains an issue. A good meta-thought but poor reasoning steps would lead to negative rewards for both. This severely limits training efficiency. The state drift because of this is an important concern. It would be interesting to see a credit assignment technique like (https://arxiv.org/abs/2410.08115 or https://arxiv.org/pdf/2412.01928) used.\n\n2. The improvements are not very high and not necessarily statistically significant. Standard RL baselines on Qwen models have often led to higher results on benchmarks than reported here.\n\n3. The method seems quite fragile to different configurations and hyperparameters.\n\nQUESTIONS:\n1. Could the authors suggest a method of incorporating intermediate rewards/learnt values to fix the credit assignment issue and try out small scale experiments for this?\n\n2. How does ReMA compare to simply prompting a single model with meta-thinking instructions and then performing RL on the subsequent model to best use these steps?\n\n3. Could the authors try replacing the meta-thinking agent with some other baseline agent as an ablation? Is it really the meta-thinking that is helpful in increasing performance or spending more inference compute?\n\nI would be happy to increase my score once these questions are answered.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}