Title: SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied AI

Agents

Generative Reward
Abstract: Self-evolution, the ability of agents to autonomously improve their reasoning and behavior, is essential for the embodied domain with long-horizon, real-world tasks. Despite current advancements in reinforcement fine-tuning (RFT) showing strong performance in enhancing reasoning in LLMs, its potential to enable self-evolving embodied intelligence with multi-modal interactions remains largely unexplored. Specifically, reinforcement fine-tuning faces two fundamental obstacles in embodied settings: (i) the lack of accessible intermediate rewards in multi-step reasoning tasks limits effective learning signals, and (ii) reliance on hand-crafted reward functions restricts generalization to novel tasks and environments. To address these challenges, we present *Self-Evolving Embodied Agents-R1*, **SEEA-R1**, the first RFT framework designed for enabling the self-evolving capabilities of embodied agents. Specifically, to convert sparse delayed rewards into denser intermediate signals that improve multi-step reasoning, we propose Tree-based group relative policy optimization (**Tree-GRPO**) integrates Monte Carlo Tree Search into GRPO. To generalize reward estimation across tasks and scenes, supporting autonomous adaptation and reward-driven self-evolution, we further introduce Multi-modal Generative Reward Model (**MGRM**). To holistically evaluate the effectiveness of SEEA-R1, we evaluate on the ALFWorld benchmark, surpassing state-of-the-art methods with scores of 85.07\% (textual) and 46.27\% (multi-modal), outperforming prior models including GPT-4o. SEEA-R1 also achieves scores of 80.3\% (textual) and 44.03\% (multi-modal) without ground truth reward, surpassing all open-source baselines and highlighting its scalability as a self-evolving embodied agent. Additional experiments and qualitative analysis further support the potential of SEEA-R1 for future research in scalable embodied intelligence. Project page is at https://seea-r1.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework for self-evolving embodied agents that utilizes reinforcement learning to overcome challenges like sparse rewards and limited generalization in complex, long-horizon tasks. SEEA-R1 integrates two key components: Tree-GRPO, which combines Group Relative Policy Optimization with Monte Carlo Tree Search to generate dense, step-wise reward signals, and Multi-modal Generative Reward Model, a learned reward model that generalizes across various tasks and environments, reducing the reliance on handcrafted reward functions. The framework operates through an iterative loop of Data Evolution and Model Evolution. The authors demonstrate SEEA-R1's effectiveness on the ALFWorld benchmark, where it achieves state-of-the-art performance in both textual and multi-modal settings, and its generalization capabilities on the EmbodiedEval benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The framework achieves state-of-the-art success rates on the ALFWorld benchmark, outperforming previous models. Furthermore, SEEA-R1 shows encouraging generalization capabilities on the out-of-distribution EmbodiedEval benchmark, highlighting its potential for broader applicability in unseen scenarios.\n- The overall motivation for generating data and learning intermediate feedback signals as a means of solving long horizon tasks is intuitive, and the authors did a good job applying this to multi-modal settings where reasoning is harder than discrete settings. \n# Weaknesses\n- While the integration of MCTS with GRPO and the development of MGRM for embodied AI is a valuable contribution, the foundational components (GRPO and MCTS) are existing techniques. The novelty primarily lies in the engineering and successful application of these components in a novel, self-evolving framework for embodied agents, rather than the introduction of fundamentally new core reinforcement learning algorithms. This means that from a perspective of pure algorithmic invention, the paper builds upon and extends established methods rather than presenting a revolutionary new approach.\n- Despite the paper's emphasis on improving \"reasoning capability\" in embodied agents, there is a notable lack of in-depth qualitative analysis of the agent's reasoning traces. The ReAct format (Thought and Action) used in the agent's interactions provides a valuable opportunity to examine how the agent's internal thought processes evolve and become more sophisticated or coherent throughout training. Without such an analysis, the claims of enhanced reasoning are primarily supported by quantitative task success rates, leaving the specific mechanisms of reasoning improvement largely unexamined.\n- The reliance on a self-trained MGRM for reward signals, especially when ground truth rewards are absent, introduces a risk of compounding errors. If the MGRM generates inaccurate reward signals, the policy model could optimize for these flawed signals, potentially leading to a degradation of true task performance over extended training iterations, a concern not fully explored in the paper. While Figure 5 shows initial improvement, the long-term stability and fidelity of the self-supervised loop need more rigorous investigation to ensure the agent doesn't converge to a suboptimal or incorrect understanding of task completion. The paper mentions that an untrained MGRM can misguide learning, but the implications of a trained but imperfect MGRM iteratively influencing policy are not deeply analyzed. If there is a misunderstanding on my part I’m more than happy to be corrected, I think in general the description of the MGRM’s training was quite brief and led to a lot of confusion when reading.\n- There is also a lack of error bars or significance reporting in tables 1 and 2. When SEEA-R1 outperforms GPT this is claimed as a “significant” improvement, and when they underperform almost by the same amount in the text-only setting this is called “on-par\". The authors should sample multiple answers per question and report significance to adequately claim significance in these cases, and analyze the error modes in both settings when SEEA-R1 underperforms.\n\nQUESTIONS:\n- Could the authors provide a more detailed explanation of the MGRM training process? Specifically, how is the \"binary rewards\" training for \"success, continue, or failure\" translated into the reward estimation used by Tree-GRPO, particularly for the A^t,i,k​ advantage function? Is there any mechanism to \"ground\" the MGRM's estimations to the true environment state beyond its own generative process, especially during self-evolution without ground truth rewards?\n- The paper emphasizes improving \"reasoning capability\" in embodied agents. Given the \"Thought\" component in the ReAct format, could the authors provide qualitative examples of how these reasoning traces evolve throughout the training process?\n- In Tables 1 and 2, when SEEA-R1 outperforms other models, it is described as \"substantially outperforming\" or achieving \"state-of-the-art\" results, while similar performance gaps when SEEA-R1 is not leading are described as \"on par.\" Could the authors provide statistical significance tests (e.g., p-values, confidence intervals) for the reported success rates in Tables 1, 2, and 3?\n- Can the authors discuss the computational costs of the training and test-time inference process of this approach compared to others? Graphs of wall clock time or token usage at test time compared to baselines would be really insightful\n\nLIMITATIONS:\nThe computational costs and potential for compounding error with longer training have not been sufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors propose SEEA-R1, a novel agent that introduces: (i) Tree-GRPO, an extension of GRPO with Monte-Carlo Tree Search (thus allowing the agent to simulate diverse trajectories at execution time); (ii) a multi-modal generative reward model (MGRM), that leverages a MLLM to predict categorical rewards (success, continue, failure). The authors evaluate SEEA-R1 on ALFWorld (using either text or image observations) and in EmbodiedEval. The results show that their agent outperforms the selected baselines in terms of average success rate. The authors also highlight how their training algorithm outperforms a naive implementation of MCTS with DPO or SFT, as well as show promising results in using the MGRM for feedback, instead of the ground truth rewards of the environment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- Integrating MCTS with GRPO is an interesting approach and (to the best of my knowledge) novel;\n- Extensive evaluation across two different environments.\n- Authors provided the code to replicate their experiments.\n\n**Weaknesses**:\n- Unclear notation employed throughout the text and missing details (See \"Comments\" below).\n- From my understanding, the reward model MGRM (which the authors claim as a contribution) is underutilized across the evaluation. In all their main results (i.e., comparing SEEA-R1 to the baselines), the authors employ ground-truth reward values and not their reward model. The only evaluation that considers MGRM is in Section 4.2.3, in which the authors show that an agent trained on the proposed reward model significantly underperforms against an agent trained on ground truth rewards.\n\n**Comments**:\n- One fundamental issue of the current paper is the lack of coherent notation across the paper. For example, in line 130 the authors define an embodied agent as a 4-tuple, and in line 177 the agent is now a parameterized policy. Another example concerns the definition of the decision-making task: the authors claim that their work concerns partial-observable MDPs (line 136), yet also claim that the state is provided to the agent (line 130, line 170). Moreover, in line 785 the authors note that the initial state of the environment $s_0$ is provided to the agent, yet this \"state\" bears little resemblance to the current state of the environment but, instead, contains information regarding the rules of the environment, few-shot examples, etc...\n- The authors claim that they \"convert sparse outcome rewards into dense process rewards\" (line 157). I believe this is a factually wrong claim: the action-value function represents the expected (discounted) reward that the agent could achieve if, starting from the current state and performing a given action, it followed the current policy. It is not a new reward, neither a densification of the reward: it is a natural artifact of standard RL algorithms whose value is estimated from the same \"sparse\" reward.\n- Some experimental results are also unclear (see Questions below)\n- There are some experimental details missing, which could hinder replicability. Example, what is the value of $\\gamma$?\n\nTypos:\n- Line 231: \"For each iteration, the model is trained on newly collected data for one epoch as steps is 4\"\n- Line 259: ground truth (GT)rewards - missing space\n- Figure 4 and Figure 5 have a significant lower resolution than the other figures and the text.\n\nQUESTIONS:\nI am willing to increase the score if the authors are able to address my comments above and the concerns below:\n\n1) Table 1 and Table 2 refer to evaluations on the ALFWorld simulation? Which one is using text-based and image observations?\n2) In Section 4.2.1 the authors claim that their method demonstrates superior \"efficiency\" (23.37 vs 25.10 average steps for GPT-4o, line 246). Where can I see those results? What about the other baselines? The same question for the evaluation of Table 2: the authors claim that the average completion steps for SEEA-R1 is 14.75 and on par with GPT-4o (line 248), yet do not present these results. Can the authors present the results of the average duration of the episode (or \"efficiency\") of SEEA-R1 against the other baselines?\n3)  How does SEEA-R1 with MGRM compare to the baselines, in terms of performance and \"efficiency\"?\n4) Which evaluation was used for Section 4.2.3? The text-based or the image-based one? It is not clear from the text.\n5) In Section 4.2.3. the performance of the agent that employs the trained MGRM seems to plateau after 4 iterations (Figure 5). Any idea why?\n6) The generalization results in Section 4.2.2 are also very interesting. SEEA-R1 seems to perform worse that the base Qwen 2.5-VL-7B-Instruct model on tasks that I would expect it to be better (i.e., Navigation and Object Interaction), as it was trained on a household environment and in tasks involving objects and navigation. Furthermore, it performances better than the base Qwen model on tasks I would expect it to be worse (i.e., Social Interaction and SpatialQA). Any comment on why you have these results?\n7) MCTS is a great idea for simulation environments, since you can easily produce counterfactual trajectories using the simulator. How could the authors extend this work for real-world embodied agents, where no such simulator is easily available?\n8) What is the computational cost of training SEEA-R1 against the baselines? How long is the training, what type of resources are needed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SEEA-R1, the first framework to adapt Reinforcement Fine-Tuning for training self-evolving embodied agents. To address the sparse reward problem, it innovatively combines Monte Carlo Tree Search to generate dense intermediate process rewards and introduces a Multi-modal Generative Reward Model (MGRM) for reward self-evolution. The framework employs a tree-structured fine-tuning mechanism named Tree-GRPO, enabling the agent to self-learn from failures without extra human annotations. On the ALFWorld benchmark, SEEA-R1's performance significantly surpasses existing methods, such as outperforming GPT-4o with a 85.2% success rate on ALFWorld tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The method is the first to adapt the RFT technique for training self-evolving embodied agents. The authors claim it has not been previously explored. Unlike prior RFT works focused on symbolic domains (e.g., DeepSeek-R1) or other embodied AI works that rely on predefined modules, SEEA-R1 establishes a more general framework by using its learned MGRM to eliminate the need for task-specific priors.\n2. The mechanism to backtrack and try upon failure via MCTS enables effective self-correction. Tree-GRPO utilizes MCTS-generated Q-values as dense process signals, providing effective guidance for the agent's multi-step reasoning.\n3. The paper demonstrates superior performance on the ALFWorld benchmark. In the multi-modal setting, SEEA-R1 achieves a 36.19% success rate compared to GPT-4o's 24%. In the text-only setting, it achieves 85.2%, surpassing GPT-4o’s 81.1%. Experiments on EmbodiedEval also demonstrate its generalization capability to new environments.\n\n## Weaknesses\n1. The reliance on MCTS may increase computational costs with larger action spaces. The paper mentions experiments were conducted on 8 A100 GPUs, indicating high computational demand that may limit its application in more complex settings.\n2. Despite performing well in simulated environments like ALFWorld, how well the method can transfer to real-world robots tasks remains unclear, a challenge the paper itself acknowledges.\n\nQUESTIONS:\nThe comparison between a \"frozen MGRM\" and a \"training MGRM\" (Figure 5) clearly shows that while an inaccurate, static reward model leads to performance degradation, the proposed self-evolution process successfully trains the MGRM to provide reliable signals. Could the authors elaborate on the failure modes of the frozen MGRM? Did it consistently reward incorrect actions, or was the feedback merely noisy?\n\nLIMITATIONS:\n1. Although the paper emphasizes efficiency in terms of human annotations, the computational resources required for extensive fine-tuning and the tree search process could be a practical limitation.\n2. The experiments are limited to a simulated household environment (ALFWorld). The transferability and effectiveness of SEEA-R1 to real-world robots and tasks remain to be investigated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors propose SEEA-R1, an RL algorithm that combines policy rollouts with MCTS through a simulated environment to improve model performance. They test it on simulated environments and report SOTA results.

I should say that I'm not qualified to give an informed review on this paper. I don't do practical ML: I have little-to-no research experience with RL, robotics / embodied agents, realistic tasks, or fine-tuning large models. Please give featherweight to my review.

STRENGTHS AND WEAKNESSES:
The strengths of this paper include:
   - it reports SOTA results on a multimodal RL task.
   - they run some ablations.
   - the explanatory figure on the first page is nice.

The weaknesses, to my understanding, include:
   - they need a simulated env to run MCTS. not clear how this would extend to the physical world, which is less amenable to rewinds.
   - the MCTS seems pretty expensive.
   - doesn't seem particularly conceptually innovative: seems like they're mostly bolting together existing techniques.

QUESTIONS:
How would you compare your notion of self-evolution in RL to biological evolution? Why does this merit the term "evolution"?

It naively seems to me that RL agents already generate their own training data by exploring their world. Is this qualitatively different, and if so, how?

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07