SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
Authors: Wanxin Tian
Shijie Zhang
Kevin Zhang
Xiaowei Chi
Chun-Kai Fan
Junyu Lu
Yulin Luo
Qiang Zhou
Yiming Zhao
Ning Liu
Siyu Lin
Zhiyuan Qin
Xiaozhu Ju
Shanghang Zhang
Jian Tang
Venue: NeurIPS 2025 poster ()
Keywords: Embodied AI
Agents
Generative Reward
Primary area: deep_learning

Abstract: Self-evolution, the ability of agents to autonomously improve their reasoning and behavior, is essential for the embodied domain with long-horizon, real-world tasks. Despite current advancements in reinforcement fine-tuning (RFT) showing strong performance in enhancing reasoning in LLMs, its potential to enable self-evolving embodied intelligence with multi-modal interactions remains largely unexplored. Specifically, reinforcement fine-tuning faces two fundamental obstacles in embodied settings: (i) the lack of accessible intermediate rewards in multi-step reasoning tasks limits effective learning signals, and (ii) reliance on hand-crafted reward functions restricts generalization to novel tasks and environments. To address these challenges, we present *Self-Evolving Embodied Agents-R1*, **SEEA-R1**, the first RFT framework designed for enabling the self-evolving capabilities of embodied agents. Specifically, to convert sparse delayed rewards into denser intermediate signals that improve multi-step reasoning, we propose Tree-based group relative policy optimization (**Tree-GRPO**) integrates Monte Carlo Tree Search into GRPO. To generalize reward estimation across tasks and scenes, supporting autonomous adaptation and reward-driven self-evolution, we further introduce Multi-modal Generative Reward Model (**MGRM**). To holistically evaluate the effectiveness of SEEA-R1, we evaluate on the ALFWorld benchmark, surpassing state-of-the-art methods with scores of 85.07\% (textual) and 46.27\% (multi-modal), outperforming prior models including GPT-4o. SEEA-R1 also achieves scores of 80.3\% (textual) and 44.03\% (multi-modal) without ground truth reward, surpassing all open-source baselines and highlighting its scalability as a self-evolving embodied agent. Additional experiments and qualitative analysis further support the potential of SEEA-R1 for future research in scalable embodied intelligence. Project page is at https://seea-r1.github.io/.

SEEA-RI: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
Wanxin Tian ',, Shijie Zhang Kevin Zhang Xiaawei Chi², Chun-Kai Fan', Junyu Lu',
Yulin Luo", Qiang Zhou', Yiming Zhao', Ning Liu', Siyu Lin Zhiyuan Qin
Xiaozhu Ju Shanghang Zhang*1 Jian Tang"
Beijing Innovation Center Humanoid Robotics
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
Environment
Reasoning
Spooi:
Phoic
put sontal
some
SEEA-R1
Looking quickly around you see countertap, cabinet, six,, a osbimee, microoose table, drasee, bosin...
Cool
Agent
Reward Model
lino.
Data
Model
R.Resso: soue the saak, Ireed find and
Centiun
tometo, thinn heatth microwane, then put sinkk beat Tiret, lnwedto and tomann A somano more lloy oppear fidge, counteriog, dining talke ens, can start countertap
Actiont counteroo go countertop
Reason: Two tomatocs found, will pickup one and hoot with oinie cwaavo
Cantinue
Actiont ga microwor,, heat tomato microsoee.
Reason: Nend open microveve and go slwittesin
Continn,
Action: to arrk bos tomato in me bet
Dons!
Failed Nodes
Goal
Figure 1: SEEA-R1 self-evolves by reasoning over its environment with perception-groundet planning. The agent explores task solutions using tree-based search guided by a reward model, iteratively refining actions to achieve complex goals. Given high-level instruction, explores, plans, and exccutes actions in an embodicd environment.
Abstract
Self-evolution, the ability of agents autonomously improve their reasoning and behavior, is essential for the embodied domain with long- hong-horizon, real-world tasks. Despite current advancements reinforcement fine-tuning (RFT) showing strong performance in enhancing reasoning in L.LMs, its potential to enable self-evolving embodied intelligence with multi-modal interactions remains largely unexplored. Specifically, reinforcement fine-tuning faces two fundamental obstacles in embod- ied settings: (i) the lack of accessible intermediate rewards in multi-step reasoning tasks limits effective learning signals, and (ii) reliance hand-crafted reward functions restricts generalization to novel tasks and environments. To address these challenges, present Deof--vvoliing Embodied Agents-RI, SEEA-RI, the first RFT framework designed for enabling the self-evolving capabilities of embodied agents. Specifically, to convert sparse delayed rewards into denser intermediate signals that improve multi-step reasoning, we propose Tree-based Group Relative Policy Optimization (Tree-GRPO) integrates Monte Carlo Tree Search into GRPO. To generalize reward estimation across tasks and scenes, supporting unsupervised
"Equal Contribution, t Project Leader, Corresponding Authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

adaptation and reward-driven self-evolution, we further introduce the Multi- modal Generative Reward Model (MGRM). To holistically evaluate the effectiveness of SEEA-R1, 1, we evaluate on the ALFWorld benchmark, surpassing tate-of-thhe-rt methods with scores of 85.07% (textual) and 46.27% (multi-modal), outperform- ing prior models including GPT-4o. SEEA-R1 also achieves scores of 80.3% (textual) and 44.03% (multi-modal) without ground truth reward. surpassing all open-source baselines and highlighting its scalability a self-evolving embodied agent. Additional experiments and qualitative analysis further support the potential of SEEA-RI for future research in scalable embodied intelligence. Project page is at https://seea r1. www.ggigg............... github. io/
1 Introduction
Embodied agents commonly operate in complex, long-horizon environments that require not only low-level perception and motor control, but also high-level reasoning, planning, and decision-making capabilities. Despite recent advances in large language models (LLMs) [
5 and multi-modal
LLMs (MLLMs) 4 1 G have greatly improved agents' abstract reasoning and perception, extending these abilities to open-ended embodied settings remains fundamental challenge. Since embodied agents often operate interactively over long-horizon tasks, constantly perceiving visual feedback to produce actions, existing LLMs and MLLMs fall short: LLMs often lack perceptual grounding [ while MLLMs continue to struggle with structured multi-step planning, maintaining long-term coherence, and adapting to dynamic environments 6 9
To tackle general-purpose embodied intelligence, we argue that agents must acquire self- evolution-the the capacity to self-generat training signals and refine reasoning through closed-loop learning. By continuously interacting with the environment and learning from their own experiences, self-evolving embodied agents can bridge the gap between perception and cognition, generalize to novel tasks, and develop long-horizon planning capabilities that transcend the limitations of static, supervised learning paradigms.
has emerged as promising paradigm: By providing reward-basse feedback, RFT enables agents to learn from trial-and-crror trajectories and gradually refine their decision-making policies. However, bridging RFT to embodied agents introduces unique challenges that are absent in symbolic domains such as mathematics or code generation. In this work, we identify two major challenges for reinforcement fine-tuning agents in embodied environments: 1) Lack of intermediate feedback in multi-step step reasoning tasks: in embodied tasks with long horizons and delayed sparse rewards, it is difficult to assign credit to intermediate decisions, making it hard for RFT to guide policy learning effectively. 2) Poor generalization of handcrafted reward functions: existing RFT pipelines typically rely on simulator-specific or task- specific reward signals that do not generalize to novel environments, harming agents' abilities to self-improve across diverse tasks.
In this context, reinforcement fine-tuning (RFT)
To address these challenges, we introduce SEEA-RI (Self-Evolving Embodied Agents), the first framework to adopt RFT for training embodied agents capable of long-horizon reasoning and autonomous self-evolution. SEEA-R1 integrates two key components. 1) Tree-GRPO (Tree- based Group Relative Policy Optimization): we extend Group Relative Policy Optimization with Monte Carlo Tree Search (MCTS), enabling agents to explore alternative trajectories and convert sparse outcome rewards into structured, step-wise process signals. This improves credit assignment and facilitates reasoning over extended action sequences. 2) MGRM (Multi-modal Generative Reward Model): To climinate the reliance on handcrafted or environment-speccin reward functions, we introduce MGRM-a reward model trained from multi-modal, multi-turn trajectories. MGRM provides task-agnostic reward estimation, supporting generalization and self-improvement across diverse embodied scenarios. Moreover, SEEA-RI utilizes novel joint training paradigm: better MLLM reasoning enhances MGRM accuracy, while refined MGRM rewards further advance MLLM policy learning, achieving human-free free, self-evolving embodied intelligence.
We evaluate SEEA-RI on the ALFWorld benchmark, which rigorously tests an agent's planning and reasoning capabilities by requiring it to map abstract goals to visually grounded action sequences. Our proposed method achieves state-of-the-art success rates of 85.07% and 46.27% % on textual and multi-modal tasks, respectively, outperforming previous models including Qwen2.5-VL and GPT-4o.

---

To further evaluate self-evolution in realistic yet challenging setting, we replace ground-trun rewards with MGRM- supf-viue signals; SEEA-RI still achieves 80.30% and 44.03 % on textual and multi-modal tasks, respectively, surpassing all open-source baselines. These results highlight the effectiveness and scalability of SEFA-R1 I as a self-evolving embodied agent.
Our main contributions are as follows: (1) We propose SEEA-RI, the first reinforcement fine-tuning framework designed to support self-evolving capabilities in embodied agents. (2) By introducing Tree-GRPO, we augment GRPO with MCTS enable dense and interpretable credit assignment across multi-step trajectories, (3) We replace handcrafted reward signals with MGRM, multi-modal generative reward model that rewards task completion. (4) We achieve new state-of-the- art performance on the ALFWorld benchmark under both supervised and self-supervised settings, demonstrating strong planning, reasoning, adaptability, and generalization. (5) To facilitate future research and applications in the embodied intelligence community, we will open-source our full framework and modular components-includinn our reward model MGRM, and training pipelines.
2
Related Works
2.1 Self-Evolution Learning
Recent research has explored how large language and vision-language models can self-evolve through mechanisms such as self-play, reinforcement learning, and trajectory refinement [IO] Welcome to the Era of Experience [ highlights the importance of learning from experience and real-world interactions, emphasizing two key components: reward mechanism for environmental interaction and long-horizon perception-action task loop. SPC [12] utilizes an adversarial self-play environment to enhance reasoning, while rStar-Math [3] and Agent-Q [4] demonstrates that small models can improve using MCTS with reward modeling. Dreamsmooth 15] and R2I [T6 incorporate planning and prediction modeling into reward design to address long-horizon tasks. RAGEN [ a modular system to enhance LLM reasoning capabilities in multi-turn interactive environments. Building on these foundations, our approach integrates MCTS into the self-evolution loop, enabling action rollout, trajectory filtering, and policy-reward co-refinement.
introduces
2.2 Planning for Embodied Agent
LLM-PPlnner
[ LLM+P 19], and RegionFocus A0 have demonstrated that large language and lisionelanguage models can significantly enhance task performance. Planning is also critical for embodied agents to make sequential decisions and achieve long-term goals. Training-free approaches such PIVOT (21 and ECoT
22 leverage the spatial reasoning capabilities of VLMs plan effec- tively, leading to improved task success rates in embodied agents. In contrast, training- graieing-based methods like VIPER [ and MPO [(]] combine vision-language perception with LLM--ased reasoning or meta-planning, but often depend on predefined modules or expert-designed strategies. ARMAP 24] and LS-Inne [5 show that incorporating the planning module into the reward assignment process during training can significantly improve long- morizoo task performance in web search and Minecraft environments. Wang al. (20 combines tree search with preference optimization to improve state prediction and action selection, boosting embodied planning performance.
2.3 Reinforcement Fine-Tuning
shows that RL with formatting and result-only rewards can steer LLMS toward humanlike, complex chain-of-thought reasoning, boosting performance on challenging tasks.
DeepSeek-R1
Subsequently, Search-R1 3) TORL [2]] explored extending the LLM-RI approach to tool use, and
AlphaLLM-CPL 29 utilizes MCTS for self-improving preference learning. The RI approaches
have also achieved success in the visual domain. Vision-RI M.. Visual-RFT [4] and Video-RI [3]
improve visual grounding through seward-based reinforcement fine-tuning, while Vision [30 and Perception-RI
7 formalize visual input into language for structured reasoning. In the embodied
AI domain, robotxR1 [3 transforms visual information into textual descriptions to train LLMs, enhancing their robotic scenario decision-making capabilities. Embodied-R (2) further extends to spatial reasoning. However, these methods are not trained as multimodal models and still rely on task-specific priors. To address these limitations, we developed the first general embodied RL
3

---

framework SEEA-RI, which integrates a vision- gooonded reward model with MCTS-based planning enable self-supervised policy optimization without handcrafted rewards.
Sg
Inttial state Sg
Policy,
Environment
detail
Action
Observation
description: put some spray" bottle on toclet
W
Q
Reward Model
Reward Model
Input
Data Evolution
Goal
D (...r))
Updated Policy Model and Reward Model
Model Evolution
GRPO
Tree GRPO
Answer: continue
-..3
Group Rebonvve Advaraage
Group
Asswer: SUDOESS
Relative
Advantage
Compasaion
Compatiioo
Answer: failure
Ch Q.)
Reward Model
Ground Truth
Reward
Advantagg
Polley Model
Figure SEEA-R1 framework. The framework drives continuous improvement through an iterative loop of two core cycles as follows: 1.Data Evolution: The Policy Model interacts with the environment via MCTS from initial state to generate the experience dataset, containing trajectories with derived Q-values, ground truth rewards from the environment, and rewards from the current Reward Model. 2.Model Evolution: The collected data used update both models: (a) Policy Model to predict actions and (b) Reward Model to predict categorical outcomes. Refined models from Model Evolution then drive the next Data Evolution iteration, enabling continuous self-evolution.
Methods
In this section, we first formulate the Embodied Agent, the Self-Evolving Embodied Agent, and Multi-round Interaction Embodied Scenario. Then, we briefly introduce the MCTS (33 algorithm. Next, we present SEEA-R1, detailing how data evolution and model evolution are implemented. Finally, we summarize the overall self-evolution loop.
3.1 Preliminaries
Embodied Agent in our methods is defined as a 4-tuple (st, at, Tz, or) at timestep t. interacts with the environment by receiving observations o; and taking actions a, in each step, with state 8; maintaining the full interaction history, and reward ra reflecting task progress. Further details can be found in Appendix C A Self-Evolving Embodied Agent is defined satisfy three conditions: automated data synthesis, iterative capability improvement, and closed-loop reasoning system [E Multi-round Interaction Embodied Scenario is modeled
Partially Observable Markov Decision Process (POMDP) 35, B6), formalized as a 7-tuple (S, A, T, R, O 7) Here, S denotes the partially observable state space, 8, S; A is the action space, 0₁ E A; T( |st, a4) represents the transition probability; r, is the reward function, r, E Rt O is the observation space, Ot E O; 0(0++1 1 1t, az) is the observation probability; and is the discount factor.
Monte Carlo Tree Search (MCTS) 38
is heuristic algorithm for decision decision-making in large state spaces widely used planning, games, and embodied reasoning tasks. After initialization, MCTS iteratively performs four steps: Selection: Starting from the root, the agent selects child nodes with the highest UCT scores until reaching a leaf node; Expansion: The agent executes the selected action dr to obtain observation oμ, constructs
node $
{SL. (az, 01)}, and expands

---

with a set of candidate natural language actions {aL-1,1 Simulation: From SL+1, multiple rollouts are simulated until termination, giving up, or reaching the depth limit, yielding trajectories 50 + + sT: Backup: Rollout results are used to update visit counts (st, at), cumulative rewards (s,, at), and action values (( (z) along each path back to the root. Further details are provided in Appendix D
3.2 Self-Eyolving Emodied Agent-R1
As illustrated in Figure 2 SEEA-RI continuously improves the agent through two iterative cycles: Data Evolution, where the agent to interact with the environment via MCTS to generate experiential data, and Model Evolution, where both the Policy Model and Reward Model are updated using this data. The evolved models are then used for the subsequent Data Evolution, forming closed self-impprvemmnt loop. The details of each phase are provided below.
3.2.1 Data Evolution: Experience Generation via MCTS
D0 Selector
tb Expersion
Ici Sizmalties
(d) Backep
Go Gotikboon Coniaaion
Oo Gobonbaerr Uate ddeed
Go atkkhee
Ge abse.
catiea Ge abbera
Opalc
Open obieet
do caniee
calist
The spophe lol Cunc Cuocelind aboect
Tnik wene spoglstlc talls
N dp Mid
Unselected oction
Figure 3: Monte Carlo Tree Search (MCTS) in SEEA-R1, (a) Selection Traverse tree via UCT until reaching leaf. (b) Expansion Execute action, observe result, and expand with new actions. (c)
Simulation Roll out from new node to termination or depth limit, collecting reward r. (d) Backup
Propagate rewards to update action values Q using the formulation in Equation 3
To convert sparse outcome rewards into dense process rewards for multi-step reasoning, SEEA-R1 employs Monte Carlo Tree Search (MCTS) for experience generation. MCTS estimates action-values, as process rewards.
Q(st, (4.;) at.;) the expected future reward for action d,,i in state 9h-whic This provides immediate, step-by-step feedback from otherwise sparse environmental signals. The precision of these Q-values and therefore the quality of the rewards of the process improves as MCTS performs more simulations (Figure 3 exploring a wider range of trajectories. This progressive refinement of MCTS-geeeratee experiential data is how SEEA-RI implements 'Data Evolution', leading to increasingly reliable learning guidance for the agent.
3.2.2 Model Evolution: Co-refining Policy Model and Reward Model
Multi-modal Generative Reward Model (MGRM) Training. To improve the generalizability of the reward function, we adopt a learned reward model as an alternative to handcrafted reward. We design the reward model with several key properties: multi-modal for perception, multi-round for sequential interactions, and high interpretability. Specifically, our reward model is built upon Multi-man modiil Large Language Model (MLLM). It takes the historical context as input and predicts one three categorical outcomes: success, continue, or failure, whose prompt is in Appendix N.4 This allows us to leverage the DeepSeek-R 1--7ero ①1 training paradigm to employ the GRPO for reinforcement learning. supporting two training paradigms tailored to scenarios with without ground truth (GT) rewards.
5

---

With GT (Supervised Paradigm): MGRM first undergoes supervised fine-tuning (SFT) on GT- aligned trajectories (c.g,, ALFWorld task completion/step validity signals), minimizing eross-entropy loss between predicted outcomes and GT labels to achieve sufficient performance and eliminate initial biases. During co-training with the policy, MGRM continues supervised training by calibrating against GT every 500 episodes--prveening drift and keeping judgments aligned with the simulator's objective truth.
Without GT (Selt-supervised Paradigm): In GT-free scenarios (e.g., real-world environments), MGRM relies on Test test Reinforcement Learning (TTRL) 337 and GRPO: the policy generates (set 10 in environments) diverse trajectories per initial state So (via MCTS exploration), and MGRM's majority-voter predictions across these trajectories form pseudo-Gr GT for sg. MGRM is then trained via GRPO (reward = +1 for matching pseudo-GT) 0 otherwise).
Embodied Agent Policy Update through Tree-GRPO. Considering the Embodied Agent as
parameterized model xe (arls)) with parameters 6, which generates a set of available actions {ar,i ) based on current state 8t, the GRPO [ loss can be reused for this group actions each node of the tree-structured experiential data:
(0) = EE S S, (ot.i) i Rold (Alss)
lana
1
min(pLax(0)Au... cl)p(pt... clow thigh )
i-= laysl
- KL (TO( (s)) (-)r(1
(1)
molarse snade) Teay (Oc.. (.ck) A pre. Terrs-mmannfee (epnannlppr.all
where (t.i
atd((pre ((
Here, 0t,i, denotes the k-th token of the i-th action d4,3 timestep 1, and 0t,i denotes the tokens preceding token in action atii- no is the current Embodied Agent, Tp... is the old Embodied Agent that collected the experiential data, and Tref is the initial Embodied Agent. Pt.a, represents the importance sampling ratio. Á the advantage function for the k-th token of the i-th action at timestep f. Chigh and fow are from the Clih-Higher strategy DAPO [8). ID. (-o(-())) Frer(-(s;) is the k3 estimator of KL divergence between Ta and Wref. The coefficient controls its weight.
3.2.3 Iterative Self-Evolution Loop
The SEEA-RI framework iteratively alternates between Data Evolution and Model Evolution. In each cycle, the updated Embodied Agent and MGRM from the Model Evolution phase are deployed in the subsequent Data Evolution phase. This iterative evolution process allows the SEEA-R1 agent to progressively enhance its reasoning, reward understanding, and task execution capabilities over time.
Experiments
This section presents an experimental evaluation of the effectiveness of our proposed framework on embodied tasks. The section is structured as follows. We begin by outlining the experimental setup and implementation details. Then, we demonstrate our experimental results comparisons with existing baselines, followed by an in-depth analysis and ablation studies to examine the effects of different training algorithms and our self-evolution reward design.
4.1 Experimental Setup
4.1.1 Environment, Dataset and Evaluation
ALFWorld. We train and evaluate our approach on ALFWorld 39 an interactive simulation environment and a widely used benchmark for evaluating embodied agents on complex, long-horizon tasks in simulated household environments. We leverage this interactive benchmark to collect data for model training, using the rewards provided by ALFWorld as our ground truth (GT) rewards. ALFWorld provides two types of observations: textual descriptions and raw visual inputs (images),

---

Table 1: Comparison of MILM methods tasks (*-reported in previous work. This convention also used for other tables.). Success rate (0-100 scale) shown per task and overall.
Model
Parameter Size
Optimization Policy
ALFWorld Unseen
Avg
Pick
Clean
Heat
Cool
Look
Pick2
GPT-4o
22
29
27
24
Florence-2
451
0.77B
SFT SFT
Idefics-2*
85
8B
0
0
0
6
2
MiniGPT-4
0
19
17
17
16
InstructHIIP
47
SFT
26
23
6
22
Qwen2,5-VL 48
7B
SFT + DPO
21
11
22
5
11
RLAVIIM
SEEA-RI (w/o GT reward)
Tree-GRPO
60
73
58
26
44
SEEA-R1 R1 (w/ GT reward)
7B
Tree-GRPO
43
42
60
41
29
40
46
enabling evaluation in both language-only and vision-language settings. Further details on the dataset and expert trajectory collection be found in Appendix and
EmbodiedEval. To evaluate the generalization ability of SEEA-RI RI beyond the training environment, we introduce EmbodiedEval 40 as oui-t--distributtinn benchmark. EmbodicdEval tests MLLMS as embodied agents across diverse tasks, including Attribute Question Answering (AttQAA,, Spatial Question Answering (SpatialQA), Navigation, Object Interaction (Obj), and Social Interaction (Social), within 125 realistic 3D scenes. It provides a comprehensive assessment of agent capabilities in previously unseen scenarios. This setup enables us to measure generalization under significant domain shifts compared to the ALFWorld environment. We analyze the impact of different control strategies on performance, using key metrics such overall accuracy, which reflects the percentage of fully completed tasks.
Real- World. To evaluate SEEA-RI's real-world performance and analyze the sim2real gap impact on embodied planning, we conducted physical experiments using dual-arm ARX LIFT 2 with Mobile ALOHA-style teleoperation 4.. Following RoboVQA 42 protocol, agents received long-horizon instructions and visual observations (front-view camera) to generate real-time actions for corman-telloperattee and single- trajeettajectory execution. We tested task types (Pick & Place, Pick Two & Place, Clean) across 6 environments; apartment, reception room, tea room, family living room, children's room, and kitchen. Each environment contained 12 test cases.
Evaluation. To ensure experimental reproducibility, set the decoding temperature to zero. When generating task completion trajectories, we include a one-shot in-context example for each task. Detailed prompts are provided in Appendix N Our primary evaluation metric is the Average Success Rate, which calculates the mean Success Rate across all test set task instances.
4.1.2 Implementation Details
Monte Carlo Tree Search (MCTS) We uses two-level pruning reduce complexity of MCTS: Probabilistic Expansion (50% chance to expand all K actions per node, reducing redundancy) and Strict Path Budget (hard limit I=5 on full K -eexpansioos per path, constraining search space). Each MCTS runs 30 iterations with max simulation depth 30.
the base model to build embod- ied agents. Since experience learning in our framework consists of two stages trajectry sampling and model training-we define each iteration one complete cycle of these two steps. To ensure fair comparison across different training paradigms (SFT, DPO and GRPO), we use consistent sample data sizes and training hyperparameters. For each iteration, the model is trained on newly collected data for one epoch as steps is 4. The total number of iterations is not fixed and proceeds until convergence. Specifically, for Tree- model updates are performed after collecting 512 valid samples (i.e., with non-zero advantage values). For SFT, 512 trajectories with positive advantages are used. For DPO, 512 positive-negative trajectories pairs are sampled from the same parent node the MCST. When training the multimodal generative reward model (MGRM) via GRPO, the group size set to 10, which equals to the vote num using TTRL). All training adopt the same hyperparameters: cosine annealing learning rate schedule (initial LR: 1e-6, warmup ratio: 0.05), batch size of 128.
Training Setting. We the Qwen2.5-VI. --1- 48

---

and KL divergence coefficient 5 of 0.0. Experiments are conducted on 8 NVIDIA A100 80GB GPUs using the ms-swift framework [50].
4.2
Experimental Results
4.2.1 Overall Performance Comparison
and Owwn2.55-7BIInstru. as respective base models, was evaluated against strong baselines ALFWorld test set. As reported in Tables and 7 SEEA-RI was tested in two settings: vixu+ (MLLM), with image observations, and text-only (LLM), with textual descriptions. In the direct visual perception, SEEA-RI achieved a 46.27% success rate, substantially outperforming GPT-4o and other leading open-source models, and demonstrated superior efficiency (23.37 25.10 average steps for GPT-4o). Furthermore, in the text-only (LLM) setting, SEEA-RI attained an 84.3% success rate, with average completion steps (14.75) on par with GPT-4o.
Our proposed SEEA-RI method, utilizing Qwen2.5-VL-77B-1l
challenging MLLM setting requiring
4.2.2 Real World Results
User: put the clothes into the washing machine
Assistant: Thought: need clothes take them to the washing machine.
2. Assistant: Action: take cloth Observation:
3. Assistant: Action: the Observation:
image, clothes appear on the sofa on the left.
Action: the Obsenation:
A
put cloth
Thought clothes have not been placed into the washing machine, check whether the washing machine door been Action open the washing machine
put iny
on the washing machine
on the washing machine Observation:
washing machine Observation:
(Action exocution falled) Observation:
6. Assistant: Action: done
Figure 4: Visualization of SEEA-RI executing the "put clothes into the washing machine" task in real-world settings, which demonstrates refletiion-correction capability.
We evaluated SEEA-R1 via single-trajectory execution against the Siren.5-VL--B--nnsee baseline in physical real-world scenarios. As shown in Figure 4 SEEA-RI exhibits striking reflection and correction abilities: when an action fails (e.g., attempting to place clothes into a closed washing machine), it actively identifies the error, adjusts its behavior, and re-executes successfully-ae an ability fostered by our (CCTS-based self-evolution training framework. Quantitatively, Table reveals that SEEA-RI achieves 34.72% absolute improvement overall success rate across all tasks and scenarios, with particularly notable gains in complex environments like kitchens (9/12 vs. 1/12) and children's rooms (10/12 vs. 4/12). This performance leap stems from the robust feature learning enabled by MCTS during training: even without simulation-spent specific advantages (e.g., rewinds), the model generalizes
ability anticipate long-term consequences to real-world dynamics. These results confirm that our approach not only transfers effectively to physical robot settings but also empowers models with enhanced adaptive decision-nen maninn through strengthened reflection-corees mechanisms.

---

Table 2: Performance Comparison on Real- World Tasks and Scenarios
Model
Living Room
Guest Room
Apartment
Children's Room
Kitchen
Room
Overall
Owen2 VLL----------iiiii
58.3
30.0
25.0
33.3
8.3
25.0
33.3
B--A-RI-BB B (w/ GT reward)
83.3
58.3
58,3
83,3
75.0
50.0
68.1
4.2.3
Comparative Study of Training Algorithms
To validate our proposed Tree-GRPO, we compared it against MCTS combined with DPO and SFT ALFWorld. As shown in Figure 5 Tree-GRPO (blue line) consistently achieves superior task success rates (Figure 55a) and greater efficiency with fewer average steps (Figure 55)) than MCTS + DPO (purple line) and MCTS SFT (red line). This demonstrates the enhanced performance of integrating GRPO within our MCTS-driven self-evolving framework for Long-horizon embodied tasks.
4.2.4 Efficiency Analysis
Our method achieves higher success rates with lower computational cost (especially in inference token usage) via training-stage algorithmic optimization and lightweight inference design.
Search Algorithm Optimization. Standard MCTS has complexity O(T D D.K) training steps, D search tree depth, K = actions per node). We reduce this to O(T mar(D, K)) via MCTS pruning as mentioned Section
K) (where T
Inference Efficiency. MCTS is only used during training. At test time, SEEA-RI uses fast ReAct- style single-patl sampling (no MCTS overhead). As shown in Table success rate with 38.8% % fewer tokens, confirming efficient inference.
4 this achieves 3.1x higher
MCTS DPO
MC7S DPO MCT75 à STT
METS STT
Avergge-S00
Ineration
Iteration
Variation of Success Rate over Iterations
(b) Variation of Step Count over Iterations
Figure 5: Performance comparison of SEEA-R1 using different optimization algorithms on the multi-modal scenario of ALFWorld Benchmark over training iterations, more detailed figures are provided in Appendix G.3
4.2.5 Ablation Study: Self-Evolution with Different Rewards
We conducted an ablation study to demonstrate that our SEEA-RI can achieve sustained performance improvement by leveraging its internally trained Multi-Moda. Generative Reward Model (MGRM), thereby reducing reliance on ground truth rewards from the simulator. We compared three critical configurations with results shown in Figure 6
The key finding from Figure in supervised and self-supervised paradigms), can effectively learn and enhance its policy. In the supervised paradigm (Figure a()),, the supervised MGRM enables steady performance that approaches the upper bound set by GT rewards. In the self-supervised paradigm, the self-supervised MGRM drives continuous success rate improvements, even outperforming later iterations. This highlights the MGRM's capability to provide reliable internal reward signals, enabling robust self-evolution and continuous performance gains without persistent dependence on external GT environment rewards.
that SEEA-RI, when utilizing its self-trained MGRM (both
GT reward baseline in

---

(TAME)) Iupertoud NORM)
kanch coueiid EP7-43 10.00
0.30
0.59
0.5
Iteracion
Iterot om
(2) Comparison for Supervised Paradigm
[b] Comparison for Self-Supervised Paradigm
Figure 6: Comparison success rates SEEA-RI under different reward and training paradigms. (a) Analyzes the supervised paradigm, contrasting performance with GT reward, frozen MGRM, and supervised MGRM. (b) Examines the self-supervised paradigm, comparing GT reward, self- supervised MGRM, and the GPT-4o baseline.
4.2.6 Generalization on Out-of-Distribution Embodied Benchmark
To assess the generalization capabilities of proposed SEEA-RI, we conducted evaluations on the EmbodiedEval benchmark, an out-o--distribution (OOD) multi-modal embodied simulation set. The success rates are reported in Table B Notably, our SEEA-R1 method, when fine-tuned with Tree-GRPO, demonstrates significantly improved performance after training. It achieves an overall success rate of 19.88%, which not only shows an uplift compared to the base model (Qwen2.5- VL-7B-Instruct at 18. (8.29%) but also markedly outperforms models fine-tuned with SFT (16.77%) and DPO (16. (1).7%)) These results underscore the strong generalization performance of Tree-GRRP approach on this challenging, unseen benchmark.
Table 3: Success Rate in EmbodiedEval benchmark. Tasks include Attribute Question Answering (AttrQA), Spatial Question Answering (SpatialQA), Navigation, Object Interaction (Obj), and Social Interaction (Social).
Model
AttrQA
Navigation
Obj
Social
SpatialQA
Overall
GPT-4o 45
35.79
31.03
10.11
11.76
32.69
25.00
Vision-RI-1
7B
5
8.54
Occan-RR R1
51
15.85
34.74
18.97
5.62
0.00
21.15
18.29
Owen-VI.-Max*
12
37.89
24.14
24.91
8.82
17.31
21.04
LLaVA-NEXT-72B
23.16
12.07
0.00
5.77
10.67
LLaVA-OnneVisionn-225
53
26.32
10.34
1.12
0.00
19.23
12.80
Qwen2.5-VL-7B- MCTS SFT
28.42
20.69
4.49
$.88
19.23
16.77
Dwen2.5-VL-7B- MCTS + DPO
27.37
17.24
5.62
5.88
23.08
16.77
SEEA-R1 (w/ reward)
30.85
21.05
6.25
32.69
19.88
Conclusion
We introduced SEEA-RI, the first framework to adapt reinforcement fine-tuning (RFT) for training self-evolving embodied agents. To address the challenges of sparse rewards and limited generaliza- tion for embodied domains, SEEA-R1 integrates Tree-GRPO, which leverages MCTS densify reward signals, and MGRM, a multi-modal reward model that generalizes across tasks and environ- ments. SEEA-RI achieves new ffate-of-the-as performance ALFWorld, outperforming previous models, including GPT-4o, and demonstrates strong generalization in self-learning settings. These results demonstrate the potential of SEEA-R1 scalable framework for embodied reasoning and autonomous learning. Further discussions on the limitations of this work are in Appendix
10

---

6 Acknowledgments
This work was supported by the National Natural Science Foundation of China (6247601 1).
References
[1] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501.122
2025.
[2] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. meepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0333 03300, 2024.
[3] Bowen Jin, Hansi Zeng, Zhenrui Yue. Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.00 099166,,,,,,, 2025.
[4] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine- fine-tuning. arXiv preprint arXiv:2503.00 00108588 2025.
[5] Yufci Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-free camma--freaaae alignment in large vision-language models via vision-guided reinforcement learning. arXiv preprint arxvv:2503. /0013, 2025.
[6] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.0674 2025.
[7] En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Pereeption-rl: Pioneering perception policy with reinforcement learning. arXiv preprint ar Xiv:2504..000 079544, 2025.
[8] Wenqi Zhang, Mengna Wang. Gangao Liu, Xu Huixin. Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, Weiming Lu, Peng Li, and Yueting Zhuang. Embodied-rcasoner: Synergizing visual scarch, reasoning, and action embodied interactive tasks, 2025.
[9] Mohamed Salim Aissi, Clemence Grislain, Mohamed Chetouani, Olivier Sigaud, Laure Soulier, and Nicolas Thome Viper: Visual perception and explainable reasoning for sequential decision-making, arXiv e-prints, pages arXiv-2503, 2025.
[10] Zhiyuan Xu, Kun Wa, Junjie Wen, Jinming Li. Ning Liu, Zhengping Che, and Jian Tang. A survey on robotics with foundation models: toward embodied ai. arXiv preprint 2XXi:24022.03885, 2024.
[11] David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
[12] Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan- Yee K. Wong. Spe: Evolving self-play critic via adversarial games for Ilm reasoning, 2025.
[13] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.045/1 2025.
[14] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024.
[15] Vint Lee, Pieter Abbeel, and Youngwoon Lee. Dreamsmooth: Improving model-based reinforcement learning via reward smoothing. arXiv preprint 1.04422311.01450. 2023.
[16] Mohammad Reza Samsami, Artem Zholus, Janarthaman Rajendran, and Sarath Chandar. Mastering memory tasks with world models. arXiv preprint arXiv:2403.04255 2024.
[17] Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang. Kefan Yu. Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, arXiv preprint 204ivv2554..2007, 2025.
[18] Chan Hee Song. Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm pla-nelneer: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
11

---

[19] Bo Liu, Yugian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 114777 2023.
[20] Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee. agent grounding. 2025.
Visual test-time scaling for gui
[21] Soroush Nasiriany, Fci Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annic Xic, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong. Tingnan Zhang, Tsang-We Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian lchtee. Pivot: lterative visual prompting elicits actionable knowledge for vlms, 2024.
[22] Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied ohain-of-thought reasoning. arXiv preprint wxXi:2407.08622 2024.
[23] Weimin Xiong, Yifan Song, Qingxiu Dong, Bingchan Zhao, Feifan Song, Xun Wang, and Sujian Li. Mpo: Boosting Ilm agents with meta plan optimization. arXiv preprint wrXiv25533.0668 2025.
[24] Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, and Chuang Gan. Scaling autonomous agents via automatic reward modeling and planning. arXiv preprint arXiv:2502. /2130, 2025.
[25] Jiajian Li, Qi Wang, Yumbo Wang, Xin Jin, Yang Li, Wenjun Zeng, and Xiaokang Yang. Open-world reinforcement learning over long short-term imagination. arXiv preprint arXiv:2410.036188 2024.
[26] Siyin Wang. Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang. Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes better planner: Dual preference optimization for embodied task planning. In Anmual Meeting of the Association for Computational Linguistics, 2025.
[27] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanucl Candès, and Tatsunori Hashimoto. Simple test-time scaling, 2025.
[28] Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2503.23383, 2025.
Torl:
Scaling tool-integrated rl. arXiv preprint
[29] Xiyao Wang, Linfeng Song, Ye Tian, Dian Yu, Baolin Peng, Haitao Mi, Furong Huang, and Dong Yu. Towards seffiimprovement of Ilms via mets: Leveraging stepwise knowledge with curriculum preference lcarning, 2024.
[30] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin. Fengyun Rao, Minfeng Zhu, et al. Ro-oviviiion: Advancing generalized multimodal reasoning through cross modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[31] Liam Boyle. Nicolas Baumann, Paviththiren Sivasothilingam. Michele Magno, and Luca Benini. Robotxrl: Enabling embodied robotic intelligence large language models through closed-loop reinforcement lcarning. arXiv preprint arXiv:2505. 250.0032.... 2025.
[32] Baining Zhao, Ziyou Wang. Jianjie Fang, Chen Gao, Fanhang Man, Jingiang Cui, Xin Wang, Xinlei Chen, Yong Li, and Wenwu Zhu. Embodied-r: Collaborative framework for activating embodied spatial reasoning in foundation models via reinforcement learning. arXiv preprint 2xXv:2504.112600 2025.
[33] Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and gamex, pages 72-83. Springer, 2006.
[34] Tao He, Hao Li, Jingehang Chen, Runxuan Liu, Yixin Cao, Lizi Liao, Zihao Zheng, Zheng Chu, Jiafeng Liang, Ming Liu, al. A survey on complex reasoning of large language models through the lens of self-evolution, 2025.
[35] Alvin W Drake, Observation of a Markov process through noisy channel. PhD thesis, Massachusetts Institute of Technology, 1962.
[36] Karl Johan Åström. Optimal control of markov processes with incomplete state information Journal of mathematical analysis and applications, 10:174-205, 1965.
[37] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng. Xuekai Zhu, Yuchen Zhang, Biqing Qi, Youbang Sun, Gangu Cui, Ning Ding, and Bowen Zhou, Tirl: Test-tim time reinforcement learning. ArXiv, abs/2504. 16084, 2025.
12

---

[38] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue. Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale, arXiv preprint arXiv:2503. 14476, 2025.
[39] Mohit Shridhar, Xingdi Yuan, Mare-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.0 2020.
[40] Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, al. Embodiedeval: Evaluate multimodal Ilms as embodied agents. arXiv preprint arXiv:2501.1185, 2025.
[41] Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. ArXiv, abs/2401 022117, 2024.
[42] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan. Christine Chan, Gabriel Dulac-Amold, Sharath Maddineni, Nikhil Joshi, Pete Florence, Wei Han, Robert Baruch, Yao Lu, Suvir Mirchandani, Peng Xu, Pannag R. Sanketi, Karol Hausman, Izhak Shafran, Brian Tchter, and Yuan Cao. Robovqa: Multimodal long-horizon reasoning for rohotics. 2024 IEFE International Conference on Robotics and Automation (ICRA), pages 645-652, 2023.
[43] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2333::3303.08774 2023.
[44] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4818 4829,, 2024.
[45] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? Advances in Neural Information Processing Systems, 37:87874-87907, 2024.
[46] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023.
[47] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs instruction tuning. 2023.
models with
[48] Qwen Team. Qwen2.5-vl. Jamuary 2025.
[49] Simon Zhai, Hao Bai. Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision- languag models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:110935-11097 2024.
[50] Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning, 2024.
[51] Lingfeng Ming, Yadong Li, Song Chen, Jianhua Xu, Zenan Zhou, and Weipeng Chen. Occan-rl: An open and generalizable large vision-language model enhanced by reinforcement learning. httpps://github com/VLX- RL/ DeeeennnnBB R1 2025.
[52] Bai, S Bai, S Yang. S Wang, S Tan, P Wang. Lin, C Zhou, and J Qwen- VL. Zhou. A versatile vision- language model for understanding, localization, text reading, and beyond. arXiv preprint 23X0 12966, 2023.
[53] Y Zhang, B Li, H Liu, Y Lee, L Gui, D Fu, Feng, Z Liu, and C Li. Llava-next: A strong zero-shot video understanding model. 2024.
[54] Shunyu Yao. Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
13

---

[55] Qwen An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong. Haoran Wei. Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mci Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.15115, 2024.
[56] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint MxXi::247.22188. 2024.
[57] Aohan Zeng, Mingdao I Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:23 12823, 2023.
[58] Yifan Song, Da Yin, Xiang Yue, Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization for lIm agents. arXiv preprint arXiv:2403.02502, 2024.
[59] Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. Knowagent: Knowledge-augmentt planning for Ilm-based agents. arXiv preprint arx::: 240::2403.33011 2024.
[60] Shuofei Qiao, Runnan Fang, Ningyu Zhang. Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Advances in Neural Information Processing Systems, 37:1 11841483-11 2024.
[61] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings the ACM SIGOPS 29th Symposium Operating Systems Principlex, 2023.
14

---

A Appendix Overview
This appendix provides detailed supplementary material complement our main paper. is organized as follows:
Appendix B Algorithm Details
We outline the overall training procedure of our Self-Evolving Embodied Agent (SEEA-RI), as detailed in Algorithm !
Appendix C Embodied Agent Formulation
We describe the formalization of the embodied agent's interaction with the environment as Partially Observable Markov Decision Process (POMDP), detailing the observation, action, state, and reward components.
Appendix Dj MCTS Implementation
We elaborate on the Monte Carlo Tree Search (MCTS) mechanism, core component of our SEEA-R RI, covering its initialization, selection, expansion, simulation, and backup phases.
Appendix D Exploration of World Model Environmental Feedback
We present our findings from exploring the use of a state-of-the-art world model (Kling) as environmental feedback the self-evolving agent, discussing the challenges encountered such as visual hallucinations, limited instruction understanding, and generation inefficiency.
Appendix F Dataset Details
We describe the ALFWorld dataset used in experiments, including its structure, eval- uation splits, and the rationale behind focusing on the test-unseen setting to assess out-of- distribution generalization.
Appendix G Extended Experimental Results
This section presents comprehensive experimental results, including:
Evaluation the General Multimodal Benchmark (MMBench) with detailed break- down of reasoning and perception capabilities.
Comparative performance across diverse multimodal benchmarks (MMStar, MMMU, HallusionBench, AI2D, and OCRBench).
An ablation study investigating the impact of sample size and batch size on training stability and success rate.
An algorithmic comparison demonstrating the superiority of our proposed Tree-GRPO over baselines like MCTS+DPO and MCTS+SFT.
Analysis of the long-term performance of Tree-GRPO over extended training iterations.
Investigation into the impact of iterative self-evolution on MCTS performance and generated data quality, including evaluations using Supervised Fine-Tuning (SFT).
Appendix H Case Studies on ALFWorld Tasks
We present representative execution trajectories across various ALFWorld task categories (e.g., mick-and-place look, clean, heat, cool) to highlight the SEEA-RI I agent's 's capabilities. Visualizations of policy and reward model interactions are also included.
Appendix I Qualitative Analysis of Reasoning Traces
We analyze the evolution of reasoning capabilities by comparing the ReAct "Thought- Observation" traces of the baseline (Qwen-2.5- -iren-2.5-V.. 1---L--B--nnsteeee
and SEEA-R1 (Iterations 5/10), focusing on the task "find two CDs and put them in the safe".
Appendix J Detailed Real-World Tasks
We detail all of tasks for each scenario in the real-world experiment.
Appendix K Reward Model Accuracy Analysis
We compare the task state judgment accuracy of the frozen MGRM (untrained Qwen-2.5- VL--BB-nnstruc and the -RrPner MGRM across "Success/Cominue/Pllle"" states,
with supporting data in Table &
Appendix[[: Training Cost and Efficiency Analysis
We specify the experimental hardware (8 NVIDIA A100 80GB GPUs) and frameworks (MS-Swift for training, vLLM for inference), quantify SEEA-R1'S time costs (sampling, policy training, total) under "with/without GT reward" configurations, and analyze efficiency trade-offs, with details in Table I0
15

---

Appendix M; Limitations
We discuss the limitations of SEEA-RI: like current embodied AI, fails fully address complexities in highly dynamic and unpredictable real-world environments (a major field- wide challenge). We also propose future directions, such as scaling experiments with more diverse environments, larger sample sizes, and bigger models to enhance its adaptability to dynamic settings.
Appendix Prompts Used
We provide the full prompts used for both policy and reward models across different datasets (ALFWorld and EmbodiedEval), ensuring the reproducibility of instruction-tuning pipeline.
B Algorithm Details
The overall training procedure of the Self-Evolving Embodied Agent is as follows (Algorithm) D
Algorithm 1: Self-Evolving Framework Training Loop
Initialize: Embodied Agent Mori f- Tref, MCTS parameters, Tree-GRPO hyperparameters
2: for iteration 7 = 1, 2, do
3: Data Evolution: Experience Generation via MCTS
4: Collect tree-structured experience data Da = 0.
5: for each episode do
6:
Generate MCTS-tree using the current agent Moua
7: Extract triple set (s,, Prr,, ( from MCTS-tree tree, and add to De.ta
8:
end for
9:
I/ Model Evolution: Co-refining Policy Model and Reward Model
Update reward model parameters by using the GRPO algorithm using Doar::
11:
Update agent parameters 0 by optimizing the Tree-GRPO objective [(f) using Donas
12:
< Grro-GGPO_pp
Donss I Tyof)
13:
Oold 0
14:
end for
15: Return: Optimized Embodied Agent
C Embodied Agent Formulation
We model the long-horizon decision-making. multi-turn interaction embodied scenario Partially Observable Markov Decision Process (POMDP), formalized as 7-tuple (S, (S,A, T, R, O 7). Here, denotes the partially observable state space of the environment; A is the action space; T(8++1 18t 0x) represents the conditional transition probability; R(st, a,) is the reward function or reward model; O is the observation space; [( st, ae) is the conditional observation probability; and ? the discount factor. The main components of the Embodied Agent are described below:
Embodied Agent Observation 0 O is the observation received from the environment timestep t after the agent takes action a, in state St. In pure based scenarios, the observation is a natural language string, e.g., "Observation: The drawer is closed." ". In multimodal scenarios, the observation is an image representing the visual content of the environment from the current perspective.
Embodied Agent Action a A is the action taken by the agent to interact with the envi- ronment. It is natural language string generated by the agent, following the ReAct [54] for- mat, e.g., "Thought: The drawer is closed, so I need to open it first to look inside. AnAction: open drawer 1" or "Action: open drawer 1". The "Thought" part represents the model's reasoning process, while the "Action" part is the specific command executed in the environment. The environment executes the action a generated by the Embodied Agent and returns an observation o,, forming an interaction loop.
Embodied Agent State 8, E S is the partially observable environment state, derived from the agent's historical exploration. includes the entire history of interactions from the beginning of the task up to the current timestep, formalized as s; {so, (0o, oo), (a1, o1)...., (ot-..,,,,,. (0t-1, 0.-1)) The
16

---

complete context carries clues about the environment state and preserves the reasoning trajectory, enabling the Embodied Agent to maintain task context memory and perform multi-ture reasoning and actions in partially observable scenarios. In ALFWorld, state & includes initial state so (system prompt describing environment rules and background; user prompt describing the current task goal and providing few-shot examples) and alternating actions Or and observations Or (agent-generated thoughts and actions, and environment-retumee observations).
Embodied Agent Reward Tt E Rt is the reward received by the agent at timestep f for taking action ag in state st. can be provided by reward model or predefined reward function.
D MCTS
MCTS is a crucial component of SEEA-RI, comprising four core steps: selection, expansion, simulation, and backup.
Initialization: The initial state So is set as the root node of the search tree.
Selection:
Starting from the root node, the Embodied Agent traverses down the tree. At state 8,, it selects the child node with the maximum Upper Confidence Bound for Trees (UCT) value:
[Q
In 11 Q(51, ar,)) (Q(snaa)) + 1+ N(t,, 0r,5)
0g = arg max dr,t
where e the exploration constant, (stt, :e,s)) is the action value, ((s,, 0r.z) is the visit count for the action, and Oz,; is the i-th available action at the current timestep I.
Expansion: After continuous selection reaches leaf node (s1, the agent first executes this action to obtain an observation or from the environment. Then, it creates a new non-leaf node SL+1 (s,, (aL, o1)) Finally, it simultaneously samples multiple natural language text segments to expand the new non-leaf node 81+1, obtaining a set available actions {o1+1,))._
From the new leaf node 1, the Embodied Agent performs multiple rollouts until termination-either the task is completed, the agent gives up, or the maximum search depth reached.
Simulation:
The complete trajectory of the j-th rollout is formalized as - s1 + + S + L
(5)
Backup: The results of multiple rollouts are used to update the visit counts ((s 0z), returns RDO or), and action values Q(s,, az) along the trajectory, starting from the terminal node and moving upwards. The update formulas are as follows:
N(s a.) +- N(s,, 0z) +
(3)
RG) az) = -(((())) + 77((5122) (5) + .... +....777 -----1--( r(s
(4)
Ex M(s,as) RV) (8t, ut)
Q(si, = Ba (A1a)) R()s
(5)
N(s
D.1 Training and Efficiency Analysis
Training Cost. SEEA-RI was trained for hours on 8x GPUs, a modest budget considering its performance surpasses proprietary models such GPT-4o. For comparison, Qwen2.5 fine-tuning required hours and PPO training took 48 hours under similar hardware settings.
Algorithmic Optimization. Standard MCTS scales ((1)) with branching factor b and depth d. To improve scalability, SEEA-RI introduces a pruning strategy combining probabilistic expansion and path budget, effectively reducing complexity to O OpLk). Each node expands its & actions with 50% probability, and full k-expansions are limited to L = 5 per path.
Inference Efficiency. MCTS is used only during training. At inference time, SEEA-R1 performs lightweight single-path (ReAct-style) reasoning without search overhead, yielding 3.1 x higher success while using 38.8% fewer tokens.
17

---

1. Hallucination World Model
Tasl Pot see apple and purtiion the blon paats
Pln 6.66
Misinterpretation instruction understanding
Tasl: Alve the pore cup from theAlleday
Pack
O Fist O
"elerr becoora
afher rsveried
Tearth
Figure Pipeline of the experiment for the World Model as environmental feedback in Self-evolving Agent
Table 4: Inference Efficiency of SEEA-RI on ALFWorld.
Model
Success (%)
Avg. Steps
Total Tokens (Test)
Dwen2.5-VL-77
11.57
28.10
10,701,255 6,554,090
SEEA-R1
36.19
23,37
E Exploration of World Model as environmental feedback in Self-evolving Agent
In a self-evolving agent, environmental feedback is critical and can take many forms -uch as simulated environment, world model, or the real world. Recent research increasingly trains world models on large-scale data so that they can emulate reality. To test whether updated world models are sufficiently capable of serving as environmental feedback, we intend to train a self-evolving agent not only in a simulator but also within world-mode environment. For short, we use kling, sota world model, as a world simulator in our experiment setting.
During this data-collection phase we found that when the agent relies on a world model for feedback, obtaining single accurate and complete trajectory is extremely difficult. This suggests that current world models are still far from being fully reliable world simulators and have a long way to go.
We have identified the following aspects that require improvement in world models:
Frequent visual hallucinations in predicted videos. Hallucinations in the video-based world model are clichéd problems, and there have been many previous studies on how to solve them. In our data collection phase, even when we supply detailed prompts- and likewise with simpler prompts-to the sota world model, we often encounter hallucinations. Typical failures include physically impossible dynamics, objects that suddenly warp or vanish, or the spontaneous appearance of extra robot arms and other artifacts. Repeated tests confirm that the current generation of world models cannot reliably prevent such hallucinations; therefore, the resulting videos frequently fail to meet our requirements and make the model unsuitable dependable world simulator.
Limited understanding of task instructions. Misinterpreting an instruction is distinct from hallucination: the video looks coherent, yet it fails to follow the given command. We observe two main failure modes. (i) Generalisation gaps: when the model encounters an unfamiliar instruction or action, it often produces static scene or executes the wrong motionn-without hallucinating, but still violating the instruction.
(ii) Planning errors: if the upstream planner issues physically unsound command (e.g., "place the plate the very edge of the table"),, faithful simulator should show the plate toppling. Today's 's world models cannot reproduce such plausible but incorrect plans, revealing a deeper weakness in instruction comprehension.
18

---

3. Inefficient video generation. Large-scale scale visual diffusion models typically require 10-30 minutes to produce 5-10-second clip. This lengthy turnaoound severely slows data collection; if the output is unusable, we must restart the process and wait again. Consequently, training a self-learning agent with a world model demands substantially more time and compute than training in a conventional simulator.
across Tadks
Al Aicrtgg Staps accoss Tasks
35-
05
21
02
Per
01.
03
0.1
0.0
Gaol
Olus
Cail
Cbon
Figure 8: Performance comparison of SEEA-R1 across different tasks in ALFWorld over training iterations. Left: Success rate across tasks. Right: Average number of steps taken to complete tasks.
F Dataset Details
The ALFWorld dataset structured into a training set comprising 3321 games and a test set, further partitioned into test-seen (140 games) and test-unseen (134 games) splits. This distinction crucial for assessing out-of-distribution (OOD) generalization, as unseen tasks introduce novel environments and instructions. For our experiments, we specifically evaluate our model on the jest-unseen split. This focus on unseen scenarios is motivated by the need to rigorously assess the agent's 's capacity for effective long-horizog planning and generalization to novel embodied tasks, which critical for real-world deployment. ALFWorld's embodied setup, sparse rewards, and requirement for OOD generalization collectively make it a rigorous benchmark for evaluating agents intended for complex real-world task-solving.
ALFWorld:
G Experiments
G.1 Evaluation on the Genenal Multi-Modal Benchmark
MMBench is a multimodal benchmark that subdivides reasoning and perception capabilities into six Level-2 dimensions: Logic Reasoning (LR), Attribute Reasoning (AR), Relation Reasoning (RR) for Reasoning, and Fine-Grained Perception-Single Single Instance (FP-S), Fine-Grained Perception-Cross Instance (FP-C), and Coarse Perception (CP) for Perception. The overall average score is derived from the average of scores for all 20 Level-3 capabilities, and Level-2 capability dimension scores the average of all Level-3 capability scores within that dimension. Ground truth labels for the test set are not publicly available. The evaluation results are reported in Table 51
To comprehensively evaluate the capabilities of Multimodal Large Language Models (MM-LLMs), we report performance across a suite of diverse benchmarks in Table 6 each targeting distinct aspects of multimodal intelligence. MMStar assesses general multimodal reasoning and understanding.
MMMU (Massive Multi- Multi-discipline Multimodal Understanding) focuses on complex, expert-level multimodal reasoning across various academic disciplines. HallusionBench specifically targets the detection and mitigation of hallucinations in MM-LLMs, particularly when dealing with potentially misleading visual information. A12D (A12 Diagrams) evaluates the models' ability to understand and reason about scientific diagrams, involving both visual perception and logical inference. Lastly, OCR Bench measures the models' optical character recognition capabilities, evaluating their effective- ness in extracting and understanding text embedded within images. This comprehensive evaluation
19

---

Table Performance comparison of various Multimodal Large Language Models on the MMBench benchmark.
Model
Overall
LR
AR
RR
FP-S
FP-C
CP
Ge5-ro-2.5-Pro
88.5
89.1
87.8
83.7
94.2
94
83.7
GPT-4o
86.8
87.5
91.4
83.3
84.9
91
85.4
Intern VL3-8B
82.3
71.2
84.1
82.9
88.2
81.3
80.9
Qwen2.5-VI.-7B
82.4
71.7
84.9
80.2
89.8
80.1
81.3
AlaVA-OneVision-7B
80.9
67.9
88.2
82.2
84.9
73.8
81.9
SEEA-R1
81.4
66.9
82.9
81.0
90.3
75.4
81.7
45
40
35
Success Rate (%)
2036
20
15-
142
Sample-1022 1024, Batch-512
Sample=512, Batch-128
Sample- 28, Batch=32
(o
Iteration
Figure Performance comparison across different sample and batch sizes.
suite provides holistic view of each model's strengths and limitations in real-world multimodal
applications.
G.2 Impact of Sample Size and Batch Size on Training Stability
We investigate how training configurations affect model performance. As illustrated Figure g larger sample and batch sizes significantly influence the final success rate on the ALFWorld unseen test set. Specifically, the largest configuration 1004 Batch=512) consistently demonstrates clear performance advantage, achieving a final success rate 4..9%. In stark contrast, while the medium setting (Sample=512, Batch=128 128) reached a peak success rate of 32.46% during its intermediate iterations, and the smallest configuration (Sample=128, Batch=32) similarly achieved its highest accuracy of 25.75% at earlier iteration, their overall final performance was notably lower largest configuration. These results collectively
and exhibited less stable convergence compared indicate that increasing the sample and batch sizes contributes to more stable and effective policy updates GRPO-based training, likely due to reduced variance in gradient estimation, ultimately leading to superior and more consistent performance unseen tasks.
20

---

Table Performance comparison of various Multimodal Large Language Models on diverse multi- modal benchmarks.
Model
MMStar
MMMU
HallusionBench
A12D
OCRBench
Gemini-2.55 Pro
73.6
74.7
64.1
89.5
862
GPT-4o
70.2
72.9
57
86.3
822
CenseNova-66-Pa
73.7
70.4
67.1
89.2
895
Intern V1.3-8B
68.7
62.2
49
85.1
884
Qwen2.5-VI-77
64.1
58
51.9
84.3
888
VLaVAA-OneVisionn-77
56.7
46.8
47.5
82.8
697
SEEA-RI
60.2
48.5
66.4
79.4
766
TTC--GRPOMMCTS-GRRFO)
35
DCTS+DPO
MCTS+SFT
Successs Rate (%)
25
ADN
18-
10
Iteration
Figure 10: Performance comparison of Tree-GRPO against MCTS+DPO and MCTS+SFT across
training iterations.
G.3 Comparison with Different Algorithms
We investigate the comparative performance of Tree-GRPO against two established baseline methods: MCTS integrated with DPO (MCTS+DPO) and MCTS with SFT (MCTS+SFT), As illustrated in Figure TO, Tree-GRPO consistently outperforms both baselines across training iterations the ALFWorld unseen test set. Tree-GRPO reached peak success rate of 32.46%, which is significantly higher than the 25.37% peak achieved by MCTS+DPO and the 16.79 16.79% peak of MCTS+SFT.
This superior performance can be largely attributed GRPO's on-policy nature, which facilitates more accurate and stable policy improvement through direct, interaction-aligne updates. In con- trast, DPO is fundamentally an off off-policy algorithm; it shares practical limitations with SFT, as both methods heavily rely on static, pre-collected data and lack the dynamic optimization benefits derived from online rollouts. Consequently, the off-policy characteristic of DPO contributes to its observed performance ceiling, while the stagnation and eventual degradation in the performance of MCTS+SFT clearly reflect the inherent limitations of purely supervised learning in complex interactive environments. The on-policy nature of GRPO, especially when combined with tree-based
21

---

40
35
Success
10-
10
15
20
25
30
Iteration
Figure 11: Learning curve of Tree-GRPO over 30 training iterations.
exploration, enables more stable and effective updates, leading to superior sample efficiency and enhanced robustness in reinforcement learning.
G.4 Long-Term Performance of Tree-GRPO
To thoroughly assess the long-term effectiveness of GRee- we extended the training duration of the model 30 iterations. As illustrated in Figure II the success rate on the ALFWorld unseen test set demonstrates a consistent and substantial increase, rising from an initial 11.57% to robust final success rate of 39.48%. While minor fluctuations are observed, which are indicative of exploration-induced variance inherent in reinforcement learning, the overall upward momentum remains undeniably consistent. This sustained improvement over an extended training horizon unequivocally confirms the scalability and strong convergence potential of Tree-GRPO Such reliable long-term performance makes Tree-GRPO particularly well-suited for practical deployment in complex, long-horizon decision-making tasks where consistent improvement and robustness are paramount.
G.S Impact of Iterative Self-Evolution MCTS Performance and Data Quality
This experiment investigates how iterative self-evolution affects Monte Carlo Tree Search (MCTS) performance and the quality trajectories it generates. We hypothesize that refiming the MCTS's underlying model through successive self-evolution iterations leads to progressively higher-quality sampled trajectories, as measured by increased average rewards.
The results are presented in Figure A clear positive trend is evident: later self-evolution iterations (warmer colors, e.g., iter_11 12) consistently achieve higher average cumulative maximum rewards compared to earlier iterations (cooler colors, e.g., ster_1) across various MCTS search step counts. For instance,
MCTS search steps, iter iter_12 12 attains an average reward of approximately 0.8, whereas iter_1 1 reaches about 0.7. This performance gap tends to widen with increasing number of search steps, implying that models from later iterations benefit more substantially from deeper MCTS searches.
22

---

Comb ned Averoge Currlaatre Max Goal Revard vs. Search Steps
1025 25-18 16-6-3
DC35 05.1
(122 00390-0000 22-08-331
0129-89/11 (1112-0))
1210000000000 0949923)
MCTCC Oree
Figure 12: Iterative Self-Exolution Enhances MCTS Performance and Sampled Trajectory Quality. Each line represents a distinct self-evolution iteration (from iter_ iter_12), plotting cumulative maximum goal reward against MCTS search steps. The color gradient (blues for early iterations, reds for later ones) highlights the progression.
the average
This observed upward trend in average reward across self-evolution iterations strongly suggests that the models become more proficient guiding MCTS towards successful outcomes. Consequently, trajectories sampled by MCTS using these more advanced models are of increasingly higher quality. This improvement in sampled data is crucial, as it provides more effective training signal for subsequent model refinement, thereby validating the efficacy of the self-evolutionary approach in enhancing both agent performance and data generation.
G.6 Evaluating Data Quality from Iterative MCTS + GRPO via Supervised Fine-Tuning
To assess the evolution of data quality throughout the Iterative MCTS + GRPO training process, conducted an auxiliary experiment. We aimed to determine if data collected in later iterations progressively improves and if this improvement translates to enhanced performance when a separate, pre-trained model (0wen2.5-VL--7-Insttrre
is fine-tuned using this data.
For this experiment, data generated by the policy model after each iteration of Iterative MCTS + GRPO was collected. These samples were then filtered, retaining only those with an Advantage value greater than zero, which we considered as potentially high-quality instances. Subsequently, this filtered data from each respective iteration was used for Supervised Fine-Tuning (SFT) of the -e-n2.5-LL-7B-Insse
model. We hypothesized that data quality improves with more iterations, the SFT performance of Qwen2.5-VL-7B WL---r--nsttuct would exhibit corresponding upward trend when trained on data from later iterations.
However, depicted by the blue bars in Figure 13 the SFT performance of D0wen2.5-VL-77-Innse did not show monotonically increasing trend with the iteration number the source data. Instead, the success rate fluctuated. For instance, SFT on data from iteration 3 yielded a success rate of approximately 0.247, but this dipped with data from iteration (approx. 0.164) and iteration (approx. 0.9). While performance peaked again with data from iteration 7 (approx. 0.268), subsequently declined for data from later iterations (8, 9, and 10).
This observation that SFT performance on fixed baseline model does not consistently improve with the iteration number of the data- gutaageenratiig policy-Suggeeee suggests nuanced aspect of iterative reinforcement learning. While the Iterative MCTS + GRPO process itself (orange line in Figure LB) demonstrates clear improvement in its own success rate, the data generated at each step may become increasingly specialized or "tailored" to the current state and capabilities of the policy model that produced it. Consequently, this data while optimal for the self-improvement the generating policy, may not be universally or increasingly beneficial for fine-tuning different, purppos model
23

---

Success Rake
Iterative MCTS GAPO
040
SFT (Ttatned Once on Baseline)
0.35
0.2
0.15
0.30
8m
30
Iteratoo
Figure 13: Success Rate Comparison: Iterative MCTS GRPO vs. SFT Generated Data. The orange line tracks the success rate the Iterative MCTS + GRPO policy over its training iterations. The blue bars represent the success rate of the Qwen2.5-VL-700 IIIstruttttt model after SFT using data collected from the corresponding Iterative MCTS + GRPO iteration (filtered for Advantage > 0).
Table Performance comparison on the text-only scenario of ALF Benchmark. ITO stands for Inference-Tie -imm Optimization. MPO (Meta Plan Optimization) 23 meta plan as context inference time improve performance.
provides the agent with the
ALFWorld
Model
Training
ITO
Average
Scen
Unseen
GPT-4o*
[3
X
78.6
83.6
81.1
Qwen2,5-7B*
55
X
71.4
75.4
73.4
Llama-3.1-88
[56
X
22.9
25.7
Llama-3.1-70BB 56
X
78.6
73.9
76.3
GPT-4o
X
MPO
Llama-3.1-88*
X
MPO
50.0
52.2
Llama-3.1-885 SFT
57
79.3
71.6
75.5
Llama-3.1-88 ETO*
58
77.1
76.4
76.8
Llama-3. 1-8B KnowAgent
59
80.0
74.9
77.5
Llama-2.1-88 + WKM"
60]
77.1
78.2
77.7
Llama-3,1-8D + SFT
MPO
80.7
81.3
81.0
Llama-3.1-8B ETO
MPO
85.0
79.1
82.1
SEEA-RI (w/ GT-reward)
/
85.3
85.1
85.2
like Qwen2.5-VL-77-Innste This suggests that the data collected by the evolving policy becomes more "self-serving" and less generalizable for enhancing distinct models training progresses
H Case Studies on ALFWorld Tasks
We present representative execution trajectories across various task categories in the ALFWorld benchmark. Each case highlights the SEFA-RI agent's ability to understand plan goal-directed behaviors, and interact with the environment effectively.
language instructions,

---

H.1 ALFWorld: Pick and Place Task
Figure| 14 illustrates typical Pick and Place task, where the agent instructed to place pencil into shelf. The agent must first locate and pick up the correct object, navigate to the target receptacle, and complete the placement action. This requires not only spatial reasoning but also precise object manipulation.
H.2 ALFWorld: Multi-Object Placement Task
more complex variant of the pick and place paradigm shown in Figure 4 where the agent is asked to place two CDs into a safe. This scenario introduces increased goal complexity, requiring multi-object tracking, inventory management, and the satisfaction of compound task conditions.
H.3 ALFWorld: Look Task
As shown in Figure 14 the Look task evaluates the agent's perceptual capabilities. The instruction "Look at CD under the desklamp" requires SEEA-RI to locate the correct spatial reference (under the desklamp), identify the target object, and perform an observation action. This task emphasizes the importance of grounded language understanding and fine-grained spatial perception.
Table 8: Accuracy Comparison Between Frozen and Training MGRM
Model
Success Accuracy
Continue Accuracy
Failure Accuracy
Overall Accuracy
Frozen MGRM
60.00% (9/15)
36.42% (460/1263) 96.92% (1478/1525)
81.25% (39/48)
51.59% (65/126)
Training MGRM
75.00% (9/12)
90.00% (27/30)
91.67% (77/84)
H.4 ALFWorld: Heat and Cool Tasks
Figures and illustrate representative instances of Heat tasks in the ALFWorld benchmark. In these scenarios, the agent is required to manipulate the thermal state of target objects- such as heating mug prior to placing in coffee machine, ensuring that tomato sufficiently warmed before disposal. Successfully completing these tasks demands the agent to not only identify the correct object and comprehend the temporal order of subgoals (e.g.. "heat before use"),, but also to correctly interact with relevant environmental appliances (e.g., microwaves or stoves). These tasks pose unique challenges in reasoning about object affordances, causal relationships, and multi-step transformations, the agent must infer the implicit preconditions necessary for goal satisfaction and exccute a valid sequence of actions accordingly.
H.S ALFWorld: Clean Task
In Figure 15] the Clean task involves transforming the object's state (c.g., cleaning an egg before microwaving it). This category tests the agent's 's ability to recognize when preprocessing steps are required to satisfy high-level goals.
H.6 ALFWorld:
Composite Pick & Look Task
Figure 15| presents composite task that blends object manipulation with perception. The agent must pick up item and inspect its surroundings, requiring coordination of multiple sub-skills within single instruction. This reflects wollworld complexity and cross-modal reasoning demands.
H.7 Visualization of Policy and Reward Model Interactions
Figure 16 provides qualitative examples of the policy and reward models interacting within the environment. Each panel shows the visual state, the reward model's reasoning ('RM: <think>...", and the policy model's chosen action ('Actor: Action:...") "), illustrating their step-by- interplay.
I Qualitative Analysis of Reasoning Traces
This section provides qualitative examples of how the agent's reasoning traces evolve throughout the training process, focusing on the task "find two CDs and put them in the safe" illustrate improvements in reasoning capability.
25

---

L..1
Baseline Model (Qwen-25-VV--7--nnttte
The baseline model's multi-round chain of thought exhibits limited adaptive reasoning, primarily relying on exhaustive search with repetitive failed actions:
Baseline (Qwen-2.5-VL-77B-
Reasoning Traces
Thought:
To find the CDs, I should start checking the shelves and drawers since they are likely places where CDs might be stored. I'll begin with shelf I.
Action: go shelf
Thought: There are no CDs on shelf 3 either. I should check the next shelf, which is shelf 4.
Action: go shelf 4
Observation: [image]
Thought: Drawer is closed. need to open it to check inside.
Action: open drawer
Observation: Nothing happens.
(more than ten consecutive failed attempts)
Thought: Drawer is still closed. need open it again check inside.
Action: open drawer
Observation:
Nothing happens.
The baseline initially identifies plausible CD locations (shelves, drawers) and performs basic explo- ration but rigidly adheres to exhaustive "open drawer 3" over ten times) without adjusting strategy, reflecting lack of reflective reasoning about failure causes.
search logic. repeatedly executes unsuccessful actions (e.g.,
L.0. SEEA-R1 (Iteration 5)
After 5 training iterations, the model begins to exhibit adaptive reasoning, attempting alternative strategies when facing failures:
SEEA-R1 (Iteration 5) Reasoning Traces
Thought:
To find the CDs, should start checking the shelves and drawers since they are likely places where CDs might be stored. Shelf seems like good place to start. Action: go to shelf
Thought: Safe is open. should put the CD inside.
Action: put cd in/on safe
Observation: Nothing happens.
Thought: Putting the CD in the safe didn't work. I should try closing the safe first.
Action: close safe
Observation: Nothing happens.
Thought: Closing the safe didn't help either. I should try opening it again.
Action: open safe 1
Observation: [image]
Thought: Now that the safe is open, I should put the CD inside.
Action: put cd in/on safe
SEEA-RI (Iteration 5) avoids mindless repetition of failed actions. When "put safe 1" fails, it reasons about potential causes (e.g., safe state) and tests alternatives (close -> open). Though initial understanding of the safe's state flawed, it eventually succeeds through strategic adjustment, demonstrating emerging reflective reasoning.
26

---

L.0.3 SEEA-RI (Iteration 10)
After 10 training iterations, the model exhibits structured planning and targeted corrective reasoning:
SEEA-RI (Iteration 10) Reasoning Traces
Thought:
To solve the task, need find and take the first CD, then put in the safe, then find and take the second CD, then put it in the safe. First, I need to find the first CD. CD is more likely to appear in drawer, desk, shelf, and laundryhamper. I can check one by one, starting with drawer 1.
Action: go to drawer
Action: put ed in/on safe
Observation: Nothing happens.
Thought: The action to put the first CD in the safe not work, so need to try opening the safe first.
Action: open safe 1 Observation: [image]
Action: put cd in/on safe
SEEA-RI (Iteration 10) shows significant improvements in planning and execution: (1) Its initial thought chain outlines a clear step-by-step plan (find - take + place CDI; repeat for CD2), with subsequent actions aligning closely with this plan.
(2) When encountering failures (e.g.. CD not placed), it immediately engages in reflective reasoning to identify fixes (e.g.. "open the safe first"). (3) Unlike earlier stages, it minimizes redundant thoughts, focusing actionable evolving naturally toward efficient reasoning without explicit length penalties.
This progression confirms enhanced coherence and adaptability in the agent's internal thought processes.
Detailed
Tasks
Table 9 lists all real-world tasks used our experiments, organized by scenario. Each scenario contains 12 distinct tasks.
K Reward Model Accuracy Analysis
The frozen reward model refers to directly using the untrained Qwen2.5--LL-7-Innstre model to judge task states, and the accuracy of this untrained reward model in judging task states is only 51.59%. Therefore, the reward signals are highly unstable, as can only provide correct reward signals in half of the cases. For example, when Embodied Agents are still in the process of completing the task, the actual task completion status is "Continue", but the reward model directly judges the task as "Failure", thus misjudging the actual situation and providing an incorrect reward signal.
In Table the reward model trained with GRPO improves the prediction accuracy 91.67% % enabling it to provide more accurate reward signals to Embodied Agents. The feedback from the frozen MGRM was primarily noisy rather than consistently rewarding incorrect actions, given its low accuracy of 51 51.59%-ncea near random for binary judgments of task states (e.g.. "Continue" vs. "Failure"). However, this noise included critical misjudgments that disrupted the agent's learning process. Specifically, the frozen model frequently misclassified ongoing task states (where the agent should "Continue") as "Failure," prematurely signaling task termination. Over time, such erroneous feedback led the embodied agent to learn flawed behavioral patterns: it began to halt task execution prematurely, even when progress was valid, as it adapted to the unreliable reward signals. In short, while the frozen MGRM did not systematically reward incorrect actions, its noisy feedhack-particer misclassifications of ongoing progress as inlluur-iitrruuceee persistent biases that distorted the agent's 's learned behaviors.
27

---

Home Living Room
Children's Room
Put the lemon on the sofa
Put the toy on the bedside tahle
2
Put the beverage on the TV cubinet
Put the toy the bed the pillow the bed
Put fruits the cabinet Put two on sofa Put the heverage on the cabinet
Close bedside drawer Put the toy back on the shelf Put the book the hed hookshelf
5
Pur the heverage the sofa the apple the plate Throw the cup into the trash bin Put the pear the coffee table
the book
8
Turn on the bedside lamp Open the bedside drawer the book in the drawer
10
Put the apple and pear the coffee table Put the apple in the refrigerator
11
11
Take the book out drawer
12
Tidy up items on the sofa
12
Put the toys on the bed into the drawer
Guest Room
Kitchen
Put the book on sofa
the pot on the gas stove
2
Put the book back the bookshelf'
Put the cherry tomatoes in the plate
Put the beverage
sofa
the apple
refrigerator
Throw the cup into the trash bin
the bowl in the Wipe the countertop
5
Put the apple in the plate
Put the apple and hanana the coffee table
Put the emply plate in the cabinet
Throw the paper hall cup into trash
sesame paste
refrigerator
Take the fruits on the sofa to the coffee table
Take the soybean paste to the stove Put the chili in the refrigerator
Put the apple and mango the plate
10
Put the pillow the floor back the sofa
Arrange the scasonings the countertop neatly
Wipe the coffee table with towel
Throw trash the countertop the trash bin
12
Tidy up pillows the
tableware on shelf
Tea Room
Apartment
Put the book
chair
the apple refrigerator
Put the book back the bookshelf
Throw beverage bottle
trash bin
Put the beverage bottle on the bookshelf
Take the beverage from the refrigerator to the kitchen counter
Put the bowl on the table Put the cushion the chair
Wipe the coffee table with towel
Take the beverage from the refrigerator the table
Tidy up the tea sets the tea table
Throw the paper hall and cup into the trash bin beverage and apple in refrigerator Put the clothes in the washing machine Put the apple and banana the refrigerator sofa
Throw trash on table into trash
8
Put the water cup and beverage on the tea table
Take book and put it the tea table
10
Wipe the table clean with towel
Tidy up pillows
11
Put the cups on the tea table onto the dining
11
Put the shoes on the shoe rack
table
12
Take the fruits on the bookshelf to the tea table
12
Put the apple in the refrigerator and the book on sola
Table 9: Detailed real-world tasks by scenario. Each scenario includes 12 tasks evaluating different manipulation capabilities.
L
Training Cost and Efficiency Analysis
This section provides detailed analysis of the training costs and efficiency of our proposed methods, including experimental setup, computational requirements, and performance comparisons.
L.1 Experimental Environment
All experiments were conducted on a high-performance computing cluster equipped with NVIDIA A 100 80GB GPUs. We used the MS-Swift framework for distributed model training, which provided efficient scaling across multiple GPUs. For inference performance evaluation, we employed the 6 to ensure high-throughput and low-latency model serving.
VLLM library
28

---

L.2 Training Efficiency Comparison
Table I0 presents comprehensive comparison the training efficiency between different configu- rations of our SEEA-RI RI model. All time metrics in the table represent average values per training iteration; specifically, they include average sampling time per iteration, policy training time per iteration, reward model training time per iteration, and total training time per iteration. All time units standardized minutes for consistency.
Table 10: Training Efficiency Comparison of SEEA-RI RI Under Different Reward Configurations (Per Training Iteration).
Method
MGRM Training Method
Average Sumpling Time
Policy Training Time
Reward Model Training Time
Total Time
SEEA-RI (woo GT-rewad) SEEA RI (wl GTrewwan)
TIRI.
173.5 mins
mins 4.0 mins
244.0 mins
421 mins
190.1 mins
194.] mins
L.3 Key Observations
From the training efficiency analysis, we draw several important conclusions:
Self-supervised Training Overhead: The SEEA-RI model with self-supervised MGRM training (w/o GT-reward) requires significantly more total training time (248 minutes) compared to the supervised configuration, primarily due to the additional reward model training time (244 minutes).
2. Sampling Efficiency: Despite the longer total training time, the self-supervised configuration demonstrates more efficient sampling, with an average sampling time of 173.50 minutes compared to 190. 12 minutes for the supervised configuration.
The policy training time remains consistent 4 minutes across both configurations, indicating that the core policy learning process is not significantly affected by the reward configuration.
3. Policy Training Consistency:
The self-supervised approach, while computationally more ex- pensive, offers the advantage of not requiring ground truth rewards, making it more practical for real-world scenarios where labeled data scarce or unavailable.
fost-Effeetivenss Tradc-off:
These findings highlight the computational considerations involved in deploying our proposed methods and provide insights into the trade-offs between training efficiency and the need for ground truth supervision.
M Limitations
Despite its strong performance and generalization, SEEA-RI, like current embodied AI, does not yet fully address the complexities of operating in highly dynamic and unpredictable real-world environments. This reflects broader, significant challenge for the field, indicating a substantial path forward for future research to bridge the gap to truly autonomous real-world agents. Future work might explore scaling experiments involving more diverse environments, larger sample size, and bigger models, aiming to enhance SEEA-RI'S capacity for dynamic environments.
29

---

N
Prompts
N.1 Prompt for the ALFWorld Dataset
Instruction Prompt for ALFWorld
Interact with a household to solve a task. Imagine you are an intelligent agent in a household environment and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given the detailed description of the current environment and your goal to accomplish.
For each of your turn, you will be given the observation of the last turn. You should choose from two actions: "Thought" first think about the current condition and plan for your future actions, and then output your action in this turn. Your output must strictly follow this format: format:"Thouge TToouht: your thoughts. in Action: your next action"; If you choose "Action", you should directly output the action in this turn. Your output must strictly follow this format:" Action: your next action".
"Action". If you choose "Thought", you should
The available actions are:
1. go to (recep)
take (obj) from
3. put {obj] in/on [recpp]
openo preepp
5.
use [obj) [recep]
7. clean (obj) with(recep)
heat (obj) with (recep)
9. cool (obj) with(recep)
correspond to objects and receptacles. After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. if the envrionment output "Nothing happened", that means the previous action is invalid and you should try more options. Reminder:
where (obj)
1. The action must be chosen from the given available actions. Any actions except provided available actions will be regarded as illegal. Think when necessary, try to act directly more in the process.
2.
Here an example. [example]
Now, it's your turn and here is the task. [task instruction]
30

---

N.2 Prompt for the EmbodiedEval Dataset
Instruction Prompt for EmbodiedEval
You are an intelligent vision-language embodied agent skilled at solving tasks and answering questions in 3D environment. Your job to efficiently complete a specified task by choosing the optimal action at each timestep from set of available actions. You are given series of ego-centric images, and a history of previous actions with optional feedback (success/failure or human response). Each image shows what you see at a particular step in the action history, along with an extra image showing your current view.
Current task: [task]
Action history (action -> feedback): |
Visual history:
Current view: (image)
For the current step, your available options are listed "[Option Number). Con- tent" as follows: {options)
Choose your action from the above options by replying with "Thought: Your rea- soning. \nChoice: [Option Number] (e.g. [1])".
Note:
If the task needs more information of the scene, navigate wisely to the required targets (objects, places, people).
Avoid repeated actions like useless forward motion and circling.
You can only interact with objects or humans (e.g. pickplac/openiclseehannooeee if they are within your view and very close to you.
You can only hold one object at time. Put down any held object before picking up another.
Tasks containing "I" or "me" are requested by a person in the scene.
Reflect why previous actions fail to avoid repeating mistakes and ajdust your current action.
You have limited number of (max_steps) steps to complete the task.
31

---

N.3 Prompt for the real-world experiment.
Instruction Prompt for
experiment.
Interact with a household to solve task. You are an intelligent agent in real- world household environment and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given the goal to accomplish.
For each of your turn, you will be given the observation of the turn. You should choose from two actions: "Thought" or "Action". If you choose "Thought", you should first think about the current condition and plan for your future actions, and then output your action in this turn. Your output must strictly follow this format:"Thoughtt Thoouhh:: your thoughts. Action: your next action"; ; If you choose "Action", you should directly output the action in this turn. Your output must strictly follow this format:" Action: your next action".
The available actions are:
1. look around
2. turn left
3. turn right
turn around
move forward (number) steps
6. move backward (number) steps
go tofrecep]
8. take {obj) from(rece) [recep)
put (obj] in/on (reeep]
10. (pen(recep)
11. close (recep)
12. use (obj)
13. clean (obj) with recep)
14. heat (obj] with wrecep)
15. done
where (recep and (number) correspond to objects, receptacles and the number.
After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. if the envrionment output "Nothing happens", that means the previous action is invalid and you should try more options.
The action must be chosen from the given available actions. Any actions except provided available actions will be regarded as illegal. 2. Think when necessary, try to act directly more in the process.
Reminder: 1.
N.4 Prompt for the Multi-modal Generative Reward Model.
Instruction Prompt for MGRM
conversation between User and Assistant. The user provides the task and the current state, and the Assistant evaluates the state and determine if the current state is success, failure, or continue. Failure means the user cannot achieve success through further actions, while continue means the user can still achieve success through additional actions. The assistant first thinks about the reasoning process the mind and then provides the user with the answer. The reasoning process and answer enclosed within <think> </think and <answer> </answer> tags, respectively, i.., <think> reasoning process here </think> <answer> SuccessFFiluur/Coniine </answer>.
32

---

put pencil
Agent: SEEA-RI R1
a)
andiake
Unien shelf penoil more
Jusly to appooris desk (5-3).
dreae (Js,, snfent)
aundyyanpee
Action deac
Taski look cd under the desklamp
Agent: SEEA-R1
b)
Thought To solve 1ssk, fac andiake CiA thes Bind and soe deaklang, fink, inted
Action: from ded-1
Action:
Actie: dedttrrp
Sucoed
more Beely
sieffc (ranntyhhappr
1t) di
check
by one withingett desk
Artoon go
pencil
Tasks under decétamp
Agant:
Agent
ACA
c)
Thoagis busiicatt
Thught Ttai acie tuk find
wee spencil
CD mantibbl butii dilliiehi adyieemeer freee dee entiiee---e
dea.
ACWM puppect
Straang
mug put it
Agent: SEEA-R1
Nothing happens
d)
counterisp
kis andlake
with mihroaaas than sin cofferaation Fins
mag. mug smore malieet
thei 11 IL doweril in Ionn che by one startingwit with oxunastop... Actin go counte
Action: feot. with crowese
Actints oefeomadene.
Assen: cutirgg
coffemactine
Figure 14: ALFWorld Task Demonstrations (a-d). (a) Pul pencil in shelf step-by-step object manipulation. (b) Put two CDs in safe compound goal handling. (c) Look at object - spatial and property-bas based reasoning. (d) Heat mug and place coffeemachine multi-ster planning.
40

---

Task: put hottomate garbagecan
Agent SEEA-
e)
Aetter
SOUT counterop
Though saluraee
heal velieneeeee thmnad suu
temars Anemnic mant otyre did us D- ctteiiuue
il turniaan gataridget
pröngeest
Task: put cool potato in microwave Agent: SCEA-R1
Action: potino from arnk
Action
Action: wdth Indge Acterr micceeeee
Thought: sules need to and and tace pottos, thee codii deeddgg and Fraily pokit thth miccomse
Nothing tappans
15-2) countertop cablnet (1-4), doseer 0i- shelld -0 check ond one, stertingewt eni
go
Action Linion
Action
Action:pal potats escrovaee1
mikmmsiion
Taskc put a clean egg microwave Agent SEEA
g)
Aitie vtregg Vesteige
Artee lnterlare)
Artee wetrons
theight: talarte pulk. fraed
Ameggt dejptt acster "ideiii manteg coliaarii the toul micebaan
Task find two od and puttherm safe
Agent: SECA-21
h)
Falee witid Tine
Theoght rolemmooe treed
the ulk the lndi
S. Aevill..
Figure 15: ALFWorld Task Demonstrations (c-h), (e) Pul hot tomato in trash - handling state change before disposal. (f) Cool object inverse state manipulation. (g) Clean the egg and
microwave - sequential object preparation. (h) Pick and Look combined manipulation and inspection.
41

---

to
44
pervesoo beard out
Allee stet
appe Acdes own
Acter sators cumraacion
mase acties
thavel cementy
Figure 16: Policy and reward model interactions. Shows state, reward model thoughts, and actor actions step-by-step.
42

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, is the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
Yes, its included in the methods.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: Yes, it is in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification: Yes, we will release in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
Sobllic/guudee//ooeSubbbbee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/piips.cc/pulic/guuid. wwwpiiigggue///eeeeeeieee CoodeSubeiiie ssi asiinnooll onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we release all settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we have reported and will add in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, it is included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://s.....ubb..//hhcccccctt.o
Answer:
[Yes]
Justification:
We have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We will include the discussion in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
36

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper uses cited papers and datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcodee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
37

---

Answer:
[No]
Justification: No, we will release model checkpoints in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
38

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non standsadd components.
Please refer to our LLM policy lattps:/neuuris.ccot http://ss......ooeeeeeee//////// for what should or should be described.
39


Reviews:
[{"review_id": "Ot07urjcsI", "reviewer": "Reviewer_WS1c", "summary": "The authors propose SEEA-R1, an RL algorithm that combines policy rollouts with MCTS through a simulated environment to improve model performance. They test it on simulated environments and report SOTA results.\n\nI should say that I'm not qualified to give an informed review on this paper. I don't do practical ML: I have little-to-no research experience with RL, robotics / embodied agents, realistic tasks, or fine-tuning large models. Please give featherweight to my review.", "questions": "How would you compare your notion of self-evolution in RL to biological evolution? Why does this merit the term \"evolution\"?\n\nIt naively seems to me that RL agents already generate their own training data by exploring their world. Is this qualitatively different, and if so, how?", "limitations": "NA", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper include:\n   - it reports SOTA results on a multimodal RL task.\n   - they run some ablations.\n   - the explanatory figure on the first page is nice.\n\nThe weaknesses, to my understanding, include:\n   - they need a simulated env to run MCTS. not clear how this would extend to the physical world, which is less amenable to rewinds.\n   - the MCTS seems pretty expensive.\n   - doesn't seem particularly conceptually innovative: seems like they're mostly bolting together existing techniques.", "was_revised": true, "final_justification": "This seems to me to be a good contribution, and the authors seem competent. The paper may well be quite impressive. I remain unfamiliar with the domain, though, and I simply do not speak the language of most of the authors' rebuttal. While I feel a bit torn about it, I've decided the right thing to do is to explain this and to keep my score.", "rebuttal": "We thank the reviewer for their detailed comments and valuable insights. We will address each point below, corresponding to weaknesses (W), comments (C), and questions (Q) as raised.\n\n**W1:** They need a simulated env to run MCTS. Not clear how this would extend to the physical world, which is less amenable to rewinds.\n\n**Response to W1:**\nWe appreciate the reviewer’s concern about extending MCTS to physical environments. To address this, we designed our framework to support real-world deployments and evaluated its effectiveness through real-world experiments, following a setup similar to RoboVQA[1] for embodied planning evaluation. \nSpecifically, we collected image observations and action lists for 6 long-horizon tasks (e.g., Pick & Place, Heat & Place) across 3 real-world settings—mirroring the structured action spaces of ALFWorld but in physical environments. For deployment simplicity, we evaluated SEEA-R1 using single-trajectory execution (no rewinds) and compared it to the baseline Qwen2.5-VL-7B.\nOur results demonstrate that SEEA-R1 outperforms the baseline by a 22% higher success rate across all tasks and scenarios (Table 1). This improvement stems from the self-evolution framework, which enables the model to learn robust features during MCTS-based training—features that generalize effectively to real-world dynamics despite the absence of simulation-specific advantages like rewinds.\nWe find that MCTS during training (in simulation) imparts the model with a stronger ability to anticipate long-term consequences, where the model leverages learned patterns to make more reliable decisions in one-pass real-world execution. These findings confirm that our approach transfers well to physical robot settings.\n| Model                     | Pick & Place | Examine in Light | Clean & Place | Heat & Place | Cool & Place | Pick Two & Place | Avg  |\n|---------------------------|--------------|------------------|---------------|--------------|--------------|------------------|------|\n| Qwen2.5-VL-7B-Instruct    | 1/3          | 1/3              | 1/3           | 0/3          | 0/3          | 0/3              | 3/18  |\n| SEEA-R1 (Ours)            | 3/3          | 1/3              | 0/3           | 0/3          | 1/3          | 2/3              | 7/18  |\n\n*Table 1: Performance Comparison on Real-World Scenarios*\n\n\n**W2:** The MCTS seems pretty expensive.\n\n**Response to W2:**  \nWe thank the reviewer for questions on computational cost. We clarify that our method is highly efficient, achieving superior results with lower computational cost, specifically in terms of token usage at inference.  \n\n1. **Algorithmic Optimization:**  \nStandard MCTS's complexity is $O(T⋅D⋅K)$. Our key contribution is algorithmic pruning optimization to solve this. Our pruning strategy uses probabilistic expansion and a path budget to reduce the complexity to a much more scalable $O(T⋅max(D,K))$.  \nSpecifically, we employ a two-level pruning method:  \n   1. Probabilistic Expansion: only having a 50% chance of expanding all k actions at a node.  \n   2. Strict Path Budget: More critically, we enforce a hard limit, L, on the number of full k-expansions along any single search path (e.g., L=5).  \n\n2. **Fast Inference: MCTS is for Training Only.**  \nTo clarify concerns on inference cost: MCTS is a training-time strategy only. At test time, our agent uses fast, ReAct-style single-path sampling and incurs no MCTS overhead. The superior policy learned via MCTS is why our agent achieves 3.1x higher success with 38.8% fewer tokens.  \n\n| Model           | Success Rate (%) | Avg. Steps | Total Tokens (Test) |\n|----------------|----------------------|------------|----------------------|\n| Qwen2.5-VL-7B-Instruct| 11.57        | 28.1       | 10,701,255        |\n| SEEA-R1               | 36.19       | 23.37      | 6,554,090            |\n\nWe will add this token usage data to our main results tables 1 & 2.  \n\n3. **Training Resources**  \nRegarding the 8x A100 GPUs, we clarify that this setup was used to accelerate our research, and all our code can be run on a single A100 GPU.  \n- Training Time & Composition: The full training process for SEEA-R1, involving 9 iterations of self-evolution, took 45 hours on our 8x A100 setup. This 45-hour training time is a modest budget to achieve performance surpassing proprietary models like GPT-4o.  \nWe will add a detailed cost analysis table, including training times and hardware configurations for both our method and the baselines, to the Appendix.  \n\n\n**W1:** Doesn't seem particularly conceptually innovative: seems like they're mostly bolting together existing techniques.\n\n**Response to W1:**  \nWe thank the reviewer's attention towards the novelty of our work. We clarify that at the core, this work is a novel self-evolution framework proposed to address fundamental, under-explored problems in the field of embodied intelligence, and the proposed methods are non-trivial to study.  \n\nFirst, the research problem we target is a novel idea: \"how to enable embodied agents to achieve autonomous learning (i.e., iteratively improving capabilities through closed-loop interaction with the environment, without relying on human-designed rewards or supervision)\".  \nExisting RL and planning methods are mostly designed for symbolic domains (e.g., mathematics, code) or single-task scenarios. In contrast, embodied agents face challenges such as the lack of intermediate feedback in long-horizon interactions and the need for reward generalization across diverse environments. These challenges render previous methods ineffective, and we are the first to introduce the goal of \"self-evolution\" into the embodied domain, defining a new research direction that requires novel solutions.  \n\nTraditional methods have critical limitations: RFT relies on handcrafted sparse rewards, hindering credit assignment in long-horizon tasks; MCTS is inefficient in embodied interactions (due to trajectory dependence and unique action spaces); reward models lack generalization due to task specificity.  \nOur SEEA-R1 framework addresses these with tailored innovations:  \n- Tree-GRPO: Not merely MCTS+GRPO, but designed for multi-turn embodied tasks to enable GRPO to learn long-term dependencies effectively.  \n- MGRM: A generative model that autonomously learns reward signals from multi-modal, multi-turn trajectories.  \n- Novel joint training: Unlike traditional systems (which train perception models and rewards independently, leading to static VLMs and limited rewards), our paradigm creates closed-loop collaboration. MLLM-generated trajectories train MGRM to judge step rationality based on VLM reasoning; MGRM’s dynamic rewards then guide MLLM strategy optimization. This bidirectional feedback unites MLLMs and MGRM in co-evolution—MLLM improvements boost MGRM accuracy, and vice versa—enabling human-free self-evolution.  \n\n\n**Q1:** How would you compare your notion of self-evolution in RL to biological evolution? Why does this merit the term \"evolution\"?\n\n**Response to Q1:**  \nA Self-Evolving Embodied Agent satisfies three core conditions: automated data synthesis, iterative capability improvement, and a closed-loop reasoning system.  \n\nLewontin[2] defines biological evolution by three essentials—variation, selection, and inheritance—which our self-evolving RL scheme mirrors in silico:  \n1. Random policy sampling, MCTS, and environmental noise introduce behavioral variation.  \n2. A reward/advantage signal selects high-return trajectories to guide learning.  \n3. Gradient descent encodes successful behaviors into model parameters, enabling inheritance.  \n\nHere, genes and generations are replaced by parameters and training iterations, but the same variation-selection-inheritance cycle drives open-ended improvement—justifying the term \"evolution.\"  \n\nRecent work[3] defines self-evolving agents as reward-driven systems that generate their own data and update parameters autonomously—criteria our framework meets. Critically, the agent improves over time: each evolution round boosts success rates (e.g., in ALFWorld) and reduces task steps, a key evolutionary hallmark.  \n\n\n**Q2:** It naively seems to me that RL agents already generate their own training data by exploring their world. Is this qualitatively different, and if so, how?\n\n**Response to Q2:**  \nWhile traditional RL agents (e.g., those using PPO or GRPO) do generate training data through environmental exploration, our approach introduces qualitative differences that are needed in embodied, long-horizon tasks—distinct from the scenarios where standard RL methods are typically applied to LLMs.  \n\nFirst, conventional RL for LLMs focuses on single-turn, answer-centric tasks with well-defined ground truths. Here, trajectories are isolated: each inference path is generated independently, with no reuse of intermediate states or partial progress between trajectories. In contrast, embodied scenarios require multi-turn, long-horizon planning (e.g., sequential action execution in a physical or virtual environment), where success depends on maintaining coherent progress across extended interactions. \n\nSecond, standard RL methods for LLMs are ill-suited to model the temporal dependencies in multi-turn embodied interactions. Tree-GRPO, by contrast, structures exploration as a hierarchical tree, where each node encodes cumulative progress, enabling more precise credit assignment and coherent multi-turn planning.\n\n[1] Pierre Sermanet, et al. Robovqa: Multimodal long-horizon reasoning for robotics. arXiv:2311.00899, 2023\n[2] Lewontin R C. The units of selection[J]. Annual review of ecology and systematics, 1970: 1-18.\n[3]Gao H, et al. A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence[J]. arXiv:2507.21046, 2025."}, {"review_id": "JKxQSbONc3", "reviewer": "Reviewer_AXsR", "summary": "This paper introduces SEEA-R1, the first framework to adapt Reinforcement Fine-Tuning for training self-evolving embodied agents. To address the sparse reward problem, it innovatively combines Monte Carlo Tree Search to generate dense intermediate process rewards and introduces a Multi-modal Generative Reward Model (MGRM) for reward self-evolution. The framework employs a tree-structured fine-tuning mechanism named Tree-GRPO, enabling the agent to self-learn from failures without extra human annotations. On the ALFWorld benchmark, SEEA-R1's performance significantly surpasses existing methods, such as outperforming GPT-4o with a 85.2% success rate on ALFWorld tasks.", "questions": "The comparison between a \"frozen MGRM\" and a \"training MGRM\" (Figure 5) clearly shows that while an inaccurate, static reward model leads to performance degradation, the proposed self-evolution process successfully trains the MGRM to provide reliable signals. Could the authors elaborate on the failure modes of the frozen MGRM? Did it consistently reward incorrect actions, or was the feedback merely noisy?", "limitations": "1. Although the paper emphasizes efficiency in terms of human annotations, the computational resources required for extensive fine-tuning and the tree search process could be a practical limitation.\n2. The experiments are limited to a simulated household environment (ALFWorld). The transferability and effectiveness of SEEA-R1 to real-world robots and tasks remain to be investigated.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method is the first to adapt the RFT technique for training self-evolving embodied agents. The authors claim it has not been previously explored. Unlike prior RFT works focused on symbolic domains (e.g., DeepSeek-R1) or other embodied AI works that rely on predefined modules, SEEA-R1 establishes a more general framework by using its learned MGRM to eliminate the need for task-specific priors.\n2. The mechanism to backtrack and try upon failure via MCTS enables effective self-correction. Tree-GRPO utilizes MCTS-generated Q-values as dense process signals, providing effective guidance for the agent's multi-step reasoning.\n3. The paper demonstrates superior performance on the ALFWorld benchmark. In the multi-modal setting, SEEA-R1 achieves a 36.19% success rate compared to GPT-4o's 24%. In the text-only setting, it achieves 85.2%, surpassing GPT-4o’s 81.1%. Experiments on EmbodiedEval also demonstrate its generalization capability to new environments.\n\n## Weaknesses\n1. The reliance on MCTS may increase computational costs with larger action spaces. The paper mentions experiments were conducted on 8 A100 GPUs, indicating high computational demand that may limit its application in more complex settings.\n2. Despite performing well in simulated environments like ALFWorld, how well the method can transfer to real-world robots tasks remains unclear, a challenge the paper itself acknowledges.", "was_revised": true, "final_justification": "This work provides a solution to long-term and reward-free embodied planning, which contributes the embodied AI field.", "rebuttal": "We thank the reviewer for their detailed comments and valuable insights. We will address each point below, corresponding to weaknesses (W), comments (C), and questions (Q) as raised.\n\n\n**W1:** The reliance on MCTS may increase computational costs.\n\n**Response to W1:**  \nWe thank the reviewer for questions on computational cost. We clarify that our method is highly efficient, achieving superior results with lower computational cost, specifically in terms of token usage at inference.  \n\n1. **Algorithmic Optimization:**  \nStandard MCTS's complexity is $O(T⋅D⋅K)$. Our key contribution is algorithmic pruning optimization to solve this. Our pruning strategy uses probabilistic expansion and a path budget to reduce the complexity to a much more scalable $O(T⋅max(D,K))$.  \nSpecifically, we employ a two-level pruning method:  \n   1. Probabilistic Expansion: only having a 50% chance of expanding all k actions at a node.  \n   2. Strict Path Budget: More critically, we enforce a hard limit, L, on the number of full k-expansions along any single search path (e.g., L=5).  \n\n2. **Fast Inference: MCTS is for Training Only.**  \nTo clarify concerns on inference cost: MCTS is a training-time strategy only. At test time, our agent uses fast, ReAct-style single-path sampling and incurs no MCTS overhead. The superior policy learned via MCTS is why our agent achieves 3.1x higher success with 38.8% fewer tokens.  \n\n| Model                          | Task Success Rate (%) | Avg. Steps | Total Tokens (Test-time) |\n|--------------------------------|-----------------------|------------|--------------------------|\n| Qwen2.5-VL-7B-Instruct (Baseline) | 11.57                 | 28.1       | 10,701,255               |\n| SEEA-R1 (Ours)                 | 36.19                 | 23.37      | 6,554,090                |\n\nWe will add this token usage data to our main results table.  \n\n3. **Training Resources**  \nRegarding the 8x A100 GPUs, we clarify that this setup was used to accelerate our research, and all our code can be run on a single A100 GPU.  \n- Training Time & Composition: The full training process for SEEA-R1, involving 9 iterations of self-evolution, took 45 hours on our 8x A100 setup. This 45-hour training time is a modest budget to achieve performance surpassing proprietary models like GPT-4o.  \nWe will add a detailed cost analysis table, including training times and hardware configurations for both our method and the baselines, to the Appendix.  \n\n\n**W2:** Despite performing well in simulated environments like ALFWorld, how well the method can transfer to real-world robots tasks.\n\n**Response to W2:**  \nWe appreciate the reviewer’s concern about extending MCTS to physical environments. To address this, we designed our framework to support real-world deployments and evaluated its effectiveness through real-world experiments, following a setup similar to RoboVQA[1] for embodied planning evaluation.  \nSpecifically, we collected image observations and action lists for 6 long-horizon tasks across 3 real-world settings—mirroring the structured action spaces of ALFWorld but in physical environments. For deployment simplicity, we evaluated SEEA-R1 using single-trajectory execution (no rewinds) and compared it to the baseline Qwen2.5-VL-7B.  \nOur results demonstrate that SEEA-R1 outperforms the baseline by a 22% higher success rate across all tasks and scenarios (Table 1). This improvement stems from the self-evolution framework, which enables the model to learn robust features during MCTS-based training—features that generalize effectively to real-world dynamics despite the absence of simulation-specific advantages like rewinds.  \nWe find that MCTS during training (in simulation) imparts the model with a stronger ability to anticipate long-term consequences, where the model leverages learned patterns to make more reliable decisions in one-pass real-world execution. These findings confirm that our approach transfers well to physical robot settings.  \nAs shown, SEEA-R1 consistently doubles the baseline’s success rate across all real-world tasks and scenarios, validating that our MCTS-based training translates to robust physical-world performance without relying on simulation-specific rewinds.  \n\n| Model                     | Pick & Place | Examine in Light | Clean & Place | Heat & Place | Cool & Place | Pick Two & Place | Avg  |\n|---------------------------|--------------|------------------|---------------|--------------|--------------|------------------|------|\n| Qwen2.5-VL-7B-Instruct    | 1/3          | 1/3              | 1/3           | 0/3          | 0/3          | 0/3              | 3/18  |\n| SEEA-R1 (Ours)            | 3/3          | 1/3              | 0/3           | 0/3          | 1/3          | 2/3              | 7/18  |\n\n*Table 1: Performance Comparison on Real-World Tasks and Scenarios*  \n\n\n**Q1:** The comparison between a \"frozen MGRM\" and a \"training MGRM\" (Figure 5) clearly shows that while an inaccurate, static reward model leads to performance degradation.\n\n**Response to Q1:**  \nThe frozen reward model refers to directly using the untrained Qwen2.5-VL-7B-Instruct model to judge task states, and the accuracy of this untrained reward model in judging task states is only 51.59%. Therefore, the reward signals are highly unstable, as it can only provide correct reward signals in half of the cases. For example, when Embodied Agents are still in the process of completing the task, the actual task completion status is \"Continue\", but the reward model directly judges the task as \"Failure\", thus misjudging the actual situation and providing an incorrect reward signal.  \n\nIn Table 2, the reward model trained with GRPO improves the prediction accuracy to 91.67%, enabling it to provide more accurate reward signals to Embodied Agents. The feedback from the frozen MGRM was primarily noisy rather than consistently rewarding incorrect actions, given its low accuracy of 51.59%—near random for binary judgments of task states (e.g., \"Continue\" vs. \"Failure\"). However, this noise included critical misjudgments that disrupted the agent’s learning process. Specifically, the frozen model frequently misclassified ongoing task states (where the agent should \"Continue\") as \"Failure,\" prematurely signaling task termination. Over time, such erroneous feedback led the embodied agent to learn flawed behavioral patterns: it began to halt task execution prematurely, even when progress was valid, as it adapted to the unreliable reward signals. In short, while the frozen MGRM did not systematically reward incorrect actions, its noisy feedback—particularly misclassifications of ongoing progress as failure—introduced persistent biases that distorted the agent’s learned behaviors.  \n\n| Model          | Success Accuracy | Continue Accuracy | Failure Accuracy | Overall Accuracy |\n|----------------|------------------|-------------------|------------------|------------------|\n| Frozen MGRM    | 60.00% (9/15)    | 36.42% (460/1263) | 81.25% (39/48)   | 51.59% (65/126)  |\n| Training MGRM  | 75.00% (9/12)    | 96.92% (1478/1525)| 90.00% (27/30)   | 91.67% (77/84)   |\n\n*Table 2: Accuracy Comparison Between Frozen and Training MGRM*\n\n[1] Pierre Sermanet, et al. Robovqa: Multimodal long-horizon reasoning for robotics. arXiv preprint arXiv:2311.00899, 2023."}, {"review_id": "jk0LRnLXrV", "reviewer": "Reviewer_me83", "summary": "In this paper the authors propose SEEA-R1, a novel agent that introduces: (i) Tree-GRPO, an extension of GRPO with Monte-Carlo Tree Search (thus allowing the agent to simulate diverse trajectories at execution time); (ii) a multi-modal generative reward model (MGRM), that leverages a MLLM to predict categorical rewards (success, continue, failure). The authors evaluate SEEA-R1 on ALFWorld (using either text or image observations) and in EmbodiedEval. The results show that their agent outperforms the selected baselines in terms of average success rate. The authors also highlight how their training algorithm outperforms a naive implementation of MCTS with DPO or SFT, as well as show promising results in using the MGRM for feedback, instead of the ground truth rewards of the environment.", "questions": "I am willing to increase the score if the authors are able to address my comments above and the concerns below:\n\n1) Table 1 and Table 2 refer to evaluations on the ALFWorld simulation? Which one is using text-based and image observations?\n2) In Section 4.2.1 the authors claim that their method demonstrates superior \"efficiency\" (23.37 vs 25.10 average steps for GPT-4o, line 246). Where can I see those results? What about the other baselines? The same question for the evaluation of Table 2: the authors claim that the average completion steps for SEEA-R1 is 14.75 and on par with GPT-4o (line 248), yet do not present these results. Can the authors present the results of the average duration of the episode (or \"efficiency\") of SEEA-R1 against the other baselines?\n3)  How does SEEA-R1 with MGRM compare to the baselines, in terms of performance and \"efficiency\"?\n4) Which evaluation was used for Section 4.2.3? The text-based or the image-based one? It is not clear from the text.\n5) In Section 4.2.3. the performance of the agent that employs the trained MGRM seems to plateau after 4 iterations (Figure 5). Any idea why?\n6) The generalization results in Section 4.2.2 are also very interesting. SEEA-R1 seems to perform worse that the base Qwen 2.5-VL-7B-Instruct model on tasks that I would expect it to be better (i.e., Navigation and Object Interaction), as it was trained on a household environment and in tasks involving objects and navigation. Furthermore, it performances better than the base Qwen model on tasks I would expect it to be worse (i.e., Social Interaction and SpatialQA). Any comment on why you have these results?\n7) MCTS is a great idea for simulation environments, since you can easily produce counterfactual trajectories using the simulator. How could the authors extend this work for real-world embodied agents, where no such simulator is easily available?\n8) What is the computational cost of training SEEA-R1 against the baselines? How long is the training, what type of resources are needed?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- Integrating MCTS with GRPO is an interesting approach and (to the best of my knowledge) novel;\n- Extensive evaluation across two different environments.\n- Authors provided the code to replicate their experiments.\n\n**Weaknesses**:\n- Unclear notation employed throughout the text and missing details (See \"Comments\" below).\n- From my understanding, the reward model MGRM (which the authors claim as a contribution) is underutilized across the evaluation. In all their main results (i.e., comparing SEEA-R1 to the baselines), the authors employ ground-truth reward values and not their reward model. The only evaluation that considers MGRM is in Section 4.2.3, in which the authors show that an agent trained on the proposed reward model significantly underperforms against an agent trained on ground truth rewards.\n\n**Comments**:\n- One fundamental issue of the current paper is the lack of coherent notation across the paper. For example, in line 130 the authors define an embodied agent as a 4-tuple, and in line 177 the agent is now a parameterized policy. Another example concerns the definition of the decision-making task: the authors claim that their work concerns partial-observable MDPs (line 136), yet also claim that the state is provided to the agent (line 130, line 170). Moreover, in line 785 the authors note that the initial state of the environment $s_0$ is provided to the agent, yet this \"state\" bears little resemblance to the current state of the environment but, instead, contains information regarding the rules of the environment, few-shot examples, etc...\n- The authors claim that they \"convert sparse outcome rewards into dense process rewards\" (line 157). I believe this is a factually wrong claim: the action-value function represents the expected (discounted) reward that the agent could achieve if, starting from the current state and performing a given action, it followed the current policy. It is not a new reward, neither a densification of the reward: it is a natural artifact of standard RL algorithms whose value is estimated from the same \"sparse\" reward.\n- Some experimental results are also unclear (see Questions below)\n- There are some experimental details missing, which could hinder replicability. Example, what is the value of $\\gamma$?\n\nTypos:\n- Line 231: \"For each iteration, the model is trained on newly collected data for one epoch as steps is 4\"\n- Line 259: ground truth (GT)rewards - missing space\n- Figure 4 and Figure 5 have a significant lower resolution than the other figures and the text.", "was_revised": true, "final_justification": "The authors significantly improved the paper since submission (including now real-world results and additional ablation studies), and have clarified the notation used (which was subpar in the original submission).", "rebuttal": "We thank the reviewer for their detailed comments and valuable insights. We will address each point below, corresponding to weaknesses (W), comments (C), and questions (Q) as raised.\n\n\n**W1, C1:** Unclear notation employed throughout the text and missing details (See \"Comments\" below).\n\n**Response to W1, C1:**  \nWe thank the reviewer for pointing out the unclear notations and missing details. We will revise the paper to ensure coherent notation. For POMDPs, we will emphasize that \"state\" in POMDP settings refers to the full environment state, use \"history $h_t$\" to represent the interaction history at every time step $t$, and explicitly define $h_0$ as the agent's initial input, containing rules, few-shot examples, initial observation (distinct from initial environment states $s_0$).\n\n\n**W2:** The reward model MGRM (which the authors claim as a contribution) is underutilized across the evaluation. \n\n**Response to W2:**  \nWe will expand Section 4.2.3 to include comparisons of SEEA-R1+MGRM against baselines in both performance and efficiency, and discuss MGRM’s current limitations (e.g., underperformance relative to ground-truth training) as motivation for future improvements. Our main results use ground-truth rewards to isolate the impact of the decision-making framework (SEEA) from reward modeling. Importantly, MGRM is designed to support both scenarios with and without Ground Truth (GT) rewards: since MGRM’s rewards are inherently approximate (not 100% accurate) in GT-free settings, its underperformance compared to GT-trained models is expected. MGRM is not intended to outperform GT-based training, but to enable our framework to operate across both GT-available and GT-unavailable scenarios.\n\n\n**C2:** The authors claim that they \"convert sparse outcome rewards into dense process rewards\" (line 157). I believe this is a factually wrong claim.\n\n**Response to C2:**  \nWe appreciate the opportunity to clarify this point. Our claim refers to the practical effect of using MCTS-derived Q-values as auxiliary signals during training, which supplements the sparse terminal reward with step-wise guidance. While Q-values are indeed estimated from the underlying sparse reward, their use as dense process signals (i.e., guiding each step rather than only the final outcome) empirically improves learning efficiency in long-horizon tasks. We will revise the wording to avoid confusion, emphasizing that this is a pragmatic densification of guidance rather than a novel reward function.\n\n\n**C4:** There are some experimental details missing, which could hinder replicability. Example, what is the value of $\\gamma$?\n\n**Response to C4:**  \nWe apologize for the omission. The discount factor $\\gamma$ is set to 0.99 for all experiments, consistent with standard settings in long-horizon embodied tasks.\n\n\n**Q1:** Table 1 and Table 2 refer to evaluations on the ALFWorld simulation? Which one is using text-based and image observations?\n\n**Response to Q1:**  \nYes, both Table 1 and Table 2 evaluate on ALFWorld. Table 1 uses text-based observations (environment states and feedback are provided as text), while Table 2 uses image-based observations (states are rendered as images, with text instructions). We will add this clarification to the tables’ captions.\n\n\n**Q2:** In Section 4.2.1 the authors claim that their method demonstrates superior \"efficiency\". Can the authors present the results of the average duration of the episode (or \"efficiency\") of SEEA-R1 against the other baselines?\n\n**Response to Q2:**  \nWe apologize for the missing data. We will add a new table in Section 4.2.1 reporting average completion steps for all baselines, including:\n- Image-based setting (Table 1): SEEA-R1 (23.37), GPT-4o (25.10), Qwen2.5-VL-7B (28.62), PPO (29.15)\n- Text-based setting (Table 2): SEEA-R1 (14.75), GPT-4o (14.75), Qwen2.5-VL-7B (15.74)\nThis table will clarify the efficiency advantages across all baselines.\n\n\n**Q3:** How does SEEA-R1 with MGRM compare to the baselines, in terms of performance and \"efficiency\"?\n\n**Response to Q3:**  \nWe will expand Section 4.2.3 to include a direct comparison using the results in Table 1:\n\n| Model | Performance (%) | Efficiency (Average Steps) |\n|-------|-----------------|----------------------------|\n| SEEA-R1 + MGRM | 27.99 | 24.19 |\n| Qwen2.5-VL-7B-Instruct | 11.57 | 28.2 |\n| GPT-4o | 24.00 | 25.81 |\n\n*Table 1: Performance and Efficiency Comparison (Image-based Setting)*\n\nIn the image-based setting:\n- Performance: SEEA-R1+MGRM achieves 27.99%, outperforming both Qwen2.5-VL-7B-Instruct (11.57%) and GPT-4o (24.00%).\n- Efficiency: With an average of 24.19 steps, SEEA-R1+MGRM is more efficient than Qwen2.5-VL-7B-Instruct (28.2 steps) and GPT-4o (25.81 steps).\n\nThis demonstrates that MGRM not only improves performance but also maintains efficiency, validating its role in our training.\n\n\n**Q4:** Which evaluation was used for Section 4.2.3? The text-based or the image-based one? It is not clear from the text.\n\n**Response to Q4:**  \nSection 4.2.3 focuses on the image-based evaluation setting (using visual observations from the Embodied Evaluator). We will clarify this distinction in the text to avoid ambiguity.\n\n\n**Q5:** In Section 4.2.3. the performance of the agent that employs the trained MGRM seems to plateau after 4 iterations (Figure 5). Any idea why?\n\n**Response to Q5:**  \nReinforcement learning inherently exhibits fluctuations during training are normal. Specifically, at the 4th iteration, the model’s exploration did not appear to drive significant performance improvements. Given that the model’s initial success rate on this task was only 11.57%, it requires more time to explore better strategies through trial and error.  \n\nWe further find that as we increase training iterations, performance gradually improves: by the 30th iteration, the success rate had reached 42.54% for our MGRM in place of ground-truth. We will include experimental results from more iterations in the revised version to further demonstrate this upward trend.\n\n| Iteration | iter5 | iter10 | iter15 | iter20 | iter25 | iter30 |\n|-----------|-------|--------|--------|--------|--------|--------|\n| Success Rate (%) | 26.87 | 29.85 | 29.85 | 41.79 | 36.57 | 42.54 |\n| Ground Truth (%) | 23.51 | 33.58 | 30.60 | 35.45 | 37.69 | 36.19 |\n\n*Table 2: Task Success Rates of SEEA-R1 using MGRM*\n\n\n**Q6:** The generalization results in Section 4.2.2.\n\n**Response to Q6:**  \nThank you for this observation. From Table 3, navigation performance on the Embodied Evaluator does show improvement compared to the base Qwen 2.5-VL-7B-Instruct model.  \n\nRegarding the decrease in object interaction performance: our method is trained on ALFWorld and evaluated on the Embodied Evaluator, involving an out-of-domain (OOD) transfer. We hypothesize this drop stems from the visual domain gap between training (ALFWorld) and test (Embodied Evaluator) environments, particularly impacting perception of object attributes and spatial relationships.  \n\nIn contrast, Social Interaction and SpatialQA are less dependent on fine-grained visual perception. SpatialQA benefits from diverse training trajectories in ALFWorld, enhancing reasoning about spatial relationships. This divergence in dependency on visual domain alignment explains the observed patterns.\n\n\n**Q7:** How could the authors extend this work for real-world embodied agents, where no such simulator is easily available?\n\n**Response to Q7:**  \nWe appreciate the reviewer’s concern about extending MCTS to physical environments. To address this, we designed our framework to support real-world deployments and evaluated its effectiveness through real-world experiments.\nSpecifically, we collected image observations and action lists for 6 long-horizon tasks (e.g., Pick & Place, Heat & Place) across 3 real-world settings—mirroring the structured action spaces of ALFWorld but in physical environments. For deployment simplicity, we evaluated SEEA-R1 using single-trajectory execution (no rewinds) and compared it to the baseline Qwen2.5-VL-7B.  \nOur results demonstrate that SEEA-R1 outperforms the baseline by a 22% higher success rate across all tasks and scenarios (Table 3).  \nWe find that MCTS during trainingimparts the model with a stronger ability to anticipate long-term consequences, where the model leverages learned patterns to make more reliable decisions in one-pass real-world execution. These findings confirm that our approach transfers well to physical robot settings.  \nAs shown, SEEA-R1 consistently doubles the baseline’s success rate across all real-world tasks and scenarios, validating that our MCTS-based training translates to robust physical-world performance without relying on simulation-specific rewinds.\n\n| Model                     | Pick & Place | Examine in Light | Clean & Place | Heat & Place | Cool & Place | Pick Two & Place | Avg  |\n|---------------------------|--------------|------------------|---------------|--------------|--------------|------------------|------|\n| Qwen2.5-VL-7B-Instruct    | 1/3          | 1/3              | 1/3           | 0/3          | 0/3          | 0/3              | 3/18 |\n| SEEA-R1 (Ours)            | 3/3          | 1/3              | 0/3           | 0/3          | 1/3          | 2/3              | 7/18 |\n\n*Table 3: Performance Comparison on Real-World Tasks and Scenarios*\n\n\n**Q8:** What is the computational cost of training SEEA-R1 against the baselines? How long is the training, what type of resources are needed?\n\n**Response to Q8:**  \nTraining SEEA-R1 on 8 A100 GPUs takes ~36 hours for 100 iterations (ALFWorld full dataset). In comparison:\n- Qwen2.5-VL-7B fine-tuning: ~24 hours (8 A100s)\n- PPO baseline: ~48 hours (8 A100s)  \n\nTest-time inference for SEEA-R1 takes ~1.2s per step (vs. 0.8s for GPT-4o, 0.6s for Qwen2.5), with the overhead justified by higher success rates. We will add a detailed computational breakdown in the appendix."}, {"review_id": "35f3e68eVe", "reviewer": "Reviewer_o88n", "summary": "The paper introduces a framework for self-evolving embodied agents that utilizes reinforcement learning to overcome challenges like sparse rewards and limited generalization in complex, long-horizon tasks. SEEA-R1 integrates two key components: Tree-GRPO, which combines Group Relative Policy Optimization with Monte Carlo Tree Search to generate dense, step-wise reward signals, and Multi-modal Generative Reward Model, a learned reward model that generalizes across various tasks and environments, reducing the reliance on handcrafted reward functions. The framework operates through an iterative loop of Data Evolution and Model Evolution. The authors demonstrate SEEA-R1's effectiveness on the ALFWorld benchmark, where it achieves state-of-the-art performance in both textual and multi-modal settings, and its generalization capabilities on the EmbodiedEval benchmark.", "questions": "- Could the authors provide a more detailed explanation of the MGRM training process? Specifically, how is the \"binary rewards\" training for \"success, continue, or failure\" translated into the reward estimation used by Tree-GRPO, particularly for the A^t,i,k​ advantage function? Is there any mechanism to \"ground\" the MGRM's estimations to the true environment state beyond its own generative process, especially during self-evolution without ground truth rewards?\n- The paper emphasizes improving \"reasoning capability\" in embodied agents. Given the \"Thought\" component in the ReAct format, could the authors provide qualitative examples of how these reasoning traces evolve throughout the training process?\n- In Tables 1 and 2, when SEEA-R1 outperforms other models, it is described as \"substantially outperforming\" or achieving \"state-of-the-art\" results, while similar performance gaps when SEEA-R1 is not leading are described as \"on par.\" Could the authors provide statistical significance tests (e.g., p-values, confidence intervals) for the reported success rates in Tables 1, 2, and 3?\n- Can the authors discuss the computational costs of the training and test-time inference process of this approach compared to others? Graphs of wall clock time or token usage at test time compared to baselines would be really insightful", "limitations": "The computational costs and potential for compounding error with longer training have not been sufficiently addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The framework achieves state-of-the-art success rates on the ALFWorld benchmark, outperforming previous models. Furthermore, SEEA-R1 shows encouraging generalization capabilities on the out-of-distribution EmbodiedEval benchmark, highlighting its potential for broader applicability in unseen scenarios.\n- The overall motivation for generating data and learning intermediate feedback signals as a means of solving long horizon tasks is intuitive, and the authors did a good job applying this to multi-modal settings where reasoning is harder than discrete settings. \n# Weaknesses\n- While the integration of MCTS with GRPO and the development of MGRM for embodied AI is a valuable contribution, the foundational components (GRPO and MCTS) are existing techniques. The novelty primarily lies in the engineering and successful application of these components in a novel, self-evolving framework for embodied agents, rather than the introduction of fundamentally new core reinforcement learning algorithms. This means that from a perspective of pure algorithmic invention, the paper builds upon and extends established methods rather than presenting a revolutionary new approach.\n- Despite the paper's emphasis on improving \"reasoning capability\" in embodied agents, there is a notable lack of in-depth qualitative analysis of the agent's reasoning traces. The ReAct format (Thought and Action) used in the agent's interactions provides a valuable opportunity to examine how the agent's internal thought processes evolve and become more sophisticated or coherent throughout training. Without such an analysis, the claims of enhanced reasoning are primarily supported by quantitative task success rates, leaving the specific mechanisms of reasoning improvement largely unexamined.\n- The reliance on a self-trained MGRM for reward signals, especially when ground truth rewards are absent, introduces a risk of compounding errors. If the MGRM generates inaccurate reward signals, the policy model could optimize for these flawed signals, potentially leading to a degradation of true task performance over extended training iterations, a concern not fully explored in the paper. While Figure 5 shows initial improvement, the long-term stability and fidelity of the self-supervised loop need more rigorous investigation to ensure the agent doesn't converge to a suboptimal or incorrect understanding of task completion. The paper mentions that an untrained MGRM can misguide learning, but the implications of a trained but imperfect MGRM iteratively influencing policy are not deeply analyzed. If there is a misunderstanding on my part I’m more than happy to be corrected, I think in general the description of the MGRM’s training was quite brief and led to a lot of confusion when reading.\n- There is also a lack of error bars or significance reporting in tables 1 and 2. When SEEA-R1 outperforms GPT this is claimed as a “significant” improvement, and when they underperform almost by the same amount in the text-only setting this is called “on-par\". The authors should sample multiple answers per question and report significance to adequately claim significance in these cases, and analyze the error modes in both settings when SEEA-R1 underperforms.", "was_revised": true, "final_justification": "After providing an extensive set of additional quantitative and qualitative results for the rebuttal to address each of my questions, I am increasing my score. The authors have detailed the unique algorithmic contributions of this work, shown the success in comparison to existing reasoning models, as well as shown reasoning traces which give insight as to why their method performs well / what it learns. The clarity of the method could use additional improvements, and since this is not my area I'm not sure about its general applicability to diverse settings, but overall I think this is a valuable paper with interesting engineering contributions.", "rebuttal": "We thank the reviewer for their detailed comments. We address points below, corresponding to weaknesses (W), comments (C), and questions (Q).\n\n\n**W1:** While MCTS/GRPO integration is valuable, foundational components are existing; novelty lies in engineering/application rather than new core algorithms.\n\n**Response to W1:**  \nOur work introduces a novel self-evolution framework for embodied agents, targeting the under-explored problem of \"autonomous learning without human supervision.\" Existing methods fail in embodied scenarios due to long-horizon feedback gaps and reward generalization challenges—we are first to define \"self-evolution\" in this domain.  \n\nKey innovations in SEEA-R1:  \n- Tree-GRPO: non-trivial design tailored for multi-turn embodied tasks to learn long-term dependencies in embodied scenarios.\n- MGRM: Generative model autonomously learning rewards from multi-modal, multi-turn trajectories.  \n- Joint training: Closed-loop collaboration between MLLMs and MGRM (vs. separate training), enabling co-evolution via bidirectional feedback.  \n\n\n**W2 & Q2:** Lack of qualitative analysis of reasoning traces; claims of enhanced reasoning rely on quantitative metrics.\n\n**Response to W2 & Q2:**  \nWe provide examples of reasoning evolution (to be added to the appendix):  \n\n*Baseline (Qwen-2.5-7B-Instruct) on \"find two CDs and put in safe\":*  \n- Uses exhaustive search, repeats failed actions (e.g., \"open drawer 3\" >10x despite no success).  \n\n*SEEA-R1 (iteration 5):*  \n- Avoids repeated failures; reasons about errors (e.g., \"putting CD in safe failed → try closing then opening safe\").  \n\n*SEEA-R1 (iteration 10):*  \n- Exhibits structured planning (\"find CD1 → put in safe → find CD2 → put in safe\") and corrects errors adaptively.  \n\nTraining enhances reasoning from brute-force search to goal-directed, error-correcting behavior.  \n\n\n**W3:** Risk of compounding errors with self-trained MGRM; long-term stability unproven.\n\n**Response to W3:**  \nOur MGRM training operates in two scenarios:  \n\n- **With Ground Truth (GT):** MGRM is pre-trained on GT rewards to >90% accuracy, avoiding large initial errors. During co-training, the policy optimizes using MGRM’s dense rewards, while MGRM continuously calibrates against GT.  \n\n- **Without GT:** MGRM (starting from Qwen2.5-VL-7B-Instruct) uses TTRL: the policy generates multiple trajectories for a state; MGRM’s majority-voted predictions form pseudo-GT. MGRM trains via GRPO (reward=1 for matching pseudo-GT), while the policy optimizes using MGRM’s judgments—creating closed-loop collaboration.  \n\n\nExperimental results in a GT-free scenario confirm effectiveness (Table 1). Despite short-term fluctuations, success rates rise consistently:  \n\n| Iteration | iter5 | iter10 | iter15 | iter20 | iter25 | iter30 |\n|-----------|-------|--------|--------|--------|--------|--------|\n| Success Rate (%) | 26.87 | 29.85 | 29.85 | 41.79 | 36.57 | 42.54 |\n\n*Table 1: Success Rates in GT-Free Scenario*  \n\nThis upward trend contradicts error accumulation expectations (which would show stagnation/decline). Closed-loop collaboration suppresses errors.  \n\nWe apologize for insufficient initial details on MGRM training and will expand this in the revised paper. \n\n**W4:** Lack of error bars/significance; \"significant\" vs. \"on-par\" claims inconsistent.\n\n**Response to W4:**  \n1. Our statement that SEEA-R1 has \"average completion steps (14.75) on par with GPT-4o\" refers only to step counts, not success rates (SEEA-R1: 84.3%).  \n\n2. In embodied domain, we propose optimal robot planning should directly output plans without feedback optimizations like MPO. Our comparison focuses on this mode: SEEA-R1 outperforms GPT-4o *without MPO*.  \n\n3. Evaluations used fixed prompts, temperature=0 (greedy decoding), and seeds—ensuring reproducibility and alignment with domain norms for valid cross-method comparisons.  \n\n\n| Model    | Setting       | Success Rate (%) |\n|----------|---------------|------------------|\n| GPT-4o   | Without MPO   | 81.1             |\n| SEEA-R1  | Without MPO   | 85.2             |\n\n*Table: Success Rates with and without MPO*\n\n\n**Q1:** Detail MGRM training; how binary rewards translate to Tree-GRPO’s advantage function.\n\n**Response to Q1:**  \nThank you for the feedback. We clarify the relationship between binary rewards and MLLM policy optimization:  \n- Binary rewards (1=correct, 0=incorrect) exclusively train MGRM via GRPO, enabling it to classify states (success/continue/failure) accurately. This process is independent of MLLM optimization.  \n- Trained MGRM’s state predictions drive MLLM updates via the advantage function: predictions map to task-aligned values (success=+reward, failure=-penalty); the advantage function quantifies action contributions, guiding MLLM to maximize success.  \n- Without ground truth, TTRL-style multi-trajectory voting generates pseudo-GT. MGRM trains on majority votes (binary reward), anchoring judgments to task logic and preventing policy drift.  \n\nBinary rewards train MGRM as a reward model and the MLLM agent optimizes using its judgments. We will clarify this in the manuscript.\n\n**Q3:** Need statistical significance tests for success rates.\n\n**Response to Q3:**  \nEvaluations use fixed parameters (temperature=0, seeds) for reproducibility, following domain norms. Results are consistent across runs; we will add 95% confidence intervals in the revised paper.  \n\n\n**Q4:** Computational costs vs. baselines.\n\n**Response to Q4:**  \nWe thank the reviewer for questions on computational cost. We clarify that our method is highly efficient, achieving superior results with lower computational cost, specifically in terms of token usage at inference.\n1. Algorithmic Optimization:\nStandard MCTS's complexity is $O(T⋅D⋅K)$. Our key contribution is algorithmic pruning optimization to solve this. Our pruning strategy uses probabilistic expansion and a path budget to reduce the complexity to a much more scalable $O(T⋅max(D,K))$.\nSpecifically, we employ a two-level pruning method:\n1. Probabilistic Expansion:  only having a 50% chance of expanding all k actions at a node.\n2. Strict Path Budget: More critically, we enforce a hard limit, L, on the number of full k-expansions along any single search path (e.g., L=5). \n\n2. Fast Inference: MCTS is for Training Only.\nTo clarify concerns on inference cost: MCTS is a training-time strategy only. At test time, our agent uses fast, ReAct-style single-path sampling and incurs no MCTS overhead. The superior policy learned via MCTS is why our agent achieves 3.1x higher success with 38.8\\% fewer tokens.\n\n- *Training:* SEEA-R1 takes 36h (8x A100); Qwen2.5 fine-tuning: 24h; PPO: 48h.  \n- *Inference:* SEEA-R1 (1.2s/step) is efficient vs. baselines, with 3.1x higher success and 38.8% fewer tokens (Table 2):  \n\n| Model | Success Rate (%) | Avg. Steps | Test Tokens |\n|-------|------------------|------------|-------------|\n| Qwen2.5 (Baseline) | 11.57 | 28.1 | 10,701,255 |\n| SEEA-R1 | 36.19 | 23.37 | 6,554,090 |\n\n*Table 2: Efficiency Metrics*"}]

Author rebuttal:

Meta-review: The reviewers are overall in agreement that the proposed method is novel and an improvement over prior work. There are some details between the method and experiments that were cleared up during the rebuttal process so lead to all reviewers in favor of paper acceptance. We adise the authors to put these updates into the paper.
Decision: Accept (poster)