Title: Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Reasoning

RL

Robot domain
Abstract: Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 learns to predict the next keypoint state required for task completion, conditioned on the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions. Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks. Despite having only 7B parameters, Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action control, such as spatial and movement reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article presents Robot-R1, a framework designed to enhance embodied reasoning for robot control by leveraging reinforcement learning. Traditional approaches often rely on Supervised Fine-Tuning (SFT) using datasets that are heuristically constructed and not specifically optimized for improving robot control, leading to issues such as catastrophic forgetting and reduced generalization performance. Robot-R1 addresses these limitations by using DeepSeek-R1 learning approach to learn to predict the next keypoint state required for task completion, based on the current scene image and environment metadata derived from expert demonstrations. Experimental results demonstrate that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The overall approach presented in the paper is relatively straightforward and easy to understand.\n2. Robot-R1 demonstrates improved performance compared to baseline methods.\n\n**Weaknesses**\n\n1. The core training methodology is derived from DeepSeek-R1, and the paper lacks further innovation in this aspect.\n2. There are some confusing statements in line 182. For example, the distinction between \"high-level action planning\" and \"subtask reasoning\" is unclear. It appears that \"high-level action\" might be synonymous with \"subtask,\" and similarly, \"primitive movement reasoning\" seems to overlap with \"subtask reasoning.\" Additionally, the categorization of \"primitive movement\" as low-level reasoning is questionable, especially since it is unclear how primitive movements alone can control a robot. The terminology is inconsistent throughout the paper (e.g., switching between \"high-level action planning\" and \"action planning,\" or \"primitive movement reasoning\" and \"primitive action reasoning\"), which makes it difficult to follow. It is recommended that the authors provide clear definitions and use consistent terminology. Visual aids, such as diagrams, could also help clarify these concepts.\n3. There are formatting issues, such as Table 3 appearing before Table 2.\n4. See Questions\n\nQUESTIONS:\n1. Since the training of Robot-R1 is conducted on MCQA constructed from expert demonstrations, (1) how is Robot-R1 adapted to Robot-R1 Bench? Or is it that Robot-R1 Bench is also in the form of MCQA, without the need for the model to generate open-ended answers? (2) How is Robot-R1 adapted to other benches such as EmbodiedBench Manipulation, which does not have answers in the form of MCQA?\n\n2. As shown in Appendix B.2 Prompt Template for Generating Training Dataset, the answer formats for Direct SFT and CoT-SFT are open-ended, while the answer format for Robot-R1 compared in the main text of the paper is in the form of MCQA. I believe this comparison is unfair. Although the appendix contains a comparison of the two answer formats for Robot-R1 (multiple-choice and open-ended), there is no comparison between the open-ended answers of Robot-R1 and the baselines.\n\n3. According to the results of EmbodiedBench Manipulation, GPT-4o achieves a 28.9% success rate in low-level manipulation, which is significantly better than Robot-R1. Should this result be included as one of the baselines in Table 4?\n\n4. Appendix B.1 Computing Cost states, \"Training the ROBOT-R1 task takes about 12 hours per epoch using 7.5K prompts.\" Regarding token costs, the phrase \"using 7.5K prompts\" is too vague. How long is a prompt? The total number of tokens consumed should be specified. Additionally, what about the cost of test? How many tokens are needed to test all tasks in the benchmark? And how many tokens do the baselines consume?\n\n5. According to the results shown in Figure 11(a) in the appendix, the open-ended QA format of Robot-R1 outperforms the MCQA format of Robot-R1 (achieving higher QA accuracy with fewer training steps). So, why not use the open-ended format of Robot-R1 in the main text of paper? Moreover, why not train for one more epoch (thus reaching the same number of training steps as the MCQA format of Robot-R1) to verify whether the open-ended format can achieve higher asymptotic performance?\n\n6. How can the accuracy of scoring using large models in the Robot-R1 Bench be ensured?\n\n7. The abstract mentions that SFT data (1) is often constructed heuristically, not explicitly optimized for improving robot control; (2) leads to issues such as catastrophic forgetting and reduced generalization performance. However, Robot-R1 (or all papers using the R1 training paradigm) still uses dataset that is the same as or similar to SFT, which has the disadvantage of heuristic construction. Moreover, how can it be ensured that the R1 training paradigm does not suffer from catastrophic forgetting and reduced generalization performance? There seems to be no literature proving this point. The paper mentions it but does not provide proof.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ROBOT-R1 that equips LVLMs with embodied reasoning abilities tailored for robot manipulation. Expert demonstrations are converted into three multiple-choice QA tasks—predicting the next waypoint, identifying the current state, and describing the primitive movement. The LVLM must first generate an explicit reasoning trace, then select an answer; the policy is optimized with GRPO using answer correctness as reward.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Using next-state prediction into MCQA plus auxiliary tasks makes RL over continuous actions tractable and sample-efficient.\n\n-  ROBOT-R1 Bench fills a gap by probing fine-grained embodied reasoning\n\n## Weaknesses\n\n- Mapping continuous keypoints to 4-option MCQA may limit precision; effect on real low-level control fidelity is unclear.\n- LLM-as-judge evaluation – Reliance on GPT-4o scoring introduces potential bias; no human cross-validation.\n- Simulation-only – No tests on physical robots; transfer from RLBench to reality or vision-noise robustness remains unverified.\n- Reward sparsity & design – Binary correctness reward ignores partial errors in continuous space; may cause brittle policies.\n\nQUESTIONS:\n1. How much precision is lost by discretising the next waypoint into four options? \n2. Can the final model be used in the action policy?\n3. Have you sampled a subset of ROBOT-R1 Bench and compared GPT-4o scores with human annotations to quantify judge bias?\n4. Real-World Validation\n5. Personally, I do not think the current RLBench contained tasks have much necessities for the manipulation reasoning, since most tasks are quite straightforward and with limited steps. I would suggest try some harder benchmarks like RoboCerebra or VLABench?\n6. Can this method extended to the rotation part, which is much harder to model with the language?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Robot-R1, an LVLM training framework leveraging reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 is trained to use the current scene image and environment metadata to predict the next action or state. The authors propose a new benchmark to evaluate the embodied reasoning capability for Robot-R1. The results show that Robot-R1 outperforms SFT methods. The authors also evaluate Robot-R1 on the EmbodiedBench Manipulation benchmark to show the improved performance in an actual robotic agent environment.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The idea of training an embodied reasoning model through reinforcement learning is interesting and worth exploring.\n2. The writing of the paper is clear and easy to understand.\n\n### Weaknesses:\n1. Lack of further analysis. The paper primarily presents extensive experimental results to showcase the performance of Robot-R1, but lacks analysis on why reinforcement learning leads to performance improvements. For example, the abstract mentions that \"SFT often leads to issues such as catastrophic forgetting and reduced generalization performance,\" yet the paper does not provide statistical evidence or concrete examples demonstrating forgetting in SFT models. It is important to show the advantages of RL training over SFT in this aspect.\nMoreover, Robot-R1 is trained on multiple-choice question answering data. How does it generalize to more general embodied reasoning tasks? The latter includes planning tasks, which appear to differ significantly in distribution from the training data. Some analysis of the training data is needed to clarify this point.\n2. Limitations in the evaluation. The main evaluation results of Robot-R1 are based on a benchmark proposed by the authors themselves and are evaluated using GPT-4o, which weakens the credibility of the results. Moreover, in the results shown in Table 2, Robot-R1 performs worse than GPT-4o, while in Table 3, the comparison is only made against GPT-4o-mini. A more comprehensive and systematic evaluation of Robot-R1 is necessary to support the claim in the abstract that it \"surpasses GPT-4o on reasoning tasks\" with greater persuasiveness.\n\nQUESTIONS:\n1. Could the performance of Robot-R1 be further enhanced by first performing supervised fine-tuning on a reasoning dataset as a cold start, followed by reinforcement learning (as done in DeepSeek-R1)?\n\n2. Is it possible to observe a gradual increase in the length of generated tokens during RL training?\n\n3. Has any decontamination been carried out to ensure that the training dataset does not overlap with Robot-R1 Bench? If so, what methodology was employed?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ROBOT-R1, a novel framework that applies Group Relative Policy Optimization to enhance embodied reasoning capabilities in Large Vision-Language Models (LVLMs) for robotic control. The key innovation lies in reformulating continuous robot control as multiple-choice question-answering tasks, where the model learns to predict next keypoint states through explicit reasoning processes. The authors train on three QA tasks: waypoint prediction, current state prediction, and primitive movement prediction, using few expert demonstrations from RLBench. Additionally, they introduce ROBOT-R1 Bench, their new benchmark for evaluating embodied reasoning in robotics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper uses DeepSeek-R1-style RL training to robotics, adapting GRPO for embodied reasoning task, where the novelty lies in reformulating continuous control as discrete multiple-choice QA, making RL tractable in high-dimensional action spaces.\n2. I appreciate the comprehensive experimental evaluation across multiple benchmarks (ROBOT-R1 Bench, EmbodiedBench Manipulation, SpatialRGPT).\n3. The approach shows strong performance gains on embodied reasoning tasks, outperforming GPT-4o despite having only 7B parameters (on EmbodiedBench Manipulation and SpatialRGPT). They show clear superiority over SFT baselines, which often fail completely on downstream tasks, demonstrating the efficacy of reasoning QA based approach that covers both high-level (planning, subtask) and low-level (movement, spatial) reasoning.\n\nWeaknesses:\n1. The experiments in the paper use single random seed with no error bars or statistical significance testing, which raises questions on the accuracy and reproducibility of results.\n2. The results are limited to table-top manipulation tasks in simulation (RLBench), with no real robot evaluation either. The results would be stronger if tested on some other navigation or manipulation tasks.\n3. The experiments use a relatively small training dataset (7.5K QA pairs across 5 tasks), which may not be enough to learn generalizable reasoning capabilities.\n4. The authors use GRPO (they mention that they are inspired from DeepSeek-R1 approach). Although GRPO has clear advantages in language prediction domain, proper comparisons with actor-critic RL approaches should be done to ascertain that GRPO is indeed a good choice as an RL algorithm for embodied reasoning tasks, which is missing in the paper.\n5. The approach requires access to a task-specific QA based dataset, which is not available for many embodied applications.\n\nQUESTIONS:\n1. Can you provide results with multiple random seeds and error bars? The lack of statistical validation significantly weakens the empirical claims. How robust are your improvements across different initializations?\n2. Have you tested ROBOT-R1 on real robotic systems? What are the expected challenges in transferring from simulation to reality, and how might performance degrade?\n3. How does performance scale with dataset size, number of tasks, and model parameters? What are the computational requirements compared to SFT?\n4. Can you provide more analysis of the failure cases of the proposed approach?\n5. How did you ensure ROBOT-R1 Bench provides unbiased evaluation? Have you considered validation by human evaluators?\n\nLIMITATIONS:\nPlease refer to the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}