Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
Authors: Dongyoung Kim
Sumin Park
Huiwon Jang
Jinwoo Shin
Jaehyung Kim
Younggyo Seo
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Reasoning
RL
Robot domain
Primary area: deep_learning

Abstract: Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 learns to predict the next keypoint state required for task completion, conditioned on the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions. Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks. Despite having only 7B parameters, Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action control, such as spatial and movement reasoning.

Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
Dongyoung Kim 144 Sumin Park' Huiwon Jang', Jinwoo Shin
Jachyung Kim*2, Younggyo Sco*3
KAIST, Yonsei University, 'UC Berkeley, 'RLWRLD
kingdy20020kaist.acc
Abstract
Large Lision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce ROBOT-R I, novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. ROBOT-RI I learns to predict the next keypoint state required for task completion, conditioned the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-PR R1 learning approach, ROBOT-RI R1 samples reasoning-basee responses and reinforces those that lead to more accurate predictions. To rigorously evaluate ROBOT-RI 1, also introduce new benchmark that demands the diverse embodied reasoning capabilities for the task. Our experiments show that models trained with Robot-R1 outperform SFT methods only 7B parameters, ROBOT-RI I even surpasses GPT-4o on reasoning tasks related low-level action control, such spatial and movement reasoning.
embodied reasoning tasks. Despite having
Introduction
Recently, Large Vision-Language Models (LVLMs) have shown significant promise in robotics [1, 21. By jointly processing visual inputs and natural language, LVLMs offer a powerful interface for interpreting complex real-world scenarios and enabling high-level reasoning robot control [3-6]. Specifically, they have been employed in robotics by providing high-level actions in the form of textual descriptions [7] and latent representations [8, 9] generate low-level actions, by directly generating low-level actions [10]. These capabilities not only improve the performance and generalization of robotic systems but also enhance the human-robot interaction interface by enabling intuitive, language-driven control.
Despite these advances, LVLMs often struggle to translate their general commonsense into the nuanced embodied reasoning required for controlling robots. For example, they often fail to accurately understand the spatial relationships crucial for low-level control [11] fail to generate high-level plans [12]. Consequently, achieving high performance in robotic tasks typically requires additional training in domain-speciff, data. A common approach involves Supervised Fine-Tuning (SFT) using question-answeringg datasets that pair task instructions with descriptions of various embodied reasoning types, such action planning and spatial reasoning [13-15]. However, embodied reasoning
'qqual advising
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A) Collect robot state from expert demo
(C) Train model
Correct
Wrong
Question
[xy,=]
Robot state [x'y'zz]
LVLM
Create various multiple-choice question answering
Question & Answers
GRPO
Next state
<think-The button is
Current state
Move right and down
Reward
Figure 1: Illustration the ROBOT-R1 framework. (a) ROBOT-R R uses robot states and image observations from expert demonstrations to create a dataset. (b) These data are reformulated into three different multiple-choice question answering (MCQA) tasks: predicting next states, current states, and movements. (c) During training, an LVLM solves MCQA tasks with reasoning which is then optimized using the GRPO algorithm [17] to reinforce reasoning pathways.
tasks for SFT are often heuristically designed and thus not fully optimized for actual robot action prediction; for instance, the linguistic expressions in SFT datasets often fail capture the precise quantitative details essential for low-level robot control. Moreover, models trained via SFT often struggle with input and output formats they were not trained on (i.e., distribution shifts), which leads to catastrophic forgetting that degrades previously acquired knowledge, including general conversational abilities [16].
Meanwhile, recent advances led by Deepseek-Ri R1 [17] have demonstrated the effectiveness of rein- forcement learning (RL) eliciting and reinforeing reasoning pathways, often achieving superior performance and generalization compared to SFT methods. Inspired by this, we introduce ROBOT-R, R1, novel framework that employs RL to effectively train LVLMs with embodied reasoning capabilities tailored specifically for robotic control (see Figure 1).
The key idea of ROBOT-RI is train LVLMs to predict the next keypoint state necessary for task completion through reasoning process, and to optimize this reasoning via RL to maximize prediction accuracy. However, since the keypoint lies in continuous space, the action space is vast, making exploration particularly challenging. This makes it difficult for the model to efficiently learn reasoning strategies through trial and error. To address this, Robot-RI reformulates the next-state prediction problem as a multiple-choice question ans (QA) problem. This discrete formulation narrows the action space, making the learning process more efficient. To further enhance state understanding. we add two auxiliary QA tasks: (i) current state prediction QA, where the model predicts the robot's s current state from visual observations, and (2) movement prediction QA, where the model predicts rule-based linguistic descriptions of state changes (e.g., "move up". "move down").
To analyze how ROBOT-RI improves embodied reasoning capabilities in trained models, design a novel benchmark called the ROBOT-R1 Bench (see Figure 2). Through this benchmark, we observe that models trained with ROBOT-RI achieve tailored for low-level action control. In particular, despite having only the 7B parameters, ROBOT- RI outperforms several major commercial models, including GPT-4o [18], in this domain. To evaluate whether the embodied reasoning abilities acquired through ROBOT-R RI transfer effectively other tasks, we conduct evaluations on two external benchmarks. First, on EmbodiedBench Manipulation [19], a vision-driven robot agent benchmark, ROBOT-RI yields 31% improvement in task performance. Moreover, on SpatialRGPTbench [20], which tests 3D spatial reasoning, name achieves approximately 40% improvement in quantitative metrics and about 60% improvement in qualitative metrics. These results contrast with conventional SFT approaches, which show limited improvement in embodied reasoning and tend to exhibit performance degradation when applied to -iitt---iittiuttionn tasks. This suggests that ROBOT-R can effectively learn generalizable and diverse embodied reasoning capabilities simply by next-state prediction.
28% improvement in embodied reasoning

---

2 Related Work
Embodied reasoning for robot control. Integrating Large Vision-Language Models (LVLMs) into robotic systems has recently emerged as a promising direction [1, 2]. These models leverage their vision-language understanding capabilities to interpret task instructions and process visual observations. Based on this understanding. LVLMs have been used to generate low-level actions directly [10, 21] or produce high-level action [3-7, 12, 22]. Recent efforts have focused on enhancing the embodied reasoning capabilities of LVLMs to improve performance in complex long- horizon tasks [23, 24]. For instance, their ability to reason over language and visual inputs has been exploited for action planning. high-level action prediction, and spatial reasoning [25-27]. A common (QA) datasets, which are designed to aid robot control [15, 11, 28-31J. These datasets are then used to fine-tund tne-ttne models via supervised fine-tuning (SFT). However, notable limitation is that such embodied reasoning datasets are often heuristically constructed and may not be explicitly optimized for robot control.
approach involves constructing embodied reasoning question-answent
Reinforcement learning for encourage reasoning. In large language models (I.LMs), generating intermediate reasoning steps - commonly known as Chain Thought (CoT) reasoning - has proven to be an effective strategy for improving performance across wide range of tasks [32]. This success has motivated substantial research into enhancing reasoning abilities through various ways, such as prompting techniques the distillation of reasoning paths from stronger models [33-36]. More recently, reinforcement learning (RL) has emerged as a strong alternative to SFT for training reasoning models. In particular, research such DeepSeek-RR R1 [17] has introduced RL-based frameworks where the model first generates a reasoning trace, then produces an answer based on this trace, and is optimized using reward signals based on answer correctness. This RL-based approach not only improves reasoning quality but also enhances sample efficiency and generalization across variety of tasks [37-39]. Consequently, it has been successfully applied domains such as mathematics [40, 41], agentic tasks [42, 43], and even vision-related problems [44, 45], yielding substantial performance gains. These advances suggest that applying RL train embodied reasoning for robot control could effectively address key limitations of SFT-based approaches.
3 Method
In this section, we present the components in the following order:
Preliminaries:
Definition of robot state, and GRPO [46] algorithm for RL training.
o Data Generation: How to prepare training dataset for ROBOT-RI.
Multiple-choice QA Base Training: How to train LVLM via RL with the generated datasets.
ROBOT-R1 Bench: Introduce new evaluation benchmark for embodied reasoning that assesses various reasoning capabilities (e.g. spatial understanding) in robotic control scenarios.
3.1 Preliminaries
Robot state. In this work, we utilize Franka Panda robot whose state is represented as 7- dimensional vector consisting of the end-effector's 3D cartesian position (x, y, 2), orientation (roll, pitch, yaw), and a binary gripper state (open/closed). For simplicity, following Liang et al. [22], we consider only the Cartesian position (x, y, z)) of the end-effector as the robot state s.
Problem setup. We denote the LLM policy as Ty that generates an answer ~ To Te(g) in response question q. reward r = freuara(() (o) is obtained by evaluating the generated answer through pre-defined reward model. This reward model is designed to assign higher values to answers that more accurately predict the next state. The LLM policy then optimized to maximize this reward.
Group relative policy optimization. We employ the Group Relative Policy Optimization (GRPO) algorithm [17] to optimize the policy. Specifically, for given query q. generate G different responses o = {01, 02, 03.,, OG from the current policy To... Each response o; is evaluated using the reward model to produce a corresponding reward ri, resulting in the set r = {ri, r2, ...r) }. For each response, we compute an advantage score A,, using the mean and standard deviation of the
rewards for that query, ie., A; Ti mean(r) Finally, GRPO updates the policy by maximizing this std(r)

---

advantage while applying a KL penalty, through the following objective:
Iczpo (0)
G
To
To(0:11) Te (oilg)) E, c.1+e A, A1) (K (To))
min
A,, clip
G
Tew (0:1g)
1=1
where and 3 hyperparameters.
3.2 Data Generation
For training ROBOT-RI, we use multiple-choice question answering (QA) data, which includes a primary task of predicting next waypoint (i.e., keypoint), along with two auxiliary tasks: (i) estimating the current state from an image and (ii) determining the necessary movements to reach the next waypoint. These tasks constructed using information extracted from expert demonstrations.
MetaData extraction. Without any contextual information, LVLMs struggle to infer low-level states. To address this, we utilize metadata M, which encodes details about the current task and the robot's low-level state, and use it to generate question prompts. The metadata M encompasses three essential types of information. First, it includes details about fixed reference points within the robot's environment, such as the center the 3D coordinate system, including the positive direction of each axis, which helps interpret how spatial changes are reflected in numerical state changes. Third, it incorporates the dimensions (æ, 2) of consistently present object, such as the robot's is end-effector, provide reference for scale estimation. This metadata M is subsequently used as conditional input during question generation.
table in table-top manipulation scenarios. Second, it defines
Waypoint prediction QA. The goal of this task is to predict future waypoint state. We represent demonstration as sequence of frames D = {(o,, so), (or, s1).... (oN. sN),, where each frame consists of observation on and its corresponding robot state st. A subset of these frames is designated as keypoints, denoted by indices K = {ko, ki...., in the trajectory completion of crucial subgoals [47]. The task is to predict the state 85* of the next keypoint following the current time step t, where h' min E K k, > t}. Each training example is thus represented as a tuple (St. or, sk-) ). To make multiple-choice question, we randomly sample three distractor states 8,1, 8,2, from the robot's valid state space. These candidates, along with the correct next waypoint Sk*, are shuffled to create the final question input Qusypoint (M, 8t, Ot, shufflet (SL 8d1, 8a2, 8d3 })) where the correct answer is A = 8ka
n......K,, kM J. which mark significant changes
Current state prediction QA. The goal to identify the correct current state 8t. The question prompt includes the metadata M and the current visual observation
To construct a multiple- choice question, we randomly sample three distractor states (8/21 2/ $113 from the state space. The question is formulated as Qatate [M, or, chuffle(f (e((((,))))))))) s'n1 5/22 8/33 1)) where the answer is A =
Movement prediction QA. The goal of this task is to predict the movement (e.g., "move up". "move right") that the robot should take to move from the current state toward the next waypoint. To extract labels for these movement, we apply rule-based heuristics based on the change in 3D Cartesian space. Specifically, we compute the difference in position between the current and next waypoint states, 8g- 8t, and identify the primary direction of movement along the x, y, and z axes. If there is a non- zero change along an axis, a corresponding directional command is generated (e.g.. "move forward" for positive change in the x-axis); this label is denoted as a, If the change along an axis is less than or equal to half of the largest change across all axes, the adverb "slightly" is added (e.g.. "Slightly move backward"). To construct a multiple-choice question, we randomly sample three adjacent movement {a'an- d'zz- d'z3). }- The question is formulated as Qnovenant (M, Ot, St, shuf flee (faa, d'n d/22- d'zx)) ))) where the answer is A = Gt-
3.3 Multiple-choice QA Base Training
Following DeepSeek-R1-Zeer [17], we train LVLMs to explicitly generate their reasoning process. During training, the policy is instructed to output its thought process enclosed within <think> and </think> tags, followed by the final answer enclosed within <answer> and </answer> tags. The

---

(A) Benchmark datasets
(C) LLM-as-judge
[GT answer] The gripper is positioned above the rubbish
Rubric
Q) What is the current spatial relationship
between the gripper and the rubbish?
(B) Generating answer
LVLM
Score: 0
A) The gripper is left of the rubbish
Figure 2: Illustration of the ROBOT-R1 Bench. (a) ROBOT-RI R1 Bench consists of human-written
questions paired with corresponding ground truth (reference) answers. (b) The LVLM under evalua- tion takes each question along with its associated image input and generates answer. (c) The generated answers are scored using GPT-4o, based on predefined rubrics and ground truth answers.
Table 1: Example of the ROBOT-RI Bench question and response.
What the immediate next meaningful subtask (i.c., a single step within a larger sequence) that should be performed given the current scene?
Question:
Model Answer <think>robot appears to be positioned above a saucepan with lid. The task is take the lid off the saucepan. The immediate next step would involve moving the gripper towards the lid grasp </think> <answer>the gripper towards the lid of the saucepan. </answer>
policy is optimized using GRPO. The reward signal Rt for RL consists of two components: a format reward ry and encourages adherence to the specified output structure. The answer reward ra provides rule-based positive feedback the model's answer, within danserr...<aasseeee exactly matches the correct option in the multiple-choice QA problem.
correctness reward ra, combined as ry + Ta- The format reward r
3.4 ROBOT-R1 Bench: A Novel Benchmark for Evaluating Embodied Reasoning
Existing visual question answering (VQA) benchmarks for embodied reasoning primarily focus general visual understanding, without explicitly evaluating the nuanced reasoning processes behind robotic behavior [48-50]. Mcanwhile, robotics-focused benchmarks often rely simulation- baseed evaluation [19, 51, 52]. priortize task successes as the main metric, or focus on tasks that are not aligned with the table-top manipulation scenarios central to work. While these benchmarks provide valuable insights, they do not directly assess a model's ability to reason about robot actions. To address this gap, introduice ROBOT-RI R1 Bench, new benchmark designed evaluate embodied reasoning through open-ended answering grounded in robot demonstrations.
Design features of ROBOT-RI Bench. In practical applications, models typically face open-ended decisions rather than selecting from pre-defined options. To better reflect this, ROBOT-R, R Bench adopts open-ended QA format that more closely resembles practical cases. All questions are based on images from expert demonstrations, ensuring visually grounded and realistic robotic scenarios. ROBOT-RI Bench supports fine-grained evaluation across four key reasoning types: planning, high-level action reasoning, movement reasoning. and spatial reasoning. Each question is crafted to assess one of these abilities, allowing for evaluation across both high-level decision making (planning and high-level action reasoning) and low-level control reasoning (movement reasoning and spatial reasoning).
Dataset construction. The ROBOT-RI Bench dataset consists of 10 tasks from RLBench [53]. For each task, we randomly sample five frames from expert demonstratrions, resulting in total of 50
5

---

Table ROBOT-R1 Bench results. Performance on embodied reasoning tasks tailored for low-level control evaluated using the ROBOT-RI R1 Bench. ROBOT-RE RI achieves the highest overall performance, outperforming leading commercial models across benchmark scores.
Movement
Spatial
Model
In
Out
Avg
Out
Avg
GPT-4o-mini [18]
0.64
0.56
0.60
1.73
1.14
1.48
GPT-4o [18]
0.92
0.52
0.72
1.70
1.07
1.43
Claude-3-Opus [59]
0.40
0.24
0.32
0.97
0.43
0.74
Claude-3.5-Haiku [60]
0.68
0.48
0.58
1,41
0.71
1.11
Claude-3.5-Sonned [61]
0.96
0.28
0.62
1.49
0.75
1.17
Claude-3.7-Sonnel [57]
0.76
0.48
0.62
1.65
1.07
1.4
Gemini-1.5-Fllate [62]
0.60
0.16
0.38
1.57
0,93
1.29
L.5-ro- [62]
0.76
0.40
0.58
1.49
1.14
1.34
Gemini-2.0-Flast [58]
0.52
0.40
0.46
1.76
1.14
1.49
Qwen2.5-WI-77-IIn [56]
0.64
0.52
0.58
1.62
1.11
1.40
w/ Direct SFT
0.12
0
0.06
0.08
0.04
0.06
w/ CoT SFT
0.84
0.56
0.70
0.46
0.07
0.29
w/ ROBOT-RI R1 (Ours)
0.96
0.56
0.76
1.76
1.18
1.51
images. For each image, experienced researchers manually created questions and detailed reference answers, targeting the four key reasoning abilities. The final dataset consists of 215 open-ended questions total: 65 for spatial reasoning and 50 for each the other three reasoning types.
Evaluation. Model responses are evaluated against reference answers. For consistent and objective scoring, we adopt an "LLM auujugee approach using GPT-4o [18, 54, 55]. Based on a predefined rubric tailored to each reasoning type, GPT-4o assesses the model's answers for accuracy, logical coherence, and completeness, and assigns a numeric score [0, 3). This allows for fine-grained quantitative analysis of the model's embodied reasoning capabilities (See Figure and Table 1).
Experiment
This section details the experimental setup for training ROBOT-RI, and the evaluation results. Our goal is to validate the effectiveness ROBOT-RI in learning embodied reasoning for robot manipulation tasks. We utilize the Dwen2.5--bbVL--In [56] as the base model for all experiments.
4.1 Experimental Setup
Training data. The training data used in our experiments are generated using the built-in data generator in RLBench [53]. We collect 50 demonstrations per task from the variation on 0 settings, rendering them at 224 x 224 resolution using OpenGL3. Waypoints from these demonstrations tracted using the waypoint extraction functionality provided in the ARM [47] repository!. The tasks selected from RLBench for training our model include: pick_ up_ cup, mush_buttonn
Table 2: Summary of ROBOT-RI
Bench results. In embodied reasoning tailored for high-level control, ROBOT- R significantly outperforms SFT, the baseline training method.
Planning
High-level
GFT--o [18]
1.96
2.02
clunde 1-ndle--7-somel111 [57]
L.72
1.58
gemini 2.0.Bash [58]
1.84
1.12
put_rubbish_in. and take _itee__id_____ of _ff_sauceaan. For generating the way- point prediction QA training data, as described in Section 3, a frame extraction interval of l = 10 was used between the current frame (Oz, 8c) and the selected
phone_on, satee__n_bbsseeee
Owen2. S.L.
1.66
1,04
SFT
ROBOT-RI (Ours)
1.44
future keypoint frame (sk-), similar demonstration augmentation proposed in ARM. Consequently, each task contains approximately 2.5K questions, resulting in a total of around 7.5K QA pairs across the three QA tasks used for training.
'https://githin http://ttttub..coo////t///////// ARM

---

Baseline models. The proposed ROBOT-RI I learns to predict the next waypoint state from expert demonstrations by generating an explicit reasoning process. To evaluate the importance of this learned reasoning, we establish two baseline models trained via Supervised Fine-tuning (SFT):
Direct SFT: This model is fine-tuned a QA dataset to directly predict the next waypoint state (sx-) ) from the current observation (or) and state (s;), without any intermediate reasoning steps. This baseline helps assess the performance gain achieved by incorporating reasoning.
This model is fine-tuned on QA dataset where the input prompt includes manually structured Chain-of-Thoughht (CoT) path. This CoT path sequentially incorporates planning. high- level action reasoning, and movement components to guide the prediction of the next waypoint state, similar to Zawalski et al. [15]. This baseline enables comparison between the reasoning learned via ROBOT-RI against reasoning guided by predefined, structured thought process.
CoT SFT:
By comparing ROBOT-RI against these baselines, we aim to highlight the importance of (i) in- corporating reasoning at all (v.s. Direct SFT) and the adaptiveness of reasoning learned through reinforcement learning (v.s. CoT SFT). See Appendix B for details.
Implementation detail. ROBOT-RE R1 is trained using GRPO as described in Section 3. We use the hyperparameter configurations in the EasyRI workspace. 2 For the training process, we utilize a batch size of 128 over 5 epoch. During GRPO updates, 5 samples were generated per prompt with a sampling temperature of 1.0. The rollout batch size is set to 512. We use a learning rate of 1.0 x 10-6 with weight decay of 1.0 x 10-2, For the SFT baselines, we use the same batch size, but learning rate is 1.0 x 10-5, We use the hyperparameters provided in the Oween-VLL-Finetung workspace."
4.2 Robot-R1 Bench
Setup. To reduce variance in benchmark results, all models generate responses with tempera- ture setting of O. To ensure the robustness of our benchmark and allow for relative performance comparisons, we evaluate the performance of several widely used commercial models across different versions, including GPT [18], Claude [57, 59-61], and Gemini [58, 62, 63]. Fur- thermore, we evaluate the performance of our own models: the base
1.2
Qwar2.5 2...2..5M--BBie of Ours
Gauds
Gamis
Qwen
Ower2 5-M -78ins
Score
0.8
0.6
0.1
11-.5---------Istture model [56], its variant fine-tuned via SFT, and the model trained using our proposed ROBOT- R1 method (see Appendix B for details).
Ae-mini
Japus S-halk
S Poommer 3 7-somee
1-5htahh
1.5
2 2-4tash
Figure 3: ROBOT-RI Bench results. In embodied reasoning tailored for low-level control, ROBOT- RI outperforms all previously reported models.
Results. Table 3 presents the results across the reasoning benchmarks focused on low-level con- trol, and Figure 3 provides its visualization to help understanding. We find that commercial model performance improves consistently with their version updates, indicating the benchmark's 's sensitivity to model quality. Notably, the ROBOT-RI -trained model achieves the highest performance across both the overall benchmark and its individual components, outperforming all commercial models evaluated. Specifically, in movement prediction, our method outperforms even the SFT-trained model, which is explicitly trained to predict accurate actions. Table 2 further show the performance in reasoning tasks tailored for high-level control. Here, the correlation between model version and benchmark performance is less pronounced. We hypothesize that this is due to the inherently high variability in valid high-level actions for robotic control. Nevertheless, we still observe a general trend of increasing benchmark performance as model quality improves. Importantly, our ROBOT-RI exhibits significant performance gains in high-level action reasoning compared the base and the SFT-trained model. This suggests that effective high-level reasoning can emerge from training exclusively on low-level control information, even without explicit high-level action supervision. Interestingly, we find that performance on planning slightly decreases, likely because our training objective focuses primarily on next keypoint prediction rather than long-horizon planning.
httpp://ggttab...co/////eeee Fil-FFnntuue hhttps:/ggithu com///1//qen2
7

---

Table EmbodiedBench Manipulation results. Success rate (%) of ROBOT-RL, SFT baselines. and various commercial models in evaluating the performance of EmbodiedBench Manipulation [19]. which evaluates LVLM performance for vision-based robotic agents.
Model
Base
Common
Complex
Spatial
Visual
Avg.
GPT-4o-mini [18]
4.2
6.3
2.1
10.4
4.6
Cluade-3.5-Hain [59]
12.5
12.5
12.5
16.7
13.9
13.6
Gemini-1.5-flan [63]
14.6
10.4
4.2
10.4
8.3
9.6
Qwen2.5-VL-7B--Ie [56]
6.3
6.3
6.3
14.6
11.1
8.92
Direct SFT
0
0
0
W CoT SFT
0
0
0
ROBO1-RI(Our))
12.5
8.3
6.3
14.6
16.7
11.68
4.3
EmbodiedBench Manipulation
Next, evaluate whether ROBOT-RI not only enhances embodied reasoning capabilities but also leads to improved performance in an actual robotic agent environment, we consider the Embod- iedBench Manipulation benchmark [19], vision-driven agent assessment platform built upon the RLBench simulation enviromment [53]. This benchmark evaluates model's ability to complete manipulation tasks by predicting low-level actions (a 7-dimension action vector) using in-context learning. A reward is awarded only if a task is completed successfully.
four different tasks: Pick & Place Objects, Stack Objects, Shape Sorter Placement, and Table Wiping. To comprehensively assess eestruction-flllowee capabilities, the benchmark categorizes task instructions into five groups: (i) Base, which requires fundamental task-solving skills, (ii) Common Sense, which requires general world knowledge to resolve indirect objective references, (iii) Complex instructions, which involves longer, possibly distracting context that obscures the primary command, (iv) Spatial reasoning, where objects are referred by their spatial relationships to other objects, and (v) Visual understanding, which requires recognizing objects based on visual attributes such as color or shape. All models are evaluated with sampling temperature of For comparison, the experimental results of GPT-4o-mini [18] and Gemini-1.5-flash [62] are adopted directly from the values reported in their paper.
Setup. We evaluate the model performance
Result. In Table 4, the model trained with the ROBOT-RI framework achieves approximate 31% performance increase over the baseline Dwen2.5--B-Innstuun model. In particular, in the Base category, which utilizes instructions similar those used during ROBOT-RI training, the model achieves nearly double the performance (6.3% 12.5%). This indicates that the embodied reasoning capabilities learned via ROBOT-RI transfer effectively downstream tasks, in setups that differ significantly from the training distribution. In contrast, the models fine-tuned with SFT failed to complete any tasks, highlighting the limitations of SFT approaches.
4.4 Ablation Study and Analysis
Spatial reasoning ability. To further evaluate whether enhanced spatial reasoning abilities of ROBOT-RI as shown in Section 4.2 and acquired through RL for optimizing reasoning paths in action prediction generalize to broader spatial understanding scenarios, we evaluate the model's performance on the SpatialRGPT benchmark [20]. This benchmark is designed to evaluate 3D spatial comprehension capabilities in LVLMs. SpP consists two main components: a qualitative evaluation, which tests understanding of relative positional relationships between objects in an image, and a quantitative evaluation, which requires accurate prediction of numerical spatial values. As shown in Table 5, the model trained with ROBOT-R RI achieves substantial performance gains, with an approximate improvement of 40% in qualitative tasks and around 60% improvement in quantitative tasks. These improvements are particularly notable when compared to the performance of models trained solely with SFT, highlighting the effectiveness of RL-optimized reasoning in developing transferrable spatial understanding.
Validation of the ROBOT-RI bench. To confirm the validity of the ROBOT-RI 1 Bench's I.L.-as-- judge, measure the Pearson correlation between LLM and human assigned score. Considering

---

Table 5: SpatialRGPT-Bench results. Accuracy (%) of ROBOT-RI R1, SET baselines in evaluating the performance of VQA tasks from SpatialRGPT-Bench [20], which measures (a) how LVLMs accurately predict the positional relations between objects, and (b) how LVLMS precisely estimate the absolute object positions, where the prediction is considered correct if it lies within +25% of the ground-truth values. In contrast to SFT models that struggle generalize to novel spatial reasoning tasks, ROBOT-RI effectively transfers these spatial reasoning tasks.
Quantitative results
Model
Direct Dist.
Heriz. Dist.
Vert. Dist.
Width
Height
Direction
Quant. Avg
Qwen2.5-VV Dwen2.5-VL-7B--1 VL--III- [56]
1.35
9.02
8.49
6.77
9.77
28.04
9.88
w Direct SFT
4.05
3.28
0.94
0.75
4.51
42.06
8.4]
CoT SFT
1.35
2.46
3.01
6.77
38.32
7.88
ROBOT- ROBOT--R(OOrr)
6.08
14.75
22.64
21.8
19,55
12.15
15.89
(b) Qualitative results
Model
Below/Above
Left/Right
Big/Small
Tall/Short
Wide/Thin
Behind/Front
Qual. Avg
Qwen2.5 5--L--LL-BBmm [56]
35
50.48
6.6
17.86
27.88
36,36
29.07
Direct SFT
17.14
5.02
CoT SFT
15.83
33.33
45.28
39.28
47.12
35.45
35.62
w ROBOT-R R1(ur)))
42.5
41.9
31.13
37.5
38.46
52.72
40.79
Table 6: Pearson correlation between human and GPT-4o [18] judgments on ROBOT-RI Bench. Computed over 80 responses (from ROBOT-R1 and GPT-4o), where human scores are defined the median of 8 human expert annotations.
Planning
High-level Action
Movement
Spatial 0.8961
Pearson Correlation
0.3315
0.8974
0.8931
the cost for collecting human annotations, sample 40 problems from the ROBOT-RI Bench and collect judgments from participants with expertise in robotics. The experiment is conducted as a blind evaluation of responses produced by ROBOT-H R1 and GPT-4o define the human score as the median across the participants. As shown Table 6, the Pearson correlations between LLM and human scores close to 0.9 for all tasks except planning, supporting the reliability of ROBOT-RI Bench. For the planning task, we observe a lower correlation, likely because its longer and more open-ended responses allow multiple valid solutions, making precise automatic judging inherently more challenging. Notably, the tasks with higher correlations, such as high-level action, movement and spatial, also where ROBOT-RI achieves the largest performance gains, further supporting the validity of the improvements achieved by ROBOT-R1.
total of 80 problems. We
Effect of different RL algorithms. We eval- uate the effectiveness of GRPO used to train ROBOT-R1 by training additional models with
Table 7: ROBOT-R1 Bench results different RL algorithms.
Model
Planning
High-level Action
Movement
Spatial
RLOO [64] and REINFORCE++ [65], two RL. algorithms commonly employed for LLM opti- REINFORCE+ [65]
GRPO [46]
1.44
0.76
[64]
0.62
mization. We use the default hyperparameters provided by the workspace and evaluate all models the Robot-R1 Bench. Table 7 shows that RLOO achieves performance comparable to GRPO; however, REINFORCE++ performs worse on average across tasks. We hypothesize that this is due to weaker reward normalization. Specifically, the
REINFORCE+ performs batch-level reward normalization, which often induces higher variance than
normalization applied in GRPO and RLOO, leading less stable learning and limited performance improvements. These findings indicate that RI. algorithms that incorporate variance-reducing mechanisms, such as GRPO or RLOO, are beneficial for ROBOT-R1.
Robustness across random seeds. To evaluate whether ROBOT-RI consistently yields performance gains across different random seeds, conduct two additional experimental runs the same dataset while changing only the random seed for online-sampling. We then evaluate all models on the Robot-RI Bench. As shown in Table 8, ROBOT-RI consistently achieves higher performance than
'nttps://giite httpps///gthhbb..oo//kyu//g/g///

---

Table Robot-R1 Bench results across random seeds. ROBOT-RI consistently improves perfor- mance over the non-finetuned original model (Qwen2.5-VL-7B- [56]) across different random seeds, showing strong robustness to seed variability.
Model
Planning
High-level Action
Movement
Spatial
Qwen2.5-VL-7B--Is [56]
1.66
1.04
0.58
1.40
ROBOT-RI(Ours) 1st Seed
1.44
1.30
0.76
1.51
W ROBOT-RI(Ours) 2nd Seed
1.70
1.46
0.64
1.66
ROBOT-RI(Ours) 3rd Seed
1.56
1.50
0.54
1.60
Seed Avg. Seed Std.
1.57
1.42
0.65
1.59
0.13
0.11
0.11
0.08
Table Ablation study. Evaluation results on the ROBOT-RI Bench comparing different dataset designs, varying auxiliary tasks, i.c., waypoint prediction (WP), current state prediction (SP), and movement prediction (MP), and the question format.
QA Type
Task Type
Low Level Control
High Level Control
Open-Ein End
MCQA
WP
SP
MP
Movement
Spatial
Avg
Planning
High-level Action
Ave
Y
x
x
0.68
1.37
1.03
1.60
1.26
1.43
/
x
0.48
1.03
0.76
1.48
1.16
1.17
d
x
0.40
1.42
1.44
1.22
1.33
/
0.54
1.51
1.50
1.24
1.37
/
/
0.76
1.51
1.44
1.30
1.37
non-finetuned model across seeds. The improvements remain significant when accounting for variance. These results demonstrate that training with ROBOT-RI is robust to seed variability.
Impact of QA design on performance. Table 9 demonstrates the effects of different QA design choices model performance. Our results show that adding auxiliary tasks progressively improves overall performance. We also evaluate an alternative approach that uses open-ended generation instead of multiple-choice question answering (MCQA). In this setup, the model directly generats next waypoint values an open-ended format and uses clip(1 L.1, 0, 1) as the reward function, where the I.1 represents the distance between the predicted and ground-truth states. While the open-ended approach initially outperforms single-task MCQA, exhibits significant performance degradation when auxiliary tasks such current-state prediction are introduced. Consequently, the MCQA framework demonstrates superior results. These findings highlight the critical role of question structure design in reinforcement learning for embodied reasoning.
Qualitative analysis. We find that the evolution of reasoning patterns during ROBOT-RI training diverges notably from trends reported in Deepseek-PP R1 [17]. While reasoning models for mathematical and coding tasks tend to develop longer reasoning chains over time, ROBOT-RI exhibits the opposite pattern of producing progressively shorter and more focused reasoning (see Appendix D for detailed examples). Qualitatively, the model initially generates broad planning or high-level action reasoning in an overview format. Over time, these are gradually replaced by more concise, logically structured reasoning traces that directly link relevant embodied reasoning components.
Conclusion
We have proposed ROBOT-RI, novel training framework that enhances the embodied reasoning capabilities of Large Languuge Models robotic domains. Our approach trains models to predict the next state based on image observations and current states from expert demonstrations, using explicit reasoning processed optimized through reinforcement learning. Our extensive experiments demonstrate that models trained with ROBOT-RI not only outperform Supervised Fine-Tuning (SFT) based approaches on embodied reasoning tasks but also exhibit consistent performance improvements across diverse embodied tasks while SFT methods often suffer from performance degradation.
10

---

Acknowledgements
This work was supported by Institute for Information communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) Nw..S-2019--11111000 Artificial Intelligence Graduate School Program(KAIST) No. RS-2024-00509279, Global AI Frontier Lab) and RLWRLD, Inc.
References
[1] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint (tt.xx::212.068//1 2022.
[2] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, al. Rt-2: Vision- languu-an action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15878 2023.
[3] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1 (2)6, 2023.
[4] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint NxXiv:2207.05600 2022.
[5] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 9119-14 PMLR, 2022.
[6] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision- laiinnlanguage planning. arXiv preprint arXiv:2311.178.4 2023.
[7] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03337, 2023.
[8] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. A lision-language-attion flow model for general robot control. arXiv preprint arXiv:24100 111110024116 24164, 2024.
[9] Qixiu Li, Yaobo Liang. Zeyu Wang, Lin Luo, Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action robotic manipulation. arXiv preprint arXiv:2411.19633 2024.
[10] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprini arXiv:2406.092. 2024.
(11) Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing lanion-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14455-14465, 2024.
[12] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01699 2022.
[13] Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open-ended instruction following with hierarchical ssioonlanguaag-actiio models. arXiv preprint arXiv:2502. 19417, 2025.
11

---

[14] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang. Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-lann language foundation models as effective robot imitators. arXiv preprini 1.X0v:2311.01378, 2023.
[15] Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of -hain-of-thought reasoning. arXiv preprint arXiv:2407.08693 2024.
[16] Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Ran Cheng, Yaxin Peng, Chaomin Shen, et al. Chatvla: Unified multimodal understanding and robot control with vision-language-actinn
model. arXiv preprint Xiv:22500. 14420, 2025.
[17] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294. 2025.
[18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[19] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang. Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint 2502.0050000 2025.
[20] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang. Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint arXiv:2406.01584 2024.
[21] Seonghyeon Ye, Joel Jang. Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410. 11758, 2024.
[22] Jacky Liang, Wenlong Huang. Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[23] Oier Mees, Lukas Hermann, Erick Rosete-Beat and Wolfram Burgard. Calvin: A benchmark for annuuag-connittione policy learning for long-horizon robot manipulation tasks. IEFE Robotics and Automation Letters, 73))7337-7334,, 2022.
[24] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023.
[25] Jiaqi Wang, Enze Shi, Huawen Hu, Chong Ma. Yiheng Liu, Xuhui Wang, Yincheng Yao, Xuan Liu, Bao Ge, and Shu Zhang. Large language models for robotics: Opportunities, challenges, and perspectives. Journal of Automation and Intelligence, 2024.
[26] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.059733 2023.
[27] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang. Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprini arXiv:2403.0963/ 2024.
[28] Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vla:: Scaling robot foundation models via unified diffusion and autoregression. arXiv preprint 2rXiv:2412.03229. 2024.
12

---

[29] Yide Shentu, Philipp Wu, Aravind Rajeswaran, and Pieter Abbeel. From lims to actions: latent codes as bridges in hierarchical robot control. In 2024 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2024.
[30] Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint wwXiv:2502.0585. 2025.
[31] Jaden Clark, Suvir Mirchandani, Dorsa Sadigh, and Suneel Belkhale. Action-free reasoning for policy generalization. arXiv preprint urXiv:2502.03729, 2025.
[32] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia. Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. 2022.
Advances in neural information processing systems,
[33] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, arXiv preprint arXiv:2203./
2022.
(34] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- 2023.
vances in neural information processing systems,
[35] Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-tf-thought fine-tuning. arXiv preprint arXiv:2305. /4045, 2023.
[36] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing IIm reasoning generalists with preference 2024.
trees. arXiv preprint 2xXiv22404.0207
[37] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.1933 2025.
[38] Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning large language models with one training example. arXiv preprint arXiv:2504.20577 2025.
[39] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 10000..4440 14476, 2025.
[40] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025.
[41] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
[42] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095// 2025.
[43] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.2 1620, 2025.
[44] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability multimodal large language models. arXiv preprint arXh::2003.06749/ 2025.
13

---

[45] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. VIm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint urXiv:2504.07615, 2025.
[46] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprini arXiv:2402.03328 2024.
[47] Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters,
2022.
[48] Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrish- nan, Christine Chan, Gabriel Dulac-Arold, Sharath Maddineni, Nikhil Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pages 645-652. IEEE, 2024.
[49] XAI. Realworldqa: A benchmark for real-world understanding, 2024. URL https:// huggingface. co/ ht////aaasss////e wwaps//aioorreeeee www.oo/gg/o/llla/...a Released alongside Grok-1.5 Vision Preview.
[50] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-10, 2018.
[51] Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Wang. VImbench: A com- positional benchmark for vision-and-lange language manipulation. Advances in Neural Information Processing Systems, 35:665-617 678, 2022.
[52] Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual foundation agents. arXiv preprint 4rXiv:2408.063227 2024.
[53] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): 3019-3026, 2020.
[54] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
[55] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus An open source lan- guage model specialized in evaluating other language models. arXiv preprint 45Xiv:2405.01535, 2024.
[56] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
[57] Authropic. Claude 3.7 sonnet. Anthropic Blog. 2024. URL news/claude-23 sonneet...
com/
[58] Demis Hassabis, Koray Kavukcuoglu, and Google DeepMind. Gemini 2.0: Google's new ai model for the agentic era. Google Blog, December 2024. URL https::/blog http://oooooooge//////oooo technology/gont hogp:/gggggegggeme//oooooggeeeee
[59] Anthropic. Introducin the claude 3 family: State-of-the-arr models intelligence, speed, and vi- sion. Anthropic Blog. 2024. URL
[60] Anthropic. Claude 3.5 haiku. Authropic Blog, October 2024. URL com//laude/haik..
[61] Anthoopic. Claude 3.5 sonnet. Anthropic Blog, 2024. URL URR.https://ww..attrpptt
news/cll news/claude- clwude-eaeeeeeeeee 5-5 sonnet.
14

---

[62] Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300, 2024.
[63] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXhv:2312.11888 2023.
[64] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in 1lms. arXiv preprint 2441::2402.147400, 2024.
[65] Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce+++: An efficient rhhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262, 2025.
[66] Homer Rich Walke, Kevin Black, Tony Zhao. Quan Vuong, Chongyi Zheng. Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, al. Bridgedata v2: dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736. PMLR, 2023.
[67] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin. Zuxuan Wu, Yu-Gang Jiang, et al. Vlabench: A large-scale benchmark for language- conditioned robotics manipulation with long-horizon reasoning tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11 1142-11152, 2025.
[68] OpenAl. Introducing gpt-4.1. OpenAl Blog, November 2024. URL wwt.hhtts//pppaaata com/ index/gpt-4-1.
[69] NVIDIA GEAR. GrOOt n1.5: improved open foundation model for generalist humanoid robots. https: httpps://resetteee //researchece nvidia. wtedii...oommm/mm com/labs/ ggear/rr0ttnn_555 , June 2025. Accessed: 2025-09-09.
[70] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776 44791, 2023.
15

---

A Limitation and Social Impact
Limitation. In this paper, we only considered the xyz positions as state, without considering the rotation of the end-effector and the manipulation of the gripper. Therefore, ROBOT-RI is hard to achieve reasoning about rotation or gripper movement, which are essential for complex tasks. Integrating these additional state dimensions would enhance ROBOT-RI R1's embodied reasoning ability for complex tasks.
Social impact. ROBOT-RI proposes a method that enhances embodied reasoning for robot control using small LVLMs (under 10B parameters) with limited data. This approach is easily accessible to research labs and small companies working on robotics. We expect ROBOT-RI to accelerate robotics research, facilitating faster application and adoption of robotic technologies in society. However, there is a potential risk that robots trained by reinforcement learning execute unintended actions to achieve their goals. This potentially leads to diminished human oversight, hence the design of complementary reward to mitigate such risk should be also explored in the future.
B Experiment Details
B.1
Computing Cost
All experiments are conducted on single node consisting of four A100 80GB GPUs. Training ROBOT-RI task approximately 12 hours for epoch using the 7.5K prompts.
B.2 Prompt Template for Generating Training Dataset
In this section, we provide the prompt templates used to generate the training datasets. To generate the Multiple-Chold Question Answering (MCQA) dataset as denoted in Section 3.2, we utilize three different prompt templates: the waypoint prediction MCQA prompt template (Figure 4), the state prediction MCQA prompt template (Figure 5), and the movement prediction MCQA prompt template (Figure 6). These templates take the current state, task instructions, and question about problem.
For generating the SFT dataset, we employ prompt structure analogous to the MCQA format. Specifically, we use two different prompt templates for both Direct SFT and CoT SFT: the waypoint prediction SFT prompt template (Figure 7) and the current state prediction SFT prompt template (Figure 8). However, the SFT dataset, unlike the ROBOT-RI R1 dataset, includes not only prompts but also answers. The Direct SFT dataset structured generate immediate answers, while the CoT SFT dataset incorporates a reasoning path for the waypoint prediction task. The CoT SFT dataset is designed to perform planning, high-level action reasoning, and movement prediction sequentially before predicting the final answer. The planning and high-level action reasoning components were defined using caman-created scripts.
16

---

Waypoint prediction MCQA
# You are Franka Robot Assistant: Task Planning and Execution System
## Task description
[task description)
## Visual Input
You will receive single combined image for understanding: <<image>: front view the workspace
## Coordinate System
The world coordinate frame follows these conventions:
This is based on the front view. (Wrist view has the Y-axis (left and right) opposite)
X--axis: Front of table (positive) back of table (negative)
Y-axis: Left (negative) right (positive)
Z-axis: Down toward floor (negative) to up toward ceiling (positive)
World origin [0.25, 0, 0.752) is at the center of the table surface
## Robot Specifications
Gripper dimensions: 0.06m width (x-direction) x 0.2 length (y-direction) 0.09 height (z-direction), with fingers 0.04 in length
## Current Robot State Position: [(x], (y). [z)]
### Choice Question
Based the provided image and current robot state, predict next waypoint position [x. y, z] Choose the most accurate option:
[[]] [A]
[[]] (B)
[[C]] (C)
[[D] (D)
Output Format
You FIRST think about the reasoning process an internal monologue and then provide the final answer.
The reasoning process MUST BE enclosed within <think> </think> tags.
final answer MUST BE enclosed within <answer> </answer> tags.
Example output format:
<think>
[detailed reasoning process]
</think>
<answer>
[[A]]
<lanswer>
Figure 4: Waypoint prediction MCQA prompt template
17

---

Current State prediction MCQA
# You are Franka Robot Assistant: Task Planning and Execution System
## Robot Task description (task_ ]
## Visual Input
You will receive single combined image for scene understanding: <image>; front view of the workspace
## Coordinate System
The world coordinate frame follows these conventions:
This is based on the front view. (Wrist view has the Y-axis (left and right) opposite)
X-axis: Front of table (positive) to back of table (negative)
Y-axis: Left (negative) to right (positive)
-7-axis: Down toward floor (negative) up toward ceiling (positive)
World origin [0.25, 0, 0.752] the center the table surface
## Robot Specifications
Gripper dimensions: 0.06m width (x-direction) x 0.2 length (y-direction) x 0.09 height (z-direction), with fingers 0.04 in length
### Choice Question
Let's predict the current robot state base on image
[[A]]
[[B]] (B)
[[C]] (C)
[[D] (D)
## Output Format
You FIRST think about the reasoning process internal monologue and then provide the final answer.
The reasoning process MUST BE enclosed within <think </think> tags.
The final answer MUST BE enclosed within <answer> </answer> tags.
Example output format:
<think>
[detailed reasoning process]
</think>
<answer>
[[A]]
<lanswer>
Figure 5: Current state prediction MCQA prompt template
18

---

Movement prediction MCQA
# You are Franka Robot Assistant: Task Planning and Execution System
## Task description [task_ )
## Visual Input
You will receive single combined image for scene understanding: <image>; front view of the workspace
## Coordinate System
The world coordinate frame follows these conventions:
This is based on the front view.
X-axis: Front of table (positive) to back of table (negative)
Y-axis: Left (negative) to right (positive)
7--axis: Down toward floor (negative) up toward ceiling (positive)
### Choice Question
What movements are needed get to next keypoint perform the task?
[[]]] (A)
[[B]] (B]
[[C]] (C)
[[D]] (D)
## Output Format
You FIRST think about the reasoning process as an internal monologue and then provide the final answer.
The reasoning process MUST BE enclosed within <think> </think> tags.
The final answer MUST BE enclosed within <answer> </answer> tags.
Example output format:
<think>
[detailed reasoning process]
</think>
<answer>
[[A]]
</answer>
Figure 6: Movement prediction MCQA prompt template
19

---

Waypoint prediction QA for SFT
Prompt
# You are Franka Robot Assistant: Task Planning and Execution System
## Task description [task_decrriptiig
## Visual Input
You will receive single combined image for scene understanding: <image>: front view of the workspace
## Coordinate System
The world coordinate frame follows these conventions:
This based on the front view. (Wrist view has Y-axis (left right) opposite)
X-axis: Front of table (positive) to back of table (negative)
Y-axis: Left (negative) right (positive)
Z-axis: Down toward floor (negative) up toward ceiling (positive)
World origin [0.25, 0.752] the center the table surface
## Robot Specifications
Gripper dimensions: 0.06m width (x-direction) x 0.2 length (y-direction) x 0.09 height (z-direction), with fingers 0.04 length
## Current Robot State Position: [Current State]
Let's determine the next rohot state to execute task
Answer for CoT SFT
Plan:
Move the gripper to align vertically and position just above the red button. 2. Move the gripper down for pressing the button. Subtask:
Move the gripper to align vertically and position just above the red button.
Move: slightly move front, slightly move left, and move down.
Answer: [Answer]
Answer for Direct SET
[Answer]
Figure 7: Waypoint prediction QA template for SFT
20

---

Current State prediction QA for SFT
Prompt
# You are Franka Robot Assistant: Task Planning and Execution System
## Task description
[task description)
## Visual Input
You will receive single combined image for scene understanding: <image>: front view of the workspace
## Coordinate System
The world coordinate frame follows these conventions:
-This based on the front view. (Wrist view has the Y-axis (left and right) opposite)
X-axis: Front of table (positive) to back table (negative)
Y-axis: Left (negative) to right (positive)
Z-axis: Down toward floor (negative) to up toward ceiling (positive)
World origin [0.25, 0.752] the center of the table surface
## Robot Specifications
Gripper dimensions: 0.06m width (x-direction) 0.2 length (y-direction) 0.09 height (z-direction), with fingers 0.04 length
Let's predict current robot state base on image
Answer
[Answer]
Figure 8: Current state prediction QA template for SFT
B.3 Benchmark Setup
To reduce variance in all benchmarks, we set the temperature of the model being evaluated to Meanwhile, the ROBOT-RI R1 Bench uses GPT-4o [18] for LLM-as-judge, and we set the temperature of GPT-4o to 1 (default temperature) for judgment tasks.
ROBOT-R1 Bench details. The evaluation dataset consists of (1) images corresponding to observa- tions, (2) questions related to embodied reasoning, and (3) human- written ground truth answers. The evaluated models receive the images and questions as input to generate responses. To improve the accuracy of answer generation, each model receives the following system prompt for every sampling process (see Figure 9). The evaluator, GPT-4o, provides scores between and points using the LLM-as-judge template prompt (see Figure 10) with the question, the response of the evaluated model, the predefined ground truth answer and the evaluation rubric. To sum up, full evaluation process is conducted as shown in Table 10.
B.4 Benchmark Setup
To reduce variance in all benchmarks, we set the temperature of the model being evaluated to Mcanwhile, the ROBOT-RI Bench uses GPT-4o [18] for LLM-assjudge, and set the temperature of GPT-4o to 1 (default temperature) for judgment tasks.
ROBOT-RI R1 Bench details. The evaluation dataset consists of (1) images corresponding to observa- tions, (2) questions related to embodied reasoning, and (3) uaman-writter ground truth answers. The evaluated models receive the images and questions as input to generate responses. To improve the
21

---

Table 10: Example of the working process ROBOT-R1 Bench
What is the complete sequence of high-level actions needed turn the lamp?
Question:
Model Raw Answer: <think> To determine the sequence of actions needed turn on the lamp, we need to consider the typical steps required for such a task. The lamp appears to be a standalone object with no visible control pancl or buttons in the image provided. Assuming the lamp be turned by remote control or switch, the following steps would be logical: Identify the lamp's location relative the robot. 2. Determine if there is remote control or switch accessible to the robot. 3. If remote control switch is present, usc the robot's arm interact with it. 4. If the lamp is directly accessible, move the robot's arm close to the lamp and interact with it. Since the image does not show a remote control or switch, and the lamp itself not directly accessible, the robot might need to perform series of actions to reach the lamp, such as moving towards interacting with it, and possibly using a tool activate it.< tu.//thi //thi/iik>>//////// <answer> I. Move the robot arm towards the lamp. 2. Interact with the lamp turn it on.</ </ns//nswe>>
Model Answer: 1. Move the robot arm lowards the lamp. 2. Interact with the lamp to turn on.
Ground Truth: 1. Position the robot's gripper near the lamp switch. 2. Press the lamp switch.
The model's answer correctly captures the intent and se- quence of actions (approaching and turning the lamp), but it uses terms ("move arm towards the lamp" and "intcract with the lamp to turn it on rather than explicitly stating positioning near the lamp switch and pressing it. While the core meaning is mostly preserved. the model omits the specific details of positioning near the switch and pressing it, making it slightly less precisc.
LLM-as-Judge Result:
general
Score: 2
Table 11: Robot-R1 Bridge Bench results
Model
Planning
High-level Action
Movement
Spatial
Gemini-1.5 Pro [62]
1.70
1.16
0.39
1.65
Qwen2.5-VL-77B-img [56]
1.20
0.89
0.46
1.53
WROBOT-RI(Ours
1.38
1.10
0.45
1.56
accuracy of answer generation, each model receives the following system prompt for every sampling process (see Figure 9). The evaluator, GPT-4o, provides scores between 0 and 3 points using the LLM-as-judge template prompt (see Figure 10) with the question, the response of the evaluated model, the predefined ground truth answer and the evaluation rubric. To up, full evaluation process is conducted as shown in Table 10.
C Additional Evaluation and Analysis
In this section, present additional evaluations and analyses further validate the effectiveness of ROBOT-RI. For efficiency, all experiments in this section are trained for one epoch before evaluation.
Robot-R1 Bench on real robot. To further investigate whether the embodied reasoning capability learned by ROBOT-R- R1 can transfer real-world robot settings, we extend the ROBOT-RI Bench to the ROBOT-R R1 Bridge Bench [66]. The problem construction and evaluation procedure remain identical to the original benchmark, but we use 100 randomly sampled images from real-robot demonstration videos in Bridge [66]. Table 11 shows the evaluation results on the ROBOT- RI Bridge Bench. Despite being trained only on RLBench simulation images, the model trained with ROBOT-R R1 demonstrates notable performance gains, indicating that the high-level embodied
22

---

Table 12: VLABench LVLM Bench results
Model
M&T
Common- Sense
Semantic
Spatial
Physical- Law
Complex
Qwen2.5-VI-7B-Ins [56]
39.02
40.78
37.64
35.83
39.20
38.51
w ROBOT-RI(Ours)
46.58
41.5
37.62
40.33
25.33
33.87
CoT-SFT
0
0
0
0
0
0
Table 13: Results of the cold start experiment on ROBOT-R1 Bench
Model
Planning
High-level Action
Movement
Spatial
Dwen2.5-VLL-7B-II [56]
1.66
1.04
0.58
1.40
+ High Quality COT-SFT
0.98
1.34
0.88
1.4
+ (ROBOT-RI(Ou))
1.22
1.44
0.88
1.43
reasoning ability acquired through ROBOT-RI successfully transfers to real-robot environments. However, improvements in the embodied reasoning tailored for low-level action are relatively limited, which we attribute to differences in camera viewpoints and coordinate systems between RLBench and V2.
Evaluation on VLABench. We further evaluate whether the model trained with ROBOT-R R gen- eralizes other robot agent environments beyond RLBench, using the VLABench [67] evaluation pipeline under the CoT setup. As shown in Table 12, the performance improves on the Mesh & Texture (M&T) and Spatial tasks, while it decreases slightly on the Physical Law and Complex Rea- soning tasks. We hypothesize that this performance drop stems from the fundamental gap between these task types and the abilities learned via ROBOT-RI. For instance, Physical Law tasks include questions such as "We have three objects with different densities. Choose the object with the smallest density," while Complex Reasoning tasks include questions such as "Please rearrange these books in chronological order starting from the first published to the latest." These tasks primarily require reasoning unrelated to embodied manipulation. In contrast, the M&T and Spatial tasks involve instructions such as "Add salt to the dish" "Add the second condiment from the bottom to the dish," which demand spatial understanding and movement prediction directly targeted by ROBOT-RI These findings demonstrate that the embodied reasoning capabilities learned through ROBOT-R R can be effectively transferred to new robotic environments beyond RLBench.
Cold-start strategy. Training with high-quality SFT reasoning data followed by RL is promising direction, as demonstrated Guo et al. [17]. To evaluate whether ROBOT-B R1 remains effective in such cold-start scenario, we conducted experiments using GPT-4.1-mii [68] to generate high- quality CoT-SFT data and then applied ROBOT-RI fine-tuning on top of the SFT trained model with human-
high-quality CoT-SFT data. Specifically, the SFT data are generated by conditioning annotated ground-truth planning and high-level action information, and prompting the model to produce CoT-style responses under the MCQA prompt used for RL. All SFT training settings followed those the main experiment. As shown in Table 13, ROBOT-RI demonstrates consistent performance improvements settings. 'However, SFT-only training still shows decreased planning task performance and limited improvement in spatial tasks, implying that even with high-quality supervision, models remain vulnerable to forgetting and poor generalization.
top of the SFT model, indicating the effectiveness cold-start
D More Quantitative Results
In this section, we present quantitative metrics observed during the ROBOT-RI training process. Specifically, we examine how the reward and response length change during training. Additionally, we include results for the Open-End approach trained ROBOT-RI in Section 4.4. Open-End approach, unlike MCQA approach, directly generats next waypoint values and use Ll distance for reward which is define between the predicted and ground-truth states. Figure la illustrates how the MCQA accuracy and the accuracy reward obtained from 1-L1 distance change during the training process.
23

---

Table 14: VLA performance on LIBERO
Model
Long
Goal
Objective
Spatial
Avg.
Qwen2.5-VI--BB- [56]
44.0
61.6
83.4
86.8
69.0
ROBOT-RI((urs)
45.8
73.8
80.4
87.0
71.8
Table 15: VLA performance on real robot
Model
PnP
PnP
PnP 3
PnP 4
Avg.
Qwen2.5-VL-7B-Ins [56]
25
0
20.83
20.83
16.67
ROBOT-RR(Ours)
25
29.17
25
16.67
23.96
These results show that GRPO training is being performed effectively.As shown in Figure 11b, ROBOT-RI also demonstrates that the response character length decreases throughout the training process. We hypothesize that this occurs the reasoning process transitions from summary format to a narrative format, during which redundant or unnecessary information is removed from the responses. Detailed analysis is provided in Section E.
E Qualitative Results
This section presents the differences between model responses with ROBOT-R1 and Qwen2.5-VI. 7b-ins (ROBOT-RI I's base model). Figure 12 shows response examples of ROBOT-RI R trained with the MCQA approach. Figure 13 shows response examples of ROBOT-R trained with the Open-End approach from Section 4.4. Commonly, Qwen2.5-VL-7b- demonstrates weak connections between reasoning process components and tends to generate responses in a summary format rather than narrative format. However, after ROBOT-RI I training, the reasoning processes become coherently connected and shift toward narrative format in their responses.
F Experiments on Vision-Language-Accioo Models
Finally, we evaluate whether training with ROBOT-R R1 only improves embodied reasoning but also enhances downstream robot control performance. We utilize the GROOT-NI.5 [69] repository for attaching a new diffusion policy head top on the LVLM backbone. Experiments are conducted in both the LIBERO simulation [70] environment and real robot.
LIBERO simulation. In LIBERO simulation, the environment is simulated using a Franka Panda Arm and includes four evaluation categories, such as, Spatial, Object, Goal, and Long, Each category consists of 10 tasks, and every task evaluated over 50 rollouts. The action policy trained using all given fine-tuning data with a batch size of 32 for 60k steps, following the original diffusion policy training setup while keeping all other hyperparameters identical the GROOT-N1.5 repository. Table 14 presents the success rates in the four LIBERO categories. The model trained with ROBOT-R outperforms the baseline on average. particular, the Goal category shows remarkable performance improvement, which we attribute to its complex manipulation requirements that are closely aligned with the reasoning enhancements encouraged by ROBOT-RI Although the Objective category shows slight performance drop, we attribute this to the limited object diversity in RLBench during ROBOT-RI training stage.
Real robot experiment. In real robot experience, we utilize Franka Research 3 arm to perform four different pick-and-place tasks, each involving four distinct objects: teddy bear, box, cup, and sponge. Each task is evaluated 24 times, resulting in 96 total trials to calculate the success rate. For training, each object has 60 demonstration trajectories, leading 240 training examples per task. For each task, we train separate action diffusion policy using its corresponding demonstrations for 30k steps with a batch setup. Table 15 shows that training the action diffusion policy with ROBOT-RI improves the average success rate from 16.67% to 23.90 23.96%, confirming ROBOT-RI effectiveness in enhancing real-world robot control performance.
of 32, while keeping other hyperparameters consistent with the simulation

---

System Prompt for Sampling in ROBOT-RI Bench
System Prompt Format
You are Al that accurately answers questions about robot actions and spatial relationships. Follow these rules strictly:
Answer ONLY what asked question.
2. Do not include any purpose or objective of actions (remove all "to...' phrases).
3. Do not include any additional descriptive information.
4. Keep answers concise and focused the information. Remove all unnccessary details about the current
conditions.
Direction should be judged based on the viewpoint in the image.
up: away from the ground
down: toward the ground
forward: toward the camera (where the image taken from)
* backward: away from the camera
right: to the right side from the camera's 's perspective
left: to the left side from the camera's 's perspective
TASK GUIDELINES:
[task description]
Additional Style Requirements:
Use simple and clear English.
Focus on semantic correctness, not stylistic variation.
Keep sentences short and remove unnecessary details.
multiple directions involved, combine them clearly (e.g., 'Move down and slightly right').
Task Description Examples
Task Type: "spatial"
Focus only on relative positions and spatial relationships between objects.
Do not describe any action or movement.
Only deseribe the current spatial configuration.
Example Answer: "The gripper is above the cup, offset the right."
Task Type: "planning"
List major actions in chronological order (I., 2., 3.,
Each step should describe only the action, without mentioning the purpose.
Example Answer: "1. Move the robot arm above the cup. 2. Lower the gripper to grasp the cup.
3. Lift the cup upward."
Task Type: "high level action"
Focus on the immediate next meaningful subtask (a single self celf-contained action).
Deseribe WHAT needs to be done, not HOW do it.
Only the immediate next step, not the final goal.
Example Answer: "Move the gripper closer to button."
Task Type: "movement"
-Specify only mechanical movements and gripper state changes.
UUe the robot's perspective for directions:
Desccribe only the very next physical movement.
Example Answer: "Move down and slightly right."
Figure 9: System prompt for sampling in ROBOT-R1 Bench
25

---

Evulation Rubric and LLM LLM-as- Judge prompt
Rubric
points: Meaning completely different from ground truth
Answer has completely different meaning from ground truth
Key concepts and ideas are misinterpreted
Contains information that contradicts ground truth
point: Partially matches but with significant meaning differences
Some key points match but main meaning is different
Contains significant misunderstandings of core concepts
Has some correct information but overall meaning incorrect
2 points: Mostly matches in meaning but with minor differences Main meaning and key points correct
Minor details or expressions are different
Overall context and intent are preserved
3 points: Meaning is equivalent to or more detailed than ground truth
Any expression that conveys the same basic meaning as ground truth
Any description that leads the functional outcome
Any expression maintains the same spatial relationship hetween objects
Any description that achieves the same goal through equivalent means
Any expression that preserves the core meaning while using different words
Any description that maintains the same context and intent
Any expression that describes the same target location and action
Additional Notes:
1. Focus on semantic equivalence rather than exact wording
2. Different expressions are acceptable if they convey the same meaning
Consider the overall context and intent of the answer
Minor differences in expression acceptable if the core meaning is maintained
Paraphrasing is considered perfect match the meaning is preserved
6. Different ways of describing the same action (c.g.. 'press' vs 'lip'' switch) should be considered equivalent
7. The specific actor (robot human) should not affect the score if the action described is functionally equivalent
Different ways of expressing the same spatial relationship (c.g., "over" vs 'toward") should be
considered cquivalent
9. Focus whether the answer achieves the same functional outcome the ground truth
10. Consider the answer if it describes the same action state using different but equivalent words
System Prompt
are an expert evaluating consistency between model's 's answer and ground truth answer. Please assign a score between 0-3 based on the given rubric and explain the reason in detail. You must respond in the format SScore: [0-3] Reason: [explanation]' in a single line. You may use line
breaks, but Score and Reason must be clearly separated and identifiable.
User Prompt
Please evaluate how well the model's matches the ground truth answer. Evaluation Criteria:
(rubric)
Question: [question]
Model's Answer: [model answerr
Ground Truth Answer: [ground truth)
Please assign a score between 0-3 and explain the reason in detail.
Figure 10: Evaluation rubric and L.MM-as-udge prompt for model answer assessment.
26

---

0.8
0.85
Mean Response Length
350
Open-Ere End QA MCQA
0.80
QA Accuracy
0.75
0.6
0.70
250
0.65
0.4
MCQA
0.60
200
Open-Eme End QA
50
60
Step
Step
(a) Avg reward change during ROBOT-RI R1 training
(b) response length change during ROBOT-R R1 training
Figure Training metrics for ROBOT-R1 across two learning approaches: Open-End QA and MCQA. We conducted 10 epoch training for Open-End End QA and epoch training for MCQA (note that all models reported in the paper are based on epoch training). Due to differences in dataset sizes between the two tasks, the graphs for the MCQA task (left axis) and LI distance for Open-End QA (right axis). (b) Shows mean response length over training steps for both task types.
plotted based on training steps. (a) Shows accuracy metrics
27

---

Question:
You are Franka Robot Assistant: Task Planning Execution System
Task deseription: lift the lid off the pan
[...]
Current Robot State: Position [0.265, -0.112, 1.154]
Choice Question: Based the provided image and current robot state, predict the waypoint position [x. y. zJ. Choose the most accurate option:
[[A]] [0.072, 0.243, 0.996]
[[B]] [-0.07, 0.247, 0.937]
[[C]] [0.254, -0.197, 0.894] [[D]] [0.187, -0.065, 1.144]
[..]]
GT Answer: [[C]]
Qwen2.5-VI-Ins Answer:
<think>
To predict the waypoint position for the robot lift the lid off the pan, need to consider the current position and the desired action. The robot is currently positioned above the pan, the lid is on top of pan. The goal is to lift the lid, which means the robot should move closer to the lid while maintaining a position that allows it to grasp the lid.
Let's analyze the options:
Option A: [0.072. 0.243, 0.996]
positioned above the lid, reasonable for grasping action
Option B: [-0.07, 0.247, 0.937]
further to the left, not efficient path
Option C: [0.254, -0.197, 0.894] further back, not aligned with lid position
Option D: [0.187, -0.065, 1.144] further up, not necessary for grasping
Option A has x and coordinates close to the lid's position, and the z-coordinate is slightly lower than the current position, which is necessary for the robot to reach the lid. Therefore, the most accurate option is A.
</think>
<answer>
[[A]]
</answer>
reward: 0
ROBOT-R1 Answer:
<think>
To predict the next waypoint position for the robot lift the lid off the pan, need to consider the current position of the pan and the lid, as well as the robot's current position and orientation. The lid is currently on the pan, and the robot is positioned above the pan.
Given the current robot position [0.265, -0.112, 1.154), the pan is located close to the robot, and the lid is on top of the pan. We need The most accurate position to predict next waypoint would be one that allows the robot to reach the lid on the pan.
the robot position where it grasp the lid and lift it off the pan.
Looking the coordinates of the pan and the lid, the most suitahle position for the robot to reach the lid would be position above the lid, but not directly top of it, to ensure the robot can grasp the lid without hitting the pan.
option [[C]] [0.254, -0.197, 0.894] is closest to while still heing above it, making it the most accurate choice for the next waypoint. </think
<answer>
[[C]]
</answer>
reward:
Figure 12: Example of model response (MCQA)
28

---

Question:
You are Franka Robot Assistant: Task Planning Execution System
Task description: chuck way rubbish on the table rubbish
[...]
Current Robot State: Position [0.275, -0.0099 1.448]
[...]
Let's determine the next robot state to execute task
GT Answer: [0.179, -0.026, 0.846]
Qwen2.5-VL-Ins Answer:
<think
The task is to chuck away rubbish on the table. From the visual input, we can see that there are two red objects (possibly apples) and crumpled piece of paper the table. The black trash bin present, indicating where the rubbish should be placed. The robot's gripper dimensions need
considered to determine it can pick up the paper. The gripper can easily pick up the paper, but the apples are larger and may require a different strategy or second task. For this task, the focus is on the crumpled paper.
The robot's current position is [0.275, +0.009, 1.448]. The next step would be move towards the crumpled paper, pick it up. and then move to the trash bin.
1. Move the rohot's 's arm towards the crumpled paper.
2. Open the gripper to pick up the crumpled paper.
Move the gripper the trash hin.
Close the gripper drop the paper into the trash bin.
5.
Move the robot's arm back to its starting position.
Let's calculate the coordinates for the paper and the trash bin:
The crumpled paper located at approximately [0.2, -0.1, 0.75]. The trash bin is located at approximately [0.4, -0.1, 0.75].
The robot will move from its current position to the paper and then to the trash bin. </think>
<answer>
-012 0.75)
</answer>
L1 distance hetween target: 0.191
ROBOT-R1 Answer:
<think>
The task is to remove any rubbish from the table. Observing the image, there apples and piece of paper the table. The paper appears to be the "rubbish" is not food item. The robot's gripper is positioned above the table, and the basket is nearby, suggesting it can be used to collect the rubbish. The robot needs to grab the paper and drop it into the basket.
The robot's current position is above the paper, and needs closer the paper to grab it. Once paper is picked up. should moved the basket and released.
Steps:
1. Move the rohot's 's gripper closer to the paper.
2. Grasp the paper.
Move the paper towards the basket.
4. Release the paper in the basket.
</thinl>
<answer>
[0.275, -0.009, 0.85] # Move closer the paper </answer>
LI distance between target: 0.117
Figure 13: Example of model response (Open-End QA)
29

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have discussed the limitations of supervised fine-tuning (SFT) for embodied reasoning and introduced ROBOT-RI 1 that enhances embodied reasoning via reinforcement learning. Throughout the paper, we have supported the effectiveness ROBOT-RI over SFT approaches with extensive experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We state it in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer:
[NA]
Justification:
This paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We disclose how to obtain all experimental data and the specific training code space and parameters at experiment section, which are all publicly accessible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All workspaces related to data generation and learning open the public. We utilize open source model for training.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(eglli/ggidee/Codeeeee
wwweinisssoooooiiiio onPolccy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublic//aeccccccccccccccc /gwpi/gg///////eeee// /Ca///eeeubbiiicc cotsSuubmisions. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental settings are described the Experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: For all the experiments, we use single fixed random seed. However, order to reduce randomness, we conducted tests by setting the temperature of model measured on all benchmarks O.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe computing resources in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttpa://neurps... cttppppicccccc///ccppppppppccccc csGuidelinest
Answer: [Yes]
Justification: This paper follows the every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include it Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
33

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research poses no risk as it focuses solely the inference of robot behavior.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes]
Justification:
All code, data, and models used in training are cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we have developed new benchmark. A detailed description of the benchmark is provided in the appendix. Additionally, we will release the dataset upon acceptance of this paper.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our research does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We state it in appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: pottps:/nntt http///nee.r.ppeeeeee for what should or should not be described.
36


Reviews:
[{"review_id": "Gb287QjQC4", "reviewer": "Reviewer_vxzH", "summary": "This paper introduces ROBOT-R1, a novel framework that applies Group Relative Policy Optimization to enhance embodied reasoning capabilities in Large Vision-Language Models (LVLMs) for robotic control. The key innovation lies in reformulating continuous robot control as multiple-choice question-answering tasks, where the model learns to predict next keypoint states through explicit reasoning processes. The authors train on three QA tasks: waypoint prediction, current state prediction, and primitive movement prediction, using few expert demonstrations from RLBench. Additionally, they introduce ROBOT-R1 Bench, their new benchmark for evaluating embodied reasoning in robotics.", "questions": "1. Can you provide results with multiple random seeds and error bars? The lack of statistical validation significantly weakens the empirical claims. How robust are your improvements across different initializations?\n2. Have you tested ROBOT-R1 on real robotic systems? What are the expected challenges in transferring from simulation to reality, and how might performance degrade?\n3. How does performance scale with dataset size, number of tasks, and model parameters? What are the computational requirements compared to SFT?\n4. Can you provide more analysis of the failure cases of the proposed approach?\n5. How did you ensure ROBOT-R1 Bench provides unbiased evaluation? Have you considered validation by human evaluators?", "limitations": "Please refer to the weaknesses section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper uses DeepSeek-R1-style RL training to robotics, adapting GRPO for embodied reasoning task, where the novelty lies in reformulating continuous control as discrete multiple-choice QA, making RL tractable in high-dimensional action spaces.\n2. I appreciate the comprehensive experimental evaluation across multiple benchmarks (ROBOT-R1 Bench, EmbodiedBench Manipulation, SpatialRGPT).\n3. The approach shows strong performance gains on embodied reasoning tasks, outperforming GPT-4o despite having only 7B parameters (on EmbodiedBench Manipulation and SpatialRGPT). They show clear superiority over SFT baselines, which often fail completely on downstream tasks, demonstrating the efficacy of reasoning QA based approach that covers both high-level (planning, subtask) and low-level (movement, spatial) reasoning.\n\nWeaknesses:\n1. The experiments in the paper use single random seed with no error bars or statistical significance testing, which raises questions on the accuracy and reproducibility of results.\n2. The results are limited to table-top manipulation tasks in simulation (RLBench), with no real robot evaluation either. The results would be stronger if tested on some other navigation or manipulation tasks.\n3. The experiments use a relatively small training dataset (7.5K QA pairs across 5 tasks), which may not be enough to learn generalizable reasoning capabilities.\n4. The authors use GRPO (they mention that they are inspired from DeepSeek-R1 approach). Although GRPO has clear advantages in language prediction domain, proper comparisons with actor-critic RL approaches should be done to ascertain that GRPO is indeed a good choice as an RL algorithm for embodied reasoning tasks, which is missing in the paper.\n5. The approach requires access to a task-specific QA based dataset, which is not available for many embodied applications.", "was_revised": true, "final_justification": "The authors satisfactorily addressed my concerns, especially the ones about extending to real robot experiments, adding multiple seeds to RL experiments, and the comparison with RL approaches apart from originally used GRPO. I believe the paper can be improved on the writing end, to add clarity, which is why I have given a final score of 4.", "rebuttal": "Dear Reviewer vxzH , \n\nWe sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.\n\n---\n\n**[W1,Q1] Reproducibility**\n\nWe agree that online reinforcement learning can exhibit variance. To address your concern, we conducted two additional experiments with different random seeds. The results are as follows, where we confirm that our method still demonstrates significant performance improvements over the baseline. We will include this result with error bars in the final draft.\n\n\\begin{array}{|l|c|c|c|c|} \\hline \n\\textbf{Method} & \\textbf{Planning} & \\textbf{Subtask} & \\textbf{Primitive} &\\textbf{Spatial} \\newline\n\\hline \n\\text{Qwen} & 1.66 & 1.04 & 0.58 & 1.40 \\newline\n\\text{GRPO seed=1} & 1.44 & 1.30 & 0.76 & 1.51 \\newline\n\\text{GRPO seed=2} & 1.70 & 1.46 & 0.64 & 1.66 \\newline\n\\text{GRPO seed=3} & 1.56 & 1.50 & 0.54 & 1.60 \\newline\n\\text{GRPO avg} & 1.57 & 1.42 & 0.65 & 1.59 \\newline\n\\hline \n\\text{RLOO} & 1.56 & 1.54 & 0.68 & 1.52 \\newline\n\\text{Reinforce++} & 1.40 & 1.08 & 0.62 & 1.38 \\newline \\hline\n\\text{GRPO with Half dataset} & 1.56 & 1.50 & 0.54 & 1.60 \\newline\n\\text{GRPO with single task} & 1.52 & 1.28 & 0.54 & 1.58 \\newline \\hline \n\\text{Qwen 3B} & 1.66 & 1.04 & 0.58 & 1.40 \\newline\n\\text{Qwen 3B + Robot-r1} & 1.58 & 1 & 1.08 & 1.23 \\newline\\hline\n\\end{array}\n\n---\n\n**[W2,Q2] Evaluation on diverse tasks and real robot system**\n\nTo address these concerns, we created a new Robot-R1 bench based on BridgeV2 real robot data and evaluated our model's performance on it. The results showed that performance improvements remained prominent across high-level actions, indicating Robot-R1 can be successfully transferred to real robot environments, despite the fact that it was trained using simulation data, RLBench. We will include the results in the final draft.\n\nHere, one can observe that low-level control reasoning did not transfer well, which we attribute to coordinate system changes. In RLBench, the camera position was always fixed facing the robot front, so we trained the model to describe robot direction and positional relationships based on images. With real robots, the camera position changed, requiring predictions of relative movements based on the robot arm as a reference. We believe this issue could be mitigated if the model were trained based on robot positioning from the beginning, which we plan to address in the final draft.\n\n\\begin{array}{c|cccc}\n\\hline\n& \\textbf{Planning} & \\textbf{Subtask} & \\textbf{Primitive} & \\textbf{Spatial} \\newline \\hline\n\\text{Qwen} & 1.20 & 0.89 & 0.46 & 1.53 \\newline\n\\text{Gemini-1.5 Pro} & 1.70 & 1.16 & 0.39 & 1.65 \\newline\n\\text{Robot-R1} &1.38 & 1.10 & 0.45 & 1.56 \\newline\n\\hline\n\\end{array}\n\n---\n\n**[W3] Dataset size issue**\n\nWe would like to clarify that models trained with reasoning incentivized RL can generalize effectively with less than 10K training data [1,2], and a recent study has even shown that generalization is possible from a single example [3].\n\nIn addition, we find that Robot-R1 trained with a 7.5K QA dataset shows consistent performance gains across diverse tasks from different benchmarks (Robot-R1 Bench, EmbodiedBench Manipulation, and SpatialRGPT-Bench), which empirically demonstrates that the 7.5K QA dataset is sufficient to learn generalizable reasoning capability.\n\n---\n\n**[W4] Traning algorithm comparison**\n\nWe agree with the reviewer in that investigating the proper RL algorithms for embodied reasoning tasks is an interesting problem. However, actor-critic RL like PPO requires excessive GPU VRAM and hence it’s hard to apply in the current constrained resources. Instead, we conduct an experiment by training Robot-R1 with widely used RL algorithms, RLOO [4] and Reinforce++ [5].\n\nAs shown in the above table in [W1,Q1], Robot-R1 trained with RLOO achieves similar performance to Robot-R1 trained with GRPO, and Reinforce++ degrades the performance. We think this is because the unstable training of Robot-R1 with Reinforce++, due to weaker reward normalization (batch reward normalization).\n\nWe will include this discussion in the final draft.\n\n---\n\n**[W5] The approach requires access to a task-specific QA based dataset, which is not available for many embodied applications.**\n\nThe main goal of Robot-R1 is to enhance VLM's embodied reasoning performance for robotic applications. The training data is constructed based on expert demonstration state trajectories, which can be readily obtained from most robotic learning datasets. \n\nThese reasoning abilities can be easily transferred to various tasks, including EmbodiedBench, real robot experiments [W2, Q2], and other benchmarks. This experimentally demonstrates that our approach is not heavily constrained by task-specific datasets.\n\n---\n\n**[Q3-1] Scaling law: How does performance scale with dataset size, number of tasks, and model parameters?**\n\nWe conducted experiments involving three modifications: reducing the training data size by half, utilizing a single task (put rubbish in bin) for training, and downsizing the model from 3B parameters. \n\nAs shown in the above table in [W1,Q1], the results demonstrated that Robot-R1 generalizes effectively even with smaller models. When the data volume was reduced by half, performance remained comparable within the margin of error. However, when limiting the number of tasks to single, we observed a slight performance degradation in subtasks. As mentioned in [W3], we attribute these results to the robust generalization capabilities of RL training.\n\n---\n\n**[Q3-2] Computing cost: What are the computational requirements compared to SFT?**\n\nTraining a model with RL (GRPO) requires around 1.5 times more training time than SFT.\n\n---\n\n**[Q4] Can you provide more analysis of the failure cases of the proposed approach?**\n\nAs mentioned in [W2,Q2], the low-level control reasoning capabilities learned from RLBench showed limited transferability to other robotic systems. This limitation is presumed to arise from the discrepancy between the coordinate systems used during the training phase and those employed during the testing phase.\n\n---\n\n**[Q5] Judging bias of LLM-as-judge evaluation: Human cross-validation experiments?**\n\nThank you for pointing this out. To address your concern, we conduct a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6].\n\n---\n\n[1] Zeng et al., Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, Arxiv 2025 \\\n[2] LMMS-Lab, Multimodal Open R1 8K Verified, Hugging Face, 2024/2025 \\\n[4] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025 \\\n[5] Ahmadian et al., Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, Arxiv 2024 \\\n[6] Hu et al., Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, Arxiv 2025\n\n---\n\nIf you have any further questions/concerns, please do not hesitate to let us know. \n\nThank you very much,  \\\nAuthors"}, {"review_id": "5qjABE7oD2", "reviewer": "Reviewer_4LvH", "summary": "This paper proposes Robot-R1, an LVLM training framework leveraging reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 is trained to use the current scene image and environment metadata to predict the next action or state. The authors propose a new benchmark to evaluate the embodied reasoning capability for Robot-R1. The results show that Robot-R1 outperforms SFT methods. The authors also evaluate Robot-R1 on the EmbodiedBench Manipulation benchmark to show the improved performance in an actual robotic agent environment.", "questions": "1. Could the performance of Robot-R1 be further enhanced by first performing supervised fine-tuning on a reasoning dataset as a cold start, followed by reinforcement learning (as done in DeepSeek-R1)?\n\n2. Is it possible to observe a gradual increase in the length of generated tokens during RL training?\n\n3. Has any decontamination been carried out to ensure that the training dataset does not overlap with Robot-R1 Bench? If so, what methodology was employed?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The idea of training an embodied reasoning model through reinforcement learning is interesting and worth exploring.\n2. The writing of the paper is clear and easy to understand.\n\n### Weaknesses:\n1. Lack of further analysis. The paper primarily presents extensive experimental results to showcase the performance of Robot-R1, but lacks analysis on why reinforcement learning leads to performance improvements. For example, the abstract mentions that \"SFT often leads to issues such as catastrophic forgetting and reduced generalization performance,\" yet the paper does not provide statistical evidence or concrete examples demonstrating forgetting in SFT models. It is important to show the advantages of RL training over SFT in this aspect.\nMoreover, Robot-R1 is trained on multiple-choice question answering data. How does it generalize to more general embodied reasoning tasks? The latter includes planning tasks, which appear to differ significantly in distribution from the training data. Some analysis of the training data is needed to clarify this point.\n2. Limitations in the evaluation. The main evaluation results of Robot-R1 are based on a benchmark proposed by the authors themselves and are evaluated using GPT-4o, which weakens the credibility of the results. Moreover, in the results shown in Table 2, Robot-R1 performs worse than GPT-4o, while in Table 3, the comparison is only made against GPT-4o-mini. A more comprehensive and systematic evaluation of Robot-R1 is necessary to support the claim in the abstract that it \"surpasses GPT-4o on reasoning tasks\" with greater persuasiveness.", "was_revised": true, "final_justification": "Weakness 1 only presents an analysis without statistical support. Therefore, I kept my rating.", "rebuttal": "Dear Reviewer 4LvH,\n\nWe sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.\n\n---\n\n**[W1-1] Lack of further analysis**\n\nThe evaluation benchmarks used in our paper, Robot-R1 bench, EmbodiedBench Manipulation, and SpatialRGPT, are significantly different from our training task of waypoint prediction. SpatialRGPT even uses completely different inputs based on real images. Robot-R1 demonstrates consistent performance improvements across all these environments, while SFT shows significant performance degradation in certain tasks (e.g., EmbodiedBench). This experimentally demonstrates the generalization capability of RL and the forgetting issues inherent in SFT.\n\nFurthermore, it is well-established in the LLM community that SFT causes catastrophic forgetting, while RL achieves better generalization with minimal data [1]. For instance, Deepseek-R1 showed that models trained with RL on coding and mathematics tasks also improve performance across other domains [2]. Additionally, research has demonstrated that RL can achieve generalization even with single samples [3].\n\n---\n\n**[W1-2] RL training is good at generalization?**\n\nWe would like to clarify that Robot-R1 is trained to explicitly perform embodied reasoning before answering the multiple-choice question. As a result, Robot-R1 generates various embodied reasoning, e.g., planning how to perform the task.\n\nHere, we hypothesize that wrong embodied reasoning leads to wrong responses to multiple-choice questions, and vice versa. As the model is trained to generate a correct response, we can expect that the model can generate more accurate embodied reasoning from the training. We have included the qualitative results regarding the generated embodied reasoning quality in Appendix D.\n\n---\n\n**[W2] Limitations in the evaluation**\n\n**[Validation of Robot-R1 bench]** To address your concern, we conducted a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6]. In particular, the high correlation observed in items related to low-level control reasoning reinforces our claim that Robot-R1 \"surpasses GPT-4o on reasoning tasks related to low-level action control.\"\n\n**[Comparison with GPT-4o]** We clarify that GPT-4o can’t be a fair baseline, as it has a significantly larger model size than Qwen-2.5 7B, which we used to train Robot-R1. As our goal is to develop the training methodology to enhance the embodied reasoning in robotics, our baseline is an initial backbone LLM (no fine-tuning) or other training methods such as SFT. For this reason, we present the performance of GPT-4o and other LLMs to provide a useful reference for better performance judgment. \n\n---\n\n**[Q1] Cold Start SFT** \n\nWe could design experiments to perform reinforcement learning on top of an SFT model that inherits Robot-R1's reasoning capabilities. We will conduct these experiments during the remaining period and aim to include the results in the final draft.\n\n---\n\n**[Q2] Is it possible to observe a gradual increase in the length of generated tokens during RL training?**\n\nWe actually observed the opposite trend that the length of the generated token is decreasing during RL training. This occurred since sentences became more concise and shifted from overview-style to descriptive formats. For specific length changes, please refer to Figure 11, and for qualitative analysis, please see Appendix C.\n\n---\n\n**[Q3] Has any decontamination been carried out to ensure that the training dataset does not overlap with Robot-R1 Bench? If so, what methodology was employed?**\n\nSince RLBench generates new demonstrations each time, we separated the data by extracting from differently generated demos to ensure no overlap between training and evaluation datasets.\n\n---\n\n[1] Chu et al., SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, ICML 2025 \\\n[2] Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, Arxiv 2025 \\\n[3] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025\n\n---\n\nIf you have any further questions/concerns, please do not hesitate to let us know. \n\nThank you very much,  \\\nAuthors"}, {"review_id": "X5hTqqw5Kz", "reviewer": "Reviewer_8V4V", "summary": "The paper introduces ROBOT-R1 that equips LVLMs with embodied reasoning abilities tailored for robot manipulation. Expert demonstrations are converted into three multiple-choice QA tasks—predicting the next waypoint, identifying the current state, and describing the primitive movement. The LVLM must first generate an explicit reasoning trace, then select an answer; the policy is optimized with GRPO using answer correctness as reward.", "questions": "1. How much precision is lost by discretising the next waypoint into four options? \n2. Can the final model be used in the action policy?\n3. Have you sampled a subset of ROBOT-R1 Bench and compared GPT-4o scores with human annotations to quantify judge bias?\n4. Real-World Validation\n5. Personally, I do not think the current RLBench contained tasks have much necessities for the manipulation reasoning, since most tasks are quite straightforward and with limited steps. I would suggest try some harder benchmarks like RoboCerebra or VLABench?\n6. Can this method extended to the rotation part, which is much harder to model with the language?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- Using next-state prediction into MCQA plus auxiliary tasks makes RL over continuous actions tractable and sample-efficient.\n\n-  ROBOT-R1 Bench fills a gap by probing fine-grained embodied reasoning\n\n## Weaknesses\n\n- Mapping continuous keypoints to 4-option MCQA may limit precision; effect on real low-level control fidelity is unclear.\n- LLM-as-judge evaluation – Reliance on GPT-4o scoring introduces potential bias; no human cross-validation.\n- Simulation-only – No tests on physical robots; transfer from RLBench to reality or vision-noise robustness remains unverified.\n- Reward sparsity & design – Binary correctness reward ignores partial errors in continuous space; may cause brittle policies.", "was_revised": true, "final_justification": "The rebuttal clears my most concerns, and I would like to keep my scores for supporting an accept.", "rebuttal": "Dear Reviewer 8V4V,\n\nWe sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below. \n\n---\n\n**[W1,W4,Q1] Reward sparsity & QA design** \n\nWe clarify that our main goal is to enhance various embodied reasoning abilities. For establishing QA design, we compared both MCQA and open-ended QA approaches and experimentally confirmed that MCQA can better enhance embodied reasoning performance. We assume that MCQA provides more opportunities for diverse embodied reasoning through the process of comparing answer candidates, compared to open-ended approaches (see details in Table 6). Based on these results, we chose the MCQA approach despite some potential precision loss.\n\n---\n\n**[W2,Q3] Judging bias of LLM-as-judge evaluation: Human cross-validation experiments?**\n\nTo address your concern, we conducted a human cross-validation experiment on a subset of the Robot-R1 bench. See details in our response to Reviewer 5ayi's [Q6].\n\n---\n\n**[W3] Evaluation on real robot setup**\n\nFollowing your constructive suggestion, we conduct an additional experiment by constructing the Robot-R1 Bench on the BridgeV2 real robot video dataset [1] and then evaluating various LVLMs, including Robot-R1 (trained on RLBench). As shown in the below table, we find that Robot-R1 consistently outperforms or is competitive with the baseline, Qwen (non-finetune), indicating Robot-R1 can be successfully transferred to real robot environments, despite the fact that it was trained using simulation data. We will include the results in the final draft.\n\nHere, one can observe that low-level control reasoning is just competitive with the non-finetuned model (Qwen), which we attribute to coordinate system changes. In RLBench, the camera position was always fixed facing the robot front, so we trained the model to describe robot direction and positional relationships based on images. With real robots, the camera position changes, requiring predictions of relative movements based on the robot arm as a reference. We believe this issue could be mitigated if the model were trained based on robot positioning from the beginning, which we plan to address in the final draft.\n\n\\begin{array}{c|cccc}\n\\hline\n& \\textbf{Planning} & \\textbf{Subtask} & \\textbf{Movement} & \\textbf{Spatial} \\newline\n\\hline\n\\text{Qwen} & 1.20 & 0.89 & 0.46 & 1.53 \\newline\n\\text{Gemini-1.5 Pro} & 1.70 & 1.16 & 0.39 & 1.65 \\newline\n\\text{Robot-R1} &1.38 & 1.10 & 0.45 & 1.56 \\newline\n\\hline\n\\end{array}\n\n---\n\n**[Q2] Can the final model be used in the action policy?** \n\nThe main goal of Robot-R1 is to enhance LVLM's embodied reasoning performance, and it would be challenging to use directly for low-level control due to insufficient precision. However, improved embodied reasoning performance often benefits VLA training [1,2], and we expect that incorporating Robot-R1 into VLA training would yield models with superior performance, which we believe is a promising direction to explore in the near future.\n\n---\n\n**[Q5] RLBench is too simple**\n\nThank you for your suggestion. However, we would like to clarify that RLBench contains many complex tasks that require multi-step reasoning, and we select them to construct the training and test datasets. For example, “put rubbish in bin” requests the robot agent to identify the trash, pick it up, carry it to the bin, and finally dispose of it. Furthermore, we already have considered EmbodiedBench Manipulation (see Table 4 in the paper), which includes multi-step reasoning like “Stack the front star on top of the right cylinder”. We also note that we find the improvement from Robot-R1 in more challenging environments, such as the Bridge dataset (see [W3]). Nevertheless, we plan to include more experimental results on various benchmarks that require complex reasoning in the final draft.\n\n---\n\n**[Q6] Can this method be extended to the rotation part?**\n\nThank you for the insightful suggestion. Although expressing rotations in language is challenging, we believe we can construct the tasks regarding rotations by using the concept of roll, pitch, and yaw. We think this extension is indeed an interesting future direction for us, and we will include relevant discussion in the final draft.\n\n---\n\n[1] Walke et al., BridgeData V2: A Dataset for Robot Learning at Scale., Arxiv 2023\n\n---\n\nIf you have any further questions/concerns, please do not hesitate to let us know. \n\nThank you very much,  \\\nAuthors"}, {"review_id": "YVZKdPaF8h", "reviewer": "Reviewer_5ayi", "summary": "This article presents Robot-R1, a framework designed to enhance embodied reasoning for robot control by leveraging reinforcement learning. Traditional approaches often rely on Supervised Fine-Tuning (SFT) using datasets that are heuristically constructed and not specifically optimized for improving robot control, leading to issues such as catastrophic forgetting and reduced generalization performance. Robot-R1 addresses these limitations by using DeepSeek-R1 learning approach to learn to predict the next keypoint state required for task completion, based on the current scene image and environment metadata derived from expert demonstrations. Experimental results demonstrate that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks.", "questions": "1. Since the training of Robot-R1 is conducted on MCQA constructed from expert demonstrations, (1) how is Robot-R1 adapted to Robot-R1 Bench? Or is it that Robot-R1 Bench is also in the form of MCQA, without the need for the model to generate open-ended answers? (2) How is Robot-R1 adapted to other benches such as EmbodiedBench Manipulation, which does not have answers in the form of MCQA?\n\n2. As shown in Appendix B.2 Prompt Template for Generating Training Dataset, the answer formats for Direct SFT and CoT-SFT are open-ended, while the answer format for Robot-R1 compared in the main text of the paper is in the form of MCQA. I believe this comparison is unfair. Although the appendix contains a comparison of the two answer formats for Robot-R1 (multiple-choice and open-ended), there is no comparison between the open-ended answers of Robot-R1 and the baselines.\n\n3. According to the results of EmbodiedBench Manipulation, GPT-4o achieves a 28.9% success rate in low-level manipulation, which is significantly better than Robot-R1. Should this result be included as one of the baselines in Table 4?\n\n4. Appendix B.1 Computing Cost states, \"Training the ROBOT-R1 task takes about 12 hours per epoch using 7.5K prompts.\" Regarding token costs, the phrase \"using 7.5K prompts\" is too vague. How long is a prompt? The total number of tokens consumed should be specified. Additionally, what about the cost of test? How many tokens are needed to test all tasks in the benchmark? And how many tokens do the baselines consume?\n\n5. According to the results shown in Figure 11(a) in the appendix, the open-ended QA format of Robot-R1 outperforms the MCQA format of Robot-R1 (achieving higher QA accuracy with fewer training steps). So, why not use the open-ended format of Robot-R1 in the main text of paper? Moreover, why not train for one more epoch (thus reaching the same number of training steps as the MCQA format of Robot-R1) to verify whether the open-ended format can achieve higher asymptotic performance?\n\n6. How can the accuracy of scoring using large models in the Robot-R1 Bench be ensured?\n\n7. The abstract mentions that SFT data (1) is often constructed heuristically, not explicitly optimized for improving robot control; (2) leads to issues such as catastrophic forgetting and reduced generalization performance. However, Robot-R1 (or all papers using the R1 training paradigm) still uses dataset that is the same as or similar to SFT, which has the disadvantage of heuristic construction. Moreover, how can it be ensured that the R1 training paradigm does not suffer from catastrophic forgetting and reduced generalization performance? There seems to be no literature proving this point. The paper mentions it but does not provide proof.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The overall approach presented in the paper is relatively straightforward and easy to understand.\n2. Robot-R1 demonstrates improved performance compared to baseline methods.\n\n**Weaknesses**\n\n1. The core training methodology is derived from DeepSeek-R1, and the paper lacks further innovation in this aspect.\n2. There are some confusing statements in line 182. For example, the distinction between \"high-level action planning\" and \"subtask reasoning\" is unclear. It appears that \"high-level action\" might be synonymous with \"subtask,\" and similarly, \"primitive movement reasoning\" seems to overlap with \"subtask reasoning.\" Additionally, the categorization of \"primitive movement\" as low-level reasoning is questionable, especially since it is unclear how primitive movements alone can control a robot. The terminology is inconsistent throughout the paper (e.g., switching between \"high-level action planning\" and \"action planning,\" or \"primitive movement reasoning\" and \"primitive action reasoning\"), which makes it difficult to follow. It is recommended that the authors provide clear definitions and use consistent terminology. Visual aids, such as diagrams, could also help clarify these concepts.\n3. There are formatting issues, such as Table 3 appearing before Table 2.\n4. See Questions", "was_revised": true, "final_justification": "- The author's response has addressed my concerns. And I will raise the score from 2 to 4.\n- I still care a lot about the consistency of the terminology used in the paper, which indeed poses a significant barrier to reading. I hope the authors can make timely corrections.", "rebuttal": "Dear Reviewer 5ayi,\n\nWe sincerely appreciate your thoughtful comments. We have carefully considered each of your questions and provide detailed responses below.\n\n---\n\n**[W1] Lack of novelty**\n\nWe clarify that our contribution is NOT in applying the RL training method, GRPO from DeepSeek-R1, to a new domain, but in proposing a novel framework specifically tailored to address the unique challenges for enhancing the embodied reasoning of LVLM. Specifically, our contributions are following three folds:\n\n1. Challenges in Applying RL Algorithms for Improving Embodied Reasoning of LVLM\n- Embodied QA lacks definitive answers, leading to suboptimal training targets.\n- Current AIs demonstrate poor embodied reasoning performance, so most annotations rely on human experts.\n\n2. Core Innovation: Waypoint-based Embodied Reasoning\n- Since the main goal of robot control is action generation, we enhance overall embodied reasoning by predicting waypoints that are fixed targets and easy to achieve.\n\n3. Methodological Advance: MCQA and Auxiliary Task Integration\n- Numeric states have limited representational power, making reasoning enhancement through state prediction alone challenging\n- We design diverse QA formats (MCQA, open-ended) and validate these approaches through experiments\n- To enhance performance, we further consider auxiliary training tasks such as state prediction to improve understanding of numeric states\n\nTo summarize, robot-r1 suggests a novel embodied reasoning training framework that enhances performance using numeric state information without any human annotation. Our framework utilizes GRPO, but overcomes many practical challenges stated above while applying it to a robot domain.\n\n---\n\n**[W2,W3] Terminology and format error**\n\nThank you for your suggestion. We will clarify the concepts by providing clear definitions of the terms consistently. In particular, we will define these terms: \n- “planning” refers to the entire sequence of all high-level actions required to complete a task.\n- “high-level action reasoning” refers to linguistic high-level action expression\n- “movement reasoning” refers to a more accurate expression that indicates a concrete change in direction within the 3d space, e.g., “move up”, “move down”.\n\nAnd we will fix the table format in the final draft.\n\n---\n\n**[Q1] Is Robot-R1 Bench MCQA-style? How does MCQA-trained Robot-R1 adapt to open-ended benchmarks?**\n\nAll evaluation environments did not use the MCQA format, and all models were evaluated through open-ended answer generation. This is because Robot-R1 enhances overall embodied reasoning performance by solving MCQA through the reasoning process during training, i.e., the knowledge is easily generalizable to various domains.\n\n---\n\n**[Q2] Unfair comparison between Robot-R1 and SFT**\n\nOur comparison is fair because we already did an open-ended generation evaluation for both Robot-R1 and SFT, as we also mentioned in our response to [Q1]. \n\n---\n\n**[Q3] GPT-4o performs better at EmbodiedBench Manipulation** \n\nWe first clarify that GPT-4o can’t be a fair baseline, as it has a significantly larger model size than Qwen-2.5 VL 7B, which we used to train Robot-R1. As our goal is to develop the training methodology to enhance the embodied reasoning in robotics, our baseline is an initial backbone LLM (no fine-tuning) or other training methods such as SFT. The reason why we present the performance of GPT-4o and other LLMs is to provide a useful reference for better performance judgment. \n\nAdditionally, EmbodiedBench is an agent benchmark, requiring not only embodied reasoning but also overall in-context learning capabilities. Meanwhile, Robot-r1 was trained to enhance embodied reasoning performance, not the performance of an LLM agent. Therefore, Robot-R1 is much unfavorable in this benchmark, compared to GPT-4o.\n\n---\n\n**[Q4] Computing cost** \n\nWe have disclosed all the prompt templates used in the experiments in B.2, allowing for easy token calculation. The average prompt length used in RL training is approximately 700 tokens, and that used in SFT is approximately 400 tokens.\n\nThe benchmark also includes all prompts publicly. Each problem for evaluation consists of approximately 200 tokens, and the model takes approximately 20 minutes for one A100 to generate all responses for a total of 215 problems. The API cost for LLM-as-judge is approximately $2 for scoring all responses in the benchmark. Each LLM-as-Judge prompt consumes approximately 800 prompt tokens and 90 response tokens. \n\n---\n\n**[Q5] Training Robot-R1 with open-ended QA format outperforms MCQA format at training accuracy**\n\nWe clarify that the results from the open-ended QA format and MCQA format are not comparable in Figure 11(a), because the applied pivot of the y-axis is different between the MCQA and open-ended formats. The open-end QA’s y-axis indicates (1 - L1 loss) while MCQA’s y-axis is QA accuracy.\n\n---\n\n**[Q6] How can the accuracy of scoring using large models in the Robot-R1 Bench be ensured?** \n\nWe first note that prior works in the literature of LLM-as-judge have observed that the evaluation with strong LLMs such as GPT-4 and human evaluation exhibit high correlation [1,2,3].\n\nNevertheless, to further investigate the validity of this approach for Robot-R1 Bench, we measured the correlation between real human scores and robot-r1 bench scores, by having human experts score responses in the same environment as the LLM-as-judge model scores responses in Robot-R1 Bench.\n\nSpecifically, we conducted the test using a subset of 40 problems from the Robot-R1 bench with 8 participants. The test was conducted as a blind evaluation of Robot-R1 and GPT-4o responses, and we defined the median score obtained from this evaluation as the human score.\n\n\\begin{array}{c|cccc}\n\\hline\n& \\textbf{Planning} & \\textbf{Subtask} & \\textbf{Movement} & \\textbf{Spatial} \\newline\n\\hline\n\\textbf{Pearson Correlation} & 0.3315 & 0.8974 & 0.8931 & 0.8961 \\newline\n\\hline\n\\end{array}\n\nThe following table shows the Pearson correlation between LLM and human scoring, which supports the reliability of the Robot-R1 bench. Meanwhile, planning shows a somewhat lower correlation coefficient. This is likely because planning tasks typically involve longer responses, which increases the possibility of multiple valid answers. This structural characteristic makes accurate evaluation inherently challenging for planning tasks.\n\nHowever, Robot-R1 particularly excelled in the other components, and we believe these results do not compromise the reliability of our experimental findings.\nWe will further scale up the reliability assessment during the remaining period and include the results in the final draft.\n\n---\n\n**[Q7] Why is RL better than SFT?** \n\nWhile the datasets used in RL training are constructed using specific rules, the model actually learns the embodied reasoning process that emerges throughout the training process, which is automatically generated and optimized.\n\nWe also evaluated across diverse environments, including Robot-R1 bench, EmbodiedBench, and SpatialRGPT, which all involve different tasks from our training objective of waypoint prediction. We observed overall performance improvements compared to pre-trained models across these varied tasks. These results experimentally demonstrate the generalization capability of our RL approach.\n\nMeanwhile, research demonstrating that SFT lacks generalization and causes forgetting compared to reassigning incentivized RL methods is a widely known fact in LLM research and has been experimentally proven across a wide range of fields, math, coding, vision etc [4,5]. For example, RL is known to achieve strong generalization performance with minimal data compared with SFT, even with a single sample [6].\n\n---\n[1] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 \\\n[2] Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, EMNLP 2023 \\\n[3] Dubois et al., Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM 2024 \\\n[4] Chu et al., SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, ICML 2025 \\\n[5] Guo et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, Arxiv 2025 \\\n[6] Wang et al., Reinforcement Learning for Reasoning in Large Language Models with One Training Example, Arxiv 2025\n\n---\n\nIf you have any further questions/concerns, please do not hesitate to let us know.\n\nThank you very much,  \\\nAuthors"}]

Author rebuttal:

Meta-review: [Summary] 

The paper proposes ROBOT-R1, a reinforcement learning (RL) framework to enhance embodied reasoning in Large Vision-Language Models (LVLMs) for robot control. The key idea is to reformulate continuous control into multiple-choice QA tasks (e.g., waypoint prediction, state prediction, primitive movement), enabling tractable RL optimization with Group Relative Policy Optimization (GRPO). The authors further introduce ROBOT-R1 Bench, a benchmark for embodied reasoning evaluation. Empirical results show consistent gains over SFT baselines and competitive performance with much larger closed-source models across several settings (ROBOT-R1 Bench, EmbodiedBench Manipulation, SpatialRGPT). Additional rebuttal experiments address reproducibility, RL algorithm comparisons, and transfer to real-robot datasets (BridgeV2).

[Strengths]

(1) Novel training reformulation: Casting robot control as multiple-choice QA is an elegant way to apply RL effectively in high-dimensional action spaces.
(2) Empirical effectiveness: ROBOT-R1 demonstrates clear improvements over SFT baselines, particularly in embodied reasoning tasks where SFT often struggles with catastrophic forgetting and poor generalization.
(3) Breadth of evaluation: Results span multiple benchmarks (including new and existing ones), with further validation on real robot data (BridgeV2). This strengthens claims of transferability beyond RLBench simulation.
(4) Thorough rebuttal engagement: Authors provided convincing additional experiments: multi-seed results, RL algorithm ablations (GRPO vs RLOO vs Reinforce++), scaling-law analysis, human cross-validation of LLM-as-judge, and preliminary VLA integration (via GR00t/LIBERO). These responses addressed the reviewers’ major methodological concerns.
(5) Clarity of contribution: Despite some terminology issues, the overall pipeline and design are relatively easy to understand, making this work accessible to a broad robotics + multimodal AI audience.

[Weaknesses]

(1) Limited scope of evaluation: Most core experiments remain restricted to tabletop manipulation in simulation. Although real-robot validation was added, low-level control transfer is still weak. Stronger evidence on diverse robotics domains (e.g., navigation, multi-step real-world manipulation) would further substantiate the claims.
(2) Benchmark credibility: Heavy reliance on an author-proposed benchmark (ROBOT-R1 Bench) and LLM-as-judge evaluation may raise concerns of bias. Human validation was conducted, but scope remains limited.
(3) Terminology and consistency issues: Multiple reviewers noted confusing or inconsistent definitions of “planning,” “subtask,” “primitive movement,” etc., which hinder readability.
(4) Comparisons: While the method improves upon SFT, direct comparisons against stronger RLHF/VLA baselines (e.g., Pi0, UniVLA, GR00t) are missing or remain preliminary. This makes it harder to position ROBOT-R1 against recent state-of-the-art.
(5) Limited gain of applying it for real robot manipulation tasks. There is clearly a gap of improving the performance on the proposed embodied reasoning benchmark and the final performance on the end-to-end manipulation tasks as indicated by the experiment results shared by the authors.

[Reviews]

All four reviewers gave borderline accept (score 4) after rebuttal. Their initial concerns like variance across seeds, lack of real-robot validation, weak comparison to other RL algorithms, and limited analysis, were largely addressed through additional experiments and clarifications. Remaining criticisms mainly concern clarity, limited scope, and over-claiming relative to SOTA. Importantly, no reviewer advocated rejection after rebuttal; instead, they acknowledged the paper’s technical soundness and the significance of its contribution to embodied reasoning in LVLMs.

[Conclusion]

While the ACs still hold some concerns about this work, this work presents an interesting idea of integrating RL into embodied multimodal reasoning. The ACs thus recommend an acceptance, but strongly recommend the authors to incorporate the discussions, related works and newly added experiments into the final version. Moreover, the authors should clearly show the transferability of performance gain on the proposed benchmark to the real robot manipulation tasks in their final version.
Decision: Accept (poster)