Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in
  Robotics
Authors: name: Dongyoung Kim, name: Sumin Park, name: Huiwon Jang, name: Jinwoo Shin, name: Jaehyung Kim, name: Younggyo Seo

Large Vision-Language Models (LVLMs) have recently shown great promise in
advancing robotics by combining embodied reasoning with robot control. A common
approach involves training on embodied reasoning tasks related to robot control
using Supervised Fine-Tuning (SFT). However, SFT datasets are often
heuristically constructed and not explicitly optimized for improving robot
control. Furthermore, SFT often leads to issues such as catastrophic forgetting
and reduced generalization performance. To address these limitations, we
introduce Robot-R1, a novel framework that leverages reinforcement learning to
enhance embodied reasoning specifically for robot control. Robot-R1 learns to
predict the next keypoint state required for task completion, conditioned on
the current scene image and environment metadata derived from expert
demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples
reasoning-based responses and reinforces those that lead to more accurate
predictions. Our experiments show that models trained with Robot-R1 outperform
SFT methods on embodied reasoning tasks. Despite having only 7B parameters,
Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action
control, such as spatial and primitive movement reasoning.