Language to Rewards for Robotic Skill Synthesis
Authors: name: Wenhao Yu, name: Nimrod Gileadi, name: Chuyuan Fu, name: Sean Kirmani, name: Kuang-Huei Lee, name: Montse Gonzalez Arenas, name: Hao-Tien Lewis Chiang, name: Tom Erez, name: Leonard Hasenclever, name: Jan Humplik, name: Brian Ichter, name: Ted Xiao, name: Peng Xu, name: Andy Zeng, name: Tingnan Zhang, name: Nicolas Heess, name: Dorsa Sadigh, name: Jie Tan, name: Yuval Tassa, name: Fei Xia

Large language models (LLMs) have demonstrated exciting progress in acquiring
diverse new capabilities through in-context learning, ranging from logical
reasoning to code-writing. Robotics researchers have also explored using LLMs
to advance the capabilities of robotic control. However, since low-level robot
actions are hardware-dependent and underrepresented in LLM training corpora,
existing efforts in applying LLMs to robotics have largely treated LLMs as
semantic planners or relied on human-engineered control primitives to interface
with the robot. On the other hand, reward functions are shown to be flexible
representations that can be optimized for control policies to achieve diverse
tasks, while their semantic richness makes them suitable to be specified by
LLMs. In this work, we introduce a new paradigm that harnesses this realization
by utilizing LLMs to define reward parameters that can be optimized and
accomplish variety of robotic tasks. Using reward as the intermediate interface
generated by LLMs, we can effectively bridge the gap between high-level
language instructions or corrections to low-level robot actions. Meanwhile,
combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive
behavior creation experience where users can immediately observe the results
and provide feedback to the system. To systematically evaluate the performance
of our proposed method, we designed a total of 17 tasks for a simulated
quadruped robot and a dexterous manipulator robot. We demonstrate that our
proposed method reliably tackles 90% of the designed tasks, while a baseline
using primitive skills as the interface with Code-as-policies achieves 50% of
the tasks. We further validated our method on a real robot arm where complex
manipulation skills such as non-prehensile pushing emerge through our
interactive system.