Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose using a two-step Code-As-Policy approach using LLMs. This two step approach first translates a natural language prompt into a set of rewards and then translates those rewards into a set of function calls to be executed. Under the hood, mujoco MPC is used to turn the rewards into a plan which is executed on the robot.
review: The authors hypothesize that producing rewards, instead of directly producing robot commands, is more natural for LLMs and is likely to make better use of their inductive bias. Their experiments on simulated quadrupeds, an arm with dexterous hand, and a real-world robot equipped with parallels jaw gripper seem to back this up. Results seem promising compared to a code-as-policy baseline and their implementation detail of making reward prediction a two step process also appears affective when compared to the baseline of Reward Coder only.

That being said, I have some questions, likely beyond the scope of this paper, about this approach scaling. The tasks used here were all fairly simple, which means they were amenable to solving via MPC. That's not to say that LLM reward generation wasn't useful, it certainly eased the design work of humans requesting tasks, but for more complex tasks finding good rewards that both sufficiently constrain behavior and allow efficient solutions is known to be difficult and this work does not address that difficulty.

The work is also lacking on details about the LLM used. It seems it was an off-the-shelf LLM with only in-context learning used.

As an aside, this seems highly related to the concurrent work VoxPoser (https://voxposer.github.io/). That's not itself a problem, but the authors should consider citing it as concurrent work.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While somewhat limited, the work illustrates an interesting and potentially useful factorized prompting strategy for generating rewards from an LLM that are of sufficient quality to synthesize (simple) behaviors. That's a useful, if slightly incremental, finding.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces a pipeline for using LLM to generate reward specification for a variety of RL tasks for a quadruped and serial robot manipulator. The authors design a two-part pipeline to prompt GPT-4 to create reliable and parameterized reward code used to train the motion controller using a MJPC planner. The work conducts an ablation study using different prompting contexts to demonstrate differences in the RL model’s success rate on each task.
review: **Significance:**
The authors propose a two-part prompt for GPT-4 to produce parameterized reward code. While this approach is interesting, I believe it lacks novelty. Much of the contribution seems to center around the prompt engineering to produce a reward parameterization. Afterwards, the paper uses a standard motion predictive controller. However, I believe that the paper does explore an important avenue within LLM + Robotic Learning / AutoRL research area.

I believe the paper would significantly benefit from evaluating the method’s performance on a vaguer “motion descriptor prompt” and a “reward coder prompt” standardized across robot types. The current method, as-is, still requires extensive domain expertise and human intervention to generate reward code.

**Quality:**
The experiments conducted by the authors were thorough and well-designed within the paper’s objective. The authors clearly explored two RL domains (quadruped robot, serial manipulator) through 17 total tasks. As an additional ablation study, I would be curious to see the RL performance when the LLM is given varying levels of context (ex: fewer variable definitions, fewer reminders, etc).

I would also be interested in seeing results from the real-world experiment. Were the success rates for each task similar to the simulated rates? 

**Clarity:**
The paper was very well written and clear. The data was presented and explained in a direct and concise manner. The sample prompts and videos in the supplemental materials were well received. There were a few spelling and grammar errors that do not impact the overall message of the paper.
- Figure 1 caption requires space between "Translator" and "that".


As summarized from the above discussion on significance, quality, and clarity...

**Strengths:**
- The paper was clear and concise. The supporting materials were helpful.
- The experimental design allowed for a thoroughly exploration of the scope of the paper.
- The exploration of interactive systems for reward tuning was a cool idea.

**Weaknesses:**
- No sensitivity study of the effects of varying the amount of context given to the LLM.
- Although interesting, the methods in the paper only produce a incremental contribution.
- Experiments largely focus on simulated results. Quantitative results on real world experiments should be included.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While the results are interesting and the experiment selection was well designed, I believe the paper still lacks in several key areas. I do not feel that the paper is significantly novel, and there is a lack of real world quantitative results. As a result, I would recommend a weak rejection for this paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: In the present work, the authors explore a task to employ LLM in robotics. In their approach, LLM defines a task by generating a reward function which is afterward optimized by model predictive control methods. The reward function is generated in two steps. In the first step, LLM receives a description of the desired state and outputs the target position of the robot. In the second step, LLM receives a description of the target position and outputs the reward function. The authors demonstrate the capabilities of the proposed method in a simulated quadruped robot and a dexterous manipulator robot and on a real robotic arm. 
review: Large language models have demonstrated remarkable capabilities in natural language understanding. Nevertheless the direct application of LLMs to control robotic devices is not possible. The authors investigate one of the possible approaches that generates reward functions from LLM given a textual description of a task. This approach seems to be quite generic since in a wide range of applications reward can be defined by simple equation. The paper clearly presents the developed approach and the evaluation results. The authors show that the proposed method performs better then code-as-policies baseline and baseline with only one LLM with returns reward directly from natural language task description. 

One possible limitation of the presented approach is simplicity of generated reward function. For example to train a robust reinforcement learning policy to control quadruped robot on needs to design a reward function with multiple terms punishing big torques, velocities, non-smooth movements. However the punishment terms may be similar for different tasks and may be handcrafted and shared between them. 

Also there are some sentences with missing spaces like "... the Reward Translatorthat interprets the user ..."
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend accepting the paper. It presents an interesting method to ground LLM for robotics tasks.
review_title: null

Meta-review: This paper addresses the problem of skill generation where an input language instruction must be translated to a reward model. The core contribution lies in the design of a reward interface that associates a high level instruction to parameters of a reward model. Once the reward model is known, a MPC is used to generate control. From the LLM perspective, the approach lowers the symbolic to real boundary by reasoning over reward parameters instead of pure symbols as used in task planning. 

During the rebuttal phase the authors provided clarifications on the experiments, engaged with closely related works and uploaded data on real world experiments. The authors are requested to update their manuscript based on the reviewer feedback.
Decision: Accept (Oral)