SayTap: Language to Quadrupedal Locomotion
Authors: Yujin Tang
Wenhao Yu
Jie Tan
Heiga Zen
Aleksandra Faust
Tatsuya Harada
Venue: CoRL 2023 Poster (2023)
Keywords: Large language model (LLM)
Quadrupedal robots
Locomotion

Abstract: Large language models (LLMs) have demonstrated the potential to perform high-level planning. Yet, it remains a challenge for LLMs to comprehend low-level commands, such as joint angle targets or motor torques. This paper proposes an approach to use foot contact patterns as an interface that bridges human commands in natural language and a locomotion controller that outputs these low-level commands. This results in an interactive system for quadrupedal robots that allows the users to craft diverse locomotion behaviors flexibly. We contribute an LLM prompt design, a reward function, and a method to expose the controller to the feasible distribution of contact patterns. The results are a controller capable of achieving diverse locomotion patterns that can be transferred to real robot hardware. Compared with other design choices, the proposed approach enjoys more than 50% success rate in predicting the correct contact patterns and can solve 10 more tasks out of a total of 30 tasks. (\url{https://saytap.github.io})


Reviews:
[{"review_id": "osMma7UILcb", "reviewer": "Reviewer_8oYu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method to control quadruped gaits using natural language.\nIt uses foot contact patterns as an interface and trains a RL controller in simulation to realize the gaits.\nThe resulting approach is evaluated on the real robot by comparing the generated gaits with the expected output.", "summary_of_recommendation": "The idea is interesting, but I would currently recommend rejecting the paper, several important clarifications are needed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths And Weaknesses\n\nThe chosen interface is interesting as it seems to be quite versatile as it should cover a large number of gaits.\nThe pipeline with the RL controller also looks like an interesting building block towards useful applications.\n\nThe idea is interesting and results on real robots are very welcome, but some points need to be clarified/improved.\n\nThe evaluation is not clear.\nIt says \"We observed the foot contact patterns generated by the LLM after accepting the commands and compared them against our expectations\".\nIs it a subjective evaluation? is it a visual evaluation or do you have some kind of edit distance between the generated and expected foot patterns?\nIt would be interesting to show a failure case.\nThe evaluation also questions the need for an LLM.\nIf a human can easily generate and compare a foot contact pattern, why would an LLM be needed? (since its output might also be unreliable).\nI agree that the RL controller is still useful in this case.\n\nIt seems that the cycle duration is almost constant (T in [24, 28]), could the authors comment on this?\nHow was the cycle duration range chosen?\nSame question for the foot contact length ratio, it seems quite important to have gaits that are feasible.\nWhat happens if a gait is not feasible? (e.g. all zeros)\n\nThe \"Random Pattern Generator\" is crucial for training, but the clarity of this section (currently in the appendix) should be improved.\n\nThe proposed interface is versatile, but also limited. For example, there is no control over the foot clearance (or ground penetration), CPGs could be an alternative in this case ([3] and [4], CPGs seem to be missing from the related work as well).\n\nA final point is the LLMs used.\nAs the output of closed LLMs can vary over time [1], it would be good for reproducibility purposes to try with an open source LLM (e.g. [2]) and at least mention the exact version used.\n\n\n[1] https://arxiv.org/abs/2307.09009\n[2] https://chat.lmsys.org/\n[3] https://arxiv.org/abs/2209.07171\n[4] https://arxiv.org/abs/2211.00458\n\n\n### Questions/Remarks\n- Fig1: what is the intermediate pattern? (between trotting and bounding, how was it generated?)\n- \"Improves the naturalness of the robot’s movement and helped shrink the sim-to-real gap\" -> please show failure\n- Only the desired foot pattern is given as input? no information about the current position in the cycle? in that case this would probably break the Markov assumption for the contact reward and make it harder to realize the desired gait.\n- The paper claims to allow low-level commands, but in practice it is not that low-level, the LLM generates the foot contact pattern, the rest is done by RL + PD for the low-level part.\n- Are unstructured/vague instructions really wanted? Especially since the evaluation is subjective, it seems that this part could be left in the appendix.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "aj_U0YGnyx", "reviewer": "Reviewer_erUF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper considers the task of controlling a quadrupedal robot with natural language commands. The authors propose to use foot contact patterns as an interface between a large language model and a locomotion controller. They contribute prompts for LLM, a reward function to train a locomotion controller, and an algorithm to generate feasible contact patterns. The authors benchmark the proposed interface against two baselines, in particular discrete gaits and sinusoidal functions. The proposed approach achieves more than a 50% success rate and can solve more tasks then the baselines. ", "summary_of_recommendation": "I recommend accepting the paper. It considers an important task of grounding LLM to control a physical robot. The proposed method is novel and achieves good results. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In the paper, LLM  was given only five gate patterns to choose from (BOUND, TROT, PACE, STAND_STILL, STAND_3LEGS). Would it be possible for LLM to generate a new meaningful gate pattern with a sufficient explanation in the prompt? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and interesting to read. The topic of grounding large language models to control robotic devices is very important for the future of robotics. The paper clearly states the task and describes the developed method. The proposed interface for interaction between LLM and locomotion controller is original and works better than the baselines. \n\nStrengths: Authors proposed a simple and quite effective interface that bridges the gap between language models and locomotion controllers. The developed method was evaluated on a real quadruped robot. \n\nWeaknesses: The expressive power of foot contact patterns may be limited for complex motions like jumping or climbing stairs. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ZnszwAzVYE", "reviewer": "Reviewer_NnfC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method to use Large Language Models (LLM) as an intermediary layer between human commands in natural language and the generation of walking patterns for a quadrupedal robot. Through the design of a prompting template, the LLM can convert a language command (e.g. Act as if you have a limping rear left leg) into a sequence of contact instructions and a desired linear velocity  for a low level controller to execute. Using a LLM in this way enables the input instruction to be free form and include aspects such as mood or appearance of the gait. \n\nI find interesting the idea of using a LLM to output this contact sequence representation, which in a way is simple for LLM generation, but that can generate behaviors by using the sequence as a walking pattern. \n", "summary_of_recommendation": "I think the paper has an interesting idea from the conceptual point of view. i.e. using LLM to generate contact sequences for gait generation. The idea can be reused for other problems as an intermediate layer between human user instructions in natural language and behavior generation. Simultaneously, the paper is relatively simple. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please clarify the naming of the random generator, perhaps there is a detail I’m not seeing. \n\nHow many example blocks (Fig. 3 right) are there in your prompt? \n\nThe low level controller for walking was trained using DRL. I was curious if it was necessary to add any specific adaptation to train the policy in the specific set up (i.e. contact sequence pattern and linear velocity inputs) versus the standard training of DRL locomotion policies? I just mean if there was any particular difficulty to make RL converge in your specific set up vs others commonly used ones. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well organized and explained, relatively straightforward to follow. \n\nI’m confused about the “random pattern generator”. I’m understanding it can output a desired contact pattern (sequence of contacts for all four legs) from a set of predefined possible templates ({BOUND, TROT, PACE, STAND_STILL, STAND_3LEGS}). Is this correct? I’m not sure why it is called “random”? \n\nPlease clarify the naming of the random generator, perhaps there is a detail I’m not seeing. \n\nHow many example blocks (Fig. 3 right) are there in your prompt? \n\nThe low level controller for walking was trained using DRL. I was curious if it was necessary to add any specific adaptation to train the policy in the specific set up (i.e. contact sequence pattern and linear velocity inputs) versus the standard training of DRL locomotion policies? I just mean if there was any particular difficulty to make RL converge in your specific set up vs others commonly used ones. ", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0fXJUIPTQg", "reviewer": "Reviewer_Ztd6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Large language models are increasingly powerful for tasks expressed in natural language. However, they are not helpful for directly generating robot motor commands, which are thought to have a substantially different structure than natural language. This paper proposes an intermediate grammar for specifying quadruped robot motions. This grammar is the robot's foot contact pattern and body velocity. Through reinforcement learning in simulation, a mapping is established between this grammar and a motor-level control policy. Then, a large language model is prompted to translate natural language into this grammar. The combined pipeline can adjust the robot control policy in response to natural language. For unstructured or vague instructions such as \"Good news, we are going to a picnic!\" or \"Go catch that squirrel on the tree\" the robot's behavior is subjectively reasonable.", "summary_of_recommendation": "This paper will be a valuable landmark in the important discussion surrounding grounding natural language in motor skills. I encourage expanding the evaluation to include failure cases which will make the limitations concrete and guide follow-up works.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Add experiments showing failure cases to mark the system limitations and facilitate follow-up work. The prompts in Table 2 are chosen intentionally to work well using the proposed method. However, the space of interesting natural language commands for quadrupeds is much larger. It would be nice to explicitly include some failure cases and describe what the robot does. I can think of lots of totally reasonable \"unstructured/vague instructions\" for a quadruped that would fail and explicitly illustrate limitations:\n\n- \"Roll over\" -> outside of grammar\n- \"Climb on top of a box directly in front of you\" -> outside of grammar\n- \"Balance on one foot\" -> dynamically infeasible\n- \"Kick the ball\" -> not enough sensory information\n- \"Bring me a Coke\" -> not enough sensory information\n\nIt's possible to reason from first principles that these prompts won't succeed due to the above. But for a casual reader, including them would help make the limitations explicit. Also, reporting those failure cases will help justify future work that accomplishes them! And, I'm curious to know what the system will actually do in these cases (maybe it'll be different than what I expect?)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths --\n- High relevance: The problem of constructing an interface from natural language to motor skills is very relevant and timely. This paper's formulation is thought-provoking and will serve as a concrete artifact to advance the discussion.\n- High originality: This is the first paper I've seen that meaningfully connects LLMs to the motor skills of a quadruped. As a researcher working with similar tools, I did not think of constructing the prompt/interface this way myself. After reading this paper, my understanding of the possibilities and limitations of motor skill-related prompting has expanded.\n- Clarity: The paper is clearly written.\n\nWeaknesses --\n- Low generality: It is essential that the (contact schedule + velocity) grammar is both (a) very low-dimensional compared to the space of motor policies and (b) spans the space of natural language descriptions well. These criteria are met through careful engineering incorporating domain knowledge specific to quadruped locomotion on flat ground. I expect these criteria could be difficult to satisfy for other domains and tasks.\n- Low direct utility: The system is mostly useful for walking in expressive locomotion styles. This is useful for entertainment but does not improve the ability of the robot to (a) move from place to place or (b) interact with its environment. If the primary downstream use case is entertainment, it would strengthen the paper to evaluate this more thoroughly. Currently, the only supporting evidence is the qualitative self-evaluation in Table 2.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary of Paper
The paper proposes a new method to connect language commands via LLMs to quadruped locomotion by utilizing foot placement patterns.

## Summary of Reviews
The paper is well written and has a neat idea. There were a few unclear points and the reviewers had some concerns about the generality of the method (both for other quadruped tasks and more general tasks).

## Influence of Rebuttal
The reviewers appreciated the modifications and clarifications as well as the added experiments on the failure cases. Reviewer 8oYu also reacted during the AC&reviewer discussion phase (not visible to authors, hence copied below). The major concerns of the reviewers have all been addressed.

## Suggestions for Improvement
Add the clarifications to the final version of the paper (largely done) and please also include the new failure cases in the paper. Making the results (more) reproducible is highly encouraged.

## Reviewer 8oYu's reply

### Thanks for the replies

Hello,

thank you for the replies and the edits.

> We submitted our prompt and command to llama-2-13b-chat and alpaca-13b. However, they both failed to give answers in the correct format. In terms of reproducibility, although, unfortunately, we didn’t use a fixed version of GPT-4 for our experiments, the current version still reproduces the results in the paper.

I raised my score to "weak accept" but I will let the AC decide, as a critical aspect for me is reproducibility, and the current reproducibility of the paper is borderline: open sourcing the code for training/evaluation (adding it to the supplementary material) and managing to reproduce the results with an open model would be a great addition.
Decision: Accept (Poster)