Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Authors: brian ichter
Anthony Brohan
Yevgen Chebotar
Chelsea Finn
Karol Hausman
Alexander Herzog
Daniel Ho
Julian Ibarz
Alex Irpan
Eric Jang
Ryan Julian
Dmitry Kalashnikov
Sergey Levine
Yao Lu
Carolina Parada
Kanishka Rao
Pierre Sermanet
Alexander T Toshev
Vincent Vanhoucke
Fei Xia
Ted Xiao
Peng Xu
Mengyuan Yan
Noah Brown
Michael Ahn
Omar Cortes
Nicolas Sievers
Clayton Tan
Sichun Xu
Diego Reyes
Jarek Rettinghouse
Jornell Quiambao
Peter Pastor
Linda Luu
Kuang-Huei Lee
Yuheng Kuang
Sally Jesmonth
Kyle Jeffrey
Rosario Jauregui Ruano
Jasmine Hsu
Keerthana Gopalakrishnan
Byron David
Andy Zeng
Chuyuan Kelly Fu
Venue: CoRL 2022 Oral (2022)
Keywords: Large Language Models
Task and Motion Planning
Grounding Models
Mobile Manipulation
Reinforcement Learning
Imitation learning

Abstract: Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so  that  the  language model  provides  high-level  knowledge about the procedures for performing complex and temporally extended instructions,  while  value  functions  associated  with  these  skills  provide  the  grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project’s website, video, and open source can be found at say-can.github.io.


Reviews:
[{"review_id": "zguhFmlRRhB", "reviewer": "Reviewer_h3Ar", "summary": "By assuming a set of primitives and their learned affordances, SayCan reweighs an LLM's generated steps in a task plan by their affordances. This enables a specific robot hardware to influence the LLM planner by only appended chosen steps after weighing by affordances. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " - Excellent figures, real world experiment results, and video.\n - Simple and effective key idea.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper demonstrates an impressive zero-shot, language-conditioned mobile manipulation system, capable of performing long horizon tasks in a robot hardware aware manner. The discussion of failure modes and challenges in building the system in the main paper and appendix were useful. The quantitative and qualitative evaluations were comprehensive. The key idea is simple and effective, and leads to a useful and plan-interpretable system. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- This work assumes the robot is given a set of skills. Appendix C.2 that choosing this set of primitive requires 1) a set of candidate tasks to see which primitives would be useful for the LLM, and 2) an attempt in training their affordance-based policy to gauge their real world consistency and success. I think this is a limitation which could be discussed in more detail in the main paper."}}, {"review_id": "caZAdy5kKs5", "reviewer": "Reviewer_Hat5", "summary": "This paper introduces a method to map high-level natural-language instructions to skills that a mobile manipulator can execute in its environment. Given an instruction, an LLM is used to predict a sequence of skills and the probability of each skill helping to complete the instruction. An affordance function is used to predict whether a given skill can be executed at the current environment state.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Similar to the concurrent work, this paper introduces a general framework that uses LLMs to map high-level natural language instruction to low-level actions. Since the LLMs are trained on a large amount of text data, they contain enough task and object knowledge to predict how to perform different high-level tasks.\n2. In addition to using LLMs to predict low-level actions, the actions are selected based on whether they are feasible given the capabilities of the robot and the environment states. This formulation provides an easy and intuitive way to ground abstract plans generated by the LLMs.\n3. The feedback to the LLM is provided by appending actions that have been successfully executed to the prompt.\n4. The framework is evaluated in both simulations and in a real-world kitchen with a mobile manipulator. Human raters are used to assess whether the tasks are planned successfully and executed correctly by the robots. Using human raters can account for the fact that many tasks have different ways to be completed. \n\nWeakness:\nI appreciate the fact that this paper introduces a very general framework that can potentially leverage different LLMs and primitive skills. However, I am worried that the main text lacks enough clarity in stating the assumptions and limitations of the current implementation.\n\n1. What are the 551 skills? It would be great if the authors can explain the definition of skills in the context of this paper. For example, is grasping an apple a different skill than grasping an orange?\n2. The main text mentions that the value function of a skill is trained via TD backups regardless of how a skill is implemented. However, this description contradicts the appendix stating that some of the value functions are defined based on heuristics. Another important detail missing from the main text is that the value functions are calibrated empirically. I imagine that this calibration is very important as the probabilities/likelihoods from the LLMs and affordance functions are not normalized. The authors should provide more details on calibrating the likelihoods and balancing the likelihoods from the LLMs and affordance functions in the main text. I also see examples (e.g., third step in Figure 12 (a) in the appendix) where the LLM and affordance function predict the highest probabilities for different actions, resulting in a very small margin between the combined probability of the correct action and of the incorrect one. How likely can this happen and cause failures? \n3. Although the goal of this paper is to leverage the generalization capability of the LLMs, the tasks tested seem to be very similar. Most tasks can be completed with the sequence of <find obj, pick up obj, go to loc, and put down obj>. Therefore, it is hard to see if the LLMs are actually required to generalize. Why is simply predicting the target object not enough? (since the location of each object is also predefined as stated in the appendix). Another question is why does the location of each object need to be pre-specified instead of being predicted by the LLMs?\n4. The only connection between the environment and the LLM is the action feedback. Therefore the LLM does not have direct access to the environment state. It would be interesting to see a discussion on this limitation. \n5. Related to the comment above, I also hope to see a comparison (even just in simulation) with existing vision-and-language navigation/object interaction methods that finetune LLMs on specific tasks. This comparison would provide insight into whether prompting or finetuning is a more effective way to balance generalization and task success. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a novel idea of combining LLMs and value functions for planning. The idea is tested with different LLMs and different implementations of primitive skills in relatively diverse settings and on real robots. This paper can be improved more by discussing the assumptions that the current implementation holds.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Besides the comments above, I have a few minor suggestions. \n1. I think the authors could provide examples of the prompts in the main text since these prompts have very specific structures that I imagine are very important.  \n2. Why is the execution success rate for language ablations and planning success rate for value function ablations not included? I am very curious to see the execution success rate for the “Generative” baseline. "}}, {"review_id": "VE1nCPTkeNN", "reviewer": "Reviewer_W6E5", "summary": "This paper proposes a method for robots to understand the instructions expressed in natural language and execute them in the real world. The sequences of low-level actions to perform instructions are estimated by using a large language model (LLM), and an appropriate prompt that extracts low-level action representations of instructions from the LLM is proposed. Furthermore, a learned value function is used to estimate how likely each action is to succeed in the current state in the real world. Combining probability provided by LLM and value functions, appropriate actions in the current state can be selected in actions that are predicted as next step actions by LLM. By appending the action representation to the prompt and querying the LLM, further next action can be predicted. Experimental results show that the proposed method can work well in the real environment. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- A novel method for robots to understand the instructions using LLM is proposed. \n- Experiment was conducted in the real world and the results show that the proposed method can work well in the real environment. \n\n# Weaknesses and suggestions\n- A lot of important information is written in the appendix. I understand due to space limitations but the contents should be selected and the important results should be written in body text. \n- Line 179: What is the definition of `skills` and `skill families` \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think it is worth accepting this paper because it presents a new potential of the use of the LLMs in robotics.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Described in the strengths and weaknesses section.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "BMEkcNat9II", "reviewer": "Reviewer_JdHL", "summary": "The paper proposes a pipeline for injecting knowledge from large language models into a robot instruction-following framework. Specifically, LLMs are used to select and sequence specific skills (i.e., actions) as to achieve the goal inferred from the language instruction.  The proposed architecture is evaluated in a real robotic system showing good performance at a variety of tasks, e.g., picking and place, object fetching, etc. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nstrengths:\n\nThe software integration effort is impressive. \n\nThe pipeline is creative and shows good performance on a variety of tasks \n\nThe evaluation on a real robot is a big plus. \n\n\nweakness:\n\nThere is a lot of \"devil is in the details\" with the details either left out or in the appendix. This paper feel like it should be expanded to explain the details and go to a journal instead. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nOverall, the paper is well written, the ideas are interesting and novel, and the work is demonstrated on a real robot. The only small drawback is that many details are missing and some comparisons with alternative approaches could be discuses better. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "\nIn the first paragraph of Sec. 2, can you describe in detail what you mean by state, as in robot's observation or something more comprehensive that takes history into account or object relations, etc.?\n\nDid you manually come up with each of the 551 skills or was there a procedure that does that?  \n\nCan you comment on to what extent these skills generalize to new objects and new configurations of objects not present in training set? It looks like the overall performance of the system is largely dependent on how well these skills work. Can you provide a subset of these in the paper? Also, are the 17 objects, 17 individual objects or are they object classes? Does that mean the robot would not do well if the objects were different from training time?\n\nThe related work is missing some works on grounding by Thomason and collaborators which is very closely related here:\n\nThomason, Jesse, et al. \"Jointly improving parsing and perception for natural language commands through human-robot dialog.\" Journal of Artificial Intelligence Research 67 (2020): 327-374.\nThomason, Jesse, et al. \"Language grounding with 3D objects.\" Conference on Robot Learning. PMLR, 2022.\nThomason, Jesse, et al. \"Guiding exploratory behaviors for multi-modal grounding of linguistic descriptions.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\nIn particular, it would be interesting to comment in the paper on how the robot would be able to interpret words like \"soft\", or \"heavy\" that require non-visual perception. \n\nIt would also be good to discuss the pros and cons of the system in relation to more traditional approaches which parse language into logical form and use symbolic knowledge bases to infer what to do. \n\nminor issues and comments. \n\nWith so many of the relevant details in appendices, this paper probably belongs in a journal. \n\nIn the last paragraph of the introduction, say exactly how many tasks, instead of \"almost 100\"\n\nFor the last sentence of paragraph 2 in Sec.2 to be true, I believe you would also need the discount factor to be 1.0 to make that equivalency. "}}]

Author rebuttal:

Decision: Accept (oral)