Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a pipeline for injecting knowledge from large language models into a robot instruction-following framework. Specifically, LLMs are used to select and sequence specific skills (i.e., actions) as to achieve the goal inferred from the language instruction.  The proposed architecture is evaluated in a real robotic system showing good performance at a variety of tasks, e.g., picking and place, object fetching, etc. 
strengths_and_weaknesses: 
strengths:

The software integration effort is impressive. 

The pipeline is creative and shows good performance on a variety of tasks 

The evaluation on a real robot is a big plus. 


weakness:

There is a lot of "devil is in the details" with the details either left out or in the appendix. This paper feel like it should be expanded to explain the details and go to a journal instead. 



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: 
Overall, the paper is well written, the ideas are interesting and novel, and the work is demonstrated on a real robot. The only small drawback is that many details are missing and some comparisons with alternative approaches could be discuses better. 

issues: 
In the first paragraph of Sec. 2, can you describe in detail what you mean by state, as in robot's observation or something more comprehensive that takes history into account or object relations, etc.?

Did you manually come up with each of the 551 skills or was there a procedure that does that?  

Can you comment on to what extent these skills generalize to new objects and new configurations of objects not present in training set? It looks like the overall performance of the system is largely dependent on how well these skills work. Can you provide a subset of these in the paper? Also, are the 17 objects, 17 individual objects or are they object classes? Does that mean the robot would not do well if the objects were different from training time?

The related work is missing some works on grounding by Thomason and collaborators which is very closely related here:

Thomason, Jesse, et al. "Jointly improving parsing and perception for natural language commands through human-robot dialog." Journal of Artificial Intelligence Research 67 (2020): 327-374.
Thomason, Jesse, et al. "Language grounding with 3D objects." Conference on Robot Learning. PMLR, 2022.
Thomason, Jesse, et al. "Guiding exploratory behaviors for multi-modal grounding of linguistic descriptions." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

In particular, it would be interesting to comment in the paper on how the robot would be able to interpret words like "soft", or "heavy" that require non-visual perception. 

It would also be good to discuss the pros and cons of the system in relation to more traditional approaches which parse language into logical form and use symbolic knowledge bases to infer what to do. 

minor issues and comments. 

With so many of the relevant details in appendices, this paper probably belongs in a journal. 

In the last paragraph of the introduction, say exactly how many tasks, instead of "almost 100"

For the last sentence of paragraph 2 in Sec.2 to be true, I believe you would also need the discount factor to be 1.0 to make that equivalency.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces a method to map high-level natural-language instructions to skills that a mobile manipulator can execute in its environment. Given an instruction, an LLM is used to predict a sequence of skills and the probability of each skill helping to complete the instruction. An affordance function is used to predict whether a given skill can be executed at the current environment state.
strengths_and_weaknesses: Strengths:
1. Similar to the concurrent work, this paper introduces a general framework that uses LLMs to map high-level natural language instruction to low-level actions. Since the LLMs are trained on a large amount of text data, they contain enough task and object knowledge to predict how to perform different high-level tasks.
2. In addition to using LLMs to predict low-level actions, the actions are selected based on whether they are feasible given the capabilities of the robot and the environment states. This formulation provides an easy and intuitive way to ground abstract plans generated by the LLMs.
3. The feedback to the LLM is provided by appending actions that have been successfully executed to the prompt.
4. The framework is evaluated in both simulations and in a real-world kitchen with a mobile manipulator. Human raters are used to assess whether the tasks are planned successfully and executed correctly by the robots. Using human raters can account for the fact that many tasks have different ways to be completed. 

Weakness:
I appreciate the fact that this paper introduces a very general framework that can potentially leverage different LLMs and primitive skills. However, I am worried that the main text lacks enough clarity in stating the assumptions and limitations of the current implementation.

1. What are the 551 skills? It would be great if the authors can explain the definition of skills in the context of this paper. For example, is grasping an apple a different skill than grasping an orange?
2. The main text mentions that the value function of a skill is trained via TD backups regardless of how a skill is implemented. However, this description contradicts the appendix stating that some of the value functions are defined based on heuristics. Another important detail missing from the main text is that the value functions are calibrated empirically. I imagine that this calibration is very important as the probabilities/likelihoods from the LLMs and affordance functions are not normalized. The authors should provide more details on calibrating the likelihoods and balancing the likelihoods from the LLMs and affordance functions in the main text. I also see examples (e.g., third step in Figure 12 (a) in the appendix) where the LLM and affordance function predict the highest probabilities for different actions, resulting in a very small margin between the combined probability of the correct action and of the incorrect one. How likely can this happen and cause failures? 
3. Although the goal of this paper is to leverage the generalization capability of the LLMs, the tasks tested seem to be very similar. Most tasks can be completed with the sequence of <find obj, pick up obj, go to loc, and put down obj>. Therefore, it is hard to see if the LLMs are actually required to generalize. Why is simply predicting the target object not enough? (since the location of each object is also predefined as stated in the appendix). Another question is why does the location of each object need to be pre-specified instead of being predicted by the LLMs?
4. The only connection between the environment and the LLM is the action feedback. Therefore the LLM does not have direct access to the environment state. It would be interesting to see a discussion on this limitation. 
5. Related to the comment above, I also hope to see a comparison (even just in simulation) with existing vision-and-language navigation/object interaction methods that finetune LLMs on specific tasks. This comparison would provide insight into whether prompting or finetuning is a more effective way to balance generalization and task success. 


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes a novel idea of combining LLMs and value functions for planning. The idea is tested with different LLMs and different implementations of primitive skills in relatively diverse settings and on real robots. This paper can be improved more by discussing the assumptions that the current implementation holds.  
issues: Besides the comments above, I have a few minor suggestions. 
1. I think the authors could provide examples of the prompts in the main text since these prompts have very specific structures that I imagine are very important.  
2. Why is the execution success rate for language ablations and planning success rate for value function ablations not included? I am very curious to see the execution success rate for the “Generative” baseline.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: By assuming a set of primitives and their learned affordances, SayCan reweighs an LLM's generated steps in a task plan by their affordances. This enables a specific robot hardware to influence the LLM planner by only appended chosen steps after weighing by affordances. 
strengths_and_weaknesses:  - Excellent figures, real world experiment results, and video.
 - Simple and effective key idea.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper demonstrates an impressive zero-shot, language-conditioned mobile manipulation system, capable of performing long horizon tasks in a robot hardware aware manner. The discussion of failure modes and challenges in building the system in the main paper and appendix were useful. The quantitative and qualitative evaluations were comprehensive. The key idea is simple and effective, and leads to a useful and plan-interpretable system. 
issues: - This work assumes the robot is given a set of skills. Appendix C.2 that choosing this set of primitive requires 1) a set of candidate tasks to see which primitives would be useful for the LLM, and 2) an attempt in training their affordance-based policy to gauge their real world consistency and success. I think this is a limitation which could be discussed in more detail in the main paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a method for robots to understand the instructions expressed in natural language and execute them in the real world. The sequences of low-level actions to perform instructions are estimated by using a large language model (LLM), and an appropriate prompt that extracts low-level action representations of instructions from the LLM is proposed. Furthermore, a learned value function is used to estimate how likely each action is to succeed in the current state in the real world. Combining probability provided by LLM and value functions, appropriate actions in the current state can be selected in actions that are predicted as next step actions by LLM. By appending the action representation to the prompt and querying the LLM, further next action can be predicted. Experimental results show that the proposed method can work well in the real environment. 
strengths_and_weaknesses: # Strengths
- A novel method for robots to understand the instructions using LLM is proposed. 
- Experiment was conducted in the real world and the results show that the proposed method can work well in the real environment. 

# Weaknesses and suggestions
- A lot of important information is written in the appendix. I understand due to space limitations but the contents should be selected and the important results should be written in body text. 
- Line 179: What is the definition of `skills` and `skill families` 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think it is worth accepting this paper because it presents a new potential of the use of the LLMs in robotics.
issues: Described in the strengths and weaknesses section.
review_title: null

Meta-review: metareview: Below is a summary of the strengths and weaknesses of the paper, according to the reviewers.

Strengths:
- The overall proposed pipeline is interesting, simple, and novel, with very general applicability.
- The engineering and integration effort that has gone into this paper is very impressive.
- There is significant evaluation with a real robot.
- The video is very interesting and impressive.

Weaknesses:
- At a high level, all the tasks tested are quite similar (find object, pick up, go to location, put down).
- There are many small details which are missing from the paper and the appendix, which make it difficult to fully understand or re-implement this work.
- It is not clear what the definition of a "skill" is, as this is often defined differently in different papers.

In the rebuttal, please address the above weaknesses, as well as the other concerns and questions raised by the reviewers.

-------------

Update after the rebuttal:

All four reviewers have very high opinions of this paper. It is clearly a strong and timely paper, with a clear novel technical contribution which covers new ground at the intersection of NLP and robotics, and the paper has the potential to become a seminal paper in the field. The extensive real-world system put together is very impressive as a demo, and is an exciting step forwards towards robots assisting humans in everyday environments. The main weakness is the lack of reproducibility, due to (1) the large resources required to re-implement the method (although this is not strictly a limitation of the contribution), and (2) the lack of implementation details in the paper (although this can be addressed through the supplementary material, as reviewers have suggested). This is one of the major robot learning papers in recent months, and it will have broad appeal across the entire robot learning community.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)