Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes language-conditioned meta-learning to learn policies that can efficiently adapt to new tools. The key insight is linguistic information about a tool can help control policies adapt faster to new tools for a given task. They show the effectiveness of their method in several manipulation tasks including pushing, lifting, sweeping, and hammering.
strengths_and_weaknesses: Strength:
* The paper is interesting to read and easy to follow.
* The authors propose an interesting and reasonable idea of using LLMs to accelerate the learning of new tools.
* The experiment is thorough and answers many most interesting questions.
* The idea of using GPT-3 to generate data for training completely removes the need for human labeling.

Suggestions and Improvements：
* I understand the focus of this paper is not real robot experiments. But the proposed algorithm is very unlikely to be deployed on real-world robots. If we perform meta-training in the real world, changing tools/tasks is extremely challenging. Resetting states is also very challenging. If we perform meta-training in simulation and apply Sim2Real transfer, the meta-testing state raises additional challenges.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'd recommend "weak accept". The paper proposes an interesting and novel idea of using language to accelerate the adaption of using tools. The method is direct and simple while achieving relatively good results.
issues: * The "Statement of Contributions" can be better organized. It is currently mixed with a brief summary of the method.
* L205, (d) SAC-LA, what is meta training objective?
* L161, could you please provide a more intuitive understanding of why both $\theta_l$ and $\theta_o$ are updated only with the critic objective
* L194, is the height manually adjusted based on different tools?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed a meta-learning framework for quickly adapting to novel tools by meta-learning a tool-description conditioned RL policy. The approach uses GPT-3 to automatically generate tool descriptions, bases their meta-learning framework off of REPTILE, and RL policy using SAC. Their ablations demonstrate that with 10 episodes in the test **task + tool**, their meta-learning framework outperforms baselines which do not perform metalearning and/or are not language-informed.
strengths_and_weaknesses: Strengths:
 - The motivation for the task and for using language in the approach is interesting and intuitive.
 - The writing is easy to follow and the figures are clear.

Weaknesses: 
 -  Experiments seem to study *tool + task adaption* rather than their claim *tool-only adaptation*. Task adaption in the context of meta-learning is not a contribution, and the lack of *tool-only adaptation* experiments means the key quantitative results of the approach is lacking. 
- Their proposed meta-learning framework is conceptually very similar to multi-task learning, yet empirically their multi-task baseline performs  significantly worse. Ablation results for design choices in their meta-learning framework is absent.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I like the motivation for the approach and the task, but there are some key experiments missing to back up their goal in studying accelerated tool learning and their meta-learning framework. 
issues:  - The authors wanted to investigate whether linguistic information could accelerate tool learning. However, to the best of my understanding, for each approach they trained one policy for all tasks and tools, which entangles accelerated tool adaptation with accelerated tool and task adaptation from language. This means all policies are not explicitly given information about which task it is performing. I think the quantitative evaluation should have trained policies for each task separately to disentangle which design decisions accelerate tool learning v.s. task learning.
- Their proposed meta-learning framework used a meta-batchsize of N=1 (L169), which is practically multi-task learning (with slow learning rates). Therefore, I'm surprised about the performance difference between multi-task baselines (e.g. SAC-LA) and the proposed approach. How much of this performance gap can be attributed to, say, training instability in multi-task baselines? A more detailed ablation on the proposed meta-learning framework can help the community better understand how each part (e.g. multiple meta updates) contributes to this performance gap.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The ability to use different tools is essential to many robotic manipulation tasks. Instead of relying demonstration data from human, this paper explores language descriptions of tools (e.g.  its geometry, common uses) can help control policies adapt faster to new tools for a given task. The paper takes the increasingly popular approach of prompting large language models, GPT3 to generate diverse descriptions of various tools. It then uses pre-trained language models, BERT to compute feature representations. The paper proposes language-conditioned meta-learning to learn policies that can quickly adapt to new tools. Relying a custom simulator, the paper tries to show that combining linguistic information and meta-learning can accelerate tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

strengths_and_weaknesses: Strengths

1. Language can increasingly help robotic tasks, e.g. learning high level policies that composes complex tasks into primitive steps, a natural way to specify tasks. This paper seeks to expand the capability to learn to use tools based on language description. The high level approach consists of (1) prompting large language models, GPT3 to generate diverse descriptions of various tools; (2) uses pre-trained language models, BERT to compute feature representations; (3) language-conditioned meta-learning (based on Reptile) to learn policies that can quickly adapt to new tools. 

2. Using a custom simulator, with a 7-DOF Franka Panda arm in the PyBullet simulator, the paper presents results on language conditioned meta-learning for learning the use of tools in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Weaknesses

1. The main problem of the paper is evaluation methodology and lack of convincing experimental results. The paper evaluates tools for four tasks, pushing, lifting, sweeping, and hammering which do not seem to agree with the tool description. For example, the language description of a plier, "a plier is a hand tool used for gripping objects It consists of a pair of metal jaws with teeth that open and close when the handles are moved. Plier typically has a long, narrow neck and a tapered head that becomes progressively thinner as it extends from the neck to the tip." It is hardly clear how this description helps the use of plier to push, lift, sweep, hammer. The robot did not actually learn to open and close the metal jaws. 

2. The paper claims tools geometry and affordance description helps. However, there are no results on whether the model actually learned to associate the geometry description to extracted visual features corresponding to geometry.

3. The video in the appendix does not support the paper's claim. There does not seem to be any "understanding" of geometry and use the tool in the way described in language.

4. The paper presents results in terms of rewards. It should provide success rate as a more direct metric. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The high level motivation of the problem is very appealing, the ability to use different tools is essential to many robotic manipulation tasks and robots should know how to use tools based on their language description. The high level approach also seems to be reasonable, prompting large LM to generate diverse descriptions of various tools, uses pre-trained language models to compute feature representations and use language-conditioned meta-learning to learn policies. However, the evaluation environment does not provide convincing evidences that the robots can learn from language to use tools as intended to accomplish manipulation tasks. 

I would recommend the authors create simulation tasks that demonstrates the intended use of these tools and redo the evaluation. Since major revision is needed, I would encourage the authors to resubmit to another venue. 
issues: 1. Please provide task success rate for the experimental results.

2. The paper claims language descriptions of a crowbar often contain phrases including “long and thin bar”, “curved”, “hook”, “used to leverage”, and “used to pry open things”. With the descriptions, ATLA (orange curve in Fig. 7(a)) enables the policy to adapt quickly to this tool unseen during meta-training. It is not clear the model actually relied on these features. Please provide more direct evidences.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method for robots to adopt the usage of unseen tools. The novelty of the proposed method is the use of linguistic information obtained from pre-trained large language models (LLMs) to adapt faster to new tools. The proposed method consists of two phases. The first phase is meta-learning with tool description. In this phase, the policies to use tools are learned from language descriptions of the tools as well as observations. The language descriptions represent the feature of objects and are obtained by question answering using LLMs. Combining the language information makes it possible to learn generalized policies of tool usage. The second phase is the adaption to unseen tools, and policies of new tools are learned from observations and their language descriptions. The meta-learning and use of the language description allow faster adaptation to unseen tools.
strengths_and_weaknesses: # Strengths
- The generalized policies can be learned by the use of language descriptions. 
- The use of language description makes the adaptation of unseen tools faster. 
- The language descriptions are obtained using LLMs, and human manual data collection is not required.

# Weaknesses and suggestions
- It becomes better to include a deeper discussion of the experimental results. For example, AT-TynyLA outperformed ATLA in some cases, and, on the other hand, there is a case where only AT-TinyLA is very poor performance. I would like to know this reason and the effects of the size of language models. 
- The reward of ATLA tends to be high, but the difference is not so large. How much does this difference in rewards affect task performance? 
- line 244: Table 4 is missing.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think it is worth accepting this paper because it presents a new potential of the use of the LLMs in robotics.
issues: Described in the strengths and weaknesses section.
review_title: null

Meta-review: metareview: Summary:

In this paper, we show that linguistic information describing a tool allows us to acquire a policy on how to use the tool rapidly. First, it uses natural language descriptions of various tools and pre-trained language models to acquire functional representations of the tools. Next, meta-learning is performed to learn policies that can be adapted to new tools. Finally, simulation experiments confirm that the combination of linguistic information and meta-learning accelerates the learning of tool operations.

Strength:

- The authors propose an interesting and reasonable idea of using LLMs to accelerate the learning of new tools
- The idea of using GPT-3 to generate data for training completely removes the need for human labeling.
- The use of language description makes the adaptation of unseen tools faster.


Weakness:

- Since the main focus is not real robot experiments, the usability in the real world is not unpredicted.
- The authors proposed meta-learning framework is conceptually very similar to multi-task learning, yet their multi-task baseline performs significantly worse empirically. Ablation results for design choices in their meta-learning framework are absent.
- The main problem of the paper is the evaluation methodology and lack of convincing experimental results. The paper evaluates tools for four tasks, pushing, lifting, sweeping, and hammering, which do not seem to agree with the tool description.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)