Leveraging Language for Accelerated Learning of Tool Manipulation
Authors: Allen Z. Ren
Bharat Govil
Tsung-Yen Yang
Karthik R Narasimhan
Anirudha Majumdar
Venue: CoRL 2022 Poster (2022)
Keywords: Language for Robotics
Tool Manipulation
Meta-learning

Abstract: Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Leveraging Language for Accelerated Learning of Tool Manipulation
Allen Z. Ren', Bharat Govil", Tsung- Yen Yang', Karthik Narasimhan 2 Anirudha Majumdar
Department of Mechanical and Aerospace Engineering. 'Department Computer Science Princeton University
(allen. rem, bgovil, ty3, karthikn, ani.man hai.majjumdrr waajuumdar/ppriinntoo...d
Abstract: Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguis- tic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for given task. We obtaín diverse descrip- tions of various tools in natural language and use pre-trained language models generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguis- tic information and meta-lcarning significantly accelerates tool learning in several manipulation tasks including pushing. lifting, sweeping. and hammering.
Keywords: Language for Roboties, Tool Manipulation, Meta-learning
Phase Meta-training with tool descriptions
Phase Adaptation to unseen tools
"A mallet typically has a long, cylindrical shape with head on one end. Mallets are used strike push objects, hammering nails"
"Most paint rollers are cylindrical in shape, with rounded end flatter end. rounded used to apply paint to the surface'
"The handle of pller tool used grip objects. It is designed to fit comfortably in the hand. The handle typically long and thin"
"A faucet generally has cylindrical shape, with hole near the top for water to come out The hole usually surrounded by ring"
Pushing
Lifting
Sweeping
Hammering
Figure 1: Rich, semantic knowledge language descriptions, as geometric features common of the tools, can help policies adapt faster to unseen tools (right) in pushing, lifting, sweeping, and hammering tasks (bottom) after meta-lcarning on training tools (left).
Introduction
The ability to quickly learn how to use a broad range of new tools is considered one of the hallmarks of human intelligence [1]. Humans are able to attain degree of proficiency with a new hand tool or a virtual tool such as a joystick or remote) within just a few minutes of interac- tion [2]. This rapid learning relies the ability understand discover the affordances [3] of new tool, Le., the ability to perform a certain action with the tool in a given environment. For exam- ple, hammer affords the opportunity grasp it,
tool (e.g.,
hammer nail, use it push another object. In contrast, a spatula affords the opportunity to flip pancake or sweep food ingredients into bowl. Endowing robots with the ability quickly discover and exploit affordances of tool in order to learn how to perform a given task has been a long-standing grand challenge in robotics [4].
Equual contribution in advising
Video showing the four manipulation tasks:
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

While humans routinely rely on geometric priors and visual observations for tool manipulation, they can also 'read' text manuals or linguistic descriptions to understand affordances of new tools and quickly adapt to using them. In this work, we investigate whether language can help robots accelerate the process of learning to use a new tool. In particular, consider the following descriptions of two types of tools:
The shape of tongs is typically that of a V. with two long, thin handles that come point at the top, and a gripping area the middle.
A spatula is kitchen utensil for flipping over food while cooking. The head of the spatuda is usually rectangular or oval in shape. The handle of the spatula usually long and thin.
Our key intuition is that such natural language descriptions of tools contain information about the affordances of the tools, how to exploit these affordances for given task, and how perceptual features of tools (e.g. their visual appearance and geometry) relate to their affordances. Moreover, language can help capture the shared structure of tools and their affordances. Thus, if one has previously learned to use a set of tools (with corresponding language descriptions), a description of a new tool can help exploit this prior knowledge in order to accelcrate learning.
Statement of Contributions. Our primary contribution is to propose ATLA - Accelerated Learn- ing of Tool Manipulation with LAnguage- models (LLMs) to accelerate learning of tool manipulation skills. The overall approach is illustrated in Fig. 1. We propose to use LLMs in two distinct ways: generate the language descriptions for tools and to obtain the corresponding feature representations. At meta-training time, the meta- learner updates base-learner that quickly fine-tunes manipulation policy; this fine-tuning process is conditioned on the LLM representations corresponding to the language descriptions of each tool. Specifically, we propose simple gradient-based meta-learning setup based on Reptile [5] that per- forms off- off-policy updates. At test time, the base-lcarner adapts to a new tool using its language descriptions and interactions with it. To our knowledge, our approach is the first to utilize LLMs to accelerate learning of new tools. We demonstrate the benefits of using language in diverse set of tool-use tasks including pushing, lifting, sweeping, and hammering.
meta- neeaa-le framework leveraging large language
2 Related Work
Tool Manipulation. Tool manipulation [6. 7, 8, 9, 10] is one of the long-standing problems in robotics research. major challenge is understanding the affordances of the tool in different tasks. Previous work has modeled and learned affordances from parameterized keypoints on the tools [7, 10], from human demonstrations [8], and from spatial-temporal parsed graphs of the tools [11]. Our work instead leverages natural language (e.g., describing affordances of the tools in words) for generalization of affordances in tool manipulation and is compatible with previous approaches.
Language-informed Control. Natural language has been applied to enable efficient robotic learn- ing through (1) generating primitive language instructions for producing control actions (i.e., in- struction following task) [12, 13, 14, 15, 16. 17, 18, 191, (2) learning language-informed reward functions for training control policies [20, 21, 22, 23, 24. 25], and (3) using language correct or adapt the behavior of the robot [26, 27J. However, these works primarily translate natural language into action policies for a specific task with the text providing information on the desired actions that optimize returns (e.g., "push the door"). This means that the text is tightly coupled with the task seen during training, making it difficult to generalize to new distribution of tasks with different dynamics. In contrast, the text in our work only provides a high-level description of the property of each tool, encouraging the agent to extract useful information to generalize to a new task. Some prior work [28, 29, 30, 31] also uses language descriptions of environment dynamics to enable gen- cralization of policies but does not leverage meta- merannca
Meta-learning. Our work uses the framework of meta-learning [32, 33, 34, 35, 36], in which the agent trained with a distribution of tasks, and later adapts quickly to a previously unseen task. "Reptile" is proposed in [5] as a simple first-order, gradient-based meta- meta-learning algorithm that learns an initialization of the neural network's 's parameters for fast adaptation at test time. Recent pa- pers [37, 38, 39, 40) have also explored providing additional context information (e.g., the property of the task) to encode task-specific knowledge for meta-learning agent. However, all these works directly provide the context information either through scalar signals or a learned task embedding, which require domain expertise or a pre-training stage. this work, we assume that the agent is provided with a text description of the tool, which is more accessible and easier to collect.
2

---

3 Problem Formulation
We consider the following goal: given a new tool and corresponding language description(s), we aim to quickly learn a policy for using the tool to achieve a given task. We pose this problem in meta-learning setting in which a policy is trained with a distribution of tools for a given task, and later adapts quickly to previously unseen tool sampled from this distribution in the same task.
Meta-training,
During meta-training, assume access to set T {T: the number of available tools. For each tool Ti, we are also provided a set of corresponding language descriptions L, = (4+ ( liy E f. where N; is the number of available descriptions for tool ti and the set of possible textual descriptions. In addition, each l.5 can describe a different property of tools such as shape and common use. Given particular robotic manipulator and particular task Markov
j_ of tools, where K is
(e.g., pushing, lifting, sweeping. or hammering), each tool 7 induces a partially-observal decision process (POMDP): A, O, Pr. Rz). Here, S, is the state of the entire environment (i.e., combined state of the robot, tool, and potentially other objects to be manipulated using the tool). The robot's action space A (e-g. corresponding to robot joint torques) and observation space (e-g., the space of RGB-D observations from a camera) are fixed across tools. The transition probabilities are given by P, S, A S + [0, 1]. and the reward function is R S, A S, [0, 1]. During meta-training, our goal is to learn policy wa O x L + A parameterized by 0 (e.g., weights of neural network) that can be quickly fine-tuned at test time.
Meta-testing, At test time, we are provided a new tool T, and corresponding language descriptions Ly = [4rي)ー111 E C. We aim to let the meta-learned policy quickly adapt to this new tool in a fixed number of interactions with the tool in order to maximize the expected cumulative reward. This is challenging task since the new tool can be quite different in terms of visual appearance and affordances as compared to previously seen tools in meta-training.
Approach
The key idea behind our approach is to collect and embed language information of the environment into meta-learning, allowing the policy to adapt faster and better to unseen environments.
4.1 Collecting Language Information Using Pre- Trained Large Language Models
A common use of language in robotics is to use it to provide an instruction the robot (e.g. "pick up the green block on the table"). Such instructions are typically specified by humans manually through crowd-sourcing, which can be labor intensive. In our setting, we consider language as additional information about the environment (e.g., "the hammer has a long handle and large head at the top"). The language here is not used to describe the goal (e.g., what do), but to provide information about properties of the environment (e.g., tool shape). This makes the text here task- agnostic, forcing the agent to learn generalizable policies. To obtain diverse set of language descriptions, we are inspired by the recent advances in LLMs that are trained with vast amounts of online data and imbued with rich, semantic knowledge of different objects. We propose using LLMs to provide language descriptions of the tools in the form of question answering. Specifically, we provide the GPT-3 [41] model with the following template prompt through the OpenAl API:
"Describe the [seature] of [name] in detailed and scientific response:"
where "feature" is selected from one of ["shape",, "geometry"] or one of ['common use". "purpose") and "name" describes the tool (e.g.. "a hammer", "a pair of tongs"). We find that adding "detailed and scientific" to the prompt significantly improves the quality of the texts generated. For each tool, generate 10 different paragraphs of descriptions for each of the four features, and then combine paragraphs in each of the four permutations of the features ("shape" and "common use", etc). Each tool 7; is thus paired with a diverse set of 800 language descriptions Lę (see Appendix Al for more examples). Each description 4jj L; is approximately 2-4 sentences long.
4.2 Obtaining Feature Representations from Large Language Models
With the collected language descriptions, we now incorporate them into policy training. One com- mon choice is to train a language module (e.g., long short-term memory (LSTM) [42]) from scratch
3

---

Prompt GPT-3 via OpenAl API
"Describe the [feature] of [name] in detailed and scientific response"
hammer hand tool that consists
Pre-trained BERT model
Language head O
Critic head
0r
Q(o ai)
handle attached to head, usualty made of metal. The head usually in the shape of a rectangle with flat surface on one side and cylindrical surface on the other, which is used for striking nails."
Cat
name: "a hammer feature: "shape", "geometry", "common use", "purpose"
Image encoder eg
Actor head
a2
0g
Fixed
Learned
Gradient critic objective
Ti
L
Ot
Gradient via actor objective
Figure 2: Model Overview. First, prompt OpenAl GPT-3 to ohtain diverse set of language descriptions T. of the tool тr. Then for each episode collected, we sample language description lyj randomly from Li, which is then fed into pre-trained BERT model to obtain the representation. The language head further distills the language information. We concatenate the representations from the language head and the image encoder, and then the features are shared by the critic head and the actor head.
to embed features of the language input, which can take substantial time and effort to tune. Instead, pre-trained LLM distill the language descriptions into feature representations. Since LLMs are trained with vast amounts of data, they can better interpret and generalize to the diverse set of long descriptions. Since the GPT-3 model is not publicly available, opt for the Google BERT- Base [43] model on HuggingFace,, which has 110. parameters and outputs a 768-dimensional vector representation for each description input. T-SNE analysis shown in Fig. 3 demonstrates that, without any fine-tuning, the model already captures semantic meanings of the descriptions among tools (e.g., hammer and mallet close to each other).
Fig. 2 shows the overall neural network architecture. We first prompt GPT-3 to obtain set L, of text descrip- tions for tool т; via the procedure in Sec. 4.1. During meta-training,
Shoehor
horseshae
randomly sample Lúj from L; for each episode to ensure that the policy sees a diverse set of de- scriptions. We then freeze the BERT model during policy training. The output from BERT is fed into a single fully- connected layer with ReLU (language head, 01). The im- age observations (possibly from two camera angles-one from overhead and one from the wrist) are passed through convolutional layers (image encoder, 0,), whose output is then concatenated with that from the language head. The
gaintroller
paayel
cllmm
poowerdrill
Austpan
maillet
Shavel
jóracictdo
/pucet
screvitifiver
Ae giinggia GoggARaPPHE
Jcoille;
1f Swiisk
squeegee
banana
actor head (0.) and critic head (0.) then output the ac- tion a, and the corresponding value for the Q function (o, az). See Appendix A3 for more details of the neural network setup.
Figure 3: T-SNE results of BERT output for multiple language descriptions of each tool.
4.3 Meta-training and Testing Language-Conut Conditioned Manipulation Policies
We hypothesize that additional language information of the tools promotes generalization, However, zero-shot transfer to unseen tools can be difficult given the distinct geometries and affordances. Thus we perform meta-training for explicitly training the policy to adapt to distinct tools within the same task. Our algorithm (shown in Algorithm 1) is based on Reptile [5], simple first-order gradient- based meta-learning algorithm, but with an additional sampling strategy to prevent overfitting. At each iteration of the meta-training, one tool is sampled from the training set T. At the base level (line 5), we run the current policy with the language description big sampled from L, We then add the collected experiences in a base replay buffer Pease, and perform B iterations of off off-policy updates using Soft Actor Critic (SAC) [44] in order to obtain the final policy parameters 0' = 10,, , 6',, 8' H'). Then at the meta level (line 12), the network is updated with gradient step towards 0':
1 new 0 (0''' 0).

---

where a is the meta-learning rate and 0 is the collection of the old policy parameters 01. fo. 6 6c. We also highlight the following remarks including differences the Reptile algorithm:
To reduce variance, the actor and critic share the parameters of language head 0, and the image encoder 8, and both modules are updated only with the critic objective.
To prevent overfitting to language descriptions of one tool during training. experiences collected from all tools are saved in a meta replay buffer meta (line 14), and Stse for each tool is initialized with random samples from Smetn (line 3). During policy update at base training, 30% of the experiences are sampled from 9meta- During test time, we do not use any experiences from mmeta- We demonstrate the effectiveness in Sec.
applications of Reptile in supervised learning, the meta update is often averaged over N sam- pled environments: Onew + 0 10+a a Σ; (0;-0) However, we find that N = trains faster and also single tool. > 1 does not offer better performance test time. See Appendix A6 for sensitivity analysis.
matches the test objective of adapting
Performing only single meta update after B 5 iterations of base update can be inefficient. Instead of using large learning rate a which causes unstable training, we perform M meta updates for each adaptation one tool, but collect experiences to frase only at the first update.
Algorithm shows the procedure of adaption at test time. First, the model is provided with target test tool and set of language descriptions. With the adaptation budget BN, we run the policy with the language description I, sampled from L,, The collected experiences are stored in the buffer Grase and used to update the policy parameters.
Algorithm ATLA: Meta-training, N
Require: = [T training set of tools; {Li)/ sets of language descriptions; 6,, t. 8a 8.: policy modules; Smeta: meta replay buffer.
1: while meta-training do
2: Sample r from T # meta level
3: Reset 9tuee with samples from fmett
4:
for m to M do
5:
for 5 = 1 to B do base level
6:
then
7:
Collect episodes each with lij ~ L.: add to Soose
8:
end if
9:
Sample from Pease and update 6., 8g G. with the critic objective
Sample from Poase and update 0a with the actor objective
11: end for
Meta update for 01, Pc, Oa with Eq.
13: end for
Add Brase to 8ietta
15:
end while
Algorithm Adaption at test time
Require: Ty: test tool; Ly: set of language descriptions; 0,, 0.. Our Og: policy modules; Brese + 0: base replay buffer
1: for b= to B, do
2: Collect episodes each with I; L,i add to P
3: Sample from Bruse and update 6,, 0t, 9g with the critic objective
4: Sample from Bouse and update 6,, with the actor objective
5: end for
5 Experiment Setup
Through different tool manipulation tasks in simulation, we aim to investigate the following ques- tions: (1) Does language information help achieve better adaptation to new tools? (2) Does meta- learning improve adaptation to new tools? (3) How does the choice of pre-trained LLLMs affect policy training? (4) Does language information help the policy utilize tools' affordances effecttvvly?
5

---

Tasks. Four different tool manipulation tasks are implemented (see bottom of Fig. 1); (1) pushing: pushing the tool to a fixed location on the table; (2) lifting: reaching and lifting the tool up from the table to some target height; (3) sweeping: using the tool to sweep a cylinder to a fixed location on the table; (4) hammering: using the tool to hammer peg further into hole in block. Solving these tasks benefits from an understanding of the geometric affordances of the tools such as the grasp location. See App. A2 for more details the task setup including the reward functions.
Robot. We build custom simulation environments with a 7-DOF Franka Panda arm in the PyBullet simulator [45]. We use RGB cameras with 128 x 128 image outputs, placed at different off-arm locations and at the arm wrist depending on the needs the tasks. For all tasks, we use 4-DOF cartesian velocity (3D translation and yaw) the action output from the policy. The arm joints then commanded with a jacobian-based velocity controller at 5Hz. The policy does not command the gripper; instead, we use the heuristic that once the gripper is below some height, the gripper closes to grasp the object. If the grasp fails, the gripper re-opens if rises above the threshold.
Tools. We collect total of 36 objects (See App. Al for the full list) from open sources. Most of the objects common tools such hammer and axc. Some of them are less used tools but have distinct geometry and affordances, such as banama whose inner curvature may help push other objects. We split the objects into training set of 27 and test set of objects with similar geometry or affordances (e.g.. hammer and mallet) into different sets.
try separate
Baselines. We compare ATLA (ours) with the following (Fig. 4): (a) AT AinyLL (ours): ATLA with smaller BERT encoder (BERT-Tiny [43] with 4.4 million parameters and 128- 228dimmnnioonal output). (b) AT: ATLA without language information. (c) AT-XL: ATLA without language information but larger networks for 0₂ and 0. (matching the number of parameters of ATLA). (d) SAC-LA: vanilla multi-environment training with SAC and language information but without meta-training objective. (e) SAC: SAC-LA but without language information. SAC-LA and SAC training follow Algorithm 3 in Appendix A4; they do not perform inner adaptation (B = 0) at training time, and the gradient experiences sampled from any environment (N oo). All meta learning baselines use = 5 and = 1. See Appendix A6 for sensitivity analyses on N and B.
update is averaged
Metric. For all experiments, we save the model checkpoint with the highest running- aveerage reward on the training dataset. After training, for each test tool load the checkpoint and run fixed number of iterations of adaptation. In Fig. 4, we report the highest reward at adaptation, averaged over seeds for each test tool. See Appendix A6 for reward in numbers for each tool and task.
Results
Pushing
Reward 600i
Crowbar Paint Roller
Plier
Hammer
Spoon
Scissors
Faucet
Trowel
Wineglass
Lifting
Reward
10
5
Crowbar Paint Roller
Plier
Hammer
Spoon
Scissors
Faucet
Trowel
Wineglass
Sweeping
Reward
25
15
5
Crowbar Paint Roller
Plier
Hammer
Spoon
Scissors
Faucet
Trowel
Wineglass
Hammering
Reward
Crowbar Paint Roller
Plier
Hammer
Spoon
Scissors
Faucet
Trowel
Wineglass
ATLA
AT-TinyLA
AT
AT-XL
SAC-LA
SAC
Figure 4: Poot- reward in mean and standard deviation over 3 seeds across 4 tasks and test tools.

---

Reward
Adaptation Iteration
ATLA
SAC-LA
Figure 5: Curves of adaptation iteration vs. reward of the test tools (columns) in four tasks (rows).
Q1: Does language information help achieve better adaptation to tools? Fig. 4 shows that ATLA and AT-TinyLA perform better than AT and AT-XL among most tools in the tasks. The differences
significant in sweeping and hammering, which more difficult and language information can better inform the affordances of the tools. Among the 9 tools, ATLA and AT- TinyLA always perform better with crowbar, plier, hammer, scissors, faucet, and trowel. ATLA and AT-TinyLA do not perform better mostly when low reward is achieved for all baselines for that tool, such wineglass in pushing and lifting and paint roller in sweeping. Fig. also shows that language helps faster learning in ATLA compared to AT in most cases, with the agent achieving higher rewards with fewer episodes of adaptation.
We also find that different tools learn better for different tasks. For example, hammer better in sweeping than plier probably due to its long bar, but worse in pushing also due the small inertia along the long bar causing instability during pushing. As ATLA performs better among most tools, language information can provide useful information about the tool affordances in different tasks.
Q2: Does meta-learning improve adaptation to new tools? Fig. 4 shows that with or with- out language information (ATLA AT-TinyLA vs. SAC-LA, or AT / AT-XL vs. SAC), meta-learning improves final performance after adaptation. Without meta-learning, SAC-LA shows smaller improvement over SAC (e.g.. plier, hammer, scissors, and trowel). This demonstrates that language information particularly helps when combined with meta-learning. Fig. 5 also compares the adaptation curves between ATLA and SAC-LA: those of SAC-LA tend stag- nate or fluctuate while those of ATLA tend to rise steadily. This indicates meta-learning trains the pol- icy to better adapt to new tools after training. In Ap-
With Ameta
16.
Training Reward
pendix A6 also perform sensitivity analysis on the number of inner adaptation during meta-trant vrea-traiinnng
B, which highlights the effectiveness of performing
Without Bmeta
a few steps of inner adaptation.
Ablation:
meta replay buffer. We inves-
lleration
500
1000
tigate the effect of re-using experiences from other tools (saved in Smeu during adaptation). For this, we run ATLA without 8meta on the pushing task. Across the 9 test tools, the (-20%, - 10%, 13%, 18%, +5%, - 25%, 18%, -18% -5%) compared to ATLA with 8meia- Note that the effect is more prominent when the reward difference between ATLA and AT is larger (e.g., 20% with crowbar and 25% with scissors), indicating that language information is more ef- fective if 3metn is applied. We also find using 3meta accelerates the meta-learning process (Fig. 6) demonstrating that sharing experiences among tools makes training more efficient.
Figure 6: Using meta replay buffer also acceler- ates training.
post-adaptation reward is mostly lower
Q3: How does the choice of pre-trained LLMs affect policy training? Fig. 4 shows that ATLA usually attains higher post-adaptation reward than AT-TinyLA, which uses smaller pre-trained BERT model, indicating that the policies benefit from the richer representation of the language descriptions that the bigger BERT model offers.
7

---

Q4: Does language information help the policy utilize tools' affordances effectively? The results above have shown that language descriptions of the geometric features and common use of the tools help policies adapt to new tools for given task. Fig. 7(a) visualizes the effect in the example of using a crowbar for sweeping. Language descriptions of a crowbar often contain phrases including "long and thin bar", "curved"" "hook", "used to leverage", and "used to pry open things". With the descriptions, ATLA (orange curve in Fig. 7(a)) enables the policy to adapt quickly to this tool unseen during meta-training towards the target. As a comparison, we replace the descriptions with only the sentence crowbar a long and thin bar," and the policy (green curve in Fig. 7(a)) does not adapt well.
the policy learns use the curved hook to better steer the cylinder
One common feature among tools is the handle. Language descriptions of a trowel includes phrases like "flat, triangular blade", "handle to be grasped", and "used for scooping". While ATLA learns to grasp at the handle (Fig. 7(b) top), when we remove "handle" from all the descriptions, the robot fails to grip firmly on the handle and loses the grip eventually (Fig. 7(b) bottom).
(a)
(b)
Adaptation Reward
22
Success
20
18
Failure
Figure 7: With language information, ATLA is able to adapt the policy to utilize the affordances of the tools (a) curved hook crowbar; (b) handle trowel.
Figure 8: Policies may fail to cor- rectly utilize affordances.
Limitations. In some cases, we observe that the policies can still fail to correctly utilize the tool affordances with language information. In the sweeping task, the paint roller is the only tool that ATLA fails to perform the best with. Fig. 8 (top) shows the grasp learned by the policy. It failed to use the bigger opening between
roller and the handle on the other side to sweep the cylinder. Fig. 8 (bottom) shows that in the hammering task, the policy fails use the faucet, specifically, its relatively flat head to push towards the nail. However, these affordances can sensitive to the initial pose of the tool and can be difficult to explore. This also highlights one of the limitations of our work. We use a relatively simple task policy setup for tool manipulation tasks, which is directly mapping image inputs to Cartesian velocity commands. This creates challenges in exploration and learning the skills even with language information. One remedy is to combine with approaches like keypoint-based methods [7] that inject additional domain knowledge into the policy. In addition, use a relatively small dataset of tools, which may limit the potential of using language information. It would be particularly interesting model revolute joints of the tools and perform more complex tasks such as picking up objects with a pair of tongs. Furthermore, our current evaluation does not consider real robot experiments, which the focus of this work.
7 Conclusion
In this work, we investigate using large language models (LLMs) to accelerate adaptation of poli- cies new tools in tool manipulation tasks. We use LLMs to both (1) generate diverse language descriptions of the tool geometry and common use, and (2) obtain vector representations of the descriptions. We then propose language-conditioned meta-learning that trains policies to quickly adapt to new tools. The results demonstrate that combining language information and meta-learning significantly improves the performance when adapting unseen tools.
8

---

Acknowledgments
The authors were partially supported by the Toyota Research Institute (TRI), the NSF CAREER Award #42044149], the Office of Naval Research IN00014-1--1-2003 N00014-18-1-2877 and the School of Engineering and Applied Science at Princeton University through the generosity of William Addy '82. This article solely reflects opinions and conclusions of authors and not NSF, ONR, Princeton SEAS, TRI or any other Toyota entity.
References
[1] A. H. Taylor and R. D. Gray. Is there a link between the crafting of tools and the evolution of cognition? Wiley Interdisciplinary Reviews: Cognitive Science, 5(6):693-703, 2014.
[2] K. R. Gibson. Evolution of human intelligence: The roles of brain size and mental construction. Brain, Behavior and Evolution, (9(1))) 59(1-2):10-20, 2002.
[3] J. Gibson. The theory of affordances. Hilldale, USA, 1(2)677-2,, 1977.
[4] M. Mason. Toward robotic manipulation. Annual Review of Control, Robotics, and Au- tonomous Systems, 1(1), 2018.
[5] A. Nichol, J. Achiam, and J. Schulman. On first-order meta- algorithms. arXiv preprint Ar0iv:1803.000 0229999 2018.
[6] C. C. Kemp, A. Edsinger, and E. Torres-Jara. Challenges for robot manipulation in human environments, IEEE Robotics & Automation Magazine (RAL), 14(1):20-29 2007.
[7] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning task- oriented grasping for tool manipulation from simulated self-supervision. The International Journal Robotics Research (IJRR), 39(2-3):202-216, 2020.
[8] A. Xie, F. Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. arXiv preprint 1904:0099000555 2019.
(9] R. Holladay, T. Lozano-Péren Pérez, and A. Rodriguez. force-and-motion constrained planning for tool use. In Proceedings of the IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
[10] M. Qin, J. Brawer, and B. Scassellati. Rapidly learning generalizable and robot-agnostic tool- use skills for wide range of tasks. Frontiers in Robotics and A1, 8, 2021.
[11] Y. Zhu, Y. Zhao, and S. Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[12] T.-Y. Yang, A. Lan, and K. Narasimhan. Robust and interpretable grounding of spatial refer- ences with relation networks. In Findings of the Association for Computational Linguistics: EMNLP, 2020.
[13] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, Hsu, J. Ibarz, B. Ichter, A. Irpan, Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Rao, J. Rettinghouse,
Reyes, P. Ser- manet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as can and not as I say: Grounding language in robotic affordances. In arXiv preprint 1XXi::204.016999 2022.
Y. Lu, L. Luu, C. Parada, P. Pastor, Quiambao,
[14] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference Artificial Intelligence, 2011.
[15] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language commands to robot control system. In Proceedings of the International Symposium on Ex- perimental Robotics, 2013.

---

[16] Y. Artzi and L. Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions actions. Transactions of the Association for Computational Linguistics, 1:49- 62, 2013.
[17] D. Chen and R. Mooney, Learning interpret natural language navigation instructions from observations. In Proceedings of AAAI Conference on Artificial Intelligence, 2011.
[18] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and grounded naviga-
A. Van Den Hengel. Vision-and--o and-language navigation: Interpreting visually-grounded tion instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[19] S. R. Branavan, H. Chen, L. Zertlemoyer, and R. Barzilay, Reinforcement learning for mapping instructions actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 2009.
[20] MacGlashan, M. Vroman, M. desJardins, M. L. Littman, S. Muresan, Squire, S. Tellex, D. Arumugam, and L. Yang. Grounding english commands to reward functions. In Proceedings of Robotics: Science and Systems (RSS), 2015.
[21] D. Bahdanau, F. Hill, J. Leike, E. Hughes, S. A. Hosseini, P. Kohli, and E. Grefenstette. Learn- ing understand goal specifications by modelling reward. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[22] P. Goyal, S. Nickum, and R. J. Mooney. Using natural language for reward shaping in rein- forcement learning. In Proceedings of the International Joint Conference on Artificial Intelli- gence (ICCAI), 2019.
[23] E. C. Williams, N. Gopalan, M. Rhee, and S. Tellex. Learning to parse natural language to grounded reward functions with weak supervision. In Proceedings of the IEEE/RSJ Interna- tional Conference on Robotics and Automation (ICRA), 2018.
[24] T.-Y. Yang, M. Y. Hu, Y. Chow, P. J. Ramadge, and K. Narasimhan. Safe reinforcement learn- ing with natural language constraints. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[25] T. R. Sumers, M. K. Ho, R. D. Hawkins, K. Narasimhan, and T. L. Griffiths. Learning rewards from linguistic feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[26] J. D. Co-Reyes, A. Gupta, S. Sanjeev, N. Altieri, J. Andreas, J. DeNero, P. Abbeel, and S. Levine. Guiding policies with language via meta-learning. In Proceedings of the Inter- national Conference on Learning Representations (ICLR), 2019.
[27] S. Karamcheti, Sadigh, and P. Liang. Learning adaptive language interfaces through de- composition. In Proceedings of the First Workshop on Interactive and Executable Semantic Parsing, 2020.
[28] A. W. Hanjie, V. Y. Zhong, and K. Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
[29] S. Branavan, D. Silver, and R. Barzilay. Learning to win by reading manuals in Monte-Carlo framework. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:
Human Language Technologies, 2011.
[30] V. Zhong, T. Rocktäschel, and F. Grefenstette. Rtfm: Generalising novel environment dynamics via reading. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2019.
[31] K. Narasimhan, R. Barzilay, and T. Jaakkola. Grounding language for transfer in deep rein- forcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018
10

---

[32] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-leariing for fast adaptation of deep networks, In Proceedings of the International Conference on Machine Learning (ICML), 2017.
[33] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. In Proceedings of the International Conference Learning Representations (ICLR), 2018.
[34] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing (NeurIPS), 2019.
[35] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[36] C. Finn, K. Xu, and S. Levine. Probabilistic madel-agnostic meta meta-leaming leamiin..nng In Advances in Neural Information Processing Systems (NeurIPS). 2018.
[37] R. Dubey, E. Grant, M. Luo, K. Narasimhan, and T. Griffiths. Connecting context-specific adaptation in humans to meta-leariing arXiv preprint arXiv:2011.13782, 2020.
[38] R. Vuorio, S.-H. Sun, H. Hu, and J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[39] S. W. Yoon, J. Seo, and Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[40] L. Lan, Z. Li, X. Guan, and P. Wang. Meta reinforcement learning with task embedding and shared policy. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[41] B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[42] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 98): 1780, 1997.
[43] I. Turc, M.-W. Chang. K. Lee, and K. Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: wtt.://9.....33 2019.
[44] T. Haarnoja, A. Zhou, Abbeel, and S. Levine. Soft actor-critic: Off off-cooicy maximum en- tropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
[45] E. Coumans and Y. Bai. Pybullet, a Python module for physics simulation for games, robotics and machine learning. 2016.
[46] Ba, R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: arXiv:1607.004453 2016.


Reviews:
[{"review_id": "kZgHzTTT02h", "reviewer": "Reviewer_uYWj", "summary": "The ability to use different tools is essential to many robotic manipulation tasks. Instead of relying demonstration data from human, this paper explores language descriptions of tools (e.g.  its geometry, common uses) can help control policies adapt faster to new tools for a given task. The paper takes the increasingly popular approach of prompting large language models, GPT3 to generate diverse descriptions of various tools. It then uses pre-trained language models, BERT to compute feature representations. The paper proposes language-conditioned meta-learning to learn policies that can quickly adapt to new tools. Relying a custom simulator, the paper tries to show that combining linguistic information and meta-learning can accelerate tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Language can increasingly help robotic tasks, e.g. learning high level policies that composes complex tasks into primitive steps, a natural way to specify tasks. This paper seeks to expand the capability to learn to use tools based on language description. The high level approach consists of (1) prompting large language models, GPT3 to generate diverse descriptions of various tools; (2) uses pre-trained language models, BERT to compute feature representations; (3) language-conditioned meta-learning (based on Reptile) to learn policies that can quickly adapt to new tools. \n\n2. Using a custom simulator, with a 7-DOF Franka Panda arm in the PyBullet simulator, the paper presents results on language conditioned meta-learning for learning the use of tools in several manipulation tasks including pushing, lifting, sweeping, and hammering.\n\nWeaknesses\n\n1. The main problem of the paper is evaluation methodology and lack of convincing experimental results. The paper evaluates tools for four tasks, pushing, lifting, sweeping, and hammering which do not seem to agree with the tool description. For example, the language description of a plier, \"a plier is a hand tool used for gripping objects It consists of a pair of metal jaws with teeth that open and close when the handles are moved. Plier typically has a long, narrow neck and a tapered head that becomes progressively thinner as it extends from the neck to the tip.\" It is hardly clear how this description helps the use of plier to push, lift, sweep, hammer. The robot did not actually learn to open and close the metal jaws. \n\n2. The paper claims tools geometry and affordance description helps. However, there are no results on whether the model actually learned to associate the geometry description to extracted visual features corresponding to geometry.\n\n3. The video in the appendix does not support the paper's claim. There does not seem to be any \"understanding\" of geometry and use the tool in the way described in language.\n\n4. The paper presents results in terms of rewards. It should provide success rate as a more direct metric. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The high level motivation of the problem is very appealing, the ability to use different tools is essential to many robotic manipulation tasks and robots should know how to use tools based on their language description. The high level approach also seems to be reasonable, prompting large LM to generate diverse descriptions of various tools, uses pre-trained language models to compute feature representations and use language-conditioned meta-learning to learn policies. However, the evaluation environment does not provide convincing evidences that the robots can learn from language to use tools as intended to accomplish manipulation tasks. \n\nI would recommend the authors create simulation tasks that demonstrates the intended use of these tools and redo the evaluation. Since major revision is needed, I would encourage the authors to resubmit to another venue. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Please provide task success rate for the experimental results.\n\n2. The paper claims language descriptions of a crowbar often contain phrases including “long and thin bar”, “curved”, “hook”, “used to leverage”, and “used to pry open things”. With the descriptions, ATLA (orange curve in Fig. 7(a)) enables the policy to adapt quickly to this tool unseen during meta-training. It is not clear the model actually relied on these features. Please provide more direct evidences. "}}, {"review_id": "YpRSYjtwxVj", "reviewer": "Reviewer_tT65", "summary": "The paper proposed a meta-learning framework for quickly adapting to novel tools by meta-learning a tool-description conditioned RL policy. The approach uses GPT-3 to automatically generate tool descriptions, bases their meta-learning framework off of REPTILE, and RL policy using SAC. Their ablations demonstrate that with 10 episodes in the test **task + tool**, their meta-learning framework outperforms baselines which do not perform metalearning and/or are not language-informed.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The motivation for the task and for using language in the approach is interesting and intuitive.\n - The writing is easy to follow and the figures are clear.\n\nWeaknesses: \n -  Experiments seem to study *tool + task adaption* rather than their claim *tool-only adaptation*. Task adaption in the context of meta-learning is not a contribution, and the lack of *tool-only adaptation* experiments means the key quantitative results of the approach is lacking. \n- Their proposed meta-learning framework is conceptually very similar to multi-task learning, yet empirically their multi-task baseline performs  significantly worse. Ablation results for design choices in their meta-learning framework is absent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the motivation for the approach and the task, but there are some key experiments missing to back up their goal in studying accelerated tool learning and their meta-learning framework. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": " - The authors wanted to investigate whether linguistic information could accelerate tool learning. However, to the best of my understanding, for each approach they trained one policy for all tasks and tools, which entangles accelerated tool adaptation with accelerated tool and task adaptation from language. This means all policies are not explicitly given information about which task it is performing. I think the quantitative evaluation should have trained policies for each task separately to disentangle which design decisions accelerate tool learning v.s. task learning.\n- Their proposed meta-learning framework used a meta-batchsize of N=1 (L169), which is practically multi-task learning (with slow learning rates). Therefore, I'm surprised about the performance difference between multi-task baselines (e.g. SAC-LA) and the proposed approach. How much of this performance gap can be attributed to, say, training instability in multi-task baselines? A more detailed ablation on the proposed meta-learning framework can help the community better understand how each part (e.g. multiple meta updates) contributes to this performance gap."}}, {"review_id": "QbN3CB0QJez", "reviewer": "Reviewer_x7HJ", "summary": "This paper presents a method for robots to adopt the usage of unseen tools. The novelty of the proposed method is the use of linguistic information obtained from pre-trained large language models (LLMs) to adapt faster to new tools. The proposed method consists of two phases. The first phase is meta-learning with tool description. In this phase, the policies to use tools are learned from language descriptions of the tools as well as observations. The language descriptions represent the feature of objects and are obtained by question answering using LLMs. Combining the language information makes it possible to learn generalized policies of tool usage. The second phase is the adaption to unseen tools, and policies of new tools are learned from observations and their language descriptions. The meta-learning and use of the language description allow faster adaptation to unseen tools.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The generalized policies can be learned by the use of language descriptions. \n- The use of language description makes the adaptation of unseen tools faster. \n- The language descriptions are obtained using LLMs, and human manual data collection is not required.\n\n# Weaknesses and suggestions\n- It becomes better to include a deeper discussion of the experimental results. For example, AT-TynyLA outperformed ATLA in some cases, and, on the other hand, there is a case where only AT-TinyLA is very poor performance. I would like to know this reason and the effects of the size of language models. \n- The reward of ATLA tends to be high, but the difference is not so large. How much does this difference in rewards affect task performance? \n- line 244: Table 4 is missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think it is worth accepting this paper because it presents a new potential of the use of the LLMs in robotics.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Described in the strengths and weaknesses section.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "94ivoOCJg4", "reviewer": "Reviewer_kdpR", "summary": "This paper proposes language-conditioned meta-learning to learn policies that can efficiently adapt to new tools. The key insight is linguistic information about a tool can help control policies adapt faster to new tools for a given task. They show the effectiveness of their method in several manipulation tasks including pushing, lifting, sweeping, and hammering.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is interesting to read and easy to follow.\n* The authors propose an interesting and reasonable idea of using LLMs to accelerate the learning of new tools.\n* The experiment is thorough and answers many most interesting questions.\n* The idea of using GPT-3 to generate data for training completely removes the need for human labeling.\n\nSuggestions and Improvements：\n* I understand the focus of this paper is not real robot experiments. But the proposed algorithm is very unlikely to be deployed on real-world robots. If we perform meta-training in the real world, changing tools/tasks is extremely challenging. Resetting states is also very challenging. If we perform meta-training in simulation and apply Sim2Real transfer, the meta-testing state raises additional challenges.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'd recommend \"weak accept\". The paper proposes an interesting and novel idea of using language to accelerate the adaption of using tools. The method is direct and simple while achieving relatively good results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* The \"Statement of Contributions\" can be better organized. It is currently mixed with a brief summary of the method.\n* L205, (d) SAC-LA, what is meta training objective?\n* L161, could you please provide a more intuitive understanding of why both $\\theta_l$ and $\\theta_o$ are updated only with the critic objective\n* L194, is the height manually adjusted based on different tools?\n\n"}}]

Author rebuttal:

Decision: Accept (poster)