Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot Manipulation
Authors: Yifan Zhou
Shubham Sonawani
Mariano Phielipp
Simon Stepputtis
Heni Amor
Venue: CoRL 2022 Poster (2022)
Keywords: Language-Conditioned Learning
Attention
Imitation
Modularity

Abstract: Language-conditioned policies allow robots to interpret and execute human instructions. Learning such policies requires a substantial investment with regards to time and compute resources. Still, the resulting controllers are highly device-specific and cannot easily be transferred to a robot with different morphology, capability, appearance or dynamics. In this paper, we propose a sample-efficient approach for training language-conditioned manipulation policies that allows for rapid transfer across different types of robots. By introducing a novel method, namely Hierarchical Modularity, and adopting supervised attention across multiple sub-modules, we bridge the divide between modular and end-to-end learning and enable the reuse of functional building blocks. In both simulated and real world robot manipulation experiments, we demonstrate that our method outperforms the current state-of-the-art methods and can transfer policies across 4 different robots in a sample-efficient manner. Finally, we show that the functionality of learned sub-modules is maintained beyond the training process and can be used to introspect the robot decision-making process.

Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot
Manipulation
Yifan Zhou Arizona State University yzhou298@asu. edu
Shubham Sonawani
Mariano Phielipp
Arizona State University
Intel AI
sdsonawa@asu.edu
mariano.j j. phielipp@intel.com
Simon Stepputtis Camegie Mellon University stepputtis@cmu edu
Heni Ben Amor
Arizona State University
hbenamor@asu, edu
"Pick up the blue soda can on the table!"
Panda
Kinova
Transferred Policy
UR5
UR5 Real
Figure 1: A human instruction is turned into robot actions via learned The neural network then successfully transferred different robots in simulation and weal-world.
policy.
Abstract: Language-conditioned policies allow robots to interpret and execute human instructions. Learning such policies requires substantial investment with regards to time and compute resources. Still, the resulting controllers are highly device-specific and cannot easily be transferred to robot with different morphology, capability, appearance dynamics. In this paper, we propose sample-efficient approach for training language-condittoned manipulation poli- cies that allows for rapid transfer across different types of robots. By introducing a novel method, namely Hicrarchical Modularity, and adopting supervised attention across multiple sub-modules, we bridge the divide between modular and end-to- end learning and enable the reuse of functional building blocks. both simulated and real world robot manipulation experiments, we demonstrate that our method outperforms the current state-of-the-art methods and can transfer policies across 4 different robots in sample-efficient manner. Finally, show that the func- tionality of learned sub-modules is maintained beyond the training process and can be used to introspect the robot decision-making process. Code is available at https:/giitub..oo//rllaaMaoo
Keywords: Language-Conditioned Learning, Attention, Imitation, Modularity.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Input Modalities
Language Conditioned Neural Network
Execution
Transfer
Camera
Encoder
Module
Joint Angle Encodar
Module
loints
Language Encoder
Grab the Pepsi!"
Modulerity through Attention
Control
Figure 2: Overview: different input modalities, i.e., vision, joint angles and language are fed into a Language-conittonge neural network to produce robot control values. The network is setup and trained in modular fashion individual modules address sub-aspects of the task. The neural net- work can efficiently be trained and transferred onto other robots and environments (e.g. Sim2Real).
1 Introduction
Creating machines that understand and physically execute natural language instructions is a long- standing vision of robotics and artificial intelligence [1, 2, 3, 4, 5J. To faithfully reflect the intentions of the human partner, robots need to interpret instructions within the current situational and behav- ioral context. For example, to execute the command "Pick up the green bottle!", a robot may have to interrelate the words "bottle" and "green" to objects in the scene, identify the corresponding po- sition in the image and, in turn, calculate the spatial relationship to its end-effector. Such inference and decision- decision-making capabilities require a deep integration multiple data modalities inference Conditioned Imitation Learning [2] addresses these challenges by jointly learning perception, language understanding, and control in end-to-end fashion. However, once trained, such innguage-conditionee policies are not applicable to other robots. Due to the monolithic nature of end-to-end policies, robot-specific aspects of the task, e.g., kinematic structure or visual appearance, cannot be individually targeted and adjusted. Vanilla retraining on the new robot can possibly address this issue but comes with the risk of catastrophic forgetting [6] and substantial computational overhead.
the intersection of vision, language and motion. Language-Conditioonl
In this paper, we address the problem of efficient training and transfer of language-conditioned poli- cies across different robot manipulators. The overarching objective is to learn language-conditione policies that can efficiently be transferred when the morphology. capability, appearance, or dynam- ics of the underlying robot changes. Instead of training an individual policy for every new robot, we aim to use methods that quickly transfer and adapt an existing master policy to new robot hardware, see Fig. for an example. To that end, is helpful to implement reusable building blocks or modules that realize specialized sub-tasks [7]. At time of transfer, robot-agnostic modules may not require any adjustments all. However, such modular approach is odds with the monolithic nature of end-to-end deep learning. We therefore introduce methodology which realizes such functionality through sub-modules that are responsible for achieving specific target functionality. The approach builds upon two parts, namely supervised attention and hierarchical modularity. Supervised atten- tion [8] was originally proposed for better alignment between two languages in the field of machine translation. In our approach, we leverage and extend supervised attention to enable the formation of modular components in networks. As results, the user is able manipulate the training process by focusing attention layers on certain input-output variables. By imposing a specific locus of at- tention, we guide individual sub-modules (or parts of attention layers) realize intended target functionality. Hierarchical modularity, on the other hand, is a training regime inspired by curricu- lum learning that aims at decomposing the overall learning process into individual subtasks. At the beginning of this process, only a single subtask is considered. After the corresponding module is trained to convergence, other tasks are addressed one-by-one within a hierarchical cascade. Com- bining both supervised attention as well as hierarchical modularity allows for neural networks to be trained in structured fashion thereby maintaining a degree of modularity and compositionality. At runtime. the user query both the overall control output of the network. well the outputs of each involved module. These queries can help to analyze the decision-making process by generating

---

the outputs of individual sub-networks. Misperforming modules can then be identified, isolated and retrained.
Our contributions can be summarized as follows: (1) we propose a sample-efficient approach for training language-conditioned manipulation policies that allows for rapid transfer across different types of robots, (2) introduce novel method, namely hierarchical modularity, which bridges the divide between modular and end-to-end learning and enables the reuse of functional building blocks, (3) we demonstrate that our method outperforms the current state-of-the-an methods (BC-Z [1] and IP [2]) and can transfer policies across different robots (including from simulation to reality).
2 Related Work
Imitation learning aims at learning agent actions from expert demonstrations and has a rich history in robotics [9, 10. 11]. Previous works [12, 13. 14] have shown the feasibility of this approach for tasks such as helicopter flight, controlling humanoid robots, or collaborative assembly. However, early imitation learning methods are usually conditioned on low-dimensional sensor states and cannot leverage rich sensor modalities such as language and vision. Recent works have leveraged the rep- resentational power of deep learning to extended the paradigm to high dimensional language and vi- sion understanding [15, 16, 17]. This progress is partly fueled by remarkable advances in the domain of image- and video-understanding [18, 19, 20, 21, 22]. For example, the work in [22] shows how vision and language can be projected into a joint embedding space. The result of these multimodal
approaches enables variety of downstream tasks, including image captioning [23, 24, 25), visual question answering systems (VQA) [26, 27], and multimodal dialog systems [28, 29]. A number of works leverage such multimodal networks inputs an autonomous agent [30, 31, 32, 33, 34, 35), thereby enabling the agents to understand semantics from raw inputs. Along this line of reasoning, language-conditionel imitation learning has been introduced to enable agents to act upon human verbal instructions [3, 2, I, 5]. For example, BC-Z [1] proposes a large multimodal dataset which trained via imitation learning. LanguagePolicies (LP) [2] uses a similar approach but describes the outputs of the policy in terms of a Dynamic Motor Primitive (DMP) [12]. More recently, Say- Can [4] was introduced which focuses on planning of longer horizon tasks and incorporates prompt engineering. Our approach follows a similar rationale to the above papers in that it aims at bridg- ing the divide between language, vision and control through ingguae--coditiooee imitation. We do so by carefully incorporating modularity into our networks. Recent works on modularity for neural networks investigate, among other things, the question of whether "modules implementing specific functionality emerge" in neural network training [36]. Similarly, the work in [37] shows that pruning can produce surprisingly modular neural networks. Both of these works focus on natural emergence of modularity, i.e., in bottom-up fashion. Our work addresses modularity from a different vantage point. In particular, we ask the question, "Can we impose a specific set and order of modules within regular neural network?". To this end, we introduce supervised attention within a hierarchical learning regime which allows for such functional modules to be implemented in a top-down manner. Supervised attention was initially introduced in machine translation by [8] for aligning two languages, and was proven effective other natural language processing and visual question answering tasks [38, 39, 40, 41J. In this paper, we show that with the help of supervised attention, modular and reusable components can be formed in hierarchical manner.
3
Conduage-Conditione Policies for Robot Manipulation
The goal of our method is to learn language-conditioned policy Te(a|s, 7) parameterized by 0, where 8 is the natural language sentence containing a human instruction and I is an image captured by an RGB camera of the entire scene. Policy To generates actions a = [ax, y, 2, r. P, y. gl E R? where (z, y, 2) and (r, v) are the task-space coordinates and orientations of the end-effector sep- arately and g is the gripper control signal. We train our policy by following the typical imitation learning paradigm by providing data set of expert demonstrations D {do, ..., d. Each demon- stration d' is sequence of tuples ((av, $o, Io).... (ar, $T, Ir)). After training, the system extracts policy me which allows it to perform new configurations of the task.
...
3

---

An overview of our method is depicted in Fig. 2. A camera image, along with natural language in- struction and rohot proprioceptive data (joint angles) first processed through modality-seccific encoders to produce embeddings. The resulting embeddings are passed as input tokens into a transformer-style [42] neural network with multiple attention layers. This neural network imple- ments the overall policy Te and generates the robot controls. A crucial element of our approach is that the neural network is trained in modular fashion, while still maintaining the advantages of end-to-end learning. The transition from training the components to training the overall network oc- curs gradually. Due to its modularity, can also be transferred to a new robot in sample-efficient manner (e.g. from Kinova to UR5). Below, we describe the elements of our approach in more detail.
3.1 Modality-Speciffe Encoding
Given an image I IR xxx33 an image encoder produces an embedding ex = fV) Inspired by [43, 44), we encode the input image into a sequence while keeping the original spatial structure. To this end, we first use convolutional neural network to decrease its resolution but expand its channels, i.e., ez RR(//s)xW/a)) resulting low-resolution pixel tokens are flattened into a sequence of tokens eg R22dd where Z (H x W)/s². The language module takes in natural language instructions and generates a language embedding of a sequence of words E W with W being the vocabulary. For computing the language em- beddings, we refine the pretrained language model from CLIP [22]. During the training process, we provide template well-formed sentences. However, during testing, we admit any free-form verbal instruction including malformed sentences, typos, bad grammar.
where is a scaling factor and is the embedding size. The
fc(s) R where = (wo, wi,..., w..] is sentence composed
3.2 Supervised Attention
Modern neural network architectures employ attention mechanisms [42] enhance performance. In general, the input to attention module consists of three sequences of tokens: queries Q, keys K and values V. The computational process underlying attention layer can be expressed as:
QK Vdl
0 Attention Q, K, V) softmax
where dy is the dimensionality of the keys. The output matrix of the above operation O, holds in every row i the weighted sum of all value tokens. The weights correspond to the similarity between the i-th query token and the respec- tive value tokens. The overall attention mechanism typi- cally learned in an end-to-end fashion without an explicit supervision signal for the attention layers. To better con- trol the information flow during learning, we propose super- vised attention specific optimization target for attention layers. The key idea underlying supervised attention is that information about optimal token pairings may be available to the user. we know which key tokens are important for the queries to look at, we can treat their similarity score as a maximization goal. Fig. 3 shows example tokens gener- ated from visual patches and the corresponding supervision labels (dark gray). In the figure, we direct the network to focus on the end-effector and target object. More formally, if the i-th query should look at the j-th key. then we need to maximize the similarity between qi and k,, i.c., we optimize for arg maxç q.k².. This process is equiva- lent to maximizing the corresponding attention map element Mij, where M = softmax( QKT Va Note that, in contrast to matrix O (Eq. 1), the calculation of the attention map does not involve the value matrix V. Since My < 1, can simply minimize the distance between Mij and
Rog
Taken
Camera Image
K Keys
Attention Map
Figure 3: Supervised attention. Su- pervision pairs (dark gray) are pro- vided by an expert direct attention.

---

LANG
EEZD
Effector
splecement
"Pick the milk carton!"
Register
Register
Register
Vision
AT3
Vision
Joints
MLP
Language
00
90 Displacement
End Effector Pos
EE3D
CTRL
TAR
TAR
2D
Hierarchy of Sub-networks Modules
Hierarchical modularity: different sub-aspects of the tasks are implemented as modules (via supervised attention). LANG identifies the target object. EE2D locates the robot end-effector. The graph the bottom the figure shows the cascade of sub-tasks that are trained.
Figure 4:
1. We assume that N supervision pairs of indices for query and key tokens are provided, i.e., S {(io- Jo), di): (1N-1--N-)) Supervision pairs contain the indices defining which queries qi should attend which corresponding keys ky. Individual supervision pairs in this set can be addressed by S(p) = (1ps jp). Accordingly, we can define the following cost function for supervised attention:
qg.k"" Vdl
C(S) = Σ softmax
where sr correspond to the indices held by the n-th supervision pair (r, = S(n). The cost function in Eq. 2 casts supervised attention as a minimization problem, i.e., we calculate the mean squared error between the attention values and one. Other cost functions such the cross- entropy [45] loss can also be employed but have yielded lower performance in practice.
3.3 Hierarchical Modularity
Using supervised attention, we can train attention layers that focus on specific input variables of in- terest. In turn, we can use this mechanism to train the attention layers to form sub-modules that real- ize target mechanism. The underlying rationale is to train part of the attention realize components (or subtasks) of the overall policy. The sub-tasks, which assigned sub-modules embedded in the atten- tion layers, are further arranged in a cascaded fash- ion, with the output token embeddings of one mod- ule being the input to the next module. Again, using supervised attention, the user can influence which variables (and which data modalities) are processed in certain module.
Algorithm 1 Hierachical Modularity: train- ing algorithm returns network weights 6.
Input: (D, (Sx {C*) (
Output: Weights 0
for subtask i +- to K do
while not converged do
Eg
-t= E (S,) + V T--Trin (D, (S)..... St Ex)
end while
end for return
Inspired by Slot Attention [44], we also define register slots tokens (as shown in Fig.4), which are trainable tokens of embeddings that can propagate information throughout attention layers. For the purposes of robot grasping and manipulation, we identified the subtasks found Tab. 1 which are implemented as individual modules within our framework. The first module LANG takes as input the sentence embedding of the human instruction and identifies the word embedding of the target object. Accordingly, when supervising the training this module, we focus the attention the lan- guage input. This process is visualized in Fig. 4. The figure shows network of four attention layers, with the first layer implementing the LANG module. The output OE (object embedding) is turned
5

---

into an input token for the next layer. The TAR2D module takes an object embedding and generates an estimate of the object position in image space. Similarly, EE2D (Fig. 4) is trained to estimate the robot end-effector position based on information from the visual modality and the robot joint encoders. EE3D and TAR3D realize a similar functionality but focus on generating 3D positions in task space. A critical component is the DISP module, which estimates the displacement (or offiset of the robot end-effector from the target position. This information is crucial in order enable closed-loop control. Finally, the output of the DISP module is used (together with all other input variable) to generate the robot control values CTRL. In our specific scenario, we predict the next 10 goal positions at every timestep, which are subsequently used by the operational space controller.
The overall learning process inherent to hi- erarchical modularization can be structured as directed graph, see Fig. 4 (bottom). Every attention layer can implement one or multiple modules whose results are, in turn, fed to the next layer. The training process can be structured such that the ele- mentary components necessary for complex Docision-making are first derived within our cascaded modules, before being used for the final goal prediction. Algorithm summa- rizes the the training process for hierachical modularity. The algorithm assumes that set of attention losses Lk. tasks-specific loss functions v and corresponding supervision signals S₃ are provided by the expert for supervised training. The algorithm then starts training the first task using L1 and S₁. which yields updated network weights. After convergence on this subtask, it appends the next cost function to the loss function E and continues training the network. This process continues until all sub tasks are learned. Note that the final task, CTRL, is exactly the overall prediction target of our training process the control signal for the robot. is important to clarify two aspects underlying methodology. First, we note that all modules are part of single overarching neural network that implements the overall Iinuuag--conittione robot policy. Modularization is purely the result of training the network with different supervised attention targets and using a cost function that succes- sively incorporates more and more sub-tasks. Second, the functionality of the modules is maintained even beyond the training process. Hence, at runtime, the user can query each of the modules (e.g. LANG, TAR2D, EE3D, etc) for their individual outputs.
Table 1: Table of subtasks for modularity.
Abbreviation
Subtask Description
LANG
Get target object from language
TAR2D
Find object patch
TAR3D
Calculate object position
EE2D
Find robot end-effector patch
EE3D
Calculate end-effector position
DISP
Find distance object to end-effector
CTRL
Predict robot positions for control
4 Evaluation
We performed a number of experiments to show our model's 's ability to follow verbal instructions, its capability to transfer to new robots, and a real-world transfer from simulation. We also compare our approach to state-of-the-as methods for language-conditioned imitation learning, namely BC-Z [1] and LP [2]. We consider three types of manipulation actions to be performed by the robot: pick up referenced object, push the object, rotate the object and put it down. There are six custom objects with different shapes and colors, which are red cube, Coke can, a Pepsi can, milk carton, a green bottle, and a loaf of bread. All 3D models of these objects are from Robosuite [46]. Evaluation Metrics: Apart from recording success rates, we also evaluate the quality of each individual module within our language-condinggg conditioned policy, see Tab. 1. More specifically, we use as metrics: 1.) Success
Rate describes the percentage of successfully executed trials among the test set, 2.) Target Object Position Error (TAR3D) provides the Euclidean 3D distance from the predicted target object position to ground truth, 3.) End Effector Position Error (EE3D) is the Euclidean 3D distance from the predicted end effector position to ground truth, 4.) Displacement Error (DISP) calculates the 3D distance between the predicted 3D displacement vector and ground truth vector.
4.1 Language-Conditionel Imitation Learning: Efficiency and Transfer
In this first set of experiments, we evaluated the ability of our approach to efficiently and accurately leam iaaguage-conditions policies in simulation. Experiments were performed in MuJoCo [47]. We collected 2400 demonstrations, of which 1600 were used for training and 400 for validation

---

Table 2: Comparison with the state-of the baseline as well ablations, in Mujoco.
Success Rate (%)
Prediction Error (cm) EE3D
Model
Pick
Pash
Putdown
Overall
TAR3D
DISP
BC-Z [1]
81.6 6.2
7.9
1
4.5
Oars (UR5 sim)
91.3 = 5.3
97.2 + 2.0
55.6 + 8.6
82.4 12.4=40 4.9
2.24 + 0.48
0.51
0.09
2.42 + 0.52
Ours w/o Sup. Attn.
88.9 4.2
92.6
= 4.7
55.1
11.6
5.8
3.18 1.80
0.42
0.10
3.10 1.61
Ours wo Hier. Mod.
44.4
3.8
39.4
+ 7.1
22.7
7.9
36.4 3.3
22.96 + 0.99
0.59
0.16
23.16 1.05
Spria Bottle
Sprite DomSa
Sprits Bottle
6:50
Figure 6: Sequence real- real-time outputs the network modules: the object name (white) and visual attention (yellow region), the length of the displacement (white text), the object pos (blue), and end- effector pos (red). All values generated from single network that also produces robot controls.
and testing, respectively. Robot motions were generated using simple motion planning technique towards the target object. Throughout robot execution, we recorded the natural language instruction (e.g.. "Pick up the green bottle!"), a real-time stream of RGB images and robot proprioception (i.e., joints angles read from sensors). We trained LPP using different hyperparameters eight times. However, we could not generate a performance better than random. This is expected performance since the size of training is substantially smaller than LP's demand [2]. We trained and tested each method three times estimate the variability. Tab. 2, shows the result of this experiment.
As shown in Tab. 2, our approach achieves a high success rate (> 90%) on both pick and push operations and outperforms BC-Z on all three tasks. The average success rate is 82.4%, compared to 73.1%. We also individually evaluated the modules EE3D, TAR3D, and DISP of the proposed network for its prediction error. We observed that the accuracy for the end-effector pose prediction (=s 0.5em) was higher than that of the target object, which is likely due the availability of joint state data of the robot. The target object position can be approximated to about 2 - 3cm, which is likely because no depth information is included in our input data (RGB only).
We also tested the relevance of super- vised attention and hierarchical modularity individ- ually. Notice that in Tab. 2, removing hierarchical modularity causes drastic drop in performance to a low of only 36.4%. In prediction error, we can see that the target and displacement error jump to over
Ablations:
Success Rate (%)
90
70
Ours (K-U)
Our (K -
20em, explaining the low success rate. As noted
BC-Z -U)
RC-Z (K -F)
before, the availability of information from the in-
30
dividual sub-networks increases the transparency of the model and our ability to introspect its behavior. Next, we removed the supervision signal on specific tokens; instead, attention is trained in an end-to-end fashion as typically done. Here too, the performance drops but only by about 2.5%. In Tab. 2, we see that the target object can still be identified via hierarchi- cal modularity but generates higher prediction error and variance.
80
160
240
320
# of Demonstrations for Transfer
Figure 5: Results of transferring policies from Kinova (K) robot to UR$ (U) and Franka (F) robots. Experiments per- formed Mujoco simulator.
Transfer: The core hypothesis of our approach that modularly trained policy network will al- low for better transfer other robots with the minimal demand of retraining model parameters. We put this hypothesis to test, by using single source model which was trained on Kinova robot and
7

---

then transferred to Franka and UR5 robots in simulation. During transfer, we fine-tune our model on a new dataset recorded on the target robot (Franka or UR5). To evaluate data efficiency, we performed four tests with datasets of size 80, 160, 240 and 320 demonstrations respectively. Fig. depicts the results of this analysis. We notice that our method achieves a success rate around 80% with only 160 demonstrations. This is not a significant drop compared to the previous training re- sults achieved on 1600 demonstrations (see Tab. 2). When using 320 demonstrations the results are on par with training on 1600 demonstrations. Further, transfer to UR5 with 160 demonstrations even outperforms BC-Z trained on 1600 demonstrations. For better comparison, we also show the re- sults of fine-tuning a BC-Z model (trained on 1600 demonstrations on Kinova and then refined with 80...., (20) in Fig. 5. It is important to note that Kinova and UR5 are 6 degrees-of of-freedom (DoF) robots, whereas Franka has 7DoF. They also have substantially different kinematic configurations and visual appearance. Transfer can effectively be performed with limited data set overcome the variations in appearance and kinematic configurations.
Transfer to Real Robot: We also tested the transfer of a network trained in simulation to a real- world robot (Sim2Real). We collected 340 demonstrations on real-world URS robot, of which 260 were used for training and 80 for validation. After transfer using our approach, we tested the transferred models on the real world setup. We compared the results to a.) using the simulation model directly on the real robot and b.) continuous training, i.e., fine-tuning of the pre-trained model on the real-robot. Each model was then tested using 30 trials on the robot. Our approach achieves 80% success rate after transfer. The continuous training setup achieves a success rate of 56.67%; difference greater than 20% compared to our proposed method. The model solely pre-trained in simulation (predictably) fails to perform any successful actions resulting in 0.0% success. This effect can be explained by the substantial variation in visual appearance of objects and end-effectors in real-world images. We also observe the noise level change in the attention maps. Continuous training without supervising the attention results in a higher noise level in the attention map compared to adopting supervised attention.
Extension of new modules: We investigated the ability to add new modules to the existing hi- erarchy by adding an obstacle avoidance module. Generalization on different scenarios: We also tested the model's performance under varying environmental and task conditions, e.g., unseen colors, object scaling, object synonyms, image occlusions and unseen object types. Linguistic vari- ation: Finally, we also asked random testers to paraphrase 30 natural language sentences for testing. Please see the appendix for details on the experiments mentioned.
5 Conclusions & Limitations
Our method demonstrates data-efficient training and multi-robot transfer of language-conditioned policies for robot manipulation. To this end, we introduce a novel method, namely Hicrarchical Modularity, and adopt supervised attention to train collection of reusable sub-modules. We also show that the learned hierarchy of sub-modules be used to introspect and visualize the robot decision-making process.
Limitations: A major assumption made in our approach is that a human expert correctly identifies components and subtasks into which a task can be divided. This process requires organizing these subtasks into a hierarchical cascade. Early results indicate that an inadequate decomposition can hamper, rather than improve, learning. Further, our method can only work with instructions con- taining a single target object. Similarly, the approach does not incorporate memory and therefore cannot perform sequential actions. In a few cases we observed a failure to stop after finishing a manipulation the robot continues with random actions. Regarding language understanding, there still exist a large number of limitations to the approach, e.g., it cannot identify objects by spatial references ("next to"") or by function ("which contains"). Apart from them, the model does not gen- eralize well on completely new objects and unseen geometric features. This is largely due to the fact that we trained our vision network from scratch on small dataset, instead of using pre- traitrd larger vision backbone.
8

---

Acknowledgments
This research was partially funded by grants NSF CNS 1932068 and IIS 1749783.
References
[1] E. Jang, A. Irpan, M. Khansari, Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learn- ing. pages 991-1002. PMLR, 2022.
[2] Stepputtis, Camphell, M. Phielipp. Lee, C. Baral, and H. Amor. Language- conditioned imitation learning for robot manipulation tasks. arXiv preprint arXiv:2010.1203. 2020.
[3] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. Proceedings of Robotics: Science and Systems. doi, 10, 2021.
[4] M. Ahn, A. Brohan, N. Brown, Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[5] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation, arXiv preprint arXiv:2109. /2098, 2021.
[6] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[7] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 39 48, 2016.
[8] L. Liu, M. Utiyama, A. Finch, and E. Sumita. Neural machine translation with supervised at- tention. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 4093-3102, Osaka, Japan, Dec. 2016. The COLING 2016 Organizing Committee. URL https: http://ccaaanoooo.rrrgg./e/ 1291.
[9] R. Dillmann and H. Friedrich. Programming by demonstration: A machine learning approach to support skill acquision for robots. In International Conference on Artificial Intelligence and Symbolic Mathematical Computing, pages 87-108. Springer, 1996.
[10] S. Schaal. Is imitation learning the route to humanoid robots? Trends Cogn Sci, 3(6):233- 242, June 1999. ISSN 1364-6613, URL http::/tt::/ww.eee ncbi nlm. mih. wwtpb//go////e////geee fcgi? oagara/eeeccctaaeapappaaaaaaoare sudd:rrrrccciiitttduud-.1377777
[11] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):4694483, 2009.
[12] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. In Adaptive motion of animals and machines, pages 261-280 Springer, 2006.
[13] A. Coates, P. Abbeel, and A. Y. Ng. Apprenticeship learning for helicopter control. Commu- nications of the ACM, 52(7))97-105, 2009.
[14] G. Maeda, M. Ewerton, R. Lioutikov, H. B. Amor, Peters, and G. Neumann. Learning interaction for collaborative tasks with probabilistic movement primitives. In 2014 IEEE-RAS International Conference Humanoid Robots, pages 527-534. IEEE, 2014.

---

[15] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho. J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc., 2017. URL httpatt/pppoceeeeagggg
pdf.
[16] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep im- itation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635, 2018. 1oi: 1001100110//IRAA22111..... 844444449....... 8461249.
[17] F. Xie. A. Chowdhury, M. C. De Paolis Kaluza, L. Zhao, L. Wong, and R. Yu. Deep imitation learning for bimanual robotic manipulation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2327-2337. Curran Associates, Inc., 2020. URL https::/prroceeddngg newrips. pdf.
[18] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for vision-and and-language tasks. Advances in neural information processing systems, 32, 2019.
[19] Y.C.. Chen, L. Li, L. Yu, A. EI Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXX, page 104-120, Berlin, Heidelberg. 2020. Seringer-Verlage ISBN 978-3-030-58576-1 doi:10.1007/ 1007 /9700 3- 030-
178-3-030-58577-- URL
3-007888---------00 58577-877
[20] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In K. Inui, J. Jiang, V. Ng. and X. W. 0001, editors, Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNNPP-JCNLPP 2019, Hong Kong. China, November 3-7, 2019, pages 5099-5110. Association for Computational Linguistics, 2019. 178- 01750777-90-11 1oi.10.11665//111111-00- 1514. URL httpp:://///et orgg/0..11000 10tr//...866///// v1/D19-1514.
[21] A. Kamath, Singh, Y. LeCun, Synnaeve, Misra, and N. Carion. Mdetr modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 1780-1799 October 2021.
[22] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. arXiv preprint 1xXIv:2003.00020. 2021.
[23] I. Laina, C. Rupprecht, and N. Navah. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), October 2019.
[24] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings the (CVPR), June 2015.
the IEEE Conference on Computer Vision and Pattern Recognition
[25] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2057. PMLR, 2015.
[26] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
10

---

[27] Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901- 2910, 2017.
[28] S. Kottur, J. M. Moura, D. Parikh, D. Batra, and M. Rohrbach. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-169, 2018.
[29] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, M. Moura, D. Parikh, and D. Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 326-335, 2017.
[30] P. Anderson, A. Shrivastava, D. Parikh, D. Batra, and S. Lee. Chasing ghosts: Instruction following as bayesian state tracking. Advances in neural information processing systems, 32, 2019.
[31] Y.-.. Kuo, B. Katz, and A. Barbu. Deep compositional robotic planners that follow natural language commands. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 4906-4912. IEEE, 2020.
[32] R. Rahmatizadch, Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference on robotics and automation (ICRA), pages 3758-3766 IEEE, 2018.
[33] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
[34] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2018.
[35] P. Abolghasemi, A. Mazaheri, M. Shah, and L. Boloni. Pay attention!--robusiifiing deep visuomotor policy through task-focused visual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4254 4262, 2019.
[36] R. Csordás, S. van Steenkiste, and J. Schmidhuber. Are neural nets modular? inspecting func- tional modularity through differentiable weight masks. In International Conference Learn- ing Representations, 2021. URL https: / openreview .nt. met/fooumm wowoooommmidiaoo id=/://puuuuuuueee gMD.
[37] D. Filan, Hod, C. Wild, A. Critch, and S. Russell. Neural networks are surprisingly modular, 2020. URL http:// org/ atsrg/3.s/000..... 04881. cite arxiv:2003.0 0488 Comment: 23 pages, 13 figures.
[38] S. Liu, Y. Chen, K. Liu, and J. Zhao. Exploiting argument information to improve event detection via supervised attention mechanisms. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1789-1798, Vancouver, Canada, July 2017. Association for Computational Linguistics. 16.53/11//// P17-1164. URL https://aclanthhlloo. htwp:chhlllgyo.g/P/11o-ee. org/P17-1164.
[39] Y. Zhao, X. Jin, Y. Wang, and X. Cheng. Document embedding enhanced event detection with hierarchical and supervised attention. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 414-419 2018
[40] Y. Zou, T. Gui, Q Zhang, and X.-J Huang. A lexicon-based supervised attention model for neural sentiment analysis. In Proceedings of the 27th international conference computa- tional linguistics, pages 868-877, 2018.

---

[41] P. Yin, H. Fang, G. Neubig, A. Pauls, E. A. Platanios, Y. Su, S. Thomson, and J. Andreas. Compositional generalization for neural semantic parsing via span-level supervised attention. Association for Computational Linguistics (ACL), 2021.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[43] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213- 229. Springer, 2020.
[44] Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, Uszkoreit,
A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. Advances in Neu- 11538, 2020.
ral Information Processing Systems,
[45] P.T. Boer, P. Kroese, S. Mannor, and R. Y. Rubinstein. tutorial on the cross-entropy method. ANNALS OF OPERATIONS RESEARCH, 134, 2004.
[46] Y. Zhu, J. Wong, A. Mandlekar, and R.
robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 2020.
[47] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012.
12


Reviews:
[{"review_id": "sfL8cV9-ei", "reviewer": "Reviewer_fi3K", "summary": "This paper presents an approach for learning transferable language-conditioned pick & place policies by exploiting the inductive bias of attention to define chainable modules for performing tasks. Because there is a separation induced by the modules that are defined (e.g., identifying object to manipulate, estimating distance from end-effector to target object, mapping this distance to a control action), the proposed approach is able to show sample-efficient transfer to new robot morphologies, as well as across environments (sim2real).\n\nIn general, this paper proposes a neat, factored architecture for modular language-conditioned manipulation, and shows that adding such structure leads to better performance compared to other language-conditioned baselines, as well as meaningful transfer.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, I like the punchline of this paper — by formulating a policy architecture with components that are human-interpretable (e.g., attention), we can leverage extra supervision to aid in sample-efficient learning. Going further, we can embed different “modules” with different functionalities within this architecture (as functions of different types of attention), and use this to guide task solving for a single robot, as well as sample-efficient transfer when generalizing to new robot morphologies and environments.\n\n---\n\nWhile the approach is straightforward, my key concern, and the biggest weakness of this paper, is the generality of the proposed modules. Just like Neural Module Networks in the computer vision & NLP communities (the end2end Neural Module networks paper should probably be cited here), the benefit of the predefined modules only exists as long as the task specification is relatively clean, and matches the hierarchy exactly — in this case, we’re mostly looking at object relocation without meaningful occlusion or other challenges, so the policy holds up.\n\nHowever, where does the existing hierarchy break down? What if the type of objects changed (e.g., deformable, fragile, requiring different grasping policies)? What if there were meaningful occlusions or obstacles (requiring moving around existing objects in order to make the grasp we want)? While traditional BC approaches (e.g., in BC-Z) are able to scale to these new scenarios in the limit of new data, it’s not clear that this approach, with a fixed hierarchy, is able to scale similarly, which is really concerning! I’d love to see experiments looking at how the proposed hierarchy (e.g., this exact set of modules) performs in various settings, and whether there’s a clean way to add new elements to the hierarchy in an efficient, elegant way.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In general, while I really like the idea of supervised attention, I’m not sure that a single hierarchy is sufficient or general enough to tackle the messiness of real-world manipulation; I’d love to see new evaluations that show if/when the current hierarchy breaks down, and whether there is a clean way to fix such problems.\n\nAs a result, I advocate for a weak accept at this time.\n\nEDIT (Post-Rebuttal): Changed score to Strong Accept!\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Mostly, I’d love to see arguments addressing the point from above; furthermore, I’m curious as to why supervised attention, without the hierarchy is performing so much worse than BC-Z (Table 2). What is this approach missing that is present in BC-Z?"}}, {"review_id": "sCjEi2mdBGWA", "reviewer": "Reviewer_piKo", "summary": "In the paper authors propose a sample efficient approach for training language-conditioned manipulation policies that can be transferred across robots with different morphology, capability, appearance or dynamics. They introduce two complementary methods: supervised attention and hierarchical modularity. Supervised attention optimizes attention map by maximizing similarity between the queries and the keys. Hierarchical modularity decomposes the task of object manipulation to a set of reusable subtasks such as object detection, end effector detection, distance calculation. Authors demonstrate in both simulated and real world robot manipulation experiments that the method outperforms the current state-of-the-art methods and can transfer policies across different robots in a sample-efficient manner. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n* supervised attention mechanism \n* decomposition to hierarchical modules which also help to introspect the robots policy\n\nweaknesses:\n* authors used supervised data to train the manipulation policy, so in that part novelty is limited \n* collection of data used for supervised training of attention and manipulation should be described in more details. \n* no examples of introspecting the robots policy ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Describe in more details data collection process and provide examples of introspecting of the robots policy. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The paper is clearly written and easy to read. Some important details are missing such as: which policy was used to collect the data, examples of introspection of the learned policy, how key-query pairs were generated for the training of the supervised attention mechanism. "}}, {"review_id": "g5fOI7nDemV", "reviewer": "Reviewer_CVpY", "summary": "This paper introduces a transformer-based policy class from learning manipulation policies for physical robots from demonstration. It shows how providing explicit supervision to the attention layers in the transformers can force each module in the network to specialize in a specific type of reasoning. The paper presents a method to introduce these supervision tasks gradually over time, starting from lowest level tasks, to build up the modular reasoning capabilities of the model. The paper demonstrates how the resulting modular architectures are better adaptable to new manipulator types with few demonstrations, compared to policies trained with traditional end-to-end task losses only.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting approach to constructing a robot policy. It’s nice that it’s a unified approach, where every module has the same architecture, and the distinctions between modules arise only from the objectives with which the modules are trained. This simplifies the design of robot policies.\n- Convincing real-robot results on transferring policies between robots with different kinematic configurations. This is an important area of study, especially for deep learning based policies.\n\nWeaknesses:\n- The description of the method is perhaps too informal and sometimes vague about what are the inputs and outputs of the different modules, what types do these inputs and outputs take, and how is information being fed into these attention networks. Could also use more info on what are the labels for attention supervision and how these labels are obtained from demonstrations.\n- Only slightly outperforms the baseline BC-Z, but with a seemingly much higher annotation burden.\n- The “language-conditioned” aspect of the paper is somewhat weak. The main benefit of language over other task specification methods (e.g. joystick control, clicking to indicate navigation or manipulation goals) is ability to support free-form natural language instructions that include complex constraints, compositional task structures, temporal structures etc. A core challenge in language-conditioned policies is supporting flexible task specifications in a large enough task space to be interesting. Experiments should demonstrate at minimum generalization to unseen instructions combining seen primitives [1, 3, 6], but recently (and ideally) also unseen objects and primitives [2]. This work seems to uniquely focus on picking tasks, so language is used only to specify a target object. Presenting this work as doing referring expression grounding for goal object specification [4] would be more accurate. Even then, there are only 6 specific target objects, so the flexibility of referring expressions is limited compared to other works in referring expression grounding.\n- The two contributions of “supervised attention” and “hierarchical modularity” are arguably overstated as contributions of this paper. “Supervised attention” has been studied multiple times in NLP research, starting with [5]. This paper may at most take credit for testing this idea on real robots. “Hierarchical modularity” simply refers to phasing in additional losses over time starting from low-level towards high-level reasoning. It’s roughly equivalent to the pre-training and joint fine-tuning paradigm that’s been commonly used in deep learning for years, and is also related to training policies with auxiliary losses, a common approach to encourage modularity in an end-to-end system [7, 8]. This paper can rightfully take credit for studying this approach (including with appropriate ablations) with attention-based networks in a robot manipulation task. This in itself is a meaningful contribution, but phrasing “hierarchical modularity” as an innovation might be overstating it.\n\nReview references:\n[1] Arumugam, Dilip, et al. \"Grounding natural language instructions to semantic goal representations for abstraction and generalization.\" Autonomous Robots 43.2 (2019): 449-468.\n[2] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. \"Cliport: What and where pathways for robotic manipulation.\" Conference on Robot Learning. PMLR, 2022.\n[3] Williams, Edward C., et al. \"Learning to parse natural language to grounded reward functions with weak supervision.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n[4] Shridhar, Mohit, Dixant Mittal, and David Hsu. \"INGRESS: Interactive visual grounding of referring expressions.\" The International Journal of Robotics Research 39.2-3 (2020): 217-232.\n[5] Liu, Lemao, et al. \"Neural Machine Translation with Supervised Attention.\" Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. 2016.\n[6] Lynch, Corey, and Pierre Sermanet. \"Grounding language in play.\" (2020).\n[7] Hermann, Karl Moritz, et al. \"Grounded language learning in a simulated 3d world.\" arXiv preprint arXiv:1706.06551 (2017).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an interesting policy model, a new method (in robotics) to train the policy for a manipulation task from demonstrations, and convincing results on physical robot. A particularly good aspect is the ability to adapt the policy to different robot manipulators. It's great to see such multi-robot experiments that are often lacking.\n\nThat said, in it's current state the paper is a little vague on some important details, and slightly oversells certain aspects related to the innovations in the paper. I would like to see these writing and presentation issues are addressed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Suggestions:\n- Line 163: Not sure what “part of the attention” is. Consider rephrasing, perhaps more formally? \n- Line 186: What are “all other input variable”? Can these be listed out or defined?\n- Section 3.3 is perhaps a bit too informal and so hard to understand what’s really going on. E.g. What is the meaning of “target position”? Is the entire network simply inferring an end-effector goal in task space? If so, why does it need this many modules to do so?\n- Algorithm 1 uses symbols that are undefined in the context in which the algorithm appears. It does not add much information.\n- This work is highly related to a line of work in instruction-following that enforces modularity in end-to-end networks using auxiliary losses, and uses that modularity for sim2real transfer of a control policy on a drone navigation task [CITE], along with follow-up work on a household navigation task [CITE].\n\n\nQuestions:\n- Line 233: Doesn’t the network predict target end-effector poses? Why is motion planning done towards a “target object”?\n- How are supervision losses L_{k} obtained for each task from the demonstrations? How is supervised attention obtained in real-world experiments? Is it manually human-specified or automatically extracted from the demonstrations?\n- Could you better explain the “w/o Hierarchical Modularity” baseline? Does this baseline only train on the task loss and not on the intermediate attention losses? Or does it train on all the same losses as the full model, but jointly all at once without gradually introducing more complex tasks once simpler tasks have converged.\n- Is this method of transfer across robots better than e.g. predicting grasp points / end-effector keypoints, and running a motion planner to control the robots?\n- Line 292: What are “register slots”? Were these defined anywhere in the paper? What kind of noise was added? Was this noise of a relative or absolute magnitude with respect to the distribution of values in these “register slots”?\n\nTypos:\n- Line 200: Typo “the the”\n- Line 201: Typo “tasks-specific”\n- Line 308: “three times estimate” -> “three times to estimate”.\n"}}]

Author rebuttal:

Decision: Accept (poster)