Imitation Learning of Robot Policies using Language, Vision and Motion
Authors: Simon Stepputtis
Joseph Campbell
Mariano Phielipp
Chitta Baral
Heni Ben Amor
Venue:  (2020)
Keywords: robot learning
imitation learning
natural language processing

Abstract: In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of a task, which in turn can be used to synthesize specific motion controllers at run-time. This multimodal approach enables generalization to a wide variety of environmental conditions and allows an end-user to influence a robot policy through verbal communication. We empirically validate our approach with an extensive set of simulations and show that it achieves a high task success rate over a variety of conditions while remaining amenable to probabilistic interpretability.

Under review as a conference paper at ICLR 2020
IMITATION LEARNING OF ROBOT POLICIES USING LANGUAGE, VISION AND MOTION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of task, which is in turn used to synthesize specific motion con- trollers at run-time. This multimodal approach enables generalization to a wide variety of environmental conditions and allows end-user to direct robot pol- icy through verbal communication. We empirically validate our approach with an extensive set of simulations and show that it achieves high task success rate over a variety of conditions while remaining amenable to probabilistic interpretability.
INTRODUCTION
significant challenge when designing robots to operate the real world lies in the generation of control policies that can adapt to changing environments. Programming such policies is a la- bor and time-consuming process which requires substantial technical expertise. Imitation learn- ing (Schaal 1999), is an appealing methodology that aims at overcoming this challenge instead of complex programming. the user only provides a set of demonstrations of the intended behavior. consequently distilled into robot control policy by learning appropriate parameter settings of the controller. Popular approaches to imitation, such as Dynamic Motor Prim- itives (DMPs) ([jspeert al. 2013) Gaussian Mixture Regression (GMR) (Calinon) 2009) largely focus on motion as the sole input and output modality, i.e., joint angles, forces or positions. Critical semantic and visual information regarding the task, such as the appearance of the target object or the type of task performed, is not taken into account during training and reproduction. The result often a limited generalization capability which largely revolves around adaptation to changes in the object position. While imitation learning has been successfully applied to a wide range of tasks including table-tennis Mülling al. (2013), locomotion [halodhorn (ChaoooionChallohhon et al.] (2007), and human-robot interaction Amor et al. (2014) an important question is how to incorporate language and vision into a differentiable end-to-end system for complex robot control.
These demonstrations
In this paper, we present an imitation learning approach that combines language, vision, and motion in order to synthesize natural kunguage-conditonn control policies that have strong generalization capabilities while also capturing the semantics of the task. The main rationale of our approach is that a teacher typically provides substantially more information than just the kind of motion to perform. Imagine athletic trainer that is demonstrating a tennis swing while also verbally explaining the involved steps, the target position, or the speed. As a result of this rich collection of information, the student can develop complex associations between (a) the observed visual features, (b) the demon- strated arm movement, and (c) the provided verbal descriptions. We argue that such a multi-modal teaching approach enables robots to acquire complex policies that generalize a wide variety of environmental conditions. To this end, we propose a neural network architecture, including several sub-networks, that can be trained in an end-to-end fashion to capture the complex relationships be- tween language, vision, and motion observed in the demonstrations. After training, the network can be provided with a camera image of the current environment and natural language description of the intended task. The description typically corresponds to verbal commands given by the current user. In turn, the network produces control parameters for a lower-level control policy that can be on robot synthesize the corresponding motion. The hierarchical nature of our approach, i.e., a high-level policy generating the parameters of a lover-level policy, allows for generalization of the trained task to variety of spatial, visual and contextual changes. Further, the ability provide

---

Under review as a conference paper at ICLR 2020
commands and instructions to the policy enables easy human-robot interaction through language. At execution time, the user can influence the behavior of the robot by simply talking to it. Our main contributions can be summarized as follows:
We propose a Multimodal Policy Network (MPN) an approach that fundamentally com- bines language, vision, and motion control in to single differentiable neural network that can learn the cross-modal relationships found in the data.
We empirically show that our model is capable of generating task sees--secific robot controllers given demonstrations of a task containing natural language and visual descriptors
In order to outline our problem statement, we contrast our approach Imitation learning (Schaal 1999) which considers the problem of learning a policy π from a given set of demonstrations D (d",.., d Each demonstration spans time horizon and contains information about the robot states and actions, e.g., demonstrated sensor values and control inputs at each time step. Robot states at each time step within a demonstration are denoted by x In contrast to other imitation learning approaches, assume that we have access to the raw camera images of the robot L at teach time step, as well as access to a verbal description of the task in natural language. This description may provide critical information about the context, goals objects involved in the task and is denoted as s. Given this information, our overall objective is to learn a policy π which imitates the demonstrated behavior, while also capturing semantics and important visual features. After training, we can provide the policy π (s, I) with a different, new state of the robot and a new verbal description (instruction) as parameters. The policy will then generate the control signals needed to perform which take the new visual input and semantic context into account.
2 BACKGROUND
A fundamental challenge in imitation learning is the extraction of policies that do not only cover the trained scenarios, but also generalize to wide range of other situations. A large body of literature has addressed the problem of learning robot motor skills by imitation (Argall et al., 2009). The majority of these approaches focus on learning functional ([jspeert et al.) 2013) or probabilistic rep- resentations (Maeda et al.] (014) of motion trajectories. Once such a model is learned, an input state vector is used to adapt the original motion changes in position, orientation, or force. However, the state vector has to be carefully designed in order to ensure that all necessary information for adaptation is available. Neural approaches to imitation learning [Pomerccuu (1989) circumvent this problem by learning feature representations that are best suited for the task. Extracting feature in- formation from rich data sources such as natural language and visual data for motion control has an extensive history. The work presented in (Arumugam et al.) 2019, Burke et al., 2019 Hristov et al., 2019 Misra et al.] (2018) focuses on sequencing manipulation tasks or choosing when to switch skill based language and/or vision input from the environment. However, these approaches assume that underlying motion primitives are available that actuate the robot in the form of motion planner or goal-directed controller. Fine grained robot control has been learned from high-level task descrip- tions in recent work presented by Chang et al.) which utilizes robot trajectories from demonstrations by learning a parameterized neural policy from visual perception of the environment. While not using natural language to specify the target, this work outlines the importance of combining robot motions with other modalities. The work presented in Sung et al. (2015) combines natural language, point-cloud perceptions of the environment and trajectories into a joint embedding that locates tasks and trajectory representations in close proximity each other in the latent space. At inference time, a control trajectory is generated by locating the task in the latent space and selecting an appropriate control policy.
Modern variants of this line of research leverage recent progress in training convolutional neural net- works in order to train increasingly complex policies from raw (visual) sensor inputs. Building upon the same basic framework, the work in innFinn et al. (2017) investigates how meta-learning can be used to learn rapidly adaptable policies. More specifically, meta-learning aims at learning policy param- eters that can quickly be fine-tuned to new tasks. While very successful in dealing with visual and spatial information, these approaches do not incorporate any semantic or linguistic component into the learning process. Creating policies that can be conditioned on natural language is one potential pathway to overcome this limitation. Several works have investigated the idea of combining natural language and imitation learning: Nicolescu & Mataric (2003): Gemignani et al. (015); Cederborg
2

---

Under review as a conference paper at ICLR 2020
Network Architecture
Network Composition
Description
W glasi
"Go towards the pink bin"
Semantic Netwock
Policy
Translation
Inpot Image I
Language Encoder
Goal
Generator
Parameterr
Encoder
Generator
Geneeraee Policy 6.6
Demorsiration
Frtel
Conrrol Signal
Robot Control Signal
Figure 1: Description of the network architecture and compositons.
Oudeyer 2013); Mericli et al. (2014); Sugita Tani (2005). However, many these approaches
assume that either sufficiently large set of motion primitives is already available or that a taxonomy of the task is available, i.e., language and motion are not trained conjunction.
Our work is most closely related to the framework introduced in Tellex al. (2014), which also focuses on the symbol grounding problem. More specifically, the work in [Tellex et al. (2014) aims at mapping perceptual features in the external world to constituents in an expert-provided natural language instruction. Our work approaches the problem of generating dynamic robot policies by fundamentally combining language, vision, and motion control in to single differentiable neural network that can learn the cross-modal relationships found the data with minimal human feature engineering. Unlike previous work, our proposed model is capable of directly generating complex low-level control policies from language and vision that reassemble robot motions demonstrated during training.
MULTIMODAL POLICY GENERATION VIA IMITATION
We motivate our approach with a simple example: consider binning task in which robot has to drop an object into one of several differently shaped and colored bowls on a table. human expert can teach the task to the robot providing a kinesthetic demonstration, i.e., physically maneuvering the robot through the necessary motion trajectory. However, in this example, it is critical to place the object in the correct bowl rather than only reproducing the control trajectories from the demon- strations. To this end, the human demonstrator may provide a verbal command, e.g.. "Move towards the blue bowl" during teaching. The trajectory generation would then have be conditioned on the blue bowl's position which, however, has to be extracted from visual sensing. Our approach auto- matically detects and extracts these relationships between vision, language, and motion modalities during learning. The result is neural network representation that integrates all available information in order make best usage of contextual information for better generalization and disambiguation.
Figure [ (left) provides an overview of our method. Our goal is to train deep neural network that can take input a task description s and and image I and consequently generates robot controls. In the remainder of this paper, we will refer to our network as the MPN Rather than immediately producing control signals, the MPN will generate the parameters for a lower-level controller. This distinction allows us to build upon well-established control schemes in robotics and optimal control. In our specific case, we use the widely used Dynamic Motor Primitives (ljspeert et al. 2013) as lower-level controller for control signal generation.
Given an image and a task description as input, first so-called semantic network is utilized to combine the information from natural language with the visual perception of the robot in order to produce a joint task embedding. The joint embedding created by converting words into a sentence embedding, which is in turn concatenated as a fourth channel to the input image. Images are provided to the network as difference images between an empty environment and the current camera image, resulting in image that highlights the objects located in the environment.
3

---

Under review as a conference paper at ICLR 2020
This step is performed simple background substraction process improve learning speed. The joint task embedding serves as robot- iettot-ddpeeeenttt description of the desired task. The embedding is forwarded to a sub-network, called the Policy Translation network, which synthesizes the parameters needed to fully define a low-level control policy. The resulting parameter vector can be used to execute the DMP and actuate the robot. While the MPN is activated only once per task yield the DMP parameters, the synthesized low-level controller is continuously utilized at every time step during task execution. The following sections will introduce each part of the MPN in more detail. An in-depth overview of our architecture can be found in Figure
3.1 SEMANTIC NETWORK
In order to extract salient information from a natural language sentence, we tokenize the sentence into vector of words The vector s is modified to have length (;; sentences with fewer than ls words are zero-padded and sentences with greater than Lg words are truncated. Each word transformed into L lu-dimensional word representation via the pre-trained Glo model (Pen- nington et al. 2014) such that we produce a word representation matrix W E Rlxt = fw (s) We then extract the relevant n-grams relating the task at hand through the of CNN as in Yang et al. (2015). In this method, the filters of the CNN are used to extract individual n-grams, such that filter with dimension X Lw produces a gram of size n. In order to de- termine which of these n-grams is relevant, we concatenate all of the convolved feature maps resulting from all filters, me = [ma,1; Me.2. N....i.......l.ee----..
then apply max pooling such that m' maxogist, -et (Mc,1, m........ m.)). The final n-gram representation is built by concate- nating the feature maps s' m/ve E C. However, the relationship between the n- n-grams still unknown; in contrast to prior work we leverage this information by further passing the n-gram map s' through a two-layer fully-connected network: e, = ReLU(K ReLLR K2s' + b2) + bi) where K, and b, represent the kernel and bias for each of the two layers. The process of converting W into e, is denoted fz(W) ) in Figure [ We expand the input image I with a fourth channel, com- posed of the sentence embedding es- To this end, we stack the sentence embedding
match the size of one input channel of the image e', = -Ie'.... e']- The resulting image I,n is used as an input ) to generate the task embedding e, which is produced with three blocks of convolutional
for (In) layers, composed of two convolutions, followed by a residual convolution each. The use of residual convolutions as proposed in He et al. (2013) allows the network utilize possible accuracy gains from increased depth without increasing the complexity of the network significantly, while main- taining the property of being easily optimized. The goal of the image network fr() is to generate a joint task representation from language and environmental perception that can be further utilized to generate low-level policies.
3.2 POLICY TRANSLATION NETWORK
The objective of the Policy Translation network is to produce the control parameters for a lowelevel controller. Hence, can be seen as a function that maps task embeddings to control parameters. Since in case the controller is DMP, we will first formally introduce the basics of this con- trol framework. A DMP is fundamentally a damped spring dynamical system which produces a trajectory of joint configurations, y E Rdd for d, actuated robot DoFs.
(1) attracted to the point g E Rdd according to the phase x, with constant coefficients ay: 3y, and Oz and the temporal scaling factor r. The forcing function determines the shape of the trajectory produced by the dynamical system, which we define as linear combination of nonlinear Gaussian basis functions, yr:
T = ag (3, (g y) 4) + (x; θ),
T = a.g
6 Σ Gill (Z(Z) D Vx(x) r(g yy))
fz: +
(2)
in which O E Xb is set of b weight coefficients for d, Dol's and yo is the initial state. Most applications of DMPs for imitation learning (Schaal 1999) directly learn a static set of weights for the forcing function from the demonstration data. At runtime these weights O and a goal position can be used to synthesize robot control signals. However, this assumes that a goal position has been generated by some other means, e.g., vision, kinematics, etc. In our approach, both the weight
4

---

Under review as a conference paper at ICLR 2020
coefficients θ as well as the goal position g are generated by the Policy Translation network. Given the task embedding e, the policy translation network generates the seper-parameters @ E 1 and g I² for the low-level DMP. The generation of the hyper-parameters is defined as
g fr(e) = fa (ReLU (Wae + ba)), fH (ReLU (Wee + bG))
(3)
that generate g and respectively after having processed e in a single perceptron with weight Wa and bias bg. One interesting advantage of using DMPs is the fact that we can leverage a large body of research regarding their behavior and stability, while also allowing other extensions of DMPs (Amor et al. [014 Paraschos et al. 2013 Khansari-Zadeh & Billard 2011) to be incorporated to our framework.
where Sa() and ('(( are multilayer--ercepiin
3.3 TRAINING
The MPN, including all of its sub-networks, is trained in an end-to-end fashion and uses a single Adam (Kingma & Ba 2015) optimizer On for the entire network. Due to the used Rectified Linear Unit activation throughout the entire network, does not suffer from the vanishing gradient problem and allows the use of a single, combined loss function, defined as follows:
Le = >e * MSE(T,)) J) + M..((T.....1.)
(4)
where MSE() denotes the Mean Squared Error loss function. The goal of the low-level controller is to re-create the shape of each trajectory d during training as well as predict the target joint config- uration g. Both of these objectives are summed in the loss function and weighted by Ae to maintain an equal contribution of both objectives to the overall loss. To allow the model for better generaliza- tion capabilities, we add Dropout each stage of the network as well as small amount of random Gaussian noise on the input image and demonstrated trajectory.
3.4 DETECTION OF INVALID TASKS
In its current state, the proposed model is forced to act in every possible scenario and language combination presented to the MPN] Situations that are not possible, e.g. moving towards an object that is not present in the current environment, may lead to dangerous behaviour of the robot. In order address this problem, we extended the current model with an additional 3-layered MLP = (e) that predicts whether or not the requested task is possible. This function is based on the embedding e and performs extension requires three additions to the previously described network structure. First, we add an adversary sentence 8a to each demonstration d that requests a task that is not possible given the current environment, We also introduce an additional loss Lv(v) that calculates the classification capabilities of the entire semantic network fE( and fv since the ability to distinguish tasks needs to be propagated through the entire network up until this point. The optimizer utilizes sparse softmax cross entropy for exclusive single class classification. The third addition is an additional optimizer OF e that optimizes the embedding network utilizing the following loss:
cisstycclassification regarding the validity of the task. This
CE = Lo + XyLv
(5)
We combine the controller and embedding network losses with an additional weight Hlper--prameen The addition of Le to CF is necessary to maintain the ability of the semantic network to generate useful embeddings for the translation network. This is also reflected in the choice of X which leaves strong weight on Lo. As last step. the former optimizer is reduced to only optimize the translation network instead of the entire network, as described in the previous section.
4 EXPERIMENTS
In this section, we describe our experimental setup and conduct and extensive set of experiments to verify the capabilities of our proposed model. We evaluate our model in simulated environment a binning task in which the goal is drop a cube in one of randomly placed multiple bins. An image sequence of the task can be seen in Figure 2
Data Collection: For data collection, we automatically generate random binning scenarios in which present the robot with three to five different bowl of different color, shape and size. In total,
5

---

Under review as a conference paper at ICLR 2020
Trajectory
Environment
Difference
Possible Sentences:
"Move to beryl
"Advance to the green bowl"
"Proceed jade pot"
"Go the olive hasin"
Figure 2: Overview of the experiment setup: (left) 3D environment with all possible objects with five colors, two sizes and two shapes. (middle) Example trajectory for going towards the green bowl. (right) Collected image and voice data for the given trajectory.
utilize five colors (yellow, red, green, blue and pink), two sizes (small and large) and two shapes (round and squared). This procedure provides us with 20 different objects and the ability to test different levels of ambiguity regarding the required amount of features needed to uniquely identify an object. As an example, when all four red objects are in the scene, unique description of an object is only possible when all three features (color, size and shape) are used at the same time to describe the task. In order to generate larger variety of different possible sentences, conducted an IRB approved human subject study in which we presented multiple colored objects to participants and asked them to explain verbally how they interacting with these objects in lock-and-place task. This allowed us to extract multiple sentence templates for going towards objects, as well as multiple synonyms for actions, colors, containers and reference points. In our study, we did restrict participants in any way, such that each individual was able to choose description that is most natural them. In addition, we expanded the list of synonyms by gathering additional words from publicly available synonym databases. The combination of the data from the IRB study and synonym database allows us to generate a large variety of natural task explanations for arbitrarily generated scenarios. A detailed description of the template generation can be found in Appendix A The visual perception of the robot is provided as a top-down image from above to robot, see Figure 2 To use the images in our model, we scale them to a resolution of 96 x 96 x 3 where the last three channel refer to the RGB values of each pixel. After generating a task and scenario, kinesthetic demonstrations
generated with physics based simulation of UR5 robot arm, taking into account inertia, weights and other properties of the robot and environment, allowing us to collect realistic movements in the simulated environment. The simulator is running with 20Hz collect a reasonable amount of samples for each trajectory. In total, we collected over 20000 generated demonstrations as described above.
4.1 GENERALIZATION
Table 2: Accuracy of detecting valid actions in random environments based on ambiguity. Low: Separation based on single feature High: Only combinations of features separate the targets.
Table I: Comparison of successfully completing the binning task by using single feature over 500 attempts.
Required Feature
Binning Success
Correctly:
None (Single Object)
96.4%
Ambiguity Low 99.3%
Accepted
Rejected
Color
97.6%
96.6%
Size
96.0%
High
70%
66.6%
Shape
79.0%
In our binning scenario, the robot needs to stop its movement above the bowl outlined the experi- ment within a radius of the bowl's center such that the dropped object from the gripper is successfully placed inside the bowl. We utilized two different bowl sizes in these experiments, large and small, diameter respectively. The object that is to be delivered, cube, has an edge length of cm. All experiments were conducted by generating new random scenarios with new environments, images, and sentences corresponding the generated task.
with 17.5 em and 12.5

---

Under review as a conference paper at ICLR 2020
0.525
0.450
0.375
0.300
0.225
0.150
0.075
"advance to cardinal dish'
"move blue bowl"
"carry to the muroon object"
'trnssport in to the azure pat"
[m]]
0.000
"move the bowl"
"transerr to the cobalt dishh
(b) Predicted goal error in meters depending on
"go the ruby basin"
"advance the indigo basin'
the target position. The area highlighted in red shows the area from which training data were sampled.
(a) Generalization to different locations (blue) and generalization with different sentences (red).
Figure 3: Generalization capabilities (Figure Bap and positioning error (Figure 3b
In our first experiment, evaluated the success rate of the object delivery by using only a single necessary feature. The results of this set of experiments can be seen in Table [ Each test was conducted on an equal amount of small and large bowl with 250 attempts each, resulting in 500 attempts for each shown feature. Except from the first row, all features were tested in scenarios with 3 bowls, one being the target and two serving as distraction. Based on the reported success rate, the robot is capable of successfully achieving a task with at least 96% probability except from when the only distinguishing feature is the shape (round or square), in which the success rate drops to 79%. This drop in successful task completion in these cases is due to the chosen image resolution of the environment in which distinguishing small round from small squared bowl is a particularly challenging task that was chosen on purpose to test the limits of our approach.
The generated parameters of the low-level DMP controller the weights and goal position - must be sufficiently accurate in order to successfully deliver the object to the specified bin. A set of weights for the first four dimensions of a DMP controller can be seen in Figure 4b The figure shows the generated weights for the movement to two different objects, one of which is closer to the robot than the other as well as being on different sides of the robots. We quantify the accuracy of the parameter generation by computing the Fuclidean distance between the ground truth target location and the end effector position of the robot, based on the predicted joint configuration. For this, we generated 6000 positions on grid that were equally distributed inside the physically reachable work space of the URS robot. The comparison between the end effector position, calculated with forward kinematics, and the target position can be seen in figure 3b Within the area used to generate train- ing data, the robot predicts the correct position with well under 5cm error, which is precise enough for the tested binning task. Additionally, the model is able to accurately generalize to target posi- tions located outside of the training area, with the error increasing as the distance from the training area increases. The proposed addition of classifying if a requested task is possible was evaluated on random environments with low and high ambiguity regarding the number of separating features. In an environment with low ambiguity, a single feature enough to tell targets apart, where as in environments with high ambiguity, multiple features are needed for each object to tell them apart. The results of this test are shown in Table In addition to generalizing to different bowl locations. the model is also capable of generalizing changes in the verbal task description. This is important when interacting with different users that may describe the same task with different words. In Fig- ure Ba we show the spatial and verbal generalization capability of our model. Since color is a key component of our verbal task descriptions, we expect that the robot is able to generalize to different color shades, something that can be caused by variations in illumination. In this experiment, we changed the colors of our green objects to different shades of green. Additionally, we also change the color components towards other bowls by increasing the red and/or green component. We em- pirically evaluated the MPN; ability to incorporate these changes in the selection of a target object. An example of the changes we made can be seen in Figure 5 In that scenario the robot chooses the dark green object over object that added blue component. However, when tasked with going to the large green object, also moves towards the larger object with increase blue component. This experiment shows that our network can combine information from multiple modalities to disam-
7

---

Under review as a conference paper at ICLR 2020
0.0
-0.0
0U0
0.3
1100
-0.3
0.04
220
-04
0.0c
Prodeed to the candimal
4.5
-0.5
-0.7
-100
-0.4
-0.2
0.0
0.2
0.2
-0.04
0.0
0.44
-200
-300
(a) Stochastic Forward Passes: Certainty of the predicted goal position for an existing object different locations (red) and an object that not existing (referred to as green, but central red).
(b) Generated DMP weights for two different
controllers approaching objects located on oppo- site sides the robot
Figure 4: Effects of language and environment on generated controllers and target prediction.
"Move the large green bowl" Component
"Move to the green howl"
Traming Color
Darker Tone
Figure 5: Sensitivity to different shades of green. During training, colors were fixed as well as the illumination. Here we show the robot's 's ability to account for small color changes.
biguate situation, i.e., the greener object is chosen when size is defined, while the slightly bluer, larger object chosen when a size is part of the verbal description.
4.2 UNCERTAINTY
We leverage recent theoretical insights in order to generate uncertainties via probabilistic outputs from our trained MPN. In particular, it was shown in Gal & Ghahramani (2015) that neural net- work learning using the Dropout method is equivalent Process modeling the training data. In each of the forward passes, we randomly drop neurons from network as done in Dropout algorithm. However, in this case, the neurons are dropped at inference time and not at training time. The generated samples form a possibly complex distribution represented as a set of outputs of the neural network. By analyzing this set we can glean impor- tant information about the uncertainty in networks outputs. Figure 4a shows the application of stochastic forward passes on the predicted goal position in Cartesian space generated by using a forward kinematics the predicted goal configuration of the robot. As be in the picture, the network is certain about the position of five red objects on the table in five different tasks. The variance of all forward passes is below 5cm, which allows for successful binning task. However, when only providing red object in the environment and asking the robot to move to a green object, the uncertainty drastically increases. This can be seen in the green scatter plot. which shows posi- tions of the green bowl over 100 stochastic passes. Based on the distribution of the predicted goals, becomes apparent that the green object is not available in the current environment. These types of analyses allow for granular decisions regarding task execution success to be made.
Bayesian approximation of a Gaussian
4.3 DYNAMIC ENVIRONMENTS
desirable for robots to be able to cope with dynamically changing environments, particularly when human is in the loop. In this experiment, evaluate the robot's ability adapt its generated
8

---

Under review as a conference paper at ICLR 2020
Vabee
Y-Valee
(a) Dynamic Environment: Changing bowl (b) Following a learned trajectory: Demonstrated (Blue). DMP location while reaching target
control (Green), Proportional control (Red)
Figure 6: Evaluation of our approach for dynamically changing environments (a) and the ability adhere to demonstrated trajectory shapes during interaction (b)
Table 3: Average success rate for placing the object 250 randomly generated environments.
Dataset Size
Syn
No Syn 20000
Objects
Required Features
20000
10000
5000
1000
20000
Small
92.8%
93.6%
45.0%
54.8%
92.8%
100.0%
Large
100.0%
96.8%
61.6%
69.2%
91.2%
100.0%
3
Color, Small
96.4%
91.6%
33.2%
20.8%
22.0%
99.6%
3
Color, Large
98.8%
98.0%
50.8%
29.2%
26.0%
100.0%
3
Size, Small
92.8%
40.0%
6.8%
4.0%
77.2%
99.6%
3 Size, Large
99.2%
37.6%
23.6%
21.2%
15.6%
100.0%
3 Shape, Small
70.0%
8.0%
14.4%
6.8%
56.0%
82.4%
3 Shape, Large 88.0%
20.4%
24.4%
9.6%
64.0%
72.0%
policy to dynamically changing environment by asking the robot to drop object in constantly moving bowl. During data collection and training, the robot was only provided with examples from static environments, such that it was enough to generate DMP once at the beginning of each interaction. However, to adapt to a changing environment, a new DMP needs to generated for each time step. Figure oa shows such scenario, in which the red bowl is moving an are from left to right around the robot by moving 1.5 em each step. For this experiment, we utilize the same model for previous experiments without having trained it for dynamically changing environments. At each time step, the same sentence s is combined with the new environment image I, generating policy by providing the parameters for updated DMP. As be seen from the image sequence in Figure 6al (a), the robot is successfully able to adapt to the changed bowl position.
4.4 TRAJECTORY RECONSTRUCTION
In our work, we chose a DMP as low-level controller for the MPN model to give the robot the ability to not just learn to approach a predicted goal position, but to also reassemble the shape of demonstrated trajectories. This ability is essential in scenarios in which the trajectory shape encodes additional information, e.g. object avoidance or a certain way in which an object needs to be approached. Figure(ob(bb) shows the MPN ability to generate trajectories that similar to what was shown during training. The dashed blue line shows the position of the tool center point (in x, y, coordinates) of demonstrated trajectory from the test set, the green line shows the respectively generated trajectory, executed by the DMP controller and the red line demonstrates the executed trajectory when using proportional controller. The movement along the Z-axis of the trajectory clearly shows a different behaviour of the robot when using a proportional controller. On average, the difference between the tool center point position and the demonstrated trajectory is 1.6em and 19.1cm when using the DMP and proportional controller, respectively.
4.5 ABLATION STUDY
Dataset Size Even though able generate large amount of artificial demonstrations in simulation, the ability the MPN to train on fewer data is desirable. For this purpose we looked the performance of the MPN when being trained with less than 20000 demonstrations, Table

---

Under review as a conference paper at ICLR 2020
Table 4: Ablation over network structure. Average success rate for placing 250 objects using n- grams of size 2, 3, 4 and 5. The last demonstrates the influence of using residual layers in the image processing pipeline.
Network Changes
Objects
Required Features
Original
NG 2
NG
NG
NG 5
NRes
Small
92.8%
31.6%
30.0%
32.8%
15.2%
56.6%
Large
100.0%
52.4%
40.4%
56.8%
22.8%
74.4%
Color, Small
96.4%
32.4%
21.2%
29.2%
18.0%
59.2%
3
Color, Large
98.8%
56.8%
43.2%
59.2%
26.8%
74.4%
Size, Small
92.8%
31.2%
26.0%
4.4%
14.0%
56.8%
3
Size, Large
99.2%
53.2%
44.0%
19.2%
26.0%
71.6%
Shape, Small
70.0%
4.8%
8.4%
5.2%
14.4%
45.2%
3
Shape, Large 88.0%
22.4%
42.0%
10.8%
24.4%
68.4%
As when testing the spatial generalization capability of the network in section 4.1 conducted experiments with various combinations of features, separated by their success rate with regards to be working better with larger objects, which again, might
the object size. The trained MPN]seems be related to the chosen image size. However, the experiments showed that when identifying the location of a single object or a scenario in which the color is sufficient to distinguish targets, it does not significantly benefit from more than 10000 training data. In addition to the amount of training data, we also trained a model without augmenting the the training data with synonyms, which is shown in the last two columns. This model was tested data using synonyms (2nd last column) and data not using synonyms (last column). As expected the problem becomes easier when no synonyms are used. However, this model shows the ability of the Glove word embeddings to embed words with a similar meaning closer to each other, resulting in a partially usable model.
Network Structure In addition the size and variety of the dataset, the structure of the network is an important component of our approach. In this section, we compare different choices with regards to the network structure. Table analyzes the performance of different n-gram sizes as compared to the original model. As expected, using a single n-gram size performs significantly worse across all tested sizes as compared to our original model using n-gram sizes of I, 2, 3 and 5 concurrently. However, the results suggest that smaller n-gram sizes are better at capturing cases in which single feature is enough to uniquely describe an object where as large n- n-grams seem to loose the ability to focus on the important part of the sentence.
In addition to process language, our approach able to ground sentences with the current envi- ronment perception. ResNet is common model for tasks related computer vision (He et al., 2015) and achieved its performance by introducing residual layers in the CNN structure. In the might-most column of table 4 we analyzed the influence of the residual sections of network by replacing them with max-pooling layers to maintain a similar output structure. As can be seen form the results, the residual units have significant influence on the overall performance of the network.
CONCLUSION
In this work, we presented an imitation learning approach combining language, vision, and motion. neural network architecture called Multimodal Policy Network was introduced which able to learn the cross-modal relationships in the training data and achieve high generalization and disam- biguation performance as a result. Our experiments showed that the model is able to generalize towards different locations and sentences while maintaining a high success rate of delivering ject to a desired bowl. addition, we discussed two extensions of the method that allow us to obtain uncertainty information from the model by either learning a separate classifier utilizing stochastic network outputs to get a distribution over the belief.
ob-
Finally, hope further expand the verbal fidelity of model by adding the ability to utilize relational object descriptions to parameterize the task. Using the full range of natural language descriptions will give the ability ground additional constraints into robot control.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Heni Ben Amor, Gerhard Neumann, Sanket Kamthe, Oliver Kroemer, and Jan Peters. Interaction primitives for human-robot cooperation tasks. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 2831-2837, IEEE, 2014.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-455 2009.
Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Edward C Williams, Mina Rhee, Law- son, LS Wong, and Stefanie Tellex. Grounding natural language instructions to semantic goal rep- resentations for abstraction and generalization. Autonomous Robots, 43:449 468, 2019. doi: 10. 9792
8. WRRLhttss:///
1007 010r///s51111-4 ore.eoorgg/g 10 wtgp/0.100011.1111.....
Michael Burke, Svetlin Penkov, and Subramanian Ramamoorthy. From explanation to syn-
thesis: Compositional program induction for learning from demonstration. feb 2019. doi: httthtt:///ara
10. 20155607/RSSS2211XXv.00 00o..orr/11.11567777 /RSS. 2019 9. 015
org/ att.aabb/...111.. 10t.....1605tttt////a
Sylvain Calinon. Robot programming by demonstration. EPFL Press, 2009.
Thomas Cederborg and Pierre-Yvee Oudeyer. From language to motor gavagai: Unified imitation
learning of multiple linguistic and nonlinguistic sensorimotor skills. IEEE Trans. on Auton. Ment. Dev., 5(3) 239, September 2013. ISSN 1943-0604. doi: 10. 0011099AAMD2220322799777
URL http: /ttt:://dx.oo doi org. 1111/111MM 2013.
Rawichote Chalodhomn David B Grimes, Keith Grochow, and Rajesh PN Rao. Learning to walk through imitation. In IJCAI, volume 7, pp. 2084-2090, 2007.
Jonathan Chang, Nishanth Kumar, Sean Hastings, Aaron Gokaslan, Diego Romeres, Devesh K Jha, Daniel Nikovski, George Konidaris, and Stefanie Tellex. Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control. Technical report.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Sergey Levine, Vincent Vanhoucke, and Ken Gold- berg (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Pro- ceedings of Machine Learning Research, pp. 357-368. PMLR, 13-15 Nov 2017. URL http:
//ppoceedings mlr. woll..rees//// w077///fnnn11/ html
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. arX6::1506.02214. 142, 2015.
Guglielmo Gemignani, Emanuele Bastianelli, and Daniele Nardi. Teaching robots parametrized
executable plans through spoken interaction. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 11, pp. 851-859, Richland, SC, 2015. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 148-1- ci 3413-6. URL httt:ttp:///dd cm. www.or///ciitatiaioio
fm? wttpccmm.m..277.722772222 2773262
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/ abs/1512.00385, 2015.
org/bss 111.. 03385
Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, and Subramanian Ramamoorthy. Disentangled Relational Representations for Explaining and Learning from Demonstration. jul 2019.
13627
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2): 328-373, 2013.
S Mohammad Khansari-Zaan -Zaeh and Aude Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics, 27(5):943-957, 2011.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015 iclr. arXiv preprint arXiv: arXiv:1412.668 41260980, 2015.
11

---

Under review as a conference paper at ICLR 2020
Guilherme Maeda, Marco Ewerton, Rudolf Lioutikov, Heni Ben Amor, Jan Peters, and Gerhard Neumann. Learning interaction for collaborative tasks with probabilistic movement primitives. In Humanoid Robots (Humanoids), 2014 14th IEEE-RAS International Conference on, pp. 527- 534, IEFE, 2014.
Cetin Mericli, Steven D. Klee, Jack Paparian, and Manuela Veloso. An interactive approach for situ- ated task specification through verbal instructions. In Proceedings of the 2014 International Con- ference on Autonomous Agents and Mulli-agent Systems, AAMAS 14, pp. 1069-1076, Richland, SC, 2014. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978- 1-4503-2738-L. URL http: http://aaaaa...otttttaioooooooo.
Dipendra Mísra, John Langford, and Yoav Artzi. Mapping Instructions and Visual Observations to Actions with Reinforcement Learning, jan 2018. URL. httpps://arxii..org/aass///////
08795
Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3): 263-279, 2013.
Monica N. Nicolescu and Maja J. Mataric. Natural methods for robot task learning: Instructive demonstrations, generalization and practice. In Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS '03, pp. 241-248, New York, NY, USA, 2003. ACM. ISBN 1-58113-683-8. doi: 10.11457860555 860614 URL http:
//doi acm. www.oogg1110 11411 atwp....rrg/...111//663333333
860614)
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic move- ment primitives. In Advances in neural information processing systems, pp. 2616-2624, 2013.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word
representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014. URL http: www. wew..wwwb....... org/ https//attoooooyy---l-a
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural
information processing systems, pp. 305-313, 1989.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-242, 1999.
Yuuya Sugita and Jun Tani. Learning Semantic Combinatoriality from the Interaction between Linguistic and Behavioral Processes. Technical report, 2005.
Jaeyong Sung. Ian Lenz, and Ashutosh Saxena. Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories. sep 2015. TRRL/atpp//aaxx.orrrrt
abs 0ets/1509.000 07831
Stefanie Tellex, Pratiksha Thaker, Joshua Joseph, Nicholas Roy. Learning perceptually grounded word meanings from unaligned parallel data. Machine Learning, 94(2):151-67, Feb 2014. ISSN 1573-0565. doi: 10.10077510944-0135333--- URL htttps::/dii.zzg//// LDC7/510944-01355300
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander Smola. Stacked attention networks for image question answering. CoRR, abs/1511.02277 2015. URL http::///aaa
02274
A VOICE TEMPLATE GENERATION
To train our neural network, require large amount of training data D of sentences s, images I and respective sensor readings vt E T from the respective robot movement. While generating random scenarios and actuating the robot in simulator random scenarios is a more challenging task. Not only do sentences need to describe a target object uniquely given all other objects in the scene, but they also need to reflect grammatically correct and
easily be done, generating sentences for
12

---

Under review as a conference paper at ICLR 2020
Table Words gathered from the human-subject study witch their common synonyms.
Base Word
Used Synonyms
round
round
circular
square
square
rectangular
small
small
tiny
smallest
petite
meager
large
large
largest big
biggest giant
grand
red
red
ruby
cardinal
crimson maroon carmine
green
green aquamarine olive
jade chartreuse
beryl
blue
blue azure cobalt indigo turquoise
yellow
yellow amber bisque
blond gold
sand
pink
pink salmon coral rose blush
cube
cube object piece dice
die
place
place put down
deposit lay down
set
down
plant
release
goto
go to move advance progress earry to
transport to
transfer to
bowl
bowl basin dish pot
left
left
port
right
right starboard
Larget>>
<color>> <size>> <shape>>
<bowl>
<sourc>>
<cube>
realistic sentences. To achieve the generation of sentences for random scenarios, we conducted human-subject study in which we presented multiple differently colored objects to the participant with the task to assemble them together while describing their actions. To prevent any bias towards certain explanations,
only showed the participants the initial state well the final result of the assembly task without further explanations. Participants were given a few tries to assemble the objects correctly before starting the experiments. The goal of the study was to collect sentences from different participants that describe common tasks like reaching, pushing, grasping, placing and insertion for variety of differently shaped and colored objects. As result, we received two general sentence templates for the reaching task as well as set of synonyms for colors, shapes and objects. The two basic templates are as follows: (1) "<place> the <source>> in <target>>" and (2) "<ooto>> to <target>>" The placeholders as well as the respective words used to replace them are shown in Table 5
Sentence generation Each random scenario contains between three and five objects. One of these objects is chosen at random as the target for the reaching task. Before being able to generate a sentence, the unique properties of the object need to be extracted. Unique properties to their global position, color, size, shape or combinations of these features. Using color, size and shape describe object will always result in unique object description. However, results from the human-subject study showed that participants used the smallest set of features necessary to uniquely describe and object. Given the type and location of all objects in the scene, we identify the smallest set of features necessary to describe the object. In case multiple sets qualify as the smallest set of features, random selection is made. The selection of features as well as an object description is used as the <target>> placeholder while replacing features as needed. Taking the different feature sets into account, we are able to generate 180,000 different sentences of which we are using 20,000 for training. Depending the randomly generated environment, the same sentence may be used multiple times for different scenarios.
be related
13


Reviews:
[{"review_id": "HJecYd1RFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of using multiple modalities for learning from demonstration. Approaches that take in task or joint space data to learn a policy for replicating that task are numerous. Doing the same with multiple modalities involved, in particular vision, language and motion, has only been recently considered, so this is a timely paper. \n\nThe core contribution is pretty well summarised by the architecture in figure 1, which involves a combination of encodings of the words and sentences, images and parameters of a DMP in order to generate movement commands from a high level instruction. \n\nUnless I have missed something in the experimental setup, all of the considered task variations are movement commands of the form <Move> to <Object>. The network setup allows for synonyms of two kinds, so <Move> can be replaced by numerous verbal synonyms such as advance and go, and the object can be specified in terms of shapes, colors and so on, but otherwise this is the only specification of the task. This has been addressed in the recent literature using neural network architectures similar to the one being proposed here, e.g., see the following papers. These papers already solve the proposed problem and provide similar explanations. It would be helpful to see comparative discussion with respect to those methods and a clear statement of novelty with respect to such prior work:\n[R1] M. Burke, S. Penkov, S. Ramamoorthy, From explanation to synthesis: Compositional program induction for learning from demonstration, Robotics: Science and Systems (R:SS), 2019.\n[R2] Y. Hristov, D. Angelov, A.Lascarides, M. Burke, S. Ramamoorthy, Disentangled Relational Representations for Explaining and Learning from Demonstration, Conference on Robot Learning (CoRL), 2019. \n\nAn interesting feature in R2 that the authors do not explicitly address here is the issue of relational specifications in the language, e.g., in addition to saying \"move to the red bowl\", we may also wish to say \"place on top of red block\". In the way that MPN is currently set up to map from the language input directly to hyperparameters of the DMP, and considering the embedding structure, it is not clear if MPN is capable of handling such specifications. If so, the claim of generalisation on the language input should be stated more clearly.\n\nThe ablation study is setup somewhat differently than what I would have expected. The authors consider the effect of changing the training set size and if the language input includes synonyms or not. Those two aspects seem to produce the expected results. It would also be interesting to see an ablation study in the sense of replacing or removing aspects of the architecture to see its relative effect on the overall model performance. So, for instance, if one did not have a DMP with the hyperparameters being estimated by a network and instead had a more straightforward encoding of where to move to - does it make a difference and how much? Likewise, how much performance benefit, if any, is being derived from an uninterpreted image I being combined as described in the embedding as opposed to an alternative that detects an objects and combines that position differently. The paper would have been stronger if such architectural choices were better justified and also demonstrated in the experiments.\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BylouRApKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary \n\nThe paper describes a new end-to-end imitation learning method combining language, vision, and motion.\nA neural network architecture called Multimodal Policy Network is proposed. That can extract internal representations from language and vision to condition the generated motions. \nIt enables an end-user to influence a robot's policy through verbal communication.\nThe experiments demonstrate the generalization performance of the method. That can generate behaviors towards different goals depending on different sentences. \n\n*Decision and supporting arguments\n\nI think the paper is just below the borderline. The reason is as follows.\n\nThe concern is about evaluation. They demonstrated the method could work, and the robot can move to appropriate goals. However, there is no comparative methods in the experiment.\nRelated to this point, the problem was not identified in the Introduction.\nThe authors might assume that introducing language into behavioral cloning itself is qualitatively new work. However, such a study has a long history. \nFor example, please refer to Tani's pioneering works.\nSugita, Yuuya, and Jun Tani. \"Learning semantic combinatoriality from the interaction between linguistic and behavioral processes.\" Adaptive behavior 13.1 (2005): 33-52.\n\nThe author should specify a current challenge or problem in pre-existing studies about imitation learning with language input, clarify their claim, and give empirical support for the claim.   \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJg1cbsTtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work uses imitations learning (from synthetic data) to train a deep model which takes a natural language instruction, and a visual representation of a robot's environment, and outputs a trajectory for the robot to follow which executes this instruction.  The work focuses on a robotic pick-and-place task, where the instruction indicates which of the available bins an item should be placed in.  In addition to the trajectory model, a second model is trained which allows the agent to predict whether a given command is actually feasible (i.e. whether the target bin exists).  Empirical results show a reasonably high success rate in placing objects in the bin specified by the instruction, though there is still room for improvement in cases where the shape o a combination of features is important to the selection of the correct bin. \n\nRather than mapping directly from instructions and observations to control signals, the model trained in this work translates from an instruction, and an image of the agent's environment, to the parameters of a DMP controller.  The network therefore outputs the entire motion for the task in a single inference pass.  This approach would have advantages and disadvantages.  The DMP formulation ensures that the resulting trajectory is relatively smooth.  It also means that the network outputs a distinct goal configuration, which the DMP should reach (assuming the goal is feasible) regardless of the other motion parameters.  The use of a DMP output space, however, limits the model to generating relatively simple, goal-directed motions, and does not allow the agent to adapt to changes in the layout of the environment (which would only be observed in the static visual input).\n\nAs other work has considered visual instruction following (e.g. Misra et. al. \"Mapping Instructions and Visual Observations to Actions with Reinforcement Learning\") it would strengthen this work considerably to see a direct comparison between this method and existing approaches.  It is likely that the approach presented in this work is better suited to the specific problem of robot control, but it would be helpful to see if learning a low-level control policy directly can be successful in this context.\n\nThe work needs to expand on the discussion in the second paragraph of section 4, where human annotators were used to generate natural language instructions for different tasks.  The paper suggests that this data was not used directly to train the model, but was instead used to build a template for generating natural language instructions.  What this template looks like, and how it was constructed based on the human-generated data, remains unclear, and needs to be described in much more detail.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The present paper addresses the problem of imitation learning in multi-modal settings, combining vision, language and motion. The proposed approach learns an abstract task representation, and the goal is to use this as a basis for generalization. This paper was subject to considerable discussion, and the authors clarified several issues that reviewers raised during the rebuttal phase. Overall, the empirical study presented in the paper remains limited, for example in terms of ablations (which components of the proposed model have what effect on performance) and placement in the context of prior work. As a result, the depth of insights is not yet sufficient for publication.
Decision: Reject