Imitation Learning of Robot Policies using Language, Vision and Motion

1 INTRODUCTION . A significant challenge when designing robots to operate in the real world lies in the generation of control policies that can adapt to changing environments . Programming such policies is a labor and time-consuming process which requires substantial technical expertise . Imitation learning ( Schaal , 1999 ) , is an appealing methodology that aims at overcoming this challenge – instead of complex programming , the user only provides a set of demonstrations of the intended behavior . These demonstrations are consequently distilled into a robot control policy by learning appropriate parameter settings of the controller . Popular approaches to imitation , such as Dynamic Motor Primitives ( DMPs ) ( Ijspeert et al. , 2013 ) or Gaussian Mixture Regression ( GMR ) ( Calinon , 2009 ) largely focus on motion as the sole input and output modality , i.e. , joint angles , forces or positions . Critical semantic and visual information regarding the task , such as the appearance of the target object or the type of task performed , is not taken into account during training and reproduction . The result is often a limited generalization capability which largely revolves around adaptation to changes in the object position . While imitation learning has been successfully applied to a wide range of tasks including table-tennis Mülling et al . ( 2013 ) , locomotion Chalodhorn et al . ( 2007 ) , and human-robot interaction Amor et al . ( 2014 ) an important question is how to incorporate language and vision into a differentiable end-to-end system for complex robot control . In this paper , we present an imitation learning approach that combines language , vision , and motion in order to synthesize natural language-conditioned control policies that have strong generalization capabilities while also capturing the semantics of the task . The main rationale of our approach is that a teacher typically provides substantially more information than just the kind of motion to perform . Imagine an athletic trainer that is demonstrating a tennis swing while also verbally explaining the involved steps , the target position , or the speed . As a result of this rich collection of information , the student can develop complex associations between ( a ) the observed visual features , ( b ) the demonstrated arm movement , and ( c ) the provided verbal descriptions . We argue that such a multi-modal teaching approach enables robots to acquire complex policies that generalize to a wide variety of environmental conditions . To this end , we propose a neural network architecture , including several sub-networks , that can be trained in an end-to-end fashion to capture the complex relationships between language , vision , and motion observed in the demonstrations . After training , the network can be provided with a camera image of the current environment and a natural language description of the intended task . The description typically corresponds to verbal commands given by the current user . In turn , the network produces control parameters for a lower-level control policy that can be run on a robot to synthesize the corresponding motion . The hierarchical nature of our approach , i.e. , a high-level policy generating the parameters of a lower-level policy , allows for generalization of the trained task to a variety of spatial , visual and contextual changes . Further , the ability to provide commands and instructions to the policy enables easy human-robot interaction through language . At execution time , the user can influence the behavior of the robot by simply talking to it . Our main contributions can be summarized as follows : • We propose a Multimodal Policy Network ( MPN ) , an approach that fundamentally combines language , vision , and motion control in to a single differentiable neural network that can learn the cross-modal relationships found in the data . • We empirically show that our model is capable of generating task-specific robot controllers given demonstrations of a task containing natural language and visual descriptors In order to outline our problem statement , we contrast our approach to Imitation learning ( Schaal , 1999 ) which considers the problem of learning a policy π from a given set of demonstrations D = { d0 , .. , dm } . Each demonstration spans a time horizon T and contains information about the robot states and actions , e.g. , demonstrated sensor values and control inputs at each time step . Robot states at each time step within a demonstration are denoted by xt . In contrast to other imitation learning approaches , we assume that we have access to the raw camera images of the robot It at teach time step , as well as access to a verbal description of the task in natural language . This description may provide critical information about the context , goals or objects involved in the task and is denoted as s. Given this information , our overall objective is to learn a policy π which imitates the demonstrated behavior , while also capturing semantics and important visual features . After training , we can provide the policy π ( s , I ) with a different , new state of the robot and a new verbal description ( instruction ) as parameters . The policy will then generate the control signals needed to perform which take the new visual input and semantic context into account . 2 BACKGROUND . A fundamental challenge in imitation learning is the extraction of policies that do not only cover the trained scenarios , but also generalize to a wide range of other situations . A large body of literature has addressed the problem of learning robot motor skills by imitation ( Argall et al. , 2009 ) . The majority of these approaches focus on learning functional ( Ijspeert et al. , 2013 ) or probabilistic representations ( Maeda et al. , 2014 ) of motion trajectories . Once such a model is learned , an input state vector is used to adapt the original motion to changes in position , orientation , or force . However , the state vector has to be carefully designed in order to ensure that all necessary information for adaptation is available . Neural approaches to imitation learning Pomerleau ( 1989 ) circumvent this problem by learning feature representations that are best suited for the task . Extracting feature information from rich data sources such as natural language and visual data for motion control has an extensive history . The work presented in ( Arumugam et al. , 2019 ; Burke et al. , 2019 ; Hristov et al. , 2019 ; Misra et al. , 2018 ) focuses on sequencing manipulation tasks or choosing when to switch skill based on language and/or vision input from the environment . However , these approaches assume that underlying motion primitives are available that actuate the robot in the form of a motion planner or goal-directed controller . Fine grained robot control has been learned from high-level task descriptions in recent work presented by Chang et al . which utilizes robot trajectories from demonstrations by learning a parameterized neural policy from visual perception of the environment . While not using natural language to specify the target , this work outlines the importance of combining robot motions with other modalities . The work presented in Sung et al . ( 2015 ) combines natural language , point-cloud perceptions of the environment and trajectories into a joint embedding that locates tasks and trajectory representations in close proximity to each other in the latent space . At inference time , a control trajectory is generated by locating the task in the latent space and selecting an appropriate control policy . Modern variants of this line of research leverage recent progress in training convolutional neural networks in order to train increasingly complex policies from raw ( visual ) sensor inputs . Building upon the same basic framework , the work in Finn et al . ( 2017 ) investigates how meta-learning can be used to learn rapidly adaptable policies . More specifically , meta-learning aims at learning policy parameters that can quickly be fine-tuned to new tasks . While very successful in dealing with visual and spatial information , these approaches do not incorporate any semantic or linguistic component into the learning process . Creating policies that can be conditioned on natural language is one potential pathway to overcome this limitation . Several works have investigated the idea of combining natural language and imitation learning : Nicolescu & Mataric ( 2003 ) ; Gemignani et al . ( 2015 ) ; Cederborg & Oudeyer ( 2013 ) ; Mericli et al . ( 2014 ) ; Sugita & Tani ( 2005 ) . However , many of these approaches assume that either a sufficiently large set of motion primitives is already available or that a taxonomy of the task is available , i.e. , language and motion are not trained in conjunction . Our work is most closely related to the framework introduced in Tellex et al . ( 2014 ) , which also focuses on the symbol grounding problem . More specifically , the work in Tellex et al . ( 2014 ) aims at mapping perceptual features in the external world to constituents in an expert-provided natural language instruction . Our work approaches the problem of generating dynamic robot policies by fundamentally combining language , vision , and motion control in to a single differentiable neural network that can learn the cross-modal relationships found in the data with minimal human feature engineering . Unlike previous work , our proposed model is capable of directly generating complex low-level control policies from language and vision that reassemble robot motions demonstrated during training . 3 MULTIMODAL POLICY GENERATION VIA IMITATION . We motivate our approach with a simple example : consider a binning task in which a robot has to drop an object into one of several differently shaped and colored bowls on a table . A human expert can teach the task to the robot providing a kinesthetic demonstration , i.e. , physically maneuvering the robot through the necessary motion trajectory . However , in this example , it is critical to place the object in the correct bowl rather than only reproducing the control trajectories from the demonstrations . To this end , the human demonstrator may provide a verbal command , e.g. , “ Move towards the blue bowl ” during teaching . The trajectory generation would then have to be conditioned on the blue bowl ’ s position which , however , has to be extracted from visual sensing . Our approach automatically detects and extracts these relationships between vision , language , and motion modalities during learning . The result is a neural network representation that integrates all available information in order to make best usage of contextual information for better generalization and disambiguation . Figure 1 ( left ) provides an overview of our method . Our goal is to train a deep neural network that can take as input a task description s and and image I and consequently generates robot controls . In the remainder of this paper , we will refer to our network as the MPN . Rather than immediately producing control signals , the MPN will generate the parameters for a lower-level controller . This distinction allows us to build upon well-established control schemes in robotics and optimal control . In our specific case , we use the widely used Dynamic Motor Primitives ( Ijspeert et al. , 2013 ) as a lower-level controller for control signal generation . Given an image and a task description as input , first a so-called semantic network is utilized to combine the information from natural language with the visual perception of the robot in order to produce a joint task embedding . The joint embedding is created by converting words into a sentence embedding , which is in turn concatenated as a fourth channel to the input image . Images are provided to the network as difference images between an empty environment and the current raw camera image , resulting in an image that highlights the objects located in the environment . This step is performed as a simple background substraction process to improve learning speed . The joint task embedding serves as a robot-independent description of the desired task . The embedding is forwarded to a sub-network , called the Policy Translation network , which synthesizes the parameters needed to fully define a low-level control policy . The resulting parameter vector can be used to execute the DMP and actuate the robot . While the MPN is activated only once per task to yield the DMP parameters , the synthesized low-level controller is continuously utilized at every time step during task execution . The following sections will introduce each part of the MPN in more detail . An in-depth overview of our architecture can be found in Figure 1 .

Summary: This work uses imitations learning (from synthetic data) to train a deep model which takes a natural language instruction, and a visual representation of a robot's environment, and outputs a trajectory for the robot to follow which executes this instruction.  The work focuses on a robotic pick-and-place task, where the instruction indicates which of the available bins an item should be placed in.  In addition to the trajectory model, a second model is trained which allows the agent to predict whether a given command is actually feasible (i.e. whether the target bin exists).  Empirical results show a reasonably high success rate in placing objects in the bin specified by the instruction, though there is still room for improvement in cases where the shape o a combination of features is important to the selection of the correct bin.