A Self-Supervised Method for Mapping Human Instructions to Robot Policies

In this paper , we propose a modular approach which separates the instruction-toaction mapping procedure into two separate stages . The two stages are bridged via an intermediate representation called a goal . The first stage maps an input instruction to a goal , while the second stage maps the goal to an appropriate policy selected from a set of robot policies . The policy is selected with an aim to guide the robot to reach the goal as close as possible . We implement the above two stages as a framework consisting of two distinct modules : an instruction-goal mapping module and a goal-policy mapping module . Given a human instruction in the evaluation phase , the instruction-goal mapping module first translates the instruction to a robot-interpretable goal . Once a goal is derived by the instructiongoal mapping module , the goal-policy mapping module then follows up to search through the goal-policy pairs to look for policy to be mapped by the instruction . Our experimental results show that the proposed method is able to learn an effective instruction-to-action mapping procedure in an environment with a given instruction set more efficiently than the baselines . In addition to the impressive data-efficiency , the results also show that our method can be adapted to a new instruction set and a new robot action space much faster than the baselines . The evidence suggests that our modular approach does lead to better adaptability and efficiency . 1 INTRODUCTION . Understanding human instructions and interpreting them into actions have long been a crucial need and research focus for autonomous robots ( Winograd , 1972 ; Thomason et al. , 2017 ) . As autonomous robots are increasingly prevalent and capable of performing versatile skills nowadays ( Antunes et al. , 2016 ; Guadarrama et al. , 2013 ; Karamcheti et al. , 2017 ; Spranger et al. , 2014 ) , developing an efficient way that enables them to quickly satisfy such a need has also become essential and necessary . Traditionally , this is treated by researchers as a semantic-parsing and instruction-to-action mapping problem . A number of earlier approaches Chen & Mooney ( 2011 ) ; Matuszek et al . ( 2010 ; 2013 ) ; Mei et al . ( 2016 ) propose to learn a semantic parser that maps natural languages or human instructions to sequences of actions executable by a robot in a supervised fashion . The technique of using a probabilistic graphical model for inferring the corresponding robot action given a human instruction has also been investigated in Tellex et al . ( 2011 ) . Although these approaches have been well explored and mostly validated in their specific problem domains ( e.g. , navigation ) ( Mei et al. , 2016 ) , they typically require a large amount of human supervision ( e.g. , data annotation ) and/or linguistic knowledge ( e.g. , the syntax of instructions ) . In case that the annotated action sequences of a robot or the relevant linguistic background is not directly available , however , developing such an instruction-to-action mapping procedure becomes no longer straightforward . Researchers in recent years attempt to relax the above constraints by adopting two alternative approaches : reinforcement learning ( RL ) and imitation learning ( IL ) . Both approaches seek to learn a behavior policy π for an agent that executes the given instructions appropriately . The authors in Branavan et al . ( 2009 ) employ an RL learner to map text instructions in documents to computer actions . A reward shaping technique is presented in Misra et al . ( 2017 ) to train an RL agent to jointly reason about linguistic and visual inputs in a fully observable simulated block world . In Hermann et al . ( 2017 ) , an RL agent is trained to follow the provided human instructions by learning a joint representation of the perceived image and the text command . Similar to Hermann et al . ( 2017 ) , the authors in Chaplot et al . ( 2017 ) propose to incorporate a multimodal fusion unit , called gated- attention , for combining the representations of the instruction and the image . On the other hand , several IL-based techniques have also been leveraged in autonomous robots for executing a diverse range of instructions in a number of tasks ( Argall et al. , 2009 ) . Although these RL- and IL-based approaches enable a robot to learn effective π ’ s to deal with the instruction-to-action mapping problem with little human supervision , they usually train their π ’ s in an end-to-end manner . This prevents π from easy adaption to unfamiliar sets of instructions or new robots with different action spaces , unless dedicated training data are provided additionally . To deal with the issues mentioned above , we propose a modular approach which separates the instruction-to-action mapping procedure into two separate stages . The two stages are bridged via an intermediate representation called a goal . The first stage maps an input instruction c to a goal g , while the second stage maps g to an appropriate policy π selected from a set of robot policies Π . The policy π is selected with an aim to guide the robot to reach g as close as possible . Different from the previous end-to-end training methods , these two stages are trained separately as long as they both agree on the same goal representation . The modular approach allows either of the two stages to be replaced by another implementation . For instance , replacing the first stage by another mapping function ( c′ ⇒ g ) allows a different set of instructions c′ ( e.g. , different languages or different instruction syntaxes ) to be used to control the robot . On the other hand , replacing the second stage enables maneuvering a different robot with an action space distinct from the original one under the same instruction set . The modular nature of our approach enables a robot to adapt to a new instruction set quickly , and allows different robots to be maneuvered by the same instruction-to-goal mapping function , as long as the goal representation is identical . In this paper , we implement the above two stages as a framework consisting of two distinct modules : an instruction-goal mapping module and a goal-policy mapping module . The first module embraces the concept of metric learning , which aims to train a metric function called the distance function as a non-linear regressor to evaluate whether a given instruction c is close to a goal g or not . The second module utilizes the concept of goal exploration process ( GEP ) ( Forestier et al. , 2017 ) to discover in advance a diversified set of policies in Π in an unsupervised manner . It keeps expanding Π during the training phase to enhance the capability of the robot to reach a wide range of goals . The goals and their corresponding policies are indexed as goal-policy pairs , and are stored in a buffer contained in the second module . Given a human instruction c in the evaluation phase , the instruction-goal mapping module first translates c to a robot-interpretable goal g. The translation is performed by using the distance function output as a metric to search for the most similar g in the goal space G. The implementation of the goal-searching procedure is dependent on the nature of the goal . For instance , a straightforward enumeration algorithm is likely an appropriate option for a discrete goal , while a more complicated sampling-based search scheme is required for a goal in the continuous space . Once a goal g is derived by the instruction-goal mapping module , the goal-policy mapping module then follows up to search through the goal-policy pairs to look for a pair containing the most similar goal to g. The selected pair is then retrieved from the buffer , and used to extract a π to serve as the policy to be mapped by c. To demonstrate the strengths and efficiency of our modular approach , we compare the performance of our method with a number of baselines in two types of benchmark tasks : goal-oriented and trajectory-oriented tasks . The former type simply requires the robot to approach a specific goal , while the latter type demands the robot to accomplish the assigned task in the way specified by the instruction . In our experiments , these two types of tasks are both implemented in a simple 2-dimensional points of mass environment as well as a more challenging locomotion control task simulated by the MuJoCo physics engine ( Todorov et al. , 2012 ) . The baselines are adapted from the RL- and IL- based approaches presented in Dhariwal et al . ( 2017 ) ; Bain & Sommut ( 1999 ) . Our experimental results show that the proposed method is able to learn an effective instruction-toaction mapping procedure in an environment with a given instruction set more efficiently than the baselines . Compared to the RL baseline , our method is able to achieve a comparable or even superior performance by fewer interactions with the environment . Moreover , our approach consumes less human demonstration data than the IL baseline . In addition to the impressive data-efficiency , the results also show that our method can be adapted to a new instruction set and a new robot action space much faster than the baselines . The above evidence suggests that our modular approach does lead to better adaptability and efficiency . The main contributions of the paper are summarized as the following : • A modular framework for mapping human instructions to robot policies via an intermediate goal representation . • A unsupervised training method that requires neither action labels of the robot nor instruction- goal pairs . • A distance function for evaluating the similarity between a human instruction and a goal . The rest of this paper is organized as follows . Section 2 introduces background material . Section 3 walks through the proposed methodology , its implementation details , and the training procedure . Section 4 presents the experimental results and the ablation study . Section 5 concludes this paper . 2 BACKGROUND . In this section , we introduce background material . We first provide an overview of multi-goal RL . We then discuss the concepts and the relevant research works on metric learning . 2.1 GOAL EXPLORATION PROCESS . Goal exploration process ( GEP ) is an unsupervised policy search algorithm proposed in Forestier et al . ( 2017 ) that aims to discover a populations of policies π ’ s corresponding to the goals g ’ s . In the framework of GEP , the goal g is often a behavioral feature characterizing the trajectory produced by performing the corresponding policy π in the environment . GEP is composed of two stages : a bootstrap stage and a random exploration . The former first samples sets of policies from Π and observes the goals g ’ s resulted from the policies π ’ s , while the latter samples a goal g from G which the algorithm looks for policy π that able to reach the specific goal . In both stages , whenever the new goal is observed , the corresponding goal-policy pair is stored . To ensure the goal g is novel to the previous ones , GEP adds random noises to perturb the paired policies . GEP explores G in an unsupervised fashion . 2.2 METRIC LEARNING . The primary goal of metric learning is to learn a metric function for measuring the relationship between different sets of data points . Theses sets may come from the same data domain or different ones . Metric learning has recently received great attention from researchers due to its applicability to a wide variety of research domains , such as RL ( Taylor et al. , 2011 ) , partition problems ( Lajugie et al. , 2014 ) , information retrieval ( Lebanon , 2006 ; Lee et al. , 2008 ; McFee & Lanckriet , 2010 ) , etc . The metric functions in the previous works are usually implemented in the form of distance functions . However , the concept of metric learning is generalizable to logical relations among data points . As a metric function is commonly trained with limited amount of training data , it is important for it to learn the general relations between its input data sets . For instance , the distance function for measuring the similarity of faces presented in Chopra et al . ( 2005 ) predicts a small value when the inputs to it come from the same person , and a large value when the inputs are from different people . Metric learning has also been used in RL to filter out features irrelevant for decision making ( Taylor et al. , 2011 ) . In this paper , we apply the concepts of metric learning for mapping human instructions to robot skills , which have yet been explored in the previous literature .

Summary: The paper proposes a modular approach to the problem of mapping instructions to robot actions. The first of two modules is responsible for learning a goal embedding of a given instruction using a learned distance function. The second module is responsible for mapping goals from this embedding space to control policies. Such a modular approach has the advantage that the instruction-to-goal and goal-to-policy mappings can be trained separately and, in principle, allow for swapping in different modules. The paper evaluates the method in various simulated domains and compares against RL and IL baselines.