Grounding Language to Autonomously-Acquired Skills via Goal Generation

We are interested in the autonomous acquisition of repertoires of skills . Languageconditioned reinforcement learning ( LC-RL ) approaches are great tools in this quest , as they allow to express abstract goals as sets of constraints on the states . However , most LC-RL agents are not autonomous and can not learn without external instructions and feedback . Besides , their direct language condition can not account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input . To resolve these issues , we propose a new conceptual approach to language-conditioned RL : the Language-Goal-Behavior architecture ( LGB ) . LGB decouples skill learning and language grounding via an intermediate semantic representation of the world . To showcase the properties of LGB , we present a specific implementation called DECSTR . DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects . In a first stage ( G→B ) , it freely explores its environment and targets selfgenerated semantic configurations . In a second stage ( L→G ) , it trains a languageconditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs . We showcase the additional properties of LGB w.r.t . both an end-to-end LC-RL approach and a similar approach leveraging non-semantic , continuous intermediate representations . Intermediate semantic representations help satisfy language commands in a diversity of ways , enable strategy switching after a failure and facilitate language grounding . 1 INTRODUCTION . Developmental psychology investigates the interactions between learning and developmental processes that support the slow but extraordinary transition from the behavior of infants to the sophisticated intelligence of human adults ( Piaget , 1977 ; Smith & Gasser , 2005 ) . Inspired by this line of thought , the central endeavour of developmental robotics consists in shaping a set of machine learning processes able to generate a similar growth of capabilities in robots ( Weng et al. , 2001 ; Lungarella et al. , 2003 ) . In this broad context , we are more specifically interested in designing learning agents able to : 1 ) explore open-ended environments and grow repertoires of skills in a self-supervised way and 2 ) learn from a tutor via language commands . The design of intrinsically motivated agents marked a major step towards these goals . The Intrinsically Motivated Goal Exploration Processes family ( IMGEPs ) , for example , describes embodied agents that interact with their environment at the sensorimotor level and are endowed with the ability to represent and set their own goals , rewarding themselves over completion ( Forestier et al. , 2017 ) . Recently , goal-conditioned reinforcement learning ( GC-RL ) appeared like a viable way to implement IMGEPs and target the open-ended and self-supervised acquisition of diverse skills . ∗Equal contribution . Goal-conditioned RL approaches train goal-conditioned policies to target multiple goals ( Kaelbling , 1993 ; Schaul et al. , 2015 ) . While most GC-RL approaches express goals as target features ( e.g . target block positions ( Andrychowicz et al. , 2017 ) , agent positions in a maze ( Schaul et al. , 2015 ) or target images ( Nair et al. , 2018 ) ) , recent approaches started to use language to express goals , as language can express sets of constraints on the state space ( e.g . open the red door ) in a more abstract and interpretable way ( Luketina et al. , 2019 ) . However , most GC-RL approaches – and language-based ones ( LC-RL ) in particular – are not intrinsically motivated and receive external instructions and rewards . The IMAGINE approach is one of the rare examples of intrinsically motivated LC-RL approaches ( Colas et al. , 2020 ) . In any case , the language condition suffers from three drawbacks . 1 ) It couples skill learning and language grounding . Thus , it can not account for goal-directed behaviors in pre-verbal infants ( Mandler , 1999 ) . 2 ) Direct conditioning limits the behavioral diversity associated to language input : a single instruction leads to a low diversity of behaviors only resulting from the stochasticity of the policy or the environment . 3 ) This lack of behavioral diversity prevents agents from switching strategy after a failure . To circumvent these three limitations , one can decouple skill learning and language grounding via an intermediate innate semantic representation . On one hand , agents can learn skills by targeting configurations from the semantic representation space . On the other hand , they can learn to generate valid semantic configurations matching the constraints expressed by language instructions . This generation can be the backbone of behavioral diversity : a given sentence might correspond to a whole set of matching configurations . This is what we propose in this work . Contributions . We propose a novel conceptual RL architecture , named LGB for Language-GoalBehavior and pictured in Figure 1 ( right ) . This LGB architecture enables an agent to decouple the intrinsically motivated acquisition of a repertoire of skills ( Goals → Behavior ) from language grounding ( Language→ Goals ) , via the use of semantic goal representation . To our knowledge , the LGB architecture is the only one to combine the following four features : • It is intrinsically motivated : it selects its own ( semantic ) goals and generates its own rewards , • It decouples skill learning from language grounding , accounting for infants learning , • It can exhibit a diversity of behaviors for any given instruction , • It can switch strategy in case of failures . Besides , we introduce an instance of LGB , named DECSTR for DEep sets and Curriculum with SemanTic goal Representations . Using DECSTR , we showcase the advantages of the conceptual decoupling idea . In the skill learning phase , the DECSTR agent evolves in a manipulation environment and leverages semantic representations based on predicates describing spatial relations between physical objects . These predicates are known to be used by infants from a very young age ( Mandler , 2012 ) . DECSTR autonomously learns to discover and master all reachable configurations in its semantic representation space . In the language grounding phase , we train a Conditional Variational Auto-Encoder ( C-VAE ) to generate semantic goals from language instructions . Finally , we can evaluate the agent in an instruction-following phase by composing the two first phases . The experimental section investigates three questions : how does DECSTR perform in the three phases ? How does it compare to end-to-end LC-RL approaches ? Do we need intermediate representations to be semantic ? Code and videos can be found at https : //sites.google.com/view/decstr/ . 2 RELATED WORK . Standard language-conditioned RL . Most approaches from the LC-RL literature define instruction following agents that receive external instructions and rewards ( Hermann et al. , 2017 ; Chan et al. , 2019 ; Bahdanau et al. , 2018 ; Cideron et al. , 2019 ; Jiang et al. , 2019 ; Fu et al. , 2019 ) , except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones ( Colas et al. , 2020 ) . In both cases , the language-condition prevents the decoupling of language acquisition and skill learning , true behavioral diversity and efficient strategy switching behaviors . Our approach is different , as we can decouple language acquisition from skill learning . The language-conditioned goal generation allows behavioral diversity and strategy switching behaviors . Goal-conditioned RL with target coordinates for block manipulation . Our proposed implementation of LGB , called DECSTR , evolves in a block manipulation domain . Stacking blocks is one of the earliest benchmarks in artificial intelligence ( e.g . Sussman ( 1973 ) ; Tate et al . ( 1975 ) ) and has led to many simulation and robotics studies ( Deisenroth et al. , 2011 ; Xu et al. , 2018 ; Colas et al. , 2019a ) . Recently , Lanier et al . ( 2019 ) and Li et al . ( 2019 ) demonstrated impressive results by stacking up to 4 and 6 blocks respectively . However , these approaches are not intrinsically motivated , involve hand-defined curriculum strategies and express goals as specific target block positions . In contrast , the DECSTR agent is intrinsically motivated , builds its own curriculum and uses semantic goal representations ( symbolic or language-based ) based on spatial relations between blocks . Decoupling language acquisition and skill learning . Several works investigate the use of semantic representations to associate meanings and skills ( Alomari et al. , 2017 ; Tellex et al. , 2011 ; Kulick et al. , 2013 ) . While the two first use semantic representations as an intermediate layer between language and skills , the third one does not use language . While DECSTR acquires skills autonomously , previous approaches all use skills that are either manually generated ( Alomari et al. , 2017 ) , handengineered ( Tellex et al. , 2011 ) or obtained via optimal control methods ( Kulick et al. , 2013 ) . Closer to us , Lynch & Sermanet ( 2020 ) also decouple skill learning from language acquisition in a goalconditioned imitation learning paradigm by mapping both language goals and images goals to a shared representation space . However , this approach is not intrinsically motivated as it relies on a dataset of human tele-operated strategies . The deterministic merging of representations also limits the emergence of behavioral diversity and efficient strategy-switching behaviors . 3 METHODS . This section presents our proposed Language-Goal-Behavior architecture ( LGB ) represented in Figure 1 ( Section 3.1 ) and a particular instance of the LGB architecture called DECSTR . We first present the environment it is set in [ 3.2 ] , then describe the implementations of the three modules composing any LGB architecture : 1 ) the semantic representation [ 3.3 ] ; 2 ) the intrinsically motivated goal-conditioned algorithm [ 3.4 ] and 3 ) the language-conditioned goal generator [ 3.5 ] . We finally present how the three phases described in Figure 1 are evaluated [ 3.6 ] . 3.1 THE LANGUAGE-GOAL-BEHAVIOR ARCHITECTURE . The LGB architecture is composed of three main modules . First , the semantic representation defines the behavioral and goal spaces of the agent . Second , the intrinsically motivated GC-RL algorithm is in charge of the skill learning phase . Third , the language-conditioned goal generator is in charge of the language grounding phase . Both phases can be combined in the instruction following phase . The three phases are respectively called G→B for Goal → Behavior , L→G for Language → Goal and L→G→B for Language → Goal → Behavior , see Figure 1 and Appendix A . Instances of the LGB architecture should demonstrate the four properties listed in the introduction : 1 ) be intrinsically motivated ; 2 ) decouple skill learning and language grounding ( by design ) ; 3 ) favor behavioral diversity ; 4 ) allow strategy switching . We argue that any LGB algorithm should fulfill the following constraints . For LGB to be intrinsically motivated ( 1 ) , the algorithm needs to integrate the generation and selection of semantic goals and to generate its own rewards . For LGB to demonstrate behavioral diversity and strategy switching ( 3 , 4 ) , the language-conditioned goal generator must efficiently model the distribution of semantic goals satisfying the constraints expressed by any language input . 3.2 ENVIRONMENT The DECSTR agent evolves in the Fetch Manipulate environment : a robotic manipulation domain based on MUJOCO ( Todorov et al. , 2012 ) and derived from the Fetch tasks ( Plappert et al. , 2018 ) , see Figure 2 . Actions are 4-dimensional : 3D gripper velocities and grasping velocity . Observations include the Cartesian and angular positions and velocities of the gripper and the three blocks . Inspired by the framework of Zone of Proximal Development that describes how parents organize the learning environment of their children ( Vygotsky , 1978 ) , we let a social partner facilitate DECSTR ’ s exploration by providing non-trivial initial configurations . After a first period of autonomous exploration , the social partner initializes the scene with stacks of 2 blocks 21 % of times , stacks of 3 blocks 9 % of times , and a block is initially put in the agent ’ s gripper 50 % of times . This help is not provided during offline evaluations .

Summary: This work proposed DECSTR, a procedure for encouraging intrinsic motivation via an intermediate semantic state-space representation. The authors propose an intermediate semantic state space that the intrinsically motivated agent learns to explore. For the environment provided (a 3-block system), the agent fully explores the symbolic state space, reaching all feasible symbolic states. In the second part of the work, the authors train a language model capable of proposing symbolic goals (in the form of symbolic states) from natural language input and shows that the previously-intrinsically-motivated agent can now be made to reach these goals, demonstrating that the symbolic-goal-conditioned policy is sufficient for instruction following in their 3-block domain.