Grounding Language to Autonomously-Acquired Skills via Goal Generation

We are interested in the autonomous acquisition of repertoires of skills . Languageconditioned reinforcement learning ( LC-RL ) approaches are great tools in this quest , as they allow to express abstract goals as sets of constraints on the states . However , most LC-RL agents are not autonomous and can not learn without external instructions and feedback . Besides , their direct language condition can not account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input . To resolve these issues , we propose a new conceptual approach to language-conditioned RL : the Language-Goal-Behavior architecture ( LGB ) . LGB decouples skill learning and language grounding via an intermediate semantic representation of the world . To showcase the properties of LGB , we present a specific implementation called DECSTR . DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects . In a first stage ( G→B ) , it freely explores its environment and targets selfgenerated semantic configurations . In a second stage ( L→G ) , it trains a languageconditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs . We showcase the additional properties of LGB w.r.t . both an end-to-end LC-RL approach and a similar approach leveraging non-semantic , continuous intermediate representations . Intermediate semantic representations help satisfy language commands in a diversity of ways , enable strategy switching after a failure and facilitate language grounding . 1 INTRODUCTION . Developmental psychology investigates the interactions between learning and developmental processes that support the slow but extraordinary transition from the behavior of infants to the sophisticated intelligence of human adults ( Piaget , 1977 ; Smith & Gasser , 2005 ) . Inspired by this line of thought , the central endeavour of developmental robotics consists in shaping a set of machine learning processes able to generate a similar growth of capabilities in robots ( Weng et al. , 2001 ; Lungarella et al. , 2003 ) . In this broad context , we are more specifically interested in designing learning agents able to : 1 ) explore open-ended environments and grow repertoires of skills in a self-supervised way and 2 ) learn from a tutor via language commands . The design of intrinsically motivated agents marked a major step towards these goals . The Intrinsically Motivated Goal Exploration Processes family ( IMGEPs ) , for example , describes embodied agents that interact with their environment at the sensorimotor level and are endowed with the ability to represent and set their own goals , rewarding themselves over completion ( Forestier et al. , 2017 ) . Recently , goal-conditioned reinforcement learning ( GC-RL ) appeared like a viable way to implement IMGEPs and target the open-ended and self-supervised acquisition of diverse skills . ∗Equal contribution . Goal-conditioned RL approaches train goal-conditioned policies to target multiple goals ( Kaelbling , 1993 ; Schaul et al. , 2015 ) . While most GC-RL approaches express goals as target features ( e.g . target block positions ( Andrychowicz et al. , 2017 ) , agent positions in a maze ( Schaul et al. , 2015 ) or target images ( Nair et al. , 2018 ) ) , recent approaches started to use language to express goals , as language can express sets of constraints on the state space ( e.g . open the red door ) in a more abstract and interpretable way ( Luketina et al. , 2019 ) . However , most GC-RL approaches – and language-based ones ( LC-RL ) in particular – are not intrinsically motivated and receive external instructions and rewards . The IMAGINE approach is one of the rare examples of intrinsically motivated LC-RL approaches ( Colas et al. , 2020 ) . In any case , the language condition suffers from three drawbacks . 1 ) It couples skill learning and language grounding . Thus , it can not account for goal-directed behaviors in pre-verbal infants ( Mandler , 1999 ) . 2 ) Direct conditioning limits the behavioral diversity associated to language input : a single instruction leads to a low diversity of behaviors only resulting from the stochasticity of the policy or the environment . 3 ) This lack of behavioral diversity prevents agents from switching strategy after a failure . To circumvent these three limitations , one can decouple skill learning and language grounding via an intermediate innate semantic representation . On one hand , agents can learn skills by targeting configurations from the semantic representation space . On the other hand , they can learn to generate valid semantic configurations matching the constraints expressed by language instructions . This generation can be the backbone of behavioral diversity : a given sentence might correspond to a whole set of matching configurations . This is what we propose in this work . Contributions . We propose a novel conceptual RL architecture , named LGB for Language-GoalBehavior and pictured in Figure 1 ( right ) . This LGB architecture enables an agent to decouple the intrinsically motivated acquisition of a repertoire of skills ( Goals → Behavior ) from language grounding ( Language→ Goals ) , via the use of semantic goal representation . To our knowledge , the LGB architecture is the only one to combine the following four features : • It is intrinsically motivated : it selects its own ( semantic ) goals and generates its own rewards , • It decouples skill learning from language grounding , accounting for infants learning , • It can exhibit a diversity of behaviors for any given instruction , • It can switch strategy in case of failures . Besides , we introduce an instance of LGB , named DECSTR for DEep sets and Curriculum with SemanTic goal Representations . Using DECSTR , we showcase the advantages of the conceptual decoupling idea . In the skill learning phase , the DECSTR agent evolves in a manipulation environment and leverages semantic representations based on predicates describing spatial relations between physical objects . These predicates are known to be used by infants from a very young age ( Mandler , 2012 ) . DECSTR autonomously learns to discover and master all reachable configurations in its semantic representation space . In the language grounding phase , we train a Conditional Variational Auto-Encoder ( C-VAE ) to generate semantic goals from language instructions . Finally , we can evaluate the agent in an instruction-following phase by composing the two first phases . The experimental section investigates three questions : how does DECSTR perform in the three phases ? How does it compare to end-to-end LC-RL approaches ? Do we need intermediate representations to be semantic ? Code and videos can be found at https : //sites.google.com/view/decstr/ . 2 RELATED WORK . Standard language-conditioned RL . Most approaches from the LC-RL literature define instruction following agents that receive external instructions and rewards ( Hermann et al. , 2017 ; Chan et al. , 2019 ; Bahdanau et al. , 2018 ; Cideron et al. , 2019 ; Jiang et al. , 2019 ; Fu et al. , 2019 ) , except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones ( Colas et al. , 2020 ) . In both cases , the language-condition prevents the decoupling of language acquisition and skill learning , true behavioral diversity and efficient strategy switching behaviors . Our approach is different , as we can decouple language acquisition from skill learning . The language-conditioned goal generation allows behavioral diversity and strategy switching behaviors . Goal-conditioned RL with target coordinates for block manipulation . Our proposed implementation of LGB , called DECSTR , evolves in a block manipulation domain . Stacking blocks is one of the earliest benchmarks in artificial intelligence ( e.g . Sussman ( 1973 ) ; Tate et al . ( 1975 ) ) and has led to many simulation and robotics studies ( Deisenroth et al. , 2011 ; Xu et al. , 2018 ; Colas et al. , 2019a ) . Recently , Lanier et al . ( 2019 ) and Li et al . ( 2019 ) demonstrated impressive results by stacking up to 4 and 6 blocks respectively . However , these approaches are not intrinsically motivated , involve hand-defined curriculum strategies and express goals as specific target block positions . In contrast , the DECSTR agent is intrinsically motivated , builds its own curriculum and uses semantic goal representations ( symbolic or language-based ) based on spatial relations between blocks . Decoupling language acquisition and skill learning . Several works investigate the use of semantic representations to associate meanings and skills ( Alomari et al. , 2017 ; Tellex et al. , 2011 ; Kulick et al. , 2013 ) . While the two first use semantic representations as an intermediate layer between language and skills , the third one does not use language . While DECSTR acquires skills autonomously , previous approaches all use skills that are either manually generated ( Alomari et al. , 2017 ) , handengineered ( Tellex et al. , 2011 ) or obtained via optimal control methods ( Kulick et al. , 2013 ) . Closer to us , Lynch & Sermanet ( 2020 ) also decouple skill learning from language acquisition in a goalconditioned imitation learning paradigm by mapping both language goals and images goals to a shared representation space . However , this approach is not intrinsically motivated as it relies on a dataset of human tele-operated strategies . The deterministic merging of representations also limits the emergence of behavioral diversity and efficient strategy-switching behaviors . 3 METHODS . This section presents our proposed Language-Goal-Behavior architecture ( LGB ) represented in Figure 1 ( Section 3.1 ) and a particular instance of the LGB architecture called DECSTR . We first present the environment it is set in [ 3.2 ] , then describe the implementations of the three modules composing any LGB architecture : 1 ) the semantic representation [ 3.3 ] ; 2 ) the intrinsically motivated goal-conditioned algorithm [ 3.4 ] and 3 ) the language-conditioned goal generator [ 3.5 ] . We finally present how the three phases described in Figure 1 are evaluated [ 3.6 ] . 3.1 THE LANGUAGE-GOAL-BEHAVIOR ARCHITECTURE . The LGB architecture is composed of three main modules . First , the semantic representation defines the behavioral and goal spaces of the agent . Second , the intrinsically motivated GC-RL algorithm is in charge of the skill learning phase . Third , the language-conditioned goal generator is in charge of the language grounding phase . Both phases can be combined in the instruction following phase . The three phases are respectively called G→B for Goal → Behavior , L→G for Language → Goal and L→G→B for Language → Goal → Behavior , see Figure 1 and Appendix A . Instances of the LGB architecture should demonstrate the four properties listed in the introduction : 1 ) be intrinsically motivated ; 2 ) decouple skill learning and language grounding ( by design ) ; 3 ) favor behavioral diversity ; 4 ) allow strategy switching . We argue that any LGB algorithm should fulfill the following constraints . For LGB to be intrinsically motivated ( 1 ) , the algorithm needs to integrate the generation and selection of semantic goals and to generate its own rewards . For LGB to demonstrate behavioral diversity and strategy switching ( 3 , 4 ) , the language-conditioned goal generator must efficiently model the distribution of semantic goals satisfying the constraints expressed by any language input . 3.2 ENVIRONMENT The DECSTR agent evolves in the Fetch Manipulate environment : a robotic manipulation domain based on MUJOCO ( Todorov et al. , 2012 ) and derived from the Fetch tasks ( Plappert et al. , 2018 ) , see Figure 2 . Actions are 4-dimensional : 3D gripper velocities and grasping velocity . Observations include the Cartesian and angular positions and velocities of the gripper and the three blocks . Inspired by the framework of Zone of Proximal Development that describes how parents organize the learning environment of their children ( Vygotsky , 1978 ) , we let a social partner facilitate DECSTR ’ s exploration by providing non-trivial initial configurations . After a first period of autonomous exploration , the social partner initializes the scene with stacks of 2 blocks 21 % of times , stacks of 3 blocks 9 % of times , and a block is initially put in the agent ’ s gripper 50 % of times . This help is not provided during offline evaluations .

Summary: This paper proposes DECSTR, a goal-driven RL framework where the goal is represented as a binary vector that encodes the semantic relationships between objects. The state is assumed to contain disentangled features for each of the objects (and other features relating to the agent’s end-effectors). The architecture is based on Deep Sets (Zaher et al., 2017), which allows the pairs of the objects to be encoded with a shared network. The paper also introduces a curriculum learning strategy similar to CURIOUS (Colas et al., 2019), which relies on metrics such as competence and learning progress (LP) in order to select goals to pursue during an episode. One key difference is that unlike CURIOUS which uses expert-defined “goal buckets”, DECSTR groups the goals based on recency of discovery. Once trained to be able to behave with respect to these semantic relationship goals, the second phase is language grounding. They learn a module (implemented as C-VAE) that converts from natural language text to the semantic configuration goal space. Experiments were conducted in the Fetch Manipulate robotic arm environment and compared with ablations of DECSTR without some of its components, demonstrating strong performance and generalization to various types of language instructions.