Help Me Explore: Minimal Social Interventions for Graph-Based Autotelic Agents

1 INTRODUCTION . During early child development , open-ended play offers a learning context for exploring and mastering various skills . From solving puzzles to manipulating objects , children continuously and autonomously acquire complex behaviors driven by their intrinsic curiosity . Recently , artificial intelligence ( AI ) researchers have been interested in this open-ended learning framework to design artificial agents that can evolve in unbounded environments and grow repertoires of skills in a selfsupervised fashion . Several argued that such artificial agents must be autotelic : they must be intrinsically motivated to represent , generate and pursue their own goals ( Steels , 2004 ; Forestier et al. , 2017 ; Colas et al. , 2020b ) . To this end , classic developmental approaches in AI relied on the agents ’ sensorimotor module to represent their goals ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ) . This was inspired by the notion of physical situatedness in developmental psychology ( Piaget , 1954 ; Needham & Libertus , 2011 ) : children first learn to represent their goals through their embodied interactions with their surroundings , then use their intrinsic motivations to sample and pursue these goals . Unfortunately , this limits the behavioral diversity , as the distribution of learnable goals is highly grounded in the distribution of previously encountered goals . Hence , classic approaches in intrinsically motivated reinforcement learning ( IMRL ) usually rely on further assumptions , curricula and biases to drive the exploration process and increase behavioral diversity ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ) . Interestingly , developmental psychology has shown that children rely on two additional features : 1 ) imaginative play and 2 ) social situatedness . On the one hand , imaginative play allows children to imagine and target goals that they did not encounter previously ; by substituting an object with another for example ( Lewis et al. , 1992 ; Stagnitti & Unsworth , 2000 ) . The IMAGINE approach attempts to leverage this feature in IMRL by using language to imagine new goals ( Colas et al. , 2020a ) . On the other hand , social situatedness is essential for the emergence of sophisticated forms of in- telligence ( Vygotsky , 1994 ; Bruner , 1973 ; Tomasello , 2009 ) . In the Zone of Proximal Development ( ZPD ) framework for instance ( Vygotsky , 1978 ) , a social partner ( SP ) boosts the learner ’ s capabilities by first ensuring the mastery of simpler and foundational skills before the more complex ones ( i.e . scaffolding ) . More generally , verbal and nonverbal interactions help acquire shared cognitive representations that make sens for both the SP and the learner . These findings have gained popularity in AI ( Lindblom & Ziemke , 2002 ; Sigaud et al. , 2021 ) . In particular , we set ourselves in the framework of Sigaud et al . ( 2021 ) , which investigates the design of artificial agents that can be both autotelic and social : autonomous teachable agents . To facilitate these cultural interactions , research in cognitive science hypothesizes that human learning operates over structured and symbolic programs ( Chater & Oaksford , 2013 ; Wang et al. , 2019 ) . These high-level representations offer a handful of efficient tools for manipulating information . Specifically , symbolic representations that are independent of languages and domains ( such as images , symbolic predicates , ... ) facilitate generalization of underlying concepts rather than memorization of specific situations . However , using these high-level tools requires a formal syntax and a cross-cultural understanding of their semantics . Fortunately , work in developmental psychology has shown that some of these requirements , precisely spatial relational predicates , are used by infants from a very young age ( Mandler , 2012 ) . This motivated recent works in developmental AI to endow artificial agents with such symbolic inductive biases ( Tellex et al. , 2011 ; Alomari et al. , 2017 ; Kulick et al. , 2013 ; Asai , 2019 ; Akakzia et al. , 2020 ) . Beyond facilitating the learning process of autotelic agents ( Akakzia et al. , 2020 ) , this additional semantic layer should simplify social interactions . Contributions . This paper makes steps towards the design of teachable autotelic agents . We argue that entangling social interventions within autotelic agents ’ individual learning facilitates the acquisition of a growing repertoire of skills . Furthermore , we show how social signals influence the learners ’ intrinsic motivations and help them break away from the constraints of their physical situatedness . To this end , our contributions are twofold : First , we propose a novel social interaction protocol , Help Me Explore ( HME ) , which intertwines social and autotelic episodes . In the former , a SP drives the agent towards its ZPD through goal proposals . They relies on a structured representation of the environment high-level goal space and a model of the agent ’ s current exploration limits . Such a model is required for a good teacher to design lessons adapted to their students capabilities ( Bruner , 1961 ) . In the latter , the agent can either train to reach its known semantic goals or rehearse social proposals using an internalization mechanism . Second , we introduce a GC-RL architecture named GANGSTR for GoAl-conditioned Neural Graph with SemanTic Representations . GANGSTR is based on the Soft-Actor-Critic algorithm ( SAC , Haarnoja et al. , 2018 ) , where both its actor and critic are implemented as Graph Neural Networks ( GNNs , Scarselli et al. , 2005 ; Gilmer et al. , 2017 ; Battaglia et al. , 2018 ) . Furthermore , it gradually constructs a semantic graph of its discovered configurations and uses it to decompose goals into sequences of intermediate sub-goals . We show that when engaged in the HME protocol , GANGSTR masters a diversity of complex goals it would not have discovered alone , spanning its entire goal space of around 70.000 semantic configurations . We also show that GANGSTR requires only light social interventions . 2 RELATED WORK . Autotelic RL . The quest for autotelic agents first emerged in the field of developmental robotics ( Steels , 2004 ; Oudeyer & Kaplan , 2007 ; Nguyen & Oudeyer , 2012 ; 2014 ; Baranes & Oudeyer , 2013 ; Forestier & Oudeyer , 2016 ; Forestier et al. , 2017 ) and recently converged with state-of-the-art deep RL research ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ; 2020a ; b ) . Such agents rely on goal-conditioned RL algorithms ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Colas et al. , 2020b ) and Automatic Currciulum Learning ( ACL ) mechanisms ( Portelas et al. , 2020 ) . Following the Piagetian tradition in psychology , most of these approaches consider individual agents interacting with their physical environment and leveraging forms of intrinsic motivations to power their exploration and learning progress ( Bellemare et al. , 2016 ; Achiam & Sastry , 2017 ; Nair et al. , 2018 ; 2020 ; Burda et al. , 2018 ; Pathak et al. , 2019 ; Colas et al. , 2019 ; Pong et al. , 2019 ) . These approaches sample goals from the set of already-known experiences , which limits their capacity to explore . Social Learning . The ZPD describes the set of learning situations a child can solve with the help of a more knowledgeable other ( Vygotsky , 1978 ) . Throughout scaffolding and supportive activities , the SP uses their knowledge to drive learners beyond what they can do alone . Interactive learning research in AI investigates and models the ways human tutors can guide their agents ’ learning process ( Thomaz & Breazeal , 2008 ; Argall et al. , 2009 ; Celemin & Ruiz-del Solar , 2015 ; Najar et al. , 2016 ; Christiano et al. , 2017 ) . A category of teaching signals requires the SP to point out sequences of actions and states using demonstrations or preferences ( Argall et al. , 2009 ; Christiano et al. , 2017 ; Fournier et al. , 2019 ) . Another category assists the development of autotelic agents by providing them with challenging goals or guiding instructions ( Grizou et al. , 2013 ) . As we want minimal interactions with the SP who does not show the agent how to act , we focus on the latter category . Semantic Goal Representations . Developmental psychology suggests that humans rely on intuitive mechanisms of affordances , analogy and relational patterns to reason about their goals ( Gibson , 1977 ; Holyoak , 2012 ) . Endowing artificial agents with such inductive biases introduces a level of abstraction that is likely to facilitate the acquisition of complex tasks and the interaction with human teachers . Our set-up and semantic representation layer is similar to that of the DECSTR agent ( Akakzia et al. , 2020 ) . However , there are several important discrepancies . First , in Akakzia et al . ( 2020 ) , the training process was split into 3 sequential phases : sensorimotor learning , language grounding and instruction following . Given this split , instructions from the tutor could not be used to drive the learning process of the agent . By contrast , here sensorimotor learning and instruction following are intertwined , making it possible to focus on teaching curriculum questions . Besides , in Akakzia et al . ( 2020 ) , the language grounding aspect was managed by a Language Goal Generator ( LGG ) component translating sentences from the tutor into semantic goals . Here , the language learning aspect is outside the scope of this study , as we consider that the tutor immediately communicates a goal in a semantic representation that the agent immediately understands . Graph-Based Exploration . Humans solve novel problems by combining familiar skills and routines . They are able to represent complex systems within their environment as compositions of entities and their interactions ( Navon , 1977 ; Marcus , 2001 ; Kemp & Tenenbaum , 2008 ) . This motivates the design of artificial agents that learn structured representations of their world , such as graphs ( Džeroski et al. , 2001 ; Toyer et al. , 2018 ) . In this paper , we aim at endowing autotelic agents with a structured representation of their goal space . Close to us , the GO-Explore algorithm uses a graph to represent a discrete archive of interesting states ( Ecoffet et al. , 2019 ; 2021 ) . They map these states to discretized cells . They first reach a selected cell , based on heuristics such as recency or visit counts in memory , then explore from that state . We take a different approach based on the semantic character of the goals we consider . Using its goal space graph embedding , GANGSTR autonomously evaluates its performances when hopping from one goal to another . It then uses it as a proxy to determine the optimal path to reach complex semantic goal configurations . 3 METHODS . In this section , we introduce the HME interaction protocol ( Section 3.1 ) and GANGSTR , a particular autotelic RL agent for manipulation domains . We first introduce the experimental environment and the underlying semantic goal space ( Section 3.2 ) . Then , we present GANGSTR ’ s object-centered architecture , its internal semantic graph construction and decomposition processes ( Section 3.3 ) . Finally , we describe the application of HME to GANGSTR ( Section 3.4 ) . The anonymous code for the environment , agent and interaction setup as well as pretrained weights can be found at https : //anonymous.4open.science/r/gangstr-2C4F .

Summary: The paper targets goal-oriented reinforcement learning tasks with possible intervention of a virtual social partner assistant. It presents experiments for exploring object manipulation with 5 different blocks, in which an agent discovers configurations based on its own intuitive exploration, as well as guidance for other configurations from which it can further explore. The mount of intervention by the social partner is varied (from 0% to 100%), showing that 2-20% are sufficient for significant improvement over exploration without social partner.