Help Me Explore: Minimal Social Interventions for Graph-Based Autotelic Agents

1 INTRODUCTION . During early child development , open-ended play offers a learning context for exploring and mastering various skills . From solving puzzles to manipulating objects , children continuously and autonomously acquire complex behaviors driven by their intrinsic curiosity . Recently , artificial intelligence ( AI ) researchers have been interested in this open-ended learning framework to design artificial agents that can evolve in unbounded environments and grow repertoires of skills in a selfsupervised fashion . Several argued that such artificial agents must be autotelic : they must be intrinsically motivated to represent , generate and pursue their own goals ( Steels , 2004 ; Forestier et al. , 2017 ; Colas et al. , 2020b ) . To this end , classic developmental approaches in AI relied on the agents ’ sensorimotor module to represent their goals ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ) . This was inspired by the notion of physical situatedness in developmental psychology ( Piaget , 1954 ; Needham & Libertus , 2011 ) : children first learn to represent their goals through their embodied interactions with their surroundings , then use their intrinsic motivations to sample and pursue these goals . Unfortunately , this limits the behavioral diversity , as the distribution of learnable goals is highly grounded in the distribution of previously encountered goals . Hence , classic approaches in intrinsically motivated reinforcement learning ( IMRL ) usually rely on further assumptions , curricula and biases to drive the exploration process and increase behavioral diversity ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ) . Interestingly , developmental psychology has shown that children rely on two additional features : 1 ) imaginative play and 2 ) social situatedness . On the one hand , imaginative play allows children to imagine and target goals that they did not encounter previously ; by substituting an object with another for example ( Lewis et al. , 1992 ; Stagnitti & Unsworth , 2000 ) . The IMAGINE approach attempts to leverage this feature in IMRL by using language to imagine new goals ( Colas et al. , 2020a ) . On the other hand , social situatedness is essential for the emergence of sophisticated forms of in- telligence ( Vygotsky , 1994 ; Bruner , 1973 ; Tomasello , 2009 ) . In the Zone of Proximal Development ( ZPD ) framework for instance ( Vygotsky , 1978 ) , a social partner ( SP ) boosts the learner ’ s capabilities by first ensuring the mastery of simpler and foundational skills before the more complex ones ( i.e . scaffolding ) . More generally , verbal and nonverbal interactions help acquire shared cognitive representations that make sens for both the SP and the learner . These findings have gained popularity in AI ( Lindblom & Ziemke , 2002 ; Sigaud et al. , 2021 ) . In particular , we set ourselves in the framework of Sigaud et al . ( 2021 ) , which investigates the design of artificial agents that can be both autotelic and social : autonomous teachable agents . To facilitate these cultural interactions , research in cognitive science hypothesizes that human learning operates over structured and symbolic programs ( Chater & Oaksford , 2013 ; Wang et al. , 2019 ) . These high-level representations offer a handful of efficient tools for manipulating information . Specifically , symbolic representations that are independent of languages and domains ( such as images , symbolic predicates , ... ) facilitate generalization of underlying concepts rather than memorization of specific situations . However , using these high-level tools requires a formal syntax and a cross-cultural understanding of their semantics . Fortunately , work in developmental psychology has shown that some of these requirements , precisely spatial relational predicates , are used by infants from a very young age ( Mandler , 2012 ) . This motivated recent works in developmental AI to endow artificial agents with such symbolic inductive biases ( Tellex et al. , 2011 ; Alomari et al. , 2017 ; Kulick et al. , 2013 ; Asai , 2019 ; Akakzia et al. , 2020 ) . Beyond facilitating the learning process of autotelic agents ( Akakzia et al. , 2020 ) , this additional semantic layer should simplify social interactions . Contributions . This paper makes steps towards the design of teachable autotelic agents . We argue that entangling social interventions within autotelic agents ’ individual learning facilitates the acquisition of a growing repertoire of skills . Furthermore , we show how social signals influence the learners ’ intrinsic motivations and help them break away from the constraints of their physical situatedness . To this end , our contributions are twofold : First , we propose a novel social interaction protocol , Help Me Explore ( HME ) , which intertwines social and autotelic episodes . In the former , a SP drives the agent towards its ZPD through goal proposals . They relies on a structured representation of the environment high-level goal space and a model of the agent ’ s current exploration limits . Such a model is required for a good teacher to design lessons adapted to their students capabilities ( Bruner , 1961 ) . In the latter , the agent can either train to reach its known semantic goals or rehearse social proposals using an internalization mechanism . Second , we introduce a GC-RL architecture named GANGSTR for GoAl-conditioned Neural Graph with SemanTic Representations . GANGSTR is based on the Soft-Actor-Critic algorithm ( SAC , Haarnoja et al. , 2018 ) , where both its actor and critic are implemented as Graph Neural Networks ( GNNs , Scarselli et al. , 2005 ; Gilmer et al. , 2017 ; Battaglia et al. , 2018 ) . Furthermore , it gradually constructs a semantic graph of its discovered configurations and uses it to decompose goals into sequences of intermediate sub-goals . We show that when engaged in the HME protocol , GANGSTR masters a diversity of complex goals it would not have discovered alone , spanning its entire goal space of around 70.000 semantic configurations . We also show that GANGSTR requires only light social interventions . 2 RELATED WORK . Autotelic RL . The quest for autotelic agents first emerged in the field of developmental robotics ( Steels , 2004 ; Oudeyer & Kaplan , 2007 ; Nguyen & Oudeyer , 2012 ; 2014 ; Baranes & Oudeyer , 2013 ; Forestier & Oudeyer , 2016 ; Forestier et al. , 2017 ) and recently converged with state-of-the-art deep RL research ( Eysenbach et al. , 2018 ; Warde-Farley et al. , 2018 ; Nair et al. , 2018 ; 2020 ; Pong et al. , 2019 ; Colas et al. , 2019 ; 2020a ; b ) . Such agents rely on goal-conditioned RL algorithms ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Colas et al. , 2020b ) and Automatic Currciulum Learning ( ACL ) mechanisms ( Portelas et al. , 2020 ) . Following the Piagetian tradition in psychology , most of these approaches consider individual agents interacting with their physical environment and leveraging forms of intrinsic motivations to power their exploration and learning progress ( Bellemare et al. , 2016 ; Achiam & Sastry , 2017 ; Nair et al. , 2018 ; 2020 ; Burda et al. , 2018 ; Pathak et al. , 2019 ; Colas et al. , 2019 ; Pong et al. , 2019 ) . These approaches sample goals from the set of already-known experiences , which limits their capacity to explore . Social Learning . The ZPD describes the set of learning situations a child can solve with the help of a more knowledgeable other ( Vygotsky , 1978 ) . Throughout scaffolding and supportive activities , the SP uses their knowledge to drive learners beyond what they can do alone . Interactive learning research in AI investigates and models the ways human tutors can guide their agents ’ learning process ( Thomaz & Breazeal , 2008 ; Argall et al. , 2009 ; Celemin & Ruiz-del Solar , 2015 ; Najar et al. , 2016 ; Christiano et al. , 2017 ) . A category of teaching signals requires the SP to point out sequences of actions and states using demonstrations or preferences ( Argall et al. , 2009 ; Christiano et al. , 2017 ; Fournier et al. , 2019 ) . Another category assists the development of autotelic agents by providing them with challenging goals or guiding instructions ( Grizou et al. , 2013 ) . As we want minimal interactions with the SP who does not show the agent how to act , we focus on the latter category . Semantic Goal Representations . Developmental psychology suggests that humans rely on intuitive mechanisms of affordances , analogy and relational patterns to reason about their goals ( Gibson , 1977 ; Holyoak , 2012 ) . Endowing artificial agents with such inductive biases introduces a level of abstraction that is likely to facilitate the acquisition of complex tasks and the interaction with human teachers . Our set-up and semantic representation layer is similar to that of the DECSTR agent ( Akakzia et al. , 2020 ) . However , there are several important discrepancies . First , in Akakzia et al . ( 2020 ) , the training process was split into 3 sequential phases : sensorimotor learning , language grounding and instruction following . Given this split , instructions from the tutor could not be used to drive the learning process of the agent . By contrast , here sensorimotor learning and instruction following are intertwined , making it possible to focus on teaching curriculum questions . Besides , in Akakzia et al . ( 2020 ) , the language grounding aspect was managed by a Language Goal Generator ( LGG ) component translating sentences from the tutor into semantic goals . Here , the language learning aspect is outside the scope of this study , as we consider that the tutor immediately communicates a goal in a semantic representation that the agent immediately understands . Graph-Based Exploration . Humans solve novel problems by combining familiar skills and routines . They are able to represent complex systems within their environment as compositions of entities and their interactions ( Navon , 1977 ; Marcus , 2001 ; Kemp & Tenenbaum , 2008 ) . This motivates the design of artificial agents that learn structured representations of their world , such as graphs ( Džeroski et al. , 2001 ; Toyer et al. , 2018 ) . In this paper , we aim at endowing autotelic agents with a structured representation of their goal space . Close to us , the GO-Explore algorithm uses a graph to represent a discrete archive of interesting states ( Ecoffet et al. , 2019 ; 2021 ) . They map these states to discretized cells . They first reach a selected cell , based on heuristics such as recency or visit counts in memory , then explore from that state . We take a different approach based on the semantic character of the goals we consider . Using its goal space graph embedding , GANGSTR autonomously evaluates its performances when hopping from one goal to another . It then uses it as a proxy to determine the optimal path to reach complex semantic goal configurations . 3 METHODS . In this section , we introduce the HME interaction protocol ( Section 3.1 ) and GANGSTR , a particular autotelic RL agent for manipulation domains . We first introduce the experimental environment and the underlying semantic goal space ( Section 3.2 ) . Then , we present GANGSTR ’ s object-centered architecture , its internal semantic graph construction and decomposition processes ( Section 3.3 ) . Finally , we describe the application of HME to GANGSTR ( Section 3.4 ) . The anonymous code for the environment , agent and interaction setup as well as pretrained weights can be found at https : //anonymous.4open.science/r/gangstr-2C4F .

Summary: The paper discusses a model for learning a task (5-block manipulation) using both "self-directed" (which the authors define as "autotelic" learning and social learning. I would say that the contribution is quite modest since the authors demonstrates the validity of their approach only through simulations of a specific game. It is difficult to evaluate the generality of the proposed approach.