Help Me Explore: Combining Autotelic and Social Learning via Active Goal Queries
Authors: Ahmed Akakzia
Hugo Caselles-Dupré
Olivier Serris
Mohamed CHETOUANI
Olivier Sigaud
Cédric Colas
Venue: Submitted to ICLR 2023 (2023)
Keywords: Exploration
Teachability
Autotelism
Social Interaction
Curriculum Learning

Abstract: Most approaches to open-ended skill learning train a single agent in a purely sensorimotor environment. But because no human child learns everything on their own, we argue that sociality will be a key component of open-ended learning systems. This paper enables learning agents to blend individual and socially-guided skill learning through a new interaction protocol named Help Me Explore (HME).
In social episodes triggered at the agent's demand, a social partner suggests a goal at the frontier of the agent's capabilities and, when the goal is reached, follows up with a new adjacent goal just beyond. In individual episodes, the agent practices skills autonomously by pursuing goals it already discovered through either its own experience or social suggestions. The idea of augmenting an individual goal exploration with social goal suggestions is simple, general and powerful. We demonstrate its efficiency on two notoriously hard exploration benchmarks: continuous mazes and a 5-block robotic manipulation task. With minimal social interventions, an HME-agent outperforms the purely social agent deprived of its autonomy, and the purely individual agent which fails to solve hard exploration problems.

Under review as a conference paper at ICLR 2023
HELP ME EXPLORE: COMBINING AUTOTELIC AND SO- CIAL LEARNING VIA ACTIVE GOAL QUERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Most approaches to open-ended skill learning train a single agent in a purely sen- sorimotor world. But because no human child learns everything on their own, we argue that sociality will be a key component of open-ended learning systems. This paper enables learning agents to blend individual and socially-guided skill learning through a new interaction protocol named Help Me Explore (IIME). In social episodes triggered at the agent's is demand, social partner suggests goal at the frontier of the agent's s capabilities and, when the goal is reached, follows up with adjacent goal just beyond. In individual episodes. the agent practices skills autonomously by pursuing goals it has already discovered through either its own experience or social suggestions. The idea of augmenting individual goal exploration with social goal suggestions is simple, general and powerful. We demonstrate its efficiency on hard exploration problems: continuous mazes and a 5-block robotic manipulation task. With minimal social interventions, the HIME agent outperforms both the purely social and purely individual agents.
INTRODUCTION
Open-ended learning is an important challenge in artificial intelligence (AI) where the goal is to design embodied artificial agents able to grow diverse repertoires of skills across their lives (Don- cieux et al., 2018; Stooke et al., 2021). Goal-conditioned reinforcement learning (GC-RL). RL), because it offers the possibility to train an agent on multiple goals in parallel, recently emerged as a key com- ponent in this quest (Schaul et al., 2015; Andrychowicz et al., 2017; Liu et al., 2022). But where do goals come from? Almost always, they are sampled from a fixed distribution over a predefined goal space; i.e. they come from an engineer. Beyond the heavy engineering burden it presupposes, this approach is fundamentally limited in realistic environments with infinite possibilities because engineers cannot foresee how the agents will learn or what they will be able to achieve. Instead, we must draw inspiration from the study of human learning and pursue a developmental approach: agents should be intrinsically motivated to learn to represent, generate, pursue and master their own goals- i.e. they must be autotelic Steels (2004); Colas et al. (2022).
In recent paper introducing the autotelic framework, Colas al. identify two challenges: 1) learn- ing goal representations; 2) exploring the corresponding goal space and mastering the associated skills (Colas al., 2022). Although, eventually, all autotelic agents must learn their own goal repre- sentations (challenge 1, Eysenbach et al. 22018); Nair et al. (20188) 8b); Colas et al. (2020)), most exist- ing approaches assume pre-defined goal space and focus on its exploration and mastery (challenge 2, Schaul et al. (2015); Colas et al. (2019); Akakzia et al. (2020)). In this second challenge the one we focus on- agents must learn to organize their own learning trajectories by prioritizing goals with the objective of maximizing long-term skill mastery.
Exploring and developing skills unknown goal spaces can be hard when they are not uniform: some goals might be easy, others hard; most of them might be unreachable. So how can an agent select which goals to practice now? Because they direct the agent's 's behavior, the selected goals im- pact the discovery of future goals (exploration), and the mastery of known ones (exploitation)-yy) another instance of the exploration- -etpiiiattellliiiiionnnn dilemma (Thrun, 1992). Existing methods belong to the family of automatic curriculum learning strategies (Portelas al., 2020). They propose to place the hard-to-optin optimize distal objective of general skill mastery with proxies such as intermediate difficulty (Florensa al., 2018). learning progress (Colas et al., 2019; Blaes al., 2019; Akakzia

---

Under review as a conference paper at ICLR 2023
et al., 2020) or novelty (Pong et al., 2019; Ecoffet et al., 2019). But all ACL strategies are far limited to generate goals from the distribution of effects the agents already experienced. Can agents demonstrate efficient exploratory capacities if they only target goals they already achieved (Campos et al., 2020) To solve this problem, we must once again draw inspiration from the study of human learning, this time from socio-cultural psychology.
Philosophers, psychologists, linguists and roboticists alike have argued for the central importance of rich socio-cultural environments in human development (Wood et al., 1976; Vygotsky & Cole, 1978; Berk, 1994; Tomasello, 1999; Lindblom & Ziemke, 2003; Lupyan, 2012; Colas, 2021). Humans are social beings wired to interact with and learn from others (Vygotsky, 1978; Tomasello, 1999; 2019). When they explore, it is often through veciall-gguieed interactions (c.g. by having parent organize the playground) and it often relies on the inter-generational population- others, a phenomenon known as the cultural ratchet (Tomasello, 1999). In guided-play interactions (Weisherg et al., 2013; Yu et al., 2018) caretakers scaffold the learning environments of children and help them practice new skills just beyond their current capacities, i.e. in Vygotsky's zone of proximal development (Vygotsky, 1978). Can Al learn from these insights? We believe so.
exploration of
This paper introduces novel social interaction protocol for autotelic lagents named Help Me Explore (HME). This consists of a succession of individual and social episodes. In individual episodes, the agent pursues self-generated goals sampled from the set of outcomes the agent already experienced during training. In social episodes, a social partner suggests a novel goal to the agent and decomposes it into consecutive sub-goals: 1) a frontier goal that the agent already discov- ered and, it is reached, 2) beyond goal never achieved by the agent but just beyond the its current abilities. The frontier goal acts as stepping stone facilitating the discovery and mastery of the be- yond goal. Combined with powerful autotelic RL algorithm, IIME makes significant steps towards teachable autotelic agents: autonomous agents that leverage external social signals and interweave them with their orla--zga skill learning processes to gain autonomy, creativity and skill mastery (Sigaud et al., 2021). Our contributions are twofold:
Help Me Explore (HME). This social interaction protocol lets agents blend social and in- dividual goal explorations through social goal suggestions. It is generally applicable to any multi-goal environment and it is tested in 5-block robotic manipulation domain, where it enables agents to master 5-block stacking, a notoriously hard exploration challenge.
A socially-sensitive autotelic agent (HME agent). We augment an existing autotelic RL agent with two social capabilities: 1) active learning mechanism allowing the agent to self-monitor its learning progress and, when it stagnates, query the social partner for a goal suggestion and 2) the ability internalize social goal suggestions and rehearse them autonomously during individual episodes. This ability is inspired from Vygotsky's 's concept of internalization describing how learners model social processes occurring in the zone of proximal development to augment their future autonomy and capacities (Vygotsky, 1978). These two mechanisms drastically reduce the amount of social interactions required to achieve skill mastery.
S0C8E pErFaT
woiilyy capeodd goll
Exteral social
adlected
+
goa
polay
lnown
TRANIE gol campler
desevendd grito
Figure I: The HME social interaction protocol and the social autotelic agent. The autotelic agent (blue) interacts with its world and discovers new configurations (grey box, right). Through its goal source selector (blue box, middle), the agent decides whether to sample from its known goals (blue box, bottom left) or to query the social partner for social goal (pink box, top left). bocially-suggeste goals are further internalized within the agent for later use during individual episodes.
2

---

Under review as a conference paper at ICLR 2023
We evaluate the social autotelic agent and the HME interaction protocol in complex block manip- ulation environment based on the Fetch robotic arm (Plappert et al., 2018). Whereas state-of-the-ann art agent successfully discovers and masters
ACL methods fail to discover difficult configurations, tens of thousands of configurations, including difficult stacks of 5 blocks and complex pyramidal structures. Thanks to its internalization and active query mechanisms, the agent requires social goal suggestions in only 0.5% of the training episodes.
2 RELATED WORK
Hard exploration problems. Environments can be said to have a hard exploration problem when their reward function is sparse and/or deceptive (Ten et al., 2022). Fighting hard exploration prob- lems requires exploratory mechanisms. Indeed, simple undirected exploration such as adding ran- dom noise on the agent's actions (Mnih et al., 2015) or policy parameters (Plappert et al.. 2017) is not enough to experience informative rewards in hard exploration problems. Directed exploration mechanisms involve the use of intrinsic motivations: auxiliary reward functions driving the agent to explore (Oudeyer & Kaplan, 2007; Ten et al., 2022). While knowledge based intrinsic motivations aim improving the agent's 'ss model of the world (Barto, 2013; Bellemare et al., 2016; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018), competence-based intrinsic motivations let agents target their own goals and pursue them as a means of exploration (Colas et al., 2022). Finally, the recent Go- -ro-xplore approach recommends to first achieve known goal that has been sparsely visited, then to start exploring from there (Ecoffet et al., 2019; Pitis et al., 2020). This paper pro- poses to solve hard exploration problems with competence-basee intrinsic motivations. Besides, argue that sparsely visited areas of the goal space considered in approaches like Go-Explore do not necessarily represent stepping stones for new goals, and consider an expert SP that proposes these intermediate checkpoints.
Autotelic reinforcement learning. Autotelic agents are autonomous agents intrinsically moti- vated to learn to represent, generate, pursue and master their own goals (competence-baset intrinsic motivations Colas et al. (2022)). More than explorers, autotelic agents are designed to build open- ended repertoires of skills across their lives. This quest first emerged in the field of developmen- tal robotics (Steels, 2004; Oudeyer & Kaplan, 2007; Nguyen & Oudeyer, 2012; 2014b; Baranes & Oudeyer, 2013; Forestier & Oudeyer, 2016; Forestier et al., 2017) and recently converged with state- of-the-art deep reinforcement learning (RL) research (Eysenbach et al., 2018; Warde-Farley et al., 2018; Nair et al., 2018b 2020; Pong et al., 2019; Colas et al., 2019; 2020; 2022). Following the Piagetian tradition in psychology. most of these approaches consider individual agents interacting with their physical environment and leveraging forms of intrinsic motivations power their explo- ration and learning progress ( (Warde-Farley et al., 2018; Nair et al., 2018b: Pong et al., 2019; Colas al., 2019; Stooke et al., 2021). A fundamental limitation of these approaches is their incapacity to generate goals outside of the distribution of known outcomes, which drastically limits their ex- ploratory capabilities. The Imagine approach proposes to leverage linguistic structures to imagine new goals out of the distribution of known outcomes but relies on a hard-coded mechanism which does not allow the generation of truly creative goals (Colas et al., 2020). In this paper, we leverage social interactions with a more knowledgeable partner to overcome this limitation.
Interactive reinforcement learning. Interactive RI. lets non-expert users communicate, teach and guide the learning processes of artificial agents (Thomaz & Breazeal, 2008; Argall al., 2009; Celemin & Ruiz-del Solar, 2015; Najar et al., 2016; Christiano et al., 2a) These methods often limit the agent's autonomy because they heavily rely on the instructor to guide the whole learning process, either through demonstrations (Nair et al., 201 Sa; Fournier et al., 2019; Hester et al., 2018), preferences (Christiano al., 2017b) or guidance (Grizou et al., 2013).
Social learning and intrinsic motivations. Combining social learning and intrinsic motivations has been studied as a way to enhance skill acquisition in artificial agents (Schaal et al., 2003; Thomaz
et al., 2006; Thomaz& & Breazeal, 2008; Peters & Schaal, 2008; Kober & Peters, 2011: Stulp & Schaal, 2011; Nguyen & Oudeyer. 2014a). The main motivation is that social guidance provided by humans can drive the learner to regions of its state space, where its own random exploration alone is usually insufficient. Intrinsic motivations can then be used to further explore these new areas. Such combination profits from both the broader aspect of intrinsic motivations (explore as
3

---

Under review as a conference paper at ICLR 2023
many goals possible) and the specialized aspect of social learning (follow specific instruction). This was studied not only in the context of learning one single skill (Schaal et al., 2003; Peters & Schaal, 2008; Kober & Peters, 2011; Stulp & Schaal, 2011), 1),, but also in achieving a variety of goals (Thomaz et al., 2006; Thomaz & Breazeal, 2008; Nguyen & Oudeyer, 2014a). Most of these works rely on human demonstrations or upstream feedback as form of social guidance. In this paper, social signals take the form of goals suggested by the SP at the demand of the learning autotelic agent.
3 METHODS
This paper studies how social autotelic agent can efficiently explore a complex environment while relying on a minimal number of interventions from a social partner. We first describe the problem of exploring an unknown goal space (Section 3.1), before introducing the proposed social interaction protocol (Section 3.2, Figure 1) and our socially-sensitive autotelic agent (Section 3.3). The pseudo- code and implementation details can be found in Appendices A.5 and A.6 respectively.
3.1 PROBLEM STATEMENT
The agent evolves in a reward-free interactive world that can perceive through states and act on through actions. This environment is modeled by an MDP: {S. A. T, po} where S is the state space, A the action space, T x A + S the transition function describing the world's physics and Po the distribution of initial states. In the autotelic framework, the goal space and associated reward functions are not specific to the environment but represented within the agent; which is why they are not modeled in the MDP (Colas al., 2022).
In this paper, the agent is given an extra representation function mapping states to high-level discrete configurations o S + C with C the set of reachable configurations. Goals are then defined as target configurations the agent attempts to reach, such that the goal space G is equal to C. The agent does not know G, but can only sample goals from the set of encountered configurations Gknown C G. Goal-specific reward functions are then defined as binary truth functions assessing the equality between the agent's current configuration c(s) and its goal g: r(g, c(s) = 9 where is the current state.
We define the exploration problem the discovery and mastery of a maximum number of goals within the configuration space. We do not assume access to the set of reachable configurations G. There might be too many and the agents could surprise us with their learning abilities. In this case, we can only use a subjective evaluation metric. We propose to measure skill mastery on a pre- defined subset of goals we, as experimenters, find interesting. In the Experiments, Section 4.1 will introduce the multi-object manipulation environment we consider, its associated goal space and the specific evaluation metrics.
3.2 THE SOCIAL INTERACTION PROTOCOL.
The Help Me Explore interaction protocol (HME) is centered around the idea of social goal proposal. It involves three elements: an environment, an autotelic agent, and a social partner (SP), see Figure 1. The environment is reward-free MDP as defined in the previous section. The autotelic agent is a goal-conditioned lagent that also embeds goal representations and associated reward functions (see description in Section 3.3).
The social partner (SP) is a hand-coded more-knowledgeable other. We assume that shares the same goal representations as the agent and that it knows a subset of the achievable goals (Gsr c G) and how to decompose them into simpler sub-goals (e-g. a stack of two is first step towards stack of three). As the agent trains, the SP constructs model of its current exploration limits by tracking which goals from GSP the agent achieved and which it did not. To suggest a goal, it samples a target configuration lying just beyond the agent's 's current abilities (Beyond goal). It then decomposes this Beyond goal by suggesting a Frontier goal: a stepping stone on the way of the Beyond goal that the agent already achieved during training. As it does so, the SP builds zone of proximal development; it helps the agent achieve goals beyond its current limits by adapting its suggestion to the agent's current knowledge. Appendix A.2 provides detailed description of the SP code.
4

---

Under review as a conference paper at ICLR 2023
The HME interaction protocol starts after an initialization phase of N, 10000 episodes where the agent performs random actions to discover its first configurations and, thus, its first potential goals. In HME, the agent engages either in individual episodes where it pursues its own goals or in social episodes where it pursues a goal suggested by the SP. Although this work uses a hand-coded SP, the discussion section addresses the idea of replacing it with human partner.
3.3 A SOCIALLY-SENSITIE AUTOTELIC AGENT
The socially-sensitive autotelic agent is built on top of powerful autotelic agent called GANGSTR presented in Akakzia & Sigaud (2022). We augment it with social capabilities: the ability to select between self-generated and social goals and the ability to internalize social goals for later reuse.
The GANGSTR agent. GANGSTR is goal-conditioned reinforcement learning agent with graph neural network (GNNS) architectures trained with the soft actor-critic algorithm (Haarnoja et al., 2018). The GNNs architectures for the policy and value network let us encode relational biases which drastically increase the agent's transfer capabilities. especially in multi-object manipulation tasks (see details in Appendix A.4 and in the original paper Akakzia & Sigaud (2022)). Because goals only rarely achieved by chance (sparse reward problem), we reuse the multi-criteria hindsight experience replay mechanism that was shown to significantly increase sample efficiency in Lanier al. (2019); Akakzia & Sigaud (2022). We further augment the autotelic agent with social capabilities: 1) the ability to decide when to ask the SP for a goal proposal and 2) the ability to internalize pocially-suggester goals for later reuse.
Active goal queries based on learning progress. The agent tracks its overall learning progress (LP) towards known goals and computes the probability to query the SP for a goal as a function of LP. The general principle is that, whenever LP is low, the agent is most likely neither learning nor discovering anything new on its own and should ask for social goal suggestion. To estimate its LP, the agent first estimates its competence. To this end, train a neural network to estimate the success or failure of reaching a goal given the goal embedding with a binary classification loss. Every N₂ 6000 training episodes, the agent estimates its competence by computing the projected success rate averaged over a subset of goals GLP uniformly sampled from the set of known goals Gknown- LP is then computed as the difference between two competence measures separated by fixed time period Ne: LP = c ct The query probability qqerry at training episode is then defined by:
P
pt query
= exp expp-3
-B
Σ LP g Sir
(I)
where 33 is a haphhhypppaaeeeeee parameter controlling the tolerance to a lower LP. As B grows, the agent becomes less susceptible to query the SP and will only do so for lower LP values.
Social goal internalization. In Vygotsky's 's theory, children learn to internalize social processes occurring in the zone of proximal development and to turn them into internal cognitive tools such that they can solve hard problems themselves in the future (Vygotsky, 1978). Our social autotelic agents model this internalization by storing the pairs of Frontier and Beyond goals suggested by the SP in their memory of known goals, even when those were not achieved in practice. This offers them the ability to resample and practice these goals on their own during individual episodes which lowers the number of social interactions required from the SP.
Training organization. At each new episode, the agent estimates its current learning progress and derives the corresponding probability to query the SP. Sampling from this probability, the agent decides whether to pursue self-generated goal (individual episode) or to query the SP for a goal suggestion (social episode). Self-generated goals are sampled uniformly from the set of known goals Gknown which includes the set of goals internalized from social suggestions. The agent then pursues the goal and computes its own rewards by comparing its reached configurations with the replay buffer,
goal configuration. After adding the set of collected transitions actor-critic algorithm (Haarnoja et al., 2018) augmented with the multi-criteria hindsight experience replay mechanism (Lanier et al., 2019) update the policy and value networks.
the soft
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
This experimental section demonstrates the benefits of blending social and individual exploratory processes with respect to purely social or purely individual explorations, including those powered by state-of-the-art ACL strategies. We first conducted preliminary experiments in simple 2D contin- uous mazes involving exploration problems (see details and experiments in Appendix A. These experiments prove the benefits of sociality for skill learning, which led us to tackle another hard exploration problem in complex robotic manipulation setting. We open-source an anonymized version of our code via Anonymous Github.
4.1 HARD EXPLORATION PROBLEM
We consider the Fetch Manipulate environment where the agent is embodied in 4-DoF robotic arm facing 5 colored blocks on a table (Akakzia & Sigaud, 2022). The agent controls the Cartesian ve- locity of its gripper and gripper closure (4 dimensions) and perceives the position and velocity of the blocks and gripper. In addition, it perceives a 35D binary representation of the blocks configuration obtained by applying the three predicates above (binary non con-symmetrie), close (binary symmetric) and on table (unary) on all possible pairs of blocks. The objective of the agent is to explore the resulting configuration space (235 configurations). All episodes start with blocks contact with the table (coplanar configuration).
This exploration problem hard for several reasons. First, the goal space incredibly large. Al- though it is hard to estimate the amount of reachable configurations, the following experiments will show that good agents can discover diversity of these goals. Second, the large majority of goals cannot be reached but the agent cannot tell priori. Third, some goals are much harder to reach than others (e.g. stacks of 5 > coplanar configurations). In fact, the hardest goals cannot be reached by chance following random exploration because they involve time-extended behaviors (stacking 5 blocks one after the other). Up to this day, learning 4+ block stacking often requires specific mechanisms such hand-defined curricula (Li al., 2019), demonstrations (Nair et al., 2018a) 8a) or reward shaping (Popov et al., 2017). Without these specific mechanisms, our purely individual agent only discovers stacks of up to blocks, which is the highest we found in the literature (Akakzia & Sigaud, 2022).
How can an agent efficiently explore that configuration space? Before we answer that question, we shall define our evaluation metrics. Indeed, we cannot list the achievable goals priori and therefore need to define subjective set of evaluation goals. We evaluate the quality of an agent along two dimensions:
Performance metrics: average success rates computed over 10 sets of interesting evalua- tion goals that humans could easily describe, including coplanar configurations, stacks of 2, 3, 5, pyramids and their combinations (see the classification in Table 1). At each eval- uation, the agent is given 20 goal configurations uniformly sampled from each of the 10 evaluation classes (200 goals). Here, we report both the 10 per-class success rate and the global success rate computed as their average (SR).
Social metric: the amount of social interactions involved in the training of socially- sensitive agents, a percentage of all training episodes.
Class
C
C2 45
S2
S3
S2&S2
S 120
P3
Ps&S2
S
Ss
Goals
20
60
60
30
60
120
120
Table 1: Classes of evaluation goals. C, goals regroup all coplanar contigurations where exactly i pairs of blocks are close. goals have exactly stacked blacks. Pt goals have pyramid made of blocks. The & symhol is logical AND. All unspecified predicates are set to false. Classes are disjoint sets and their union does not cover the whole configuration space.
In all subsequent experiments, we train our agents for a fixed budget of 20 hours and periodically evaluate them offline without exploration noise. For all conditions, report the mean and standard deviation computed over seeds. Stars indicate statistical differences with respect to the first condi- tion (leftest in the legend) using Welch's t-test with Ho: difference in the means and 0.05.

---

Under review as a conference paper at ICLR 2023
4.2 DOES THE AGENT NEED SOCIAL PARTNER?
How can we tackle this hard exploration problem? The standard autotelic approach would be to let the agent sample goals uniformly from the set of configurations it already discovered (autotelic baseline). If this does not work, one could augment the agent with an ACI. strategy to refine the goal selection mechanism. Here, we consider two baselines inspired from state-of-the-art ACI. algo- rithms: 1) the selection of goals associated with high learning progress (LP baseline, based on Colas et al. (2019)) and 2) the selection of goals the agent is the most uncertain about (value-disagreement sampling baseline (VDS), based on Zhang et al. (2020)). In the LP baseline, we train a neural net- work to predict the success or failure for any given goal (binary classification, see Section 3) and use to estimate LP. After having estimated LP for a set of 50 goals sampled uniformly from Gknown: the agent pursues the one with the highest I.P. In the VDS baseline, we train 3 Q-value networks instead of one and use their disagreement as a measure of the agent's uncertainty. After having estimated the value disagreement for 1000 goals uniformly sampled from Gknown. the agent samples goal using the softmax distribution of their disagreement scores. Hyperparameters were taken from the original papers (Colas et al., 2019; Zhang et al., 2020).
Figure 2a shows that our proposed algorithm outperforms both the standard autotelic baseline and the two ACL baselines inspired by sfate-of-the-at algorithms (Colas et al., 2019; Zhang al., 2020). Although all conditions seem to demonstrate similar skill learning capacities at first, non-social baselines reach plateau around top row of Figure 3, ACL agents do not even discover the hardest configurations. They are indeed focused on the goals that are currently the most challenging for the agent, which do not represent good stepping stones for the discovery of more complex configurations.
of 0.75 while HME reaches score of 0.93. As shown in the
Social interactions do not only suggest interesting stepping stones (frontier goal), they also point to the adjacent novel goal when the stepping stone is reached (beyond goal). To test the importance of the frontier goals, we introduce HME-50 B, an ablation of our method where the SP only suggests beyond goals. Figure 2b depicts the counts of stepping stones during training for the full HMF-50 and its ablation. By stepping stones, we refer to configurations reached by the agent that are adjacent to unknown configurations in the structured goal space of the SP. Although both methods perform similarly in terms of success rates (orange and blue curves in Figure 2a), HMME50 explores faster than HME-50 B. This suggests that the role of the frontier goal is enable the agent to reach the beyond goal from the first time. The Go- ProExxllore approach from Ecoffet et al. (2019) leverages a similar decomposition. In Go-Explore the stepping stone is selected using a novelty proxy and the following exploration is obtained with random actions. Could this work in our hard exploration problem?
To test this, we compared four ways of selecting stepping stones: 1) uniformly from the set of known configurations; 2) with a bias towards sparse regions of the goal space (novelty proxy); 3) with a bias towards high-LP goals (LP proxy) and 4) in the same way as IIME; i.c. on the way to a beyond goal (social bias). Just like in Go-Explore, all variants first pursue a stepping stone, then explore with random actions from there. Variant 2 is the closest to the original Go-Explor implementation from Ecoffet et al. (2019). Figures 2a and 3 show that the HME agent outperforms
HIME-50
Go-Cae Rand
Go-Dag UP
SMUCE
022
02
Ep Lododds
150
380
Eploodee D000
(a)
(b)
Figure 2: a) Global success rates for HME and different basclines. b) Count of stepping stones for HME-50 with and without the frontier goal suggestions.
7

---

Under review as a conference paper at ICLR 2023
HME-50
LP baseline
VDS beseline
Individual
000
100
100
200
25
EopExp Rand
Go-Exp 55
Go-Exp Nov
100
Go-Exp
75
15-
Figure 3: Exploration metrics for HME, ACI. baselines and Go-Explore baselines. Counts of configurations discovered across training for the hardest classes of evaluation goals.
all these variants and discover wider diversity of contigurations. Comparing condition 4) with conditions 1-3) indicates that the social selection of stepping stone is not enough to power an efficient exploration. Interestingly, we find that HME outperforms condition 4, its variant which replaces exploration towards the beyond goal with random actions. Altogether, these results indicate that HME agents benefit from directing their exploration phase towards socially-suggested beyond goal instead of exploring with random actions.
The results presented in this section show that ACL methods and go-explore strategies cannot place social goal suggestions. The increased exploratory capabilities originate in the combination of returning to known stepping stone and performing subsequent exploration directed towards beyond goal. Only social interactions can give agents access to such knowledge.
4.3 How MUCH SOCIAL INTERACTION DOES THE AGENT NEED?
The active query mechanism lets the agent decide when it needs social goal suggestion from the SP. The amount of social episodes is thus a function both the sensitivity of the agent to stagnation in its perceived LP (8 in Equation equation 1) and of the agent's actual progress in reaching its goals. For 3 = O, the agent purely social and asks for suggestions
the time (social baseline). For 3 the agent is purely individual and never asks for suggestions (individual baseline). In between, the agent is both individual and social (HME-8 for / in [20, 50, 200].
Table 2:
Evaluation metrics for HME variants as a function of their social sensitivity 8.
Metrics
\ Agent
Social (8 0)
RMB 20
RME: 30
HME 50 w/o intern.
HME: 100
HME 200
Indiv, (8 oo)
Social
100
51.62 2.49
6.98 =: 0.58
9.12
0.06
0.001 0.001
Global sR
0.81 = 0.01
0.92 = 0.01
0.93 =: 0.02
0.89 = 0.01
0.87 = 0.03
0.72 0.03
0.75 + 0.05
Table presents the empirical share of social episodes and the global SR for all variants of the HME algorithm and the extreme baselines (purely individual and purely social). Surprisingly, the highest global performance is recorded by HME-3 = 50, an agent experiencing only 6.98% of social episodes. HME-3 100 only requires 0.5% of social episodes to outperform both the purely social and the purely autotelic agents. Figure 4 shows the evolution of per-class SR as a function of training time. The more individual variants (right) struggle in learning the most complex goals such as the S₃ and S₃ classes. If the purely social baselines makes some progress in these classes, it remains less performant and sample-efficies they blend individual and social episodes, these variants manage to learn the most complex goals. The 3 100 variant, especially, makes important progress on the hardest tasks with only 0.5% of social episodes.
than intermediate variants of HME for [20 100]. Because
Figure 5 presents the distribution configurations achieved by the agents during training. We focus on the most complex configurations that can hardly be discovered via random actions: Ss. Sz&Sz, S4 and Ss. The individual baseline, the social baseline and HME- = 200 fail to discover and achieve these goals. On the other hand, HME-3 20, 50, 100 agents able reach these configurations more often. Note that HME-3 50 reaches them often although it benefits from social episodes only 6.98% of the time.
8

---

Under review as a conference paper at ICLR 2023
6.5.
Social
HME-20
HME-50
HNE-100
HME-200
Individual
1.00
1.00
30.75
0.75
0.75
0.75
1.500
0.50
0.50
0.50
9.25
0:20
0.20
300g
100
200
100
200
TUU
200
Episodes (x10")
Episodes (x10")
Episodes (x20")
Episodes bx20")
Episodes (x10")
Episodes (X20")
Figure 4: Per-class SR across training episodes for different social sensitivities, from high (low 3) to low (high ().
S3
Sg & Sg
54
Social
HME-20
HME-59
HME-1000
HME-200
Individual
800
400
400
200
25
500
200-
25
25
Figure 5: Exploration metrics for HME variants. Counts of configurations discovered across training for the hardest classes of evaluation goals
DISCUSSION AND CONCLUSION
This paper makes step towards teachable aulonomous agents: the cognitive development of arti- ficial agents must take its root in both social situatedness and the free exploration of the physical contributions: 1) an autotelic agent that learns dis-
world (Sigaud et al., 2021). It introduces cover and master thousands of semantic configurations, and 2) social interaction protocol where social partner efficiently catalyzes the development the learner.
The agent decides when to query the social partner, which then provides a goal just beyond the cur- rent agent's capabilities and offers an easy step on the way (frontier goal). This protocol efficiently models Vygotsky' concept of zone of proximal development; specific type of interactions where more knowledgeable other scaffolds the experience of a learner to implicitly teach them achieve new kinds of goals and explore their world (Vygotsky, 1978). As we show, these social interactions bring significant advantages in complex worlds where exploration is hard. On the other hand, ACL methods alone are limited to sampling within the set of known goals, which drastically limits the exploration capabilities of autotelic agents.
More generally, human exploration is mnter-generational population- based process supported by social interactions and cultural evolution, phenomenon known as the cultural ratchet (Tomasello, 1999). If we them interact with social partners. From them, they can learn which goals are culturally important; they can discover new goals they never reached and learn recognize good stepping stones towards harder goals.
reproduce human exploration abilities in artificial agents, we need to let
Although the social partner is hand-coded in this work, we hope to let real humans fill this role for our agents. Humans indeed have the natural tendency to adapt their social interactions with young learners as function of the learner's capabilities, knowledge and goals (Vygotsky, 1978). Humans do not represent configurations with similar binary vectors in their mind, but could easily demon- strate a target configuration (beyond goal) and stepping stone configuration that they believe the agent already knows (frontier goal). From these configurations we could extract binary target con- figurations to suggest to the learning agent. The principal limiting factor is the amount required feedback. Although we drastically limit the number of required interactions in this work (0.5% already leads to significant improvements). currently remains too high to involve actual humans (250k episodes gives 12500 social episodes). Better learning architectures with higher transfer pabilities and improved sample efficiency may help bring this number down and open the way for experiments with humans in the loop.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning, arXiv preprint arXiv:1703.0173: 2017.
Ahmed Akakzia and Olivier Sigaud. Learning object-centered autotelic behaviors with graph neural networks. arXiv preprint a0Xhv:2244.05111,0000 2022.
Ahmed Akakzia, Cédric Colas, Pierre Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acqquiee skills via goal generation. arXiv preprint arXiv:2006.07185. 2020.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay, arXiv preprint www.iv:1077.0444 2017.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Breut Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 4765):469-483, 2009.
Adrien Baranes and Pierre- Yves Oudeyer. Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots. Robotics and Autonomous Systems, 61(1):49-74 -3, 2013. Publisher: Elsevier.
Andrew G Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learn- ing in natural and artificial systems, pp. 17-47. Springer, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 1991491-447, 2016.
Laura E Berk. Why children talk to themselves. Scientific American, 271(5):78-83 1994.
Sebastian Blaes, Marin Vlastelica Pogančić, Jiajie Zhu, and Georg Martius. Control what you can: Intrinsically motivated task -okannlnnnng agent, Advances in Neural Information Processing Systems, 32, 2019.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study curiosity-driven learning. arXiv preprint 1rXiv:1808.04333 2018
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Carlos Celemin and Javier Ruiz-del Solar. Coach: Learning continuous actions from corrective ad- vice communicated by humans. 2015 International Conference on Advanced Robotics (ICAR), pp. 581-586. IEEE, 2015.
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv: 2017a.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances neural information processing systems, pp. 4299-4307, 2017b.
Cédric Colas. Towards Vygotskian Autotelic Agents: Learning Skills with Goals, Language and Intrinsically Motivated Deep Reinforcement Learning. PhD thesis, Université de Bordeaux, 2021.
Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre-Y Oudeyer. Cu- rious: intrinsically motivated modular multi-goal reinforcement learning. International con- ference machine learning, pp. 1331-134H PMLR, 2019.
Cédric Colas, Tristan Karch, Nicolas Lair, Sean-Michel Dussoux, Clément Moulin-Frier. Peter Ford Dominey, and Pierre Yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity- driven exploration. arXiv preprint arXiv:2002.09253. 2020.
10

---

Under review as a conference paper at ICLR 2023
Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre- Yves Oudeyer. Autotelic agents with intrin- sically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74:1159 1591 11159-1199 1199, 2022.
Lisandro D. Dalcin, Rodrigo R. Paz, Pablo A. Kler, and Alejandro Cosimo. Parallel distributed computing using python. Advances in Water Resources, 34(9):1124-1133 2011.
Stephane Doncieux, David Filliat, Natalia Slaz-Rodriguez Timothy Hospedales, Richard Duro, Alexandre Coninx, Diederik M. Roijers, Benoît Girard, Nicolas Perrin, and Olivier Sigaud. Open- ended learning: a conceptual framework based on representational redescription. Frontiers in Robotics and AI, 12, 2018. doi: 10.3389/mbot.2088.0000
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprini arXiv: /802.06070, 2018.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. In International conference on machine learning, pp. 1515- 1528, 2018
Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning. arXiv preprint /tl.:/708.02//0 2017.
Sébastien Forestier and Pierre-Yve Oudeyer. Modular Active Curiosity-Driver Discovery of Tool Use. In Intelligent Robots and Systems (IROS), 2016 IEFF/RSI International Conference on, pp- 3965-3972. IREF, 2016.
Pierre Fournier, Cédric Colas, Mohamed Chetouani, and Olivier Sigaud. CLIC: Curriculum learning and imitation for object control in non-rewarding environments. IEEE Transactions Cognitive and Developmental Systems, 13:239-248, 2019.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. arXiv preprint urXiv:1704.01211 2017.
Jonathan Grizou, Manuel Lopes, and Pierre-Yves Oudeyer. Robot learning simultaneously task and how to interpret human instructions. In 2013 IEEE third joint international conference on development and learning and epigenetic robotics (ICDL), pp. 1-8. IEEE, 2013.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint 1.Xi:18000.01990 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Machine learning, 84(1): 171-203, 2011.
John B. Lanier, Stephen MeAleer, and Pierre Baldi. Curiosity-driven multi-criteria hindsight expe- rience replay. CoRR, abs/1906.03710, 2019.
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manip- ulation using relational reinforcement learning. arXiv preprint arXiv: /912 1032, 2019.
Jessica Lindblom and Tom Ziemke. Social situatedness of natural and artificial intelligence: Vygot- sky and beyond. Adaptive Behavior, 11(2):79-96, 2003.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint urXiv:220/.0829) 2022.
Gary Lupyan. Linguistically modulated perception and cognition: The label-feedback hypothesis. Frontiers in psychology, 3:54, 2012.
11

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforeement learning, Nature, 518(7540):529-533 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on roboties and automation (ICRA), pp. 6292-6299 IEEE, 2018a.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv preprint arXiv:1807 1807.04442 2018b.
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervisee robotic learning. In Conference on Robot Learning, pp- 530-539. PMIR, 2020.
Anis Najar, Olivier Sigaud, and Mohamed Chetouani. Training robot with evaluative feedback and unlabeled guidance signals. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communicution (RO-MAN), pp. 261-266. IEEE, 2016.
Sao Mai Nguyen and Pierre-Yvee Oudeyer. Active Choice of Teachers, Learning Strategies and Goals for Socially Guided Intrinsic Motivation Learner. Paladyn, 3(3):136-146, 2012. Pub- lisher: Springer.
Sao Mai Nguyen and Pierre-Yvel Oudeyer. Socially guided intrinsic motivation for robot learning of motor skills. Autonomous Robots, 36(3):273-294, 2014a.
Sao Mai Nguyen and Pierre-Yves Oudeyer. Socially Guided Intrinsic Motivation for Robot Learning of Motor Skills. Autonomous Robots, 36(3):273-294, 2014b. Publisher: Springer.
Pierre-Yve Oudeyer and Frederic Kaplan. What Is Intrinsic Motivation? A Typology of Computa-
tional Approaches. Frontiers in neurorobotics, 1:6, 2007. Publisher: Frontiers.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. arXiv preprint arXiv:1705.05363 2017.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 2144):682-697, 2008
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: /00::7066.0 00000 2017.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv: NNX2::/800.09444 2018.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03:. 2019.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXiv:1704.0307. 2017.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yver Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint wrxiv:2033006664 2020.
Stefan Schaal, Auke Ijspeert, and Aude Billard. Computational approaches to motor learning by im- itation. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 358(1431):537-57,, 2003.
12

---

Under review as a conference paper at ICLR 2023
Tom Schaul, Daniel Horgan, Karol Gregor. and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
Olivier Sigaud, Hugo Caselles-Dupré, Cédric Colas, Ahmed Akakzia, Pierre-Y Oudeyer, and Mohamed Chetouani. Towards teachable autonomous agents. arXiv preprint arXiv:2105.11977 2021.
Luc Steels. The autotelic principle. In Embodied artificial intelligence, pp. 231-244 Springer, 2004.
Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz. Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2 12808 2021.
Freek Stulp and Stefan Schaal. Hierarchical reinforcement learning with movement primitives. In 2011 11th IEEE-RAS International Conference on Humanoid Robots, pp. 231-238. IEEE, 2011.
Alexandr Ten, Pierre- Yves Oudeyer, and Clément Moulin-Frier, Curiosity- exploration: Di- versity mechanisms and functions. In The Drive for Knowledge: The Science of Human Infor- mation Seeking. 2022. doi: 10. 1009026949.
Andrea L Thomaz and Cynthia Breazeal. Teachable robots:
Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence,
2008.
Andrea Lockerd Thomaz, Cynthia Breazeal, et al. Reinforcement learning with human teachers: Ev- idence of feedback and guidance with implications for learning performance. Aaai, volume 6, pp. 1000-1005. Boston, MA, 2006.
Sebastian B. Thrun. Efficient exploration in reinforcement learning. Technical report, Carnegie Mellon University, 1992.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Michael Tomasello. The human adaptation for culture. Annual review of anthropology, 28, 1999.
Michael Tomasello. Becoming human. In Becoming Human. Harvard University Press, 2019.
L. S. Vygotsky. Tool and Symbol in Child Development. In Mind in Society, chapter Tool and Symbol in Child Development, pp. 19-30. Harvard University Press, 1978. ISBN 0674576292. doi: (0.2307/j.ctvff92244
Lev Semenovich Vygotsky and Michael Cole. Mind in society: Development of higher psychologi- cal processes. Harvard university press, 1978.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv: /8/ 1359, 2018
Deena Skolnick Weisberg, Kathy Hirsh-Pasek, and Roberta Michnick Golinkoff. Guided play:
Where curricular goals meet a playful pedagogy. Mind, Brain, and Education, 1(2):104-112, 2013.
David Wood, Jerome Bruner, and Gail Ross. Psychology & Psychiatry & Allied Disciplines, 1976.
The role of tutoring in problem solving. Child
Yue Yu, Patrick Shafto, Elizabeth Bonawitz, Scott C-H Yang, Roberta M Golinkoff, Kathleen H Corriveau, Kathy Hirsh-Pasek, and Fei Xu. The theoretical and methodological opportunities afforded by guided play with young children. Frontiers in psychology, 9:1 152, 2018.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic Curriculum Learning through Value Disagreement. arXiv:2006.0944 096404 [cs, stat), June 2020. URL http: / /twxiv.vvvv..///////...///aaatvv orgg abs/ 2006 09641.
arXiv: 2006.09641
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
PROOF OF CONCEPT EXPERIMENTS ON MAZES
In our preliminary experiments, we ran the HME agent in set of continuous mazes that present hard exploration problems. These experiments act proof of concept for the more complete experiments presented in the main paper.
Environments. We consider a set of three maze navigation environments generated with an ex- isting implementation' based on the MuJoCo physics engine (Todorov et al. (2012), see Figure 6). The agent is represented as a moving point (orange point) that navigates specific areas (blue areas) delimited by thick walls (grey areas). Observations include the agent's position, velocity and ori- entation (6D). Actions control the velocity and orientation of the agent (3D). Goals include target positions for the agent to achieve (2D). Rewards are sparse such that the agent receives a reinforeing signal of +1 whenever they are within threshold 8 = 1cm from the target position.
Four Rooms
Bottleneck Maze
Hard Maze
Figure 6: The 3 mazes considered in our study. They are of increasing difficulty in exploration.
The agent is given an extra representation function ó mapping the state to discrete representation corresponding to the identity of an area or cell.
Experiments. These experiments demonstrate the benefit of using the HIME interaction protocol. We use simpler version of the HME agent that replaces the active query mechanism with fixed probability to query the social partner and which does not internalize socially suggllss goals. On the one hand, we use simple flat MLP networks to model the agent's internal models, including its actor and critic. On the other hand, to model the SP's suggested goals, we construct and annotate cells from the bottom row upwards (see Figure 6). The goals that the SP propose correspond to the centers of these annotated cells. As the goal space is not known a priori, the agent has to 1) explore discover goals and 2) exploit to learn to reach goals. At the beginning of each episode, the agent starts at the center cell number 1.
We experiment with different levels of sociality: SP p% where p is the probability query the social partner (SP) with p E {0, 1, 10, 100)% SP 0% and SP 100% represent the individual and social baseline respectively. We three metrics to evaluate the strategies: 1) the global success rate across all the goal space (known & unknown goals), 2 the local success rates for each goal (heat maps on the mazes) and 3) the goal space coverage (ratio of discovered goals).
Results. Figure 7, presents the results for the first metric (global success across all goal space). The results unequivocally show that the presence of the SP helps the agent reach better perfor- mance. With only 10% of social episodes. the agent able to master all goals on Corridor and Four Rooms. Increasing the number of social episodes does not further improve performance, as the agent with 100% and 10% social episodes performs similarly. Notably, using only 1 1 of social episodes is enough to show satisfying performance, further highlighting the importance of the social partner to discover new goals.
In order to illustrate that the discovery of new goals drives performance, we analyzed the exploration metries across trained agents in Figures 8 and 9. Note that adding as little 1% of social episodes in
14

---

Under review as a conference paper at ICLR 2023
SP 100%
SP 10%
SP 1%
SP 0%
Succezs Rate
Four Rooms
Bottleneck Maze
Hard Maze
Figure 7: Global success rate across all the goal space (known & unknown goals) the four tested mazes.
all mazes drastically improves the local success rate on goals that go beyond the first room. Indeed, thanks the SP. the agents are able pass the bottleneck of the maze and discover the other rooms, while SP 0% stays in the first room because it is not able to explore efficiently. The ratio of discovered goals as function of time plotted in Figures and shows that the agents with social episodes rapidly discover most of the goals while the agent without social episodes does not. For the Hard maze (Figure 9, which considered the most complex among the of mazes we have chosen, I 1% of social episodes enables the ratio of discovered goals to catch up with the ones with 10% and 100% of social episodes. However, the agent still unable to maximize its on all the cells (middle part of Figure 9).
SP 0%
SP 1%
Ratio of discovered Goals
SP 100%
SP 10%
SP 1%
SP 0%
Oyces (4007)
Cyche
Figure 8: Performance Metrics on the Four Rooms (first row) and Bottleneck mazes (sccond row). Left: per cell with 0% social interventions; Middle: SR per cell with 1% social interventions; Right: Coverage of the goal space computed as the ratio of discovered goals.
A.2 REPRESENTING THE SOCIAL PARTNER'S 'S KNOWLEDGE
We model the social partner (SP) by a hard-coded program endowed with the set of goals that are potentially achievable by agent within the five-object manipulation benchmark. We choose to rep- resent this domain knowledge as a directed semantic graph. This facilitates determining stepping stones in the agent's capabilities. However, it is impractical configurations and decide whether pair should be linked or not in graph. Semantic configurations can be infeasible for reasons: 1) they are semantically impossible to obtain-ea, e.g. two objects cannot both be above each other 2) they are physically impossible to achieve e.g. in the case of inverted pyramids.
manually enlist all the imaginable
15

---

Under review as a conference paper at ICLR 2023
SP 0%
SP 1%
Ratio of discovered Goals
100%
SP 10%
13
Cydee
Figure 9: Performance Metrics on the Hard maze. Left: SR per cell with 0% social interventions; Middle: sR per cell with 1% of social interventions; Right: Coverage of the goal space computed as the ratio of discovered goals.
Choose Block
Choose column
configuration new?
Move
MOvE
Move
Figure Simplified example with blocks in grid of oracle tree construction procedure
To avoid enlisting all semantic and physical constraints, we define two types of the SP's goal space:
Oracle Goal Space: we design a 3D grid within which each cell can contain one object. Initially, all the blocks are initialized in different columns of the grid so that they are all far from each other (root node). From this state, we select one object and move it to another column. If that column already contains another object, than the first one will be stacked above the second. By doing action at each step, can extract the current semantic configuration and link to the previous one in the oracle graph. Iteratively repeating this process yields the described process for two blocks. The number of nodes within the oracle goal space is valued 12666.
starting from the rool node. See Figure 10 for 2D illustration of
Minimal Goal Space: this represents a sub-graph of the oracle goal space defined above. It consists of set of goal configurations exclusively including the path from stacks all to stacks of 5 objects for each permutation of objects (No stack at all -> Stack 2 + Stack of 5). The total number of nodes within this minimal goal space is valued 321.
In our main experiments, we use the minimal goal space to represent the SP's knowledge in all our agents except for the social baseline. In fact, our experiments have shown that the oracle goal space
16

---

Under review as a conference paper at ICLR 2023
needed when training includes exclusively social episodes. Note that this engineered process only serves to evaluate the capacities the agent and is not used by the agent itself in any way.
A.3
EVALUATION CLASSES
Close
Close
Stack
Stack
Stack 2&2
Stack 2&3
Pyramid
Pyr 3&Stack2
Stack
Stack
Table 3:
The different semantic classes used in evaluation. The class Close regroups all semantic configura- tions where i pairs of blocks are close.
Table 3 illustrates the evaluation classes presented in the main paper. For the sake of simplicity here, only represent the blocks that concerned by the underlying predicates. All the predicates associated with the other blocks have values set to O. We use a hard-coded function to generate random configuration given the identifier of the considered class. We also dictionary where keys are configurations and values are identifiers of the classes to keep counts of either the sP proposed goals or the agent's encountered and achieved goals.
A.4 OBJECT-CENTEREH ARCHITECTURE IN FETCH MANIPULATE
In the Feich Manipulate environment with 5 blocks we use the GANGSTR agent proposed in Akakzia & Sigaud (2022). It perceives:
1. the low-level geometric states. Since the 5 objects share the same attributes dimensions (positions, velocities, orientations), the behavior with respect to an object should be inde- pendent from the object's 's attributes.
2. the high-level semantic configurations. Since the relations between all the pairs of objects share the same predicates (close, above), the behavior with respect to a binary semantic relation should be independent from the considered pair.
Thus, is natural to encode both object-centered and relational inductive biases in architecture. To do so, we model both the policies and critics of the agents as message passing graph neural networks (MPGNNs) (Gilmer al., 2017). We consider graph of nodes, each representing single object. All the nodes are interconnected, yielding a compact graph of 20 directed edges. Furthermore, we consider the agent's 's body attributes as global features of the policy networks and both the agent's body attributes and the actions global features of the critic networks. A single forward pass through this graph consists in three steps:
1. Message computation performed for each edge. The features of the considered edge are concatenated with the features of the edge's source and target nodes before being fed to first shared neural network N Nedgee
2. Node-wise aggregation is performed for each node. The features of the considered node are concatenated with an aggregation of the updated features of all the incoming edges. The resulting vectors are then concatenated with the global features of the graph before being fed to a second shared neural network N N
3. Graph-wise aggregation is performed once for all the graph. The updated features of all the nodes of the graph are aggregated and fed to third neural network N Needoutt
The aggregating function needs to be permutation invariant. We use pooling for the node-wise aggregation and summation for the graph-wised aggregation. The final output of N N is either the action (in the case of the actor) the Q-value (in the case of the critic).
17

---

Under review as a conference paper at ICLR 2023
A.5 PSEUDO CODE
Algorithms and 2 present the high-level pseudo-code for the individual and social learning
episodes.
Algorithm Individual Learning
Algorithm 2 Social Learning
1:
Require Env E, Initialize policy I1, semantic graph Ga. path estimator PE,, buffer B.
1: Require Env E, social partner SP
2:
2:
Initialize policy II, semantic graph G., path estimator PE, buffer B. loop
3: loop
3:
4:
4 G..sample_nodee node()
g SP ww.prroose-goalee
5: path +- -ample caamppe..pah(g.... path(g, G.)
5: path +- wasssmmll_ppahppee 9.)
6:
loop y: E palh
loop 9: path
7:
trajectory +- Erollout(g.)
7:
trajectory + E.rollout(g:)
8:
g..updat((trjjectory))
8:
G.uudaat((rujectoryy)
9:
(FFuudate(traaecdory))
9:
P (E.upate(((uujecoorr)
10:
B.update((rajectoryy)
10:
B. suppate(trajectory)
11:
H.update(B)
11:
if is a stepping stone and is reached then
12:
return II, PE, 9.
12:
9b SP SP .roppooe w.e.ppoos..unnoowiiee wnw.own(n....................... g)
13:
path < P.E.sample.ppathee G.)
13:
14:
loop 9r E path
14:
15:
trajeclory <- E.ooloou(gg))
16:
G.uundate(rraeectrr)))
16:
17:
PE PE.update(trajeetoyy
18:
B.update(trajectory)
18:
19:
ILupdate(B)
20:
20:
return PE,
A.6
IMPLEMENTATION DETAILS
This part includes details necessary to reproduce the results. An anonymous version of our code will be made available via Anonymous Github.
GNN-based networks. Our object-centered architecture uses two shared networks, N Nodgge and N Naode- respectively for the message computation and node-wise aggregation. Both are 1-hidden- layer networks of hidden size 256. Taking the output dimension to be equal to 3x the input di- mension for the shared networks showed better results. All networks use ReLU activations and the Xavier initialization. We use Adam optimizers, with learning rates 10- The list of hyperparame- ters is provided in Table
Parallel implementation of SAC-HRRRR Our experiments rely on Message Passing Interface (Dalcin al., 2011) to exploit multiple processors. Each of the 24 parallel workers maintains its own replay buffer of size 10t and performs its own updates. Updates updated actor and critic networks are broadcast to all workers. Each worker alternates between 10 episodes of data collection and 30 updates with batch size 256. To form epoch, this cycle is repeated 50 times and followed by the offline evaluation of the agent.
summed over the 24 actors and the
18

---

Under review as a conference paper at ICLR 2023
Table 4: Hyperparameters used in GANGSTR.
Hyperparam.
Description
Values.
nb_mpis
Number of workers
24
nb nb_cycles
Number of repeated cycles per epoch
50
mbbrollouts.pp
Number of rollouts per worker
10
rollouts
Number of episode steps per rollout
40
mb updates
Number of updates per cycle
30
replay strategy
HER replay strategy
final
kepplay
Ratio of HER data to data from normal experience
4
batch_size
Size of the batch during updates
256
y
Discount factor to model uncertainty about future decisions
0.99
Polyak coefficient for target critics smoothing
0.95
Ir iacoor
Actor learning rate Critic learning rate Entropy coefficient used in SAC
10-3
Iy crilicc
10-3
0.2
QEMA
EMA coefficient for SR edge estimation Default value for edges' SR Number of shortest paths sample from Ratio of alternation between shortest and safest paths
0.01
edge prior
0.5
shortest,aths
5
Shortest.sa.fest.uue
0.5
19


Reviews:
[{"review_id": "pzOx3ZNgbUd", "reviewer": "Reviewer_JZLF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn a goal conditioned policy based on goals generated from a pre-defined goals and the agent's own goals. \n", "summary_of_the_review": "Based on the weaknesses listed, I recommend to reject this paper", "strength_and_weaknesses": "Weaknesses:\n1. The paper starts with grandiose claims of tackling \"open-ended learning\". However, open-ended learning involves learning to perform across diverse environments. But the definition of open-ended learning in this work seems restricted only to learning different skills in a given environment. In experiments, it is mostly restricted to goal conditioned environments and learning a goal conditioned policy.\n2. \"But where do goals come from? Almost always, they are sampled from a fixed distribution over a predefined goal space; i.e. they come from an engineer.\" There are numerous works where the goals are NOT generated from a fixed distribution (listed in the references below)\n3. The previous statement makes us believe that in this work, the goals are not generated from a fixed distribution. However, a few paragraphs later, the authors note that \"In this second challenge — the one we focus on — agents must learn to organize their own learning trajectories by prioritizing goals with the objective of maximizing long-term skill mastery.\" i.e, this work focuses on learning a goal conditioned policy from pre-defined goals. \n4. \"In social episodes, a social partner suggests a novel goal to the agent and decomposes it into two consecutive sub-goals: 1) a frontier goal that the agent already discovered and, if it is reached, 2) a beyond goal never achieved by the agent but just beyond the its current abilities.\" The social agent keeps a list of all the goals discovered so far and a list of all the goals to be reached. This is not tractable in most environments.\n5. One of the contributions listed is: \"an active learning mechanism allowing the agent to self-monitor its learning progress and, when it stagnates, query the social partner for a goal suggestion\". This seems like a standard active learning setting and not a novel contribution. \n\nReferences:\n[1] Learning with AMIGo: Adversarially Motivated Intrinsic Goals. Campero et al, 2020\n[2] Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. Sukhbaatar et al, 2017\n[3] Asymmetric self-play for automatic goal discovery in robotic manipulation. OpenAI et al, 2021\n[4] An automatic curriculum for learning goal-reaching tasks. Zhang et al, 2021\n[5] Automatic curriculum learning through value disagreement. Zhang et al, 2020\n[6] Exploration via hindsight goal generation. Ren et al, 2019\n[7]  Automatic goal generation for reinforcement learning agents. Florensa et al, 2018\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "gSIQggeEcy", "reviewer": "Reviewer_qrSi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a goal-shaping method to overcome sparse exploration challenges. The key idea is to provide a goal-conditioned learning agent with both “frontier” and “beyond” goals; but only on demand, when unaided learning stagnates. Empirical demonstrations show this can achieve stacks of 5 blocks in a simulated robotics task. ", "summary_of_the_review": "This is a well-written paper with an interesting new idea, backed up by solid empirical evidence. It has a number of major flaws, and lots of minor ones, however, and I really hope the post-rebuttal version will address them, so that I can advocate for its acceptance with good conscience.\n\n--- update post rebuttal ---\nUnfortunately, the authors did not address my questions in any satisfying way, nor update the submission.", "strength_and_weaknesses": "Strengths\n- The problem is well-framed and motivated\n- The empirical validation is thorough and the results are backing up the proposed methods\n- The head-line result of stacking 5 blocks is compelling\n- The links to social learning are interesting and likely to inspire follow-up work\n\nWeaknesses\n- The specific “social partner” is broadly named, but quite tailored to the specific stacking task: how general is the approach if there is a less obvious decomposition into sequential stepping stones (e.g. how could this system teach an agent how to play StarCraft?)\n- I think the paper should discuss its limitations more explicitly: e.g. how important is the full observability assumption made in Sec 3.1? \n- There is a lack of clarity on some important aspects, e.g. around episode boundaries, termination, whether reward is accumulated after a goal is achieved, whether each episode is reserved for a single goal, etc.\n- The reliable success on 5-block stacking is great: why not try 6 blocks to see where/whether the method breaks?\n- The hand-crafted evaluation set is a bit disappointing, would there not be a way to define criteria for “interesting, non-overlapping, complex goals” and have the set emerge from data? That would boost generality of the methodology?\n- The on-demand SP querying section seems very ad-hoc: why does Eq (1) contain an exponential if its argument never strays far from 0 (i.e it is essentially linear)? How does the LP difference take into account the fact that the early and later goal sets may differ? How important/justified is the indirection to compress success rates into a neural network instead of just using counts to determine LP? (and how much does its generalization rely on the goal representation?) Also, how do you prevent biased predictions in that network, caused by under-training on rare goals?\n\nSmaller concerns\n- It might be nice to tie back the conclusion to the first sentence of the intro: what type of open-ended learning is feasible now that wasn’t before?\n- Don’t introduce the identical mathcal{C} and mathcal{G} under two notations, pick one and clarify that paragraph\n- The last paragraph of Sec 3.1 is both vague (exploration, discovery, mastery are all undefined) and incorrect: you don’t seek the “maximum number of goals”, but rather mastery of a specific set of hard-to-reach ones.\n- Fig 1: how do discovered goals inform the social partner (top arrow)?\n- End of Sec 1: mention this is a simulated robotics environment, not an actual robotic arm\n- The references are quite heavily relying on a particular Bordeaux research group, you might want to re-balance those a little.\n- Sec 3.1: I’d recommend using indicator function notation for the reward, instead of the pythonic “==”.\n- Is the query probability re-assessed “each new episode” as stated in Sec 3.3, or held constant actress chunks of 6000 episodes?\n- Sec 4.1: how do you sample 20 goals from the C1 set that just contains 10 (Table 1)?\n- VDS: how do the 3 nets preserve diversity? What’s the softmax temperature? Why once do max-over-50 and once softmax-over-1000?\n- Sec 4.2 “indeed [not] good stepping stones”: can you show this?\n- Can you add a plot of P_query over the course of training?\n- Fig 3 could benefit from log-scale, and definitely is confusing to have one out of 8 subplot use a different y-scale to the rest…\n- Table 2 would probably be clearer as an xy-plot?\n- Table 2: what is “w/o intern”? The results you would have gotten without the help of an intern :-)\n- Please use 7% instead of 6.98% – given the large error-bars, they are equally correct, but one of them is readable too.\n- Sec 5: 0.5% is 1250 episodes, not 12500?\n- There are multiple duplicate citations\n- Appendix A1: either refer to what it adds to the paper from the main text, or omit the section\n- Table 4: what are “shortest and safest paths”? Is is a table that belongs into another paper?\n\nTypos\n- “lagents” (twice)\n- numerous missing commas, namely in front of “which” clauses\n- “leftest” -> “left-most”\n- discoverS a wider\n- Vygotsky’S concept\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NYoVNrbo547", "reviewer": "Reviewer_sim2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an exploration method inspired by the importance of social learning in humans. The method ‘Help Me Explore’ combines individual epodes for practicing already known goals with active learning via ‘social queries’ for goals just beyond the agent’s current abilities, as suggested by a social partner. Specifically, social queries are made when the learning progress on known goals is low. The primary task the paper focuses on is a 5 block rearrangement task.", "summary_of_the_review": "I recommend a 5. While the idea is interesting, the paper would be strengthened with a wider set of experimental environments (where it would be harder to hard-code a social partner) and/or querying humans as social partners. Currently the comparison against ACL methods seem a little unfair, as those methods do not make use of expert knowledge of what the zone of proximal development would be, but HME does, with the hard-coded social partner.", "strength_and_weaknesses": "Strengths\n- The underlying intuition of querying a ‘social partner’ for goal suggestions just beyond the agent’s capabilities makes sense, and learning when to query for that information with a mostly autotelic agent is an interesting problem. \n- The chosen task seems difficult and interesting, and the authors show ablations across various settings of their method to justify the mixture of social and autotelic learning.\n\nWeaknesses\n- It’s not clear how well this method extends beyond the block stacking environment, as (from my understanding) the social partner has to be hard-coded with underlying knowledge of the task to create a zone of proximal development which is used to select the ‘beyond’ goals. This creates a bit of an unfair comparison with the baseline ACL methods, which propose goals without any underlying expert knowledge of the task / difficulty progression.\n- Not clear what authors mean by ‘current ACL strategies are all limited to generate goals from distribution of effects agent has already experienced’ — eg. methods like GoalGAN (Florensa et al., 2018) try to generate goals of intermediate difficulty, which may not be goals the agent has seen/achieved before.\n- Furthermore, in some tasks it may not be straightforward to create a graph showing progression of goal difficulties, as with block stacking — e.g. locomotion across changing/mixed terrain. Rather than directly giving expert knowledge, perhaps another way of showing the usefulness of social learning without necessarily using expert knowledge is to query a human teacher instead, or to simulate imperfect social partners by injecting some noise into the proposed ‘beyond’ goals?\n\nAdditional Feedback\n- Figure 1 could be made larger, currently the text is hard to read. Same with Figure 2.\n- Fourth line under 3.2 typo: goal-conditioned ‘lagent’", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-b1RvT8J9u", "reviewer": "Reviewer_LbgX", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper argues that artificial agents may benefit from socially-guided skill discovery. This paper presents a protocol for active learning that enables an agent to query a partner for a goal. The partner is an auxiliary agent that maintains a model of the probability that an agent could reach goals in the goal-space. The paper claims that this agent outperforms existing baselines, including curriculum-based agents.", "summary_of_the_review": "* In several places, the claims of the paper extend beyond the evidence\n* plots are not clearly explained, making it a challenge to interpret the empirical claims\n\nFor these reasons, I suggest a strong reject.", "strength_and_weaknesses": "Strengths:\n\n1. This paper takes care to embed its understanding social agents within the broader interdisciplinary literature.\n\nConcerns:\n\n1. **In figure 2a, the success rate of multiple agents is depicted across episodes. There are a few aspects of this plot that I find concerning**\n\ni. I am concerned that the experiment was run over hours rather than a standardised number of time-steps or episodes. By limiting the learning by wall-time, both the algorithm and the implementation of an agent are being evaluated. This means that code-level implementation choices that are independent of the learning method being evaluated will influence the reported performance. This is not a fair comparison.\n\nii. I am concerned that the results are reported over 5 seeds. Reporting the standard deviation over results for so few seeds is misleading. Moreover, given the results are overlapping in many locations, it's difficult for me to conclude that HME outperforms the comparators.\n\niii. I am struggling to understand the star notation in figure 2a. In this case, I'm not sure what the \"left-most\" algorithm is. I am assuming it is HME-50. It's unusual to report statistical differences across time-series data as presented. It would be better to report over summary statistics (error after learning). I have doubts that with 5 seeds that the data meets the requirements for a T-test (e.g., normality). The paper should provide information that demonstrates that this data meets the assumptions for the tests that have been done. \n\nyou're more likely to find a statistically significant difference when you're making many comparisons. To account for this, you should use a Bonferroni correction (or similar). In that case, it would require dividing by the number of comparisons made. As a result alpha will be less than 0.05, I believe. This would mean that the evaluation episodes for which there is a statistically significant difference would decrease. \n\nReporting the actual P values would be helpful in this case.\n\niv. At this point, I might wonder why the confidence intervals were not plotted, given the comparisons made. If you have the confidence intervals, why would standard deviation be reported for the error bars?\n\nv. It is claimed that \"non-social baselines reach a plateau of around 0.75\". Examining the figure, it seems that all but go-exp ss achieve scores higher than 0.75, and that the average final performance of most are somewhere around 0.8-0.9. This looks like a misrepresentation of the data. \n\nvi. It looks like HME-50 and HME-50b perform equitably in terms of success rate. This suggests that there is little difference in final performance between suggesting goals that are exclusively beyond the agent's perceived skill level, and suggesting goals that are at the frontier of the agent's skill.\n\n2. **some aspects of the evaluation strategy are unclear**\n\nThe agent is evaluated on \"interesting evaluation goals that humans could easily describe\". What makes a goal interesting or easy to describe isn't elaborated on, making it challenging to assess whether this is a reasonable evaluation strategy.\n\n3. **Figure 2b could be interpreted in several ways**\n\nI am struggling to understand what is being reporting in figure 2b. To my understanding, It counts the number of positions that are reached by the agent that are---according to the SP agent---adjacent to unknown configurations (configurations that have yet to be achieved?). In this case, HME-50 B reaches more stepping stone configurations across most episodes with lower variance(?) than HME-50. If I understand this correctly, it means that the agent is reaching more positions that are perceived by the SP to be at the frontier of the underlying agent's ability. \n\nit is concluded that *\"this suggests that the role of the frontier goal is to enable the agent to reach the\nbeyond goal from the first time.\"* If I understand this claim correctly, we cannot conclude this from the data presented. All we can say is that the agent that is proposing goals beyond its skill level is able to reach the frontier more regularly. \n\nA plot that depicts the number of new goals an agent reaches, or the rate at which the frontier expands, would be better able to support the claims being made in the paper, to my understanding.\n\ni. Figure 3 is mentioned twice in the text, from what I can tell. There is no Y axis labeled, and there is no description in the caption: I cannot tell what is being plotted here without hunting through the text. In the text an average success metric, and a social request metric are defined. I don't know what it means for the agents to have a score of 600 in one of the categories. I do not know what the error bars are. \n\nLabelling is absent from figure 5 as well.\n\nii. The axes are different for each sub-plot, making them difficult to visually compare.\n\niii.  \"As shown in the top row of Figure 3, ACL agents do not even discover the hardest configurations.\" In the top row of figure 3, I see HME-50 included. It would be helpful if you direct the reader's attention to specific sub-figures.\n\n4. **It is unclear why 50 was the beta parameter chosen for the baseline comparisons.**\nExamining figure 5  seems to suggest that HME-20 has the best performance (although there is higher variance).\n\n5. (stylistic suggestion) Figure 4 is challenging to interpret. In this figure, the success rate for all beta values chosen is presented. In this case, I cannot readily see the difference between the beta values---especially for the less complex stacking classes---because all of the lines overlap. Grouping the lines by category and presenting all the beta values on a single plot would help with the interpretability. Right now, it is challenging for me to see the difference between the success rates. For instance, one plot with HME 20, 50, 100, 200 plotted for the success rate of S4.\n\n6. **Frequently the claims of the paper extend beyond the truth.**\n\nIn a couple of locations *very* strong claims are made.\n\n* *agents must learn to organize their own learning trajectories by prioritizing goals\nwith the objective of maximizing long-term skill mastery*\n\n*Must* is a very strong word. Agents may benefit from prioritising goals, but it's not a necessity. Certainly many sparse reward problems have been solved by other means. \n\n* *[e-greedy action selection] is not enough to experience informative rewards in hard exploration problems*\n\nIf an agent is following a random behaviour policy, it can experience reward in sparse environments (as defined in the paper). It might be highly improbable, but it is certainly not impossible. \n\nMinor points:\n\n1. Terms are used without being defined (e.g., SP is used several times before being defined in 3.2)\n2. Some of the citations feel a bit strange. For instance, Mnih et al. 2015 is used as a citation for epsilon-greedy action selection.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject