Grounding Language to Autonomously-Acquired Skills via Goal Generation
Authors: Ahmed Akakzia
Cédric Colas
Pierre-Yves Oudeyer
Mohamed CHETOUANI
Olivier Sigaud
Venue: ICLR 2021 Poster (2021)
Keywords: Deep reinforcement learning
intrinsic motivations
symbolic representations
autonomous learning

Abstract: We are interested in the autonomous acquisition of repertoires of skills. Language-conditioned reinforcement learning (LC-RL) approaches are great tools in this quest, as they allow to express abstract goals as sets of constraints on the states. However, most LC-RL agents are not autonomous and cannot learn without external instructions and feedback. Besides, their direct language condition cannot account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input. To resolve these issues, we propose a new conceptual approach to language-conditioned RL: the Language-Goal-Behavior architecture (LGB). LGB decouples skill learning and language grounding via an intermediate semantic representation of the world. To showcase the properties of LGB, we present a specific implementation called DECSTR. DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects. In a first stage G -> B, it freely explores its environment and targets self-generated semantic configurations. In a second stage (L -> G), it trains a language-conditioned  goal generator to generate semantic goals that match the constraints expressed in language-based inputs. We showcase the additional properties of LGB w.r.t. both an end-to-end LC-RL approach and a similar approach leveraging non-semantic, continuous intermediate representations. Intermediate semantic representations help satisfy language commands in a diversity of ways, enable strategy switching after a failure and facilitate language grounding.

Published as a conference paper at ICLR 2021
GROUNDING LANGUAGE TO AUTONOMOUSLY- ACQUIRED SKILLS VIA GOAL GENERATION
Ahmed Akakzia"
Cédric Colas" Inria
Sorbonne Université
ahmed. a akakzia upme. fr
cedric.colas@img
Peere--Yees Oudeyer
Mohamed Chetouani Sorbonne Université
Olivier Sigaud
Inria
Sorbonne Université
ABSTRACT
We are interested in the autonomous acquisition of repertoires of skills. Language- conditioned reinforcement learning (LC-RL) approaches great tools this quest, as they allow to express abstract goals as sets constraints on the states. However, most LC-RI. agents not autonomous and cannot learn without ternal instructions and feedback. Besides, their direct language condition can- not account for the goal-directed behavior of pre-verbal infants and strongly lim- its the expression of behavioral diversity for given language input. To resolve these issues, we propose a new conceptual approach language-conditioned RL: the BaaaageeGGal-BBhhave
architecture (LGB). LGB decouples skill learning and language grounding via an intermediate semantic representation of the world. To showcase the properties of LGB, present specific implementation called DECSTR. DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical ob- jects. In a first stage (G-B), it freely explores its environment and targets self- generated semantic configurations. In second stage (L->G), it trains language- conditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs. We showcase the additional properties of LGB w.r.t. both an end-to-end LC-RL approach and similar approach leverag- ing non-semantic, continuous intermediate representations. Intermediate seman- tic representations help satisfy language commands in a diversity of ways, enable strategy switching after failure and facilitate language grounding.
INTRODUCTION
Developmental psychology investigates the interactions between learning and developmental pro- cesses that support the slow but extraordinary transition from the behavior of infants to the sophis- ticated intelligence of human adults (Piaget 1977; Smith & Gasser, (005).. Inspired by this line of thought, the central endeavour of developmental robotics consists in shaping set of machine learning processes able to generate similar growth of capabilities in robots (Weng et al. 2001] Lungarella al. 2003). In this broad context, learning agents able to: 1) explore open-ended environments and grow repertoires of skills in self-supervised way and learn from
more specifically interested in designing
via language commands.
The design of intrinsically motivated agents marked a major step towards these goals. The Intrin- sically Motivated Goal Exploration Processes family (IMGEPs). for example, describes embodied agents that interact with their environment at the sensorimotor level and are endowed with the ability represent and set their own goals, rewarding themselves over completion (Forestier et al., 2017). Recently, coal-conditioned reinforcement learning (GC-RI.) appeared like viable way to implement IMGEPs and target the open-ended and self-supervised acquisition of diverse skills.
"Equal contribution.

---

Published as a conference paper at ICLR 2021
coal-conditioned RL approaches train condiconnd policies to target multiple goals (Kaelbling 1993 Schaul et al. (2013). While most GC-RL approaches express goals as target features (e.g. target block positions (Andrychowiez al. 2017), agent positions in maze (Schaul et al. 2013) or target images (Nair et al. 2018)). recent approaches started to use language to express goals, as language can express sets of constraints on the state space (e.g. open the red door) in more abstract and interpretable way (Luketina et al. 2019).
However, most GC-RL approaches and language- ones (LC-RL) in particular are not intrin- sically motivated and receive external instructions and rewards. The IMAGINE approach is one of the rare examples of intrinsically motivated LC-RL approaches (Colas et al. 2020). In any case, the lan- guage condition suffers from three drawbacks. 1) It couples skill learning and language grounding. Thus, cannot account for goal-directed behaviors in pre-verbal infants (Mandler 1999). 2) Direct conditioning limits the behavioral diversity associated to language input: a single instruction leads to low diversity of behaviors only resulting from the stochasticity of the policy or the environment. 3) This lack of behavioral diversity prevents agents from switching strategy after failure.
To circumvent these three limitations, one can decouple skill learning and language grounding via an intermediate innate semantic representation. On one hand, agents can learn skills by targeting configurations from the semantic representation space. On the other hand, they can learn to generate valid semantic configurations matching the constraints expressed by language instructions. This generation can be the backbone of behavioral diversity: a given sentence might correspond to a whole set of matching configurations. This is what propose in this work.
Contributions. We propose a novel conceptual RL architecture, named LGB for Language- Behavior and pictured in Figure (right). This LGB architecture enables an agent to decouple the intrinsically motivated acquisition of a repertoire of skills (Goals > Behavior) from language grounding (Language Goals), via the use of semantic goal representation. To our knowledge, the LGB architecture is the only one to combine the following four features:
is intrinsically motivated: it selects its own (semantic) goals and generates its own rewards, It decouples skill learning from language grounding, accounting for infants learning,
can exhibit a diversity of behaviors for any given instruction,
It can switch strategy in case of failures.
Besides, we introduce an instance LGB, named DECSTR for DEep sets and Curriculum with SemanTic goal Representations. Using DECSTR, showcase the advantages of the conceptual decoupling idea. In the skill learning phase, the DECSTR agent evolves in a manipulation envi- ronment and leverages semantic representations based predicates describing spatial relations be- tween physical objects. These predicates are known to be used by infants from a very young age (Mandler 2012). DECSTR autonomously learns to discover and master all reachable configurations in its semantic representation space. In the language grounding phase, we train a Conditional Vari- ational Auto-Encoder (C-VAE) to generate semantic goals from language instructions. Finally, we can evaluate the agent in an instriction-oolowinn phase by composing the two first phases. The experimental section investigates three questions: how does DECSTR perform in the three phases? How does it compare to end-to-end LC-RI. approaches? Do we need intermediate representations to be semantic? Code and videos can be found at htttts://sites.gooost wwt wttp://...c////c/e//s///tt
RELATED WORK
Standard language-condittooee RL. Most approaches from the LC-RL literature define instruc- tion following agents that receive external instructions and rewards (Hermann et al. 2017 Chan et al. 2019 BBahaanau et al. 2018,, Cideron et al.] 2019 Jiang et al.] 2019 Fu et al. 2019). except the IMAGINE approach which introduced intrinsically motivated agents able to set their own goals and to imagine new ones (Colas et al. 2020). In both cases, the language-condition prevents the decoupling of language acquisition and skill learning, true behavioral diversity and efficient strar- egy switching behaviors. Our approach is different, skill learning. The langunge-conditione goal generation allows behavioral diversity and strategy switching behaviors.
can decouple language acquisition from
2

---

Published as a conference paper at ICLR 2021
Language - Behavior (LB) or language-conditioned RL
Language Goal Behavior (LGB)
Known Semantic Goals
Langage Encoder
language
Langage Encoder
Inst
language embbedding poal
Semantic Goal
OR semantic goal
initial state
Generator
Policy
Policy
state
state
action
LG phase: language grounding
6B phase: skill lcaming
(anguage - semansic goals)
[semantic goals - behavior)
LB phase: instruction following (language + behavior)
LGB phase: followin--
(aaguage + semantic goals
behaviori
Figure 1: standard language-conditione RL architecture (left) and our proposed LGB architecture (right).
Goal-conditioned RL with target coordinates for block manipulation. Our proposed imple- mentation of I.BB, called DECSTR, evolves block manipulation domain. Stacking blocks is one of the earliest benchmarks in artificial intelligence (e.g. [Sussman (1973); Tate et (197)) and has led to many simulation and robotics studies (Deisenroth et al. 201 Xu et al. [0118] Colas et al.) 2019a). Recently, Lanier et al. (2019) and Li al. 2019) demonstrated impressive results by stack- ing up to and blocks respectively. However, these approaches are not intrinsically motivated, involve hand-defined curriculum strategies and express goals specific target block positions. In contrast, the DECSTR agent is intrinsically motivated, builds its own curriculum and uses semantic goal representations (symbolic or language-based) based spatial relations between blocks.
Decoupling language acquisition and skill learning. Several works investigate the use of seman- tic representations to associate meanings and skills (Alomari al., 2017 Tellex et al. 2011 Kulick et al. 2013). While the two first use semantic representations as an intermediate layer between lan- guage and skills, the third one does not use language. While DECSTR acquires skills autonomously, previous approaches all use skills that are either manually generated (Alomari et al. (007)), hand- engineered TTllex et al. 2011) Th or obtained via optimal control methods (Kulick et 2013). Closer to us. |Lynch & Sermanet (2020) also decouple skill learning from language acquisition in a goal- conditioned imitation learning paradigm by mapping both language goals and images goals a shared representation space. However, this approach is not intrinsically motivated as it relies dataset of human tele-operated strategies. The deterministic merging of representations also limits the emergence of behavioral diversity and efficient wrategy-switchine behaviors.
METHODS
This section presents proposed Sanguage-Goal-Behaae architecture (LGB) represented in Fig- [ 3.T) and a particular instance the LGB architecture called DECSTR. first present the environment it is set in [3.2] then describe the implementations of the three modules com- posing any LGB architecture: 1) the semantic representation [3.3]: 2) the intrinsically motivated
goal-conditioned algorithm 3.4 and 3) the language-conditionnd goal generator 3.5
We finally
present how the three phases described Figure are evaluated 3.6
3.1 THE LANGUAGE-GGALL-OO
ARCHITECTURE
The LGB architecture composed of three main modules. First, the semantic representation defines the behavioral and goal spaces of the agent. Second, the intrinsically motivated GC-RL algorithm in charge of the skill learning phase. Third, the Iagguage-conditionn of the language grounding phase. Both phases can be combined in the instruction following phase. The three phases are respectively called G-+B for Goal + Behavior, L-> for Language - Goal and L- for Language Goal + Behavior, see Figure D and Appendix A Instances of the LGB architecture should demonstrate the four properties listed in the introduction: 1) be intrinsically motivated; 2) decouple skill learning and language grounding (by design); 3) favor behavioral di- versity; 4) allow strategy switching. We argue that any LGB algorithm should fulfill the following constraints. For LGB to be intrinsically motivated (1), the algorithm needs integrate the generation
goal generator in charge
3

---

Published as a conference paper at ICLR 2021
and selection of semantic goals and to generate its own rewards. For LGB demonstrate behavioral
diversity and strategy switching (3, 4), the language-conditigg conditiond goal generator must efficiently model the distribution of semantic goals satisfying the constraints expressed by any language input.
3.2 ENVIRONMENT
The DECSTR agent evolves in the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO (Todorov et al. 2012) and derived from the Fetch tasks (Plappert al.. 2018),
Figure 2 Actions are 4-dimensional: 3D gripper velocities and grasping velocity. Observations include the Cartesian and angular positions and velocities of the gripper and the three blocks. In- spired by the framework of Zone of Proximal Development that de- scribes how parents organize the learning environment of their chil- dren (Vygotsky 1978), we let social partner facilitate DECSTR's exploration by providing non-trivial initial configurations. After a first period of autonomous exploration, the social partner initializes the scene with stacks of blocks 21% of times, stacks of 3 blocks 9% of times, and a block is initially put in the agent's gripper 50% of times. This help is not provided during offline evaluations.
Figure 2:
Example configura- tions. Top-right: (111000100).
3.3 SEMANTIC REPRESENTATION
Semantic predicates define the behavioral space. Defining the list of semantic predicates is defining the dimensions of the behavioral space explored by the agent. replaces the traditional definition of goal spaces and their associated reward functions. We believe it is for the best, as does not require the engineer fully predict all possible behaviors within that space, know which behaviors can be achieved and which ones cannot, nor to define reward functions for each of them.
Semantic predicates in DECSTR. We assume the DECSTR agent to have access to innate seman- tic representations based on list of predicates describing spatial relations between pairs of objects in the scene. We consider two of the spatial predicates infants demonstrate early in their develop- ment (Mandler 2012): the close and the above binary predicates. These predicates are applied to all permutations of object pairs for the 3 objects we consider: 6 permutations for the above predicate and 3 combinations for the close predicate due to its order-invariance. the concatenation of the evaluations of these 9 predicates and represents spatial relations between objects in the scene. In the resulting semantic configuration space {0, 1)9, the agent can reach 35 physically valid configurations, including stacks of 2 or 3 blocks and pyramids, Figure 2 The binary reward function directly derives from the semantic mapping: the agent rewards itself when its current configuration Cp matches the goal configuration Cp g. Appendix B provides formal definitions and properties of predicates and semantic configurations.
semantic configuration is
examples in
3.4 INTRINSICALLY MOTIVATED GOAL-CONDITTONER REINFORCEMEN LEARNING
This section describes the implementation of the intrinsically motivated goal-conditioned RL module in DECSTR. is powered by the Soft-Actor Critic algorithm (SAC) (Haarnoja et al.. 2018) that takes as input the current state, the current semantic configuration and the goal configuration, for both the critic and the policy. use Hindsight Experience Replay (IER) to facilitate transfer between goals (Andyychowicz al. 2017). DECSTR samples goals via its curriculum strategy, collects experience in the environment, then performs policy updates via SAC. This section describes two particularities RI. implementation: the self-generated goal selection curriculum and the object-centered network architectures. Implementation details and hyperparameters can be found in Appendix C
of
Goal selection and curriculum learning. The DECSTR agent can only select goals among the set of semantic configurations it already experienced. We automatic curriculum strategy
4

---

Published as a conference paper at ICLR 2021
(Portelas et al. 2020) inspired from the CURIOUS algorithm (Colas et al. 2019a). The DECSTR agent tracks aggregated estimations of its competence (C) and learning progress (LP). Its selection of goals to target during data collection and goals to learn about during policy updates (via HER) is biased towards goals associated with high absolute I.P and low C.
Automatic bucket generation. To facilitate robust estimation, LP is usually estimated sets of goals with similar difficulty or similar dynamics (Forestier et al. 2017 Colas et al. 2019a). While previ- ous works leveraged expert-defined goal buckets, we cluster goals based on their time of discovery, as the time of discovery is a good proxy for goal difficulty: easier goals are discovered earlier. Buckets are initially empty (no known configurations). When an episode ends in new configura- tion, the N₂ = 5 buckets are updated. Buckets are filled equally and the first buckets contain the configurations discovered earlier. Thus goals change buckets as new goals are discovered.
Tracking competence, learning progress and sampling probabilities. Regularly, the DECSTR agent evaluates itself on goal configurations sampled uniformly from the set of known ones. For each bucket, it tracks the recent history of past successes and failures when targeting the corresponding goals (last W = 1800 self-evaluationns. C is estimated as the success rate over the most recent half of that history C = Creent- LP is estimated as the difference between Greeet and the one evaluated over the first half of the history (Ceatlirr) This is a crude estimation of the derivative of the C curve Cearlier. The sampling probability P1 for bucket i is:
w.r.t. time: LP Crocent
(1 (1- C+) * L.P
P.
E,(1- * ILPI))
In addition the usual I.P bias (Colas et al. 2019a), this formula favors lower C when LP is similar. The absolute value ensures resampling buckets whose performance decreased (e.g. forgetting).
Object-centered architecture. Instead of fully-connected or recurrent networks, DECSTR uses for the policy and critic an object-centered architecture similar to the ones used in Colas et al. (2020); Karch et al. (2020), adapted from Deep-Sets (Zabeer et al., (017).. For each pair of objects. a shared network independently encodes the concatenation of body and objects features and current and target semantic configurations, see Appendix Figure 4 This shared network ensures efficient transfer of skills between pairs of objects. A second inductive bias leverages the symmetry of the behavior required to achieve above(o;, 0z) and above(oj, o;). To ensure automatic transfer between the two, present half of the features (e.g. those based on pairs (o;, 0;) where i < .)) with goals containing one side of the symmetry (all above(o;, 0y) for i < j) and the other half with the goals containing the other side (all above(o), o:) for < j). As a result, the above(o,, 0,) predicates fall into the same slot of the shared network inputs as their symmetric counterparts above(oj, 0,), only with different permutations of object pairs. Goals are now of size close and above predicates, corresponding to one side of the above symmetry. Skill transfer between symmetric predicates are automatically ensured. Appendix C.I further describes these inductive biases and our modular architecture.
3.5 LANGUAGE GOAL GENERATION
The inguage-connitioons goal generation module (LGG) is generative model of semantic repre- sentations conditioned by language inputs. It is trained to generate semantic configurations matching the agent's initial configuration and the description of a change in one object-pair relation.
A training dataset is collected via interactions between a DECSTR agent trained in phase G-BB and social partner. DECSTR generates semantic goals and pursues them. For each trajectory, the social partner provides a description d of one change in objects relations from the initial configuration da to the final one ey. The set of possible descriptions contains 102 sentences, each describing, in a simplified language, a positive or negative shift for one of the 9 predicates (c.g. get red above green). This leads to a dataset of 5000 triplets: (cs, d, cr). From this dataset, the LGG is learned using a conditional Variational Auto Encoder (C-VAE) (Sohn et al., 2015). Inspired by the context- conditioned goal generator from| et al.) 2019). we add an extra condition on language instruction to improve control on goal generation. The conditioning instruction is encoded by recurrent net- work that is jointly trained with the VAE via a mixture of Kullback-Leible and cross-entropy losses. Appendix C.2 provides the list of sentences and implementation details. By repeatedly sampling the LGG, a set of goals is built for any language input. This enables skill diversity and strategy switching: if the agent fails, it can sample another valid goal fulfill the instruction, effectively
5

---

Published as a conference paper at ICLR 2021
switching strategy. This also enables goal combination using logical functions of instructions: and is an intersection, or is an union and not is the complement within the known set of goals.
3.6 EVALUATION OF THE THREE LGB PHASES
Skill learning phase G->B: DECSTR explores its semantic representation space, discovers achiev- able configurations and learns to reach them. Goal-specific performance is evaluated offline across learning as the success rate (SR) over 20 repetitions for each goal. The global performance is measured across either the set of 35 goals or discovery-organizze buckets of goals, see Section
Language grounding phase L-+G: DECSTR trains the LGG to generate goals matching constraints expressed via language inputs. From a given initial configuration and given instruction, the LGG should generate all compatible final configurations (goals) and just these. This is the source of behavioral diversity and strategy switching behaviors. To evaluate LGG, we construct synthetic, oracle dataset O of triplets (q. d. Cy(c,, d)). where Cy(c, d) is the set of all final configurations compatible with (ci, d). On average, Cj in O contains 16.7 configurations, while the training dataset D only contains 3.4 (20%). We are interested two metrics: 1) The Precision is the probability that goal sampled from the LGG belongs to Cy (true positive / all positive); 2) The Recall is percentage of elements from C that were found by sampling the LGG 100 times (true positive all true). These metries computed on 5 different subsets of the oracle dataset, each calling for a different type of generalization (see full lists of instructions in Appendix
1. Pairs found in D, except pairs removed to form the following test sets. This calls for the extrap- olation of known initialization-effeet pairs (c,, d) to new final configurations ey (D contains only 20% of Cj on average).
2. Pairs that were removed from D, calling for a recombination of known effects d on known
3. Pairs for which the o was entirely removed from D. This calls for the transfer of known effects
d on unknown C;.
4. Pairs for which the was entirely removed from D. This calls for generalization in the language space, to generalize unknown effects d from related descriptions and transpose this to known c;,
5. Pairs for which both the &; and the d were entirely removed from D. This calls for the general- izations 3 and 4 combined.
Instruction following phase L-+G>B: DECSTR is instructed modify an object relation by one of the 102 sentences. Conditioned on its current configuration and instruction, it samples compati- ble goal from the LGG, then pursues it with its goal-conditioned policy. We consider three evaluation settings: 1) performing single instruction; 2) performing sequence of instructions without failure; 3) performing a logical combination instructions. The transition setup measures the success rate of the agent when asked to perform the 102 instructions 5 times each, resetting the environment each time. In the expression setup, the agent is evaluated on 500 randomly generated logical functions of sentences, the generation mechanism in Appendix C In both setups, evaluate the perfor- mance in I-shot (SR,) ) and 5-shot (SR5) settings. In the 5-shot setting, the agent can perform strategy switching, sample new goals when previous attempts failed (without reset). In the sequence setup, the agent must execute 20 sequences of random instructions without reset (5-shot). We also test be- havioral diversity. We ask DECSTR to follow each of the 102 instructions 50 times each and report the number of different achieved configurations.
4 EXPERIMENTS
How does DECSTR perform in the three phases? [4.2]: : How does compare to end-to-end language-conditionel approaches?
Our experimental section investigates three questions: 4.1
4.3 -Do we
need intermediate representations to be semantic?
4.1 HOW DOES DECSTR PERFORM IN THE THREE PHASES?
This section presents the performance of the DECSTR agent in the skill learning, language grounding, and instruction following phases.

---

Published as a conference paper at ICLR 2021
Skill learning phase G->B: Figure shows that DECSTR successfully masters all reachable figurations in its semantic representation space. Figure Sa shows the evolution of SR computed per bucket. Buckets are learned in increasing order, which confirms that the time of discovery is good proxy for difficulty. Figure 3b reports C, I.P and sampling probabilities P computed online using self-evaluations for example agent. The agent leverages these estimations select its goals: first focusing on the casy goals from bucket 1, it moves on towards harder and harder buckets as easier ones are mastered (low LP, high C). Figure 3e presents the results of ablation studies. Each condition removes one component of DECSTR: 1) Flar replaces our object-centered modular architectures by flat ones; 2) w/o Curr. replaces our automatic curriculum strategy by uniform goal selection; 3) w/o Sym. does not use the symmetry inductive bias; 4) In w/o SP, the social partner does not provide non-trivial initial configurations. In the Expert buckets condition, the curriculum strategy is applied on expert-defined buckets, see Appendix [D.]] The full version of L.GB performs par with the Expert buckets oracle and outperforms significantly all its ablations. Appendix E.3 presents more examples of learning trajectories, and dissects the evolution of bucket compositions along training.
Buskett
Buskett
Exp. Buckets
1.00F
Asym.
0055
Su00e55 Rate
6.755
0.0
0.05
0.0.
6.00
25n
#pisades 19t0
0.0g
1700
0.00
Fpiidee
(a)
(b)
(c)
Figure 3: Skill Learning: (a) SR bucket. (b): C, LP and P estimated by a DECSTR agent. (c): ablation
study. Medians and interquartile ranges over 10 seeds for DECSTR and 5 seeds for others (a) and (c). Stars indicate significant differences to DECSTR as reported by Welch's t-tests with 0.05 (Colas et al. 2019b).
Table I: L->G phase. Metrics are averaged over 10 seeds,
Table 2: L->G>BB phase. Mean + stdev
stdev < 0.06 and 0.07 respectively.
over 10 seeds.
Metrics
Test
Test 2
Test 3
Test 4
Test
Metr.
Transition
Expression
Precision
0.97
0.93
0.98
0.99
0.98
SR
0.89 0.05
0.74 0.08
Recall 0.93
0.94
0.95
0.90
0.92
SRş
0.99 + 0.01
0.94 + 0.06
Language grounding phase L->G: The LGG demonstrates the 5 types of generalization from Table []] From known configurations, agents can generate more goals than they observed in training data (1, They can do so from new initial configurations (3). They can generalize to new sentences (4) and even to combinations of new sentences and initial configurations (5). These results assert that DECSTR generalizes well in variety of contexts and shows good behavioral diversity.
Instruction following phase L-+G+B: Table presents the 1-shot and 5- results in the transi- tion and expression setups. In the sequence setups, DECSTR succeeds in T. = 14.9 + 5.7 successive instructions (mean-tsddyy over 10 seeds). These results confirm efficient language grounding. DEC- STR follow instructions sequences of instructions and generalize their logical combinations. Strategy switching improves performance (SR; SR1). DECSTR also demonstrates strong behavioral diversity: when asked over 10 seeds to repeat 50 times the same instruction, it achieves least 7.8 different configurations, 15.6 on average and up to 23 depending on the instruction.
4.2 DO WE NEED AN INTERMEDIATE REPRESENTATION?
This section investigates the need for intermediate semantic representation. To this end, we in- troduce an end-to-end LC-RL baseline directly mapping Language to Behavior (I->B) and compare its performance with DECSTR in the instruction following phase (L-+G++B.
The LB baseline. To limit the introduction of confounding factors and muder-tuning concerns, we base this implementation the DECSTR code and incorporate defining features of IMAGINE, state-

---

Published as a conference paper at ICLR 2021
of af-the-art language conditioned RL agent (Colas et al. 2020). We keep the same HER mechanism, object-centered architectures and RL algorithm as DECSTR. We just replace the semantic goal space by the 102 language instructions. This baseline can be algorithm where the reward function is assumed perfect, but without the imagination mechanism.
oracle version of the IMAGINE
Comparison in the instruction following phase L->B vs L->G-BB After training the LB baseline for 14K episodes, compare its performance DECSTR s in the centtructioo-oolooinn setup. In the transition evaluation setup, LB achieves SR1 = 0.76=0.001: it always manages to move blocks close or far from each other, but consistently fails stack them. Adding more attempts does not help: SRş = 0.76 + 0.001. The LB baseline cannot be evaluated in the expression setup because it does not manipulate goal sets. Because cannot stack blocks, LB only succeeds in 3.01 + 0.43 random instructions in a row, against 14.9 for DECSTR (sequence setup). We then evaluate LB's diversity on the set of instructions succeeds in. When asked repeat 50 times the same instruction, it achieves at least 3.0 different configurations, 4.2 on average and up to 5.2 depending on the instruction against 7.8, 17.1, 23 on the same set of instructions for DECSTR. We did not observe grrateyyswitcchig behaviors in LB, because it either always succeeds (close/far instructions) or fails (stacks).
The introduction of an intermediate semantic representation helps DECSTR decou- ple skill learning from language grounding which, in turns, facilitates instruction-folloiig when language-conditionee conditionnd learning of LB. This leads to improved scores in the transition and sequence setups. The direct language-conditionnn of LB prevents the general- ization to logical combination and leads to a reduced diversity in the set of mastered instructions. Decoupling thus brings significant benefits to I.GB architectures.
Conclusion.
compared to the end-to-end language-
4.3 DO WE NEED A SEMANTIC INTERMEDIATE REPRESENTATION?
This section investigates the need for the intermediate representation to semantic. To this end, we introduce the LGB-C baseline that leverages continuous goal representations in place of semantic ones. We compare them on the two first phases.
The LGB-C baseline. The LGB-C baseline uses continuous goals expressing target block coor- dinates in place of semantic goals. The skill learning phase is thus equivalent to traditional goal- conditioned RL setups in block manipulation tasks (Andrychowicz al., 2017 Colas et al., 2019a]] Li et al., 2019; Lanier et al. 2019). Starting from the DECSTR algorithm, LGB-C adds translation module that samples set of target block coordinates matching the targeted semantic configuration which is then used as the goal input to the policy. In addition, integrate defining features of the state-of-the thrt approach from Lanier et al. (2019): non-binary rewards (+1 for each well placed block) and multi-criteria HER, see details in Appendix D.2
Comparison in skill learning phase G-+B: The LGB-C baseline successfully learns discover and master all 35 semantic configurations by placing the three blocks to randomly-sampled target coordinates corresponding to these configurations. It does faster than DECSTR: 708 10 episodes to reach SR= 95%, against 1238- 10 for DECSTR, see Appendix Figure 6 This can be explained by the denser learning signals gets from using HER on continuous targets instead of discrete ones. In this phase, however, the agent only learns one parameterized skill: to place blocks at their target po- sition. It cannot build repertoire of semantic skills because it cannot discriminate between different block configurations. Looking at the sum of the distances travelled by the blocks or the completion time, we find that DECSTR performs opportunistic goal reaching: it finds simpler configurations of the blocks which satisfy its semantic goals compared to LGB-C. Blocks move less (Adist = 26 = cm). and goals are reached faster (Astess 13 4, ttd across goals with p p-vauues > 1.3- 10 and 3.2 10 19 respectively).
Table LGB-C performance the L-+ phase. Mean over 10 seeds. Stdev < 0.003 and 0.008 respectively.
Metrics
Test 1
Test 2
Test
Test 4
Test 5
Precision
0.66
0.78
0.39
0.0
0.0
Recall
0.05
0.02
0.06
0.0
0.0
8

---

Published as a conference paper at ICLR 2021
Comparison in language grounding phase L-+G: We train the LGG to generate continuous tar- get coordinates conditioned on language inputs with a mean-squared loss and evaluate it in the same setup DECSTR's 's LGG, see Table [] Although it maintains reasonable precision in the first two testing sets, the LGG achieves low recall i.e. diversity on all sets. The lack of semantic represen- tations of skills might explain the difficulty of training nanguage-conditoone goal generator.
Conclusion. The skill learning phase of the LGB-C baseline competitive with the one of DECSTR. However, the poor performance in the language grounding phase prevents this baseline to perform instruction following. For this reason, and because semantic representations enable agents to perform opportunistic goal reaching and to acquire repertoires for semantic skills, we believe the semantic representation is an essential part of the LGB architecture.
5 DISCUSSION AND CONCLUSION
This paper contributes LGB, a new conceptual RT. architecture which introduces an intermediate se- mantic representation to decouple sensorimotor learning from language grounding. To demonstrate its benefits, we present DECSTR, a learning agent that discovers and masters all reachable configu- rations in manipulation domain from set of relational spatial primitives, before undertaking an efficient language grounding phase. This was made possible by the use of object-centered inductive biases, new form of automatic curriculum learning and novel manuuag--condtionee goal genera- tion module. Note that our main contribution is in the conceptual approach, DECSTR being only an instance to showcase its benefits. We believe that this approach could benefit from any improvement in GC-RL (for skill learning) or generative models (for language grounding).
Semantic representations. Results have shown that using predicate- representations was sufficient for DECSTR to efficiently learn abstract goals in an opportunistic manner. The proposed semantic configurations showcase promising properties: 1) they reduce the complexity of block manipulation where most effective works rely on heavy hand-crafte curriculum (Li et al., 2019 Lanier et al. (019) and specific curiosity mechanism (Li et al.) 2019): 2) they facilitate the ground- ing of language into skills and 3) they enable decoupling skill learning from language grounding, as observed in infants (Piaget, 1977). The set of semantic predicates is, of course, domain-dependent as it characterizes the space of behaviors that the agent explore. However, we believe it is easier and requires less domain knowledge to define the set of predicates, i.c. the dimensions of the space of potential goals, than it is craft a list of goals and their associated reward functions.
new approach to language grounding. The approach proposed here is the first simultancously enabling to decouple skill learning from language grounding and fostering diversity of possible be- haviors for given instructions. Indeed, while an instruction following agent trained on goals like put red close_to green would just push the red block towards the green one, our agent can generate many matching goal configurations. It could build a pyramid, make a blue-green--ee pile or target a dozen other compatible configurations. This enables it to switch strategy, to find alternative approaches to satisfy a same instruction when first attempts failed. Our goal generation module can also gen- eralize to new sentences or transpose instructed transformations to unknown initial configurations. Finally, with the goal generation module, the agent can deal with any logical expression made of instructions by combining generated goal sets. would be of interest to simultaneously perform language grounding and skill learning, which would result in "overlapping waves" of sensorimotor and linguistic development (Siegler 1998).
Semantic configurations of variable size. Considering a constant number of blocks and, thus, fixed-size configuration spaces a current limit of DECSTR. Future implementations of LGB may handle inputs of variable sizes by leveraging Graph Neural Networks in Li et al. (2019). Corre- sponding semantic configurations could be represented as a set of vectors, each encoding informa- tion about a predicate and the objects it applies to. These representations could be handled by Deep Sets (Zaheer et al. 2017). This would allow to target partial sets of predicates that would not need to characterize all relations between all objects, facilitating scalability.
Conclusion In this work, we have shown that introducing abstract goals based on relational predi- cates that are well understood by humans can serve pivotal representation between skill learning

---

Published as a conference paper at ICLR 2021
and interaction with user through language. Here, the role of the social partner was limited to: 1) helping the agent to experience non-trivial configurations and 2) describing the agent's 's behavior in simplified language. In the future, intend to study more intertwined skill learning and language grounding phases, making it possible to the social partner to teach the agent during skill acquisition.
ACKNOWLEDGMENTS
This work was performed using HPC resources from GENCI-IDRIS (Grant 20XX-AP0106116677, the MeSU platform at Aeaboonn--nnversite and the PlaFRIM experimental testbed. Cédric Colas is partly funded by the French Ministère des Armées Direction Générale de I' I'Armmeent.
10

---

Published as a conference paper at ICLR 2021
REFERENCES
Muhannad Alomari, Paul Duckworth, David C Hogg, and Anthony G Cohn. Natural language acquisition and grounding for embodied robotic systems. Thirty-Firs AAAI Conference on Artificial Intelligence, 2017.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay, arXiv preprint arXiv:1107.01455 2017.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosscini. Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint /806.// 2018.
Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, and Jimmy Ba. Actree: Augmenting experience via teacher's advice for multi-goal reinforcement learning. arXiv preprint arXiv:1902. 1902.04466, 2019.
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Self-educated lan- guage agent with hindsight experience replay for instruction following. arXiv preprint arXiv:1910.09451, 2019.
Cédric Colas, Pierre-Yvel Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani. CU- RIOUS: Intrinsically motivated multi-task, multi-goal reinforcement learning. In International Conference on Machine Learning (ICML), pp. 1331-1340, 2019a.
Cédric Colas, Olivier Sigaud, and Pierre-Y Oudeyer. A hitchhiker's guide to statistical compar- isons reinforcement learning algorithms. arXiv preprint wxXiv:1944009977 2019b.
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Friee Peter Ford Dominey, and Pierre-Y Yves Oudeyer. Language cognitive tool imagine goals in curiosity- driven exploration. arXiv preprint 1rxiv:2002.09253 2020.
Mare Peter Deisenroth, Carl Edward Rasmussen, and Dieter Fox. Learning to control a low-cost manipulator using data-efficient reinforcement learning. Robotics: Science and Systems VII, pp. 57-64, 2011.
Sébastien Forestier, Yoan Mollard, and Pierre- Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning. arXiv preprint arXiv: 0277778..2110 0900 2017.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to
goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01299 2018.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint arXiv:1706.06551, 2017.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language abstrac- tion for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 9414-9424 2019.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099, 1993.
Tristan Karch, Cédric Colas, Laetitia Teodorescu, Clément Moulin-Friee and Pierre-Yver Oudeyer. Deep sets for generalization in RI. arXiv preprint 20Xi::2003309443 2020.
Johannes Kulick, Mare Toussaint, Tobias Lang, and Manuel Lopes. Active learning for teaching robot grounded relational symbols. In Twenty-Thine International Joint Conference on Artificial Intelligence, 2013.
11

---

Published as a conference paper at ICLR 2021
John B. Lanier, Stephen McAleer, and Pierre Baldi. Curiosity multi hindsss experi- ence replay. CoRR, 1ess/1990.,,,1,, 2019. URL httt:::/aaxxi..oo//abb//////////
Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manip- 2019.
ulation using relational reinforcement learning. arXiv preprint
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. arXiv preprint arXiv: 1906.0326, 2019.
Max Lungarella, Giorgio Metta, Rolf Pfeifer, and Giulio Sandini. Developmental robotics: survey. Connection Science, 15(4):151-190, 2003.
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 2020.
Jean M. Mandler. Preverbal representation and language. Language and space, pp. 365, 1999.
Jean M Mandler. On the spatial foundations of the conceptual system and its enrichment. Cognitive science, 36(3):421-451, 2012.
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. arXiv preprint arXiv:1910.11670 2019.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pp. 9191-9200, 2018
Jean Piaget. The development of thought: Equilibration of cognitive structures. Viking, 1977. (Trans
A. Rosin).
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09440 2018.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint MxXiv:2003.04664 2020.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pp. 1312-1320, 2015.
Robert S Siegler. Emerging minds: The process of change in children's 's thinking. Oxford University Press, 1998.
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-2, 2005.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp- 3483-3491, 2015.
Gerald J. Sussman. computational model of skill acquisition. Technical report, HIT Technical Report AI TR-297, 1973.
Austin Tate et al. Interacting goals and their use. In IJCAI, volume 10, pp. 215-218, 1975.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Approaching the symbol grounding problem with probabilistic graphi- cal models. Al magazine, 32(4):64-76, 2011.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8026-5033 IEEE, 2012.
12

---

Published as a conference paper at ICLR 2021
L. S. Vygotsky. Tool and Symbol in Child Development. In Mind in Society, chapter Tool and Symbol in Child Development, pp. 19-30. Harvard University Press, 1978. ISBN 0674576292. doi: (0.2307/j.ctvjf99244..
Weng, McClelland, A. Pentland, O. Sporns, Stockman, M. Sur, and E. Thelen. Autonomous mental development by robots and animals. Science, 2916504))599-600, 2001.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg. Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391- 3401, 2017.
13

---

Published as a conference paper at ICLR 2021
A LGB PSEUDO-CODE
Algorithm and present the high-level pseudo-code of any algorithm following the LGB architec- ture for each of the three phases.
Algorithm LGB architecture G-B phase
Algorithm 2 LGB architecture
L-+G and L->G-B phases
Goal - Behavior phase
Language - Goal phase
1: Require Env E
1: Require II, E, G.. social partner SP
2: Initialize policy II. goal sampler G.. buffer B
2: Initialize language goal generator LGG
3: dataset +- SP ..P.intreact(E. II, G.)
3: loop
4: I.GG.undate(dattaset)
4:
4- G-Gg.sampee0)
5: return LGG
5:
(s, s's Cp. Sprraj + E.rollout())
Language - Behavior phase
6:
(..uudate(c
6: Require E, 11, LGG, SP
7:
B.update((s: (s, a, s', a,s', IL.update(B))
cp. 4p)raa))
7: loop
8: instr. + -SP.listen()
8:
loop
Strategy switching loop
9:
return IT, G.
10:
g L.G..sampleiinstrr c²)
10:
11:
o
pp -- F.oollou(())
12:
12:
if of then break
B SEMANTIC PREDICATES AND APPLICATION TO FETCH MANIPULATE
In this paper, we restrict the semantic representations to the use of the close and above binary pred- icates applied to M objects. The resulting semantic configurations are formed by:
Cp [c(o1, 0z); c(02,03) a(oi, 02), a(02, 01), a(o1, 03); a(03; 01), a(02,03) a(03, 02)]; where c() and a() refer to the close and above predicates respectively and (01, 02, 03) are the red, green and blue blocks respectively.
Symmetry and asymmetry of close and above predicates. We consider objects 0₁ and 02-
close is symmetric:
"o, is close to 02" + "02 is close to 01". The corresponding semantic mapping function is based the Euclidean distance, which symmetric.
above is asymmetric: "o, is above Og" + not "Oy is above 01". " The corresponding semantic mapping function evaluates the sign of the difference of the object Z-axis coordinates.
C THE DECSTR ALGORITHM
C.I INTRINSICALLY MOTIVATED GOAL-CONDITIONED RI.
Overview. Algorithm presents the pscudo-code of the sensorimotor learning phase (G->B of DECSTR. It alternates between two steps:
Data acquisition. A DECSTR agent has no prior the set of reachable semantic config- urations. Its first goal is sampled uniformly from the semantic configuration space. Using this goal, it starts interacting with its environment, generating trajectories of sensory states 8, actions and configurations Cp. The last configuration of achieved in the episode after T time steps is considered stable and is added to the set of reachable configurations. As it in- teracts with the environment, the agent explores the configuration space, discovers reachable configurations and selects new targets.
Internal models updates. A DECSTR agent updates two models: its curriculum strategy and its policy. The curriculum strategy can seen as an active goal sampler. It biases the selection of goals to target and goals to learn about. The policy is the module controlling the agent's behavior and is updated via RL.
14

---

Published as a conference paper at ICLR 2021
Algorithm 3 DECSTR:
sensorimotor phase G-+B.
1: Require: env E, # buckets N₁. # episodes before biased init. Nunb, self-evaluation probability
Pself.eval, noise function o()
2: Initialize: policy II, buffer B, goal sampler G2, bucket sampling probabilities Ps, language
module LGG.
3: loop
4:
eval + random() < PselCeval 5: 9 g- C...smple(ssll__eee pb)
bi True then evaluate competence
6:
biased init + epoch < Tlunb
> Bias initialization only after Yuun epochs D en: Initial semantic configuration
7:
s S. E.rese((biaseddinn))
8:
for [= 1 do a' f policy(s', .'', if not self eval then a² aa + +o( gtt1, dett - E.step(a')
9:
10:
12:
13:
episode - (s,c,a,s',
14:
G. G..update(c")
15:
B.update(episode)
16:
g +- (amppe())))))
17:
batch + B.ampl)(g))
18:
Tupddate(batch) if' sell then
19:
pb - G.dupa LPO OPO
Policy updates with a goal-conditioned Soft Actor-Critic. Readers familiar with Markov Deci- sion Process and the use of SAC and HER algorithms can skip this paragraph.
We want the DECSTR agent to explore semantic configuration space and master reachable configurations in it. We frame this problem as a goal-conditioned MDP (Schaul et al. 2015): M (8, Sp: A, T, 7), where the state space is the usual sensory space augmented with the configuration space Cp. the goal space Gp is equal to the configuration space 9p = Cp- A is the action space, T S x A sparse reward function and y E [0, 1] is the discount factor.
+ [0, 1] is the unknown transition probability, R S x {0, 1] is
Policy updates are performed with Soft Actor-Critic (SAC) (Haarnoja et al. 2018), a state-of-the-art
off-policy actor-critic algorithm. We also use Hindsight Experience Replay (HER) (Andrychowicz ict a (017). This mechanism enables agents to learn from failures by reinterpreting past trajectories in the light of goals different from the ones originally targeted. HER designed for continuous goal spaces, but can be directly transposed to discrete goals (Colas et al.] (019a). In our setting, we simply replace the originally targeted goal configuration by the currently achieved configuration in the transitions fed to SAC. We also use our automatic curriculum strategy: the LP-C-based probabil- ities are used sample goals to learn about. When goal g is sampled, search the experience buffer for the collection of episodes that ended in the configuration Cp = From these episodes, we sample transition uniformly. The HER mechanism substitutes the original goal with one of the configurations achieved later in the trajectory. This substitute has high chances of being the sampled one. At least, a configuration on the path towards this goal, as it is sampled from a trajectory leading to it. The HER mechanism is thus biased towards goals sampled by the agent.
Object- Ceject-Centere Inductive Biases. In the proposed Fetch Manipulate environment, the three blocks share the same set of attributes (position, velocity, color identifier). Thus, natural to encode relational inductive bias in our architecture. The behavior with respect to pair of objects should be independent from the position of the objects in the inputs. The architecture used for the policy is depicted in Figure 4
A shared network (N encodes the concatenation of: 1) agent's body features; 2) object pair features: 3) current configuration (cp) and 4) current goal g. This done independently for all object pairs. No matter the location of the features of the object pair the initial observations, this shared network ensures that the same behavior will be performed, thus skills transferred between object
15

---

Published as a conference paper at ICLR 2021
pairs. A sum is then used to aggregate these outputs, before final network (N (NoNellcy maps the aggregation to actions a. The critic follows the same architecture, where a final network N ertic maps the aggregation an action-value Q. Parallel encoding of each pair-specifie imputs can be as different modules trying to reach the goal by only seeing these pair-specific inputs. The intuition is that modules dealing with the pair that should be acted upon reach the goal will supersede others in the sum aggregation.
body
obj
obl.-
obd
body
abjy
NN shardi
NN
Semantic
polley
8
mapping
abi
funetion
obis
Figure 4: Object-centered modular architecture for the policy.
Although in principle our architecture could work with combinations of objects (3 modules), we found permutations to work better in practice modules). With combinations, the shared network would need to learn to put block A on block B to achieve a predicate above(o,, oj), and would need learn the reverse behavior (put B A) achieve the symmetric predicate above(o,, oz). With permutations, the shared network can simply learn one of these behaviors (e.g. A on B). Considering the predicate above(o., 0B), at least one of the modules has objects organized so that this behavior is the good one: if the permutation (on, OAA) is not the right one, permutation (оA, oB) is. The symmetry bias explained Section 3.4 It leverages the symmetry of behaviors required to achieve the predicates above(o,, 0ÿ) and above(o), o,). As result, the two goal configurations are:
91 02), c(01, c(o2, 03), a(01, 02), a(01; 03), a(0z, 0z)l;
92 02), c(01,03), c(o2, 03), a(02, 01), alos, 01), a(03, 02)];
where 91 is used in association with object permutations (o;, 9,) with < j and g2 is used in association with object permutations (oj, 0;) with i < j. As a result, the shared network automatically ensures transfer between predicates based on symmetric behaviors.
Implementation Details. This part includes details necessary to reproduce results. The code is available at https://site.gooolll.oo//iiiiii view/decstr/
Parallel implementation of SAC-IIRR. We use parallel implementation of SAC (Haarnoja et al. 2018). Each of the 24 parallel worker maintains its own replay buffer of size 10 and performs its own updates. Updates are summed over the 24 actors and the updated network are broadcast to all workers. Each worker alternates between 2 episodes of data collection and 30 updates with batch size 256. form an epoch, this cycle is repeated 50 times and followed by the offline evaluation of the agent each reachable goal. An epoch is thus made of 50 2 x 24 2400 episodes.
Goal sampler updates. The agent performs self-evaluations with probability self-eval During these runs, the agent targets uniformly sampled discovered configurations without explo- ration noise. This enables the agent to self-evaluate on each goal. Goals are organized into buckets. automatic bucket generation mechanism. Once buckets formed,
0.1.
Main Section 3.4 presents we compute LP and P, based on windows of the past W = 1800 self-evaluation interactions for each bucket.
16

---

Published as a conference paper at ICLR 2021
Modular architecture. The shared network of our modular architecture N is 1-hidden layer network of hidden size 256. After all pair-specific inputs have been encoded through this module, their output (of size 84) are summed. The sum is then passed through final network with a hidden layer of size 256 to compute the final actions (policy) or action-values (critic). All networks use ReLU activations and the Xavier initialization. We use Adam optimizers, with learning rates 10-3 The list of hyperparameters is provided in Table 4
Table 4: Sensorimotor learning hyperparameterss used in DECSTR.
Hyperparam.
Description
Values.
nb_ mb_mpis
Number of workers
24
nb ab_cycles
Number repeated cycles per epoch
50
mb.rollouts-per-ee
Number of rollouts per worker
2
nb updates
Number of updates per cycle
30
start bias inil
Epoch from which initializations
biased
100
W
Curriculum window size
1800
self-evdl
Self evaluation probability
0.1
No
Number of buckets
5
strateyy
HER replay strategy
future
kereplay
Ratio of HER data to data from normal experience
4
batch
Size of the batch during updates
256
7
Discount factor to model uncertainty about future decisions
0.98
T
Polyak coefficient for target critics smoothing
0.95
Ir actor
Actor learning rate Critic learning rate Entropy coefficient used SAC
10-3
Ir critic
10
a
0.2
automatic_entroye
Automatically tune the entropy coefficient
False
Computing resources. The sensorimotor learning experiments contain 8 conditions: 2 10 seeds and 6 of 5 seeds. Each run leverages 24 cpus (24 actors) for about 72h for total of 9.8 cpu years. Experiments presented this paper requires machines with at least 24 cpu cores. The language grounding phase runs single epu and trains in few minutes.
C.2 LANGUAGE-CONDITIONE) GOAL GENERATOR
Languagg-Condittonee Goal Generator Training. We use conditional Variational Auto- Encoder (C-VAE) (Sohn et al. 2015). Conditioned on the initial configuration and a sentence describ- ing the expected transformation of one object relation, it generates compatible goal configurations. After the first phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 3 From these interactions, triplets: initial configuration, final configuration and sentence describing one change of predicate from the initial to the final configuration. The list of sentences used by the synthetic social partner provided in Table Note that red, green and blue refer to objects 01, 02, 03 respectively.
obtain a dataset of 5000
Content of test sets. We describe the test sets:
Test set is made of input pairs (cts s) from the training set, but tests the coverage of all compatible final configurations Cy. 80% of which are not found in the training set. In that sense, it is partly test set.
2. Test set 2 contains two input { 0 00000000000000000000000000000000 0j. put green below red} corresponding and 24 compatible final config- urations respectively.
1000000000 0), put biue close_o green} and
3. Test set 3 corresponds to all pairs including the initial configuration c; = 110000000
(29 pairs), with average of 13 compatible final configurations.
Test set 4 corresponds to all pairs including one the sentences put green on top of red and put blue far from red, i.e. 20 pairs with average of 9.5 compatible final configurations.
17

---

Published as a conference paper at ICLR 2021
5. Test set 5 is all pairs that include both the initial configuration of test set 3 and one of the sentences of test set 4, i.c. 2 pairs with 6 and 13 compatible goals respectively. Note that pairs of set 5 are removed from sets 3 and 4.
Table 5: List of instructions. Each of them specifies shift of one predicate, either from false true (0 1) or true to false (1 + 0). block block represent two different blocks from (red, blue, green).
Transition type
Sentences
Close 0
Put block A close to block B, Bring block A and block together,
(x3)
Get block A and block B close_ from each_other, Get block A close block B.
Close I -
Put block A from block B, Get block A jar from block B,
(x3)
Get block A and block B far from each_otherr Bring block A and block B apart,
Above - 1
Put block A above block B, Put block A on top of block B, Put block B under block A, Put block B below block A. Remove block A from above block B, Remove block A from block B Remove block B from_below block A, Put block B and block on same samee-lanee tamee Put block A and block B ohe same plane.
(x6)
Above 1 0
(x6)
Testing on logical expressions of instructions. To evaluate DECSTR on logical functions of in- structions, we generate three types of expressions:
100 instructions of the form "A and B" where and B are basic instructions corresponding to shifts of the form above 0 > 1 (see T). These intersections correspond to stacks of 3 or pyramids.
2. 200 instructions of the form "A and B" where A and B are above and close instructions respectively. can be replaced by "not B" with probability 0.5.
3. 200 instructions of the form "(A and B) or (C and D))", where A, C, tions: A and C are above instructions while B and D are close instructions. Here also, any instruction can be replaced by its negation with probability 0.5.
D are basic instruc-
Implementation details. The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations. It takes as input the concatenation of the final binary configuration and its two conditions: the initial binary configuration and an embedding of the NL sentence. The NL sentence embedded with an recurrent network with embedding size 100, tanh non-linearities and biases. The encoder outputs the mean and log-variance of the latent distribution of size 27. The decoder also a fully-connected network with two hidden layers of size 128 and Rel.U activations. It takes input the latent code and the same conditions as the encoder. As it generates binary vectors, the last layer uses sigmoid activations. We train the architecture with a mixture of Kullback- Leibler divergence loss Diss) w.r.t a standard Gaussian prior and binary Cross-
loss ( The combined loss is BC + 3 X K Dioss with B = 0.6. We use an Adam optimizer, a learning rate of 5 10m4, batch size of 128 and optimize for 150 epochs. As training is fast (xx min on a single cpu), we conducted a quick hyperparameter search over 3, layer sizes, learning rates and latent sizes (see Table 6). We found robust results for various layer sizes, various 3 below 1. and latent sizes above 9.
Table 6: LGG hyperparameter search. In bold are the selected hyperparameters.
Hyperparam.
Values,
B
0.5, 0.6, 0.7, 0.8, 0.9, 1.J [128, 256
layers size learning rate
[0.01, 0.005, 0.001]
latent sizes
[9, 18, 27)
D BASELINES AND ORACLE
The language-conditionee LB baseline is fully described in the main document.
18

---

Published as a conference paper at ICLR 2021
D.1 EXPERT BUCKETS ORACLE
In the EXPERT BUCKETS oracle, the automatic bucket generation of DECSTR is replaced with an expert-predefined set of buckets using a priori measures of similarity and difficulty. To define these buckets, one needs prior knowledge of the set of unreachable configurations, which are ruled out. The 5 predefined buckets contain all configurations characterized by:
Bucket I: single close relation between pair of objects and no above relations (4 configu- rations).
Bucket 2: 2 or 3 close relations and above relations (4 configurations).
Bucket 3: 1 stack of blocks and a third block that is either away or close to the base, but is not close to the top of the stack (12 configurations).
Bucket 4: 1 stack of 2 blocks and the third block close to the stack, as well as pyramid configurations (9 configurations).
Bucket 5: stacks of 3 blocks (6 contigurations).
These buckets are the only difference between the EXPERT BUCKETS baseline and DECSTR.
D.2 LGB-C BASELINE
The LGB-C baseline represent goals not as semantic configurations but as particular 3D targets po- sitions for each block, as defined for example in Lanier et al. (2019) and Li et al. (2019). The goal vector size is also 9 and contains the 3D target coordinates of the three blocks. This baselines also implements decoupling and, thus, can be compared to DECSTR in the three phases. We keep as many modules as possible common with DECSTR to minimize the amount of confounding factors and reduce the under-fitting bias. The goal selection is taken from DECSTR, but converts semantic configuration into specific uundomly-sampe its current semantic configuration nor its semantic goal configuration. For this reason, we do not apply the symmetry bias. The binary reward is positive when the maxi- mal distance between block and its target position is below 5 cm, i.e. the size of block (similar to [Andrychowicz et al. 2017)). ). To make this baseline competitive, we integrate methods from a state of the block manipulation algorithm (Lanier al.) 2019). The agent receives positive re- wards of 1, 2, 3 when the corresponding number of blocks are well placed. We also introduce the multi-criteria HER from Lanier et al. (2019). Finally, add an additional object-centered inductive bias by only considering, for each Deep Sets module, the 3D target positions of the corresponding pair.The is, for each object pair, we ignore the 3D positions of the remaining object, yielding to a vector of size 6. Language grounding is based on a C-VAE similar to the one used by DECSTR. We only replace the cross-entropy loss by a mean-squared loss due to the continuous nature of the target goal coordinates. We use the exact same training and testing sets as with semantic goals.
target coordinates for the blocks, see Figure 5 The
agent is not conditioned
Figure 5: The LGB-C baseline samples target positions for cach block (example for a pyramid here).
19

---

Published as a conference paper at ICLR 2021
E ADDITIONAL RESULTS
E.1 COMPARISON DECSTR I.GB-C IN SKILL LEARNING PHASE
Figure 6 presents the average success rate over the 35 valid configurations during the skill learning phase for DECSTR and the LGB-C baseline. Because L.BB-C cannot pursue semantic goals as such, randomly sample specific instance of this semantic goal: target block coordinates that satisfy the constraints expressed by it. Because LGB-C is not aware of the original semantic goal, we cannot measure success as the ability to achieve it. Instead, success is defined as the achievement of the corresponding specific goal: bringing blocks to their respective targets within an error margin of 5 em each. In short, DECSTR targets semantic goals and is evaluated on its ability to reach them. LGB-C targets specific goals and is evaluated on its ability to reach them. These two measures do not match exactly. Indeed, L.GB-C sometimes achieves specific goal but, because of the error margins, does not achieve the original semantic goal.
DECSTR
LGB-C
1.00
Success Rate
0.75-
0.50
0.25
0.00
200
400
600
800
1000
1200
1400
Episodes (x10²)
Figure Comparison DECSTR and I.GB-C in the skill learning phase.
E.2 AUTOMATIC BUCKET GENERATION.
Figure 7 depicts the evolution of the content of buckets along training (epochs 1, 50 and 100). Each pie chart corresponds to reachable configuration and represents the distribution of configurations into buckets across 10 different seeds. Blue, orange, green, yellow, purple represent buckets to respectively and grey are undiscovered configurations. At each moment, the discovered configura- tions are equally spread over the buckets. A given configuration may thus change bucket as new configurations are discovered, so that the ones discovered earlier assigned buckets with lower indexes. Goals are organized by their bucket assignments in the Expert Buckets condition (from top bottom).
After the first epoch (left), DECSTR has discovered all configurations from the expert buckets 1 and 2, and some have discovered few other configurations. After 50 epochs, more configurations have been discovered but they are not always the same across runs. Finally, after 100 epochs, all configurations are found. Buckets are then steady and can be compared to expert-defined buckets. It seems that easier goals (top-most group) are discovered first and assigned in the first-easy buckets (blue and orange). Hardest configurations (stacks of bottom-mos group) seem be discovered last and assigned the last-hardes bucket (purple). In between, different runs show different composi- tions, which are not always aligned with expert-defined buckets. Goals from expert-defined buckets 3 and 4 (third and fourth group from the top) seem to be attributed different automatic buckets in different runs. This means that they are discovered in different orders depending on the runs. In summary, easier and harder goals from expert buckets 1 2 and 5 respectively seem to be well de- tected by our automatic bucket generations. Goals medium-level expected difficulty as defined by expert buckets seem not show any significant difference in difficulty for our agents.
E.3 DECSTR LEARNING TRAJECTORIES
Figure & shows the evolution of internal estimations of the competence c, the learning progress I.P and the associated sampling probabilities Note that these metrics are computed online by
20

---

Published as a conference paper at ICLR 2021
Automatic Buckets
Bucket
Bucket
Bucket
Bucket
Bucket
Net discovered
Expert
Expert Bucket 2
Expert
Expert Bucket
Expert 5
Figure 7: Evolution the content of buckets from automatic bucket generation: epoch (2400 episodes, left), 50 (middle) and 100 (right). Each pic chart corresponds to one of the 35 valid configurations. It represents the distribution of the bucket attributions of that configuration across 10 runs. Blue. orange. green, yellow, purple represent automatically generated buckets 5 respectively (increasing order of difficulty) and grey represents undiscovered configurations. Goals are organized according to their expert bucket attributions in the Expert Buckets condition (top-bottom organization).
DECSTR, it self-evaluates random discovered configurations. Learning trajectories seem be uniform across different runs, and buckets are learned in increasing order. This confirms that the time of discovery is good proxy for goal difficulty. In that case, configurations discovered first end in the lower index buckets and are indeed learned first. Note that failing automatic bucket generation would assign goals random buckets. This would result in uniform measures of learning progress across different buckets, which would be equivalent to uniform goal sampling. As Main Figure [3 shows, DECSTR performs much better than the random goals conditions. This proves that our automatic bucket algorithm generates useful goal clustering.
22
82
024
025
015
5.334
54.4
Spsodee k010)
Epsstes (018
Spoodee
3,104
0.2
4.2
Epaodes X1D0)
Ep Epaodes (400%)
Figure 8: Learning trajectories of DECSTR agents.
21


Reviews:
[{"review_id": "y5zTu5ve6X", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed DECSTR, a procedure for encouraging intrinsic motivation via an intermediate semantic state-space representation. The authors propose an intermediate semantic state space that the intrinsically motivated agent learns to explore. For the environment provided (a 3-block system), the agent fully explores the symbolic state space, reaching all feasible symbolic states. In the second part of the work, the authors train a language model capable of proposing symbolic goals (in the form of symbolic states) from natural language input and shows that the previously-intrinsically-motivated agent can now be made to reach these goals, demonstrating that the symbolic-goal-conditioned policy is sufficient for instruction following in their 3-block domain.\n\nThe work is generally interesting, and seems to address a simple version of a broader class of problems that embodied agents typically struggle with, particularly in the absence of clear goals. However, the approach presented in the behavior (in particular the form of the semantic representation that is claimed as one of the primary contributions of the work) is very specific to the single problem used for demonstrations in the paper, limiting the potential impact of the work.\n\nFirst---and I think the most significant issue with the submission---is that many critical experimental details are included only in the lengthy appendix. Much of this information, including the information provided to the learning algorithm at every step and how that information is encoded such that it allows for a relatively object-agnostic representation, is only available in sufficient detail in the appendix. Relatedly, visualizations of the approach and experimental setup also only appear in the appendix, yet are extremely helpful (if not essential) for understanding. Detail critical to understanding the approach should be included in the body of the text.\n\nSecond, it is unclear exactly what problem is being solved in this work or what its primary contribution is. A clearer statement of its motivations will be necessary before publication. /What problem is the robot or system designers trying to overcome?/ Right now, the paper seems to come up with three potential answers to this question, none of which necessarily rises above the others. Here are what I think the main contributions of the work could be:\n\n1. *The proposed semantic representation* The semantic goal representation used to define the space of intrinsic motivation seems to be a novel contribution. However, if the paper were to focus on this aspect of the contribution, it would need to do a better job understating why this representation were useful beyond a relatively small manipulation task. Critically: using only one problem setting with only three blocks is insufficient to convince the reader that this representation is useful more generally (as might be suggested by much of the talk about Inductive Bias).\n2. *State of the art state-space exploration in intrinsic motivation.* This might be true, though I find such a thing hard to measure. In addition, it seems that many if not all of the tools used in the learning process are not novel. (Perhaps a combination of this and point 1. is the primary contribution.)\n3. *State of the art performance on language-driven block manipulation tasks.* This might be true as well, but the results are so-far unconvincing. All baselines are varied forms of the proposed agent, which makes it difficult to compare against other approaches (e.g. something like Li et al 2019).\n\nThe paper currently seems to claim that the combination of progress in these three areas is a novel contribution; I am sympathetic to this idea (as I do not believe that every paper needs to be \"state of the art\" in one single thing), though it is sufficiently unclear at the moment what the takeaway message of the paper is that I cannot recommend it be published in its current state. In particular, the authors need to work on honing the message of the paper. It is also not unlikely that one or two more experiments will need to be added to support the focused narrative.\n\nSmaller comments\n- The name of algorithm should appear in the body of the text, not a footnote. Relatedly, it is unclear how the proposed approach uses the \"Deep Sets\" work in such a way that it justifies inclusion in the name of the proposed technique.\n- The paper/Introduction would benefit from a summary of contributions: even after reading, it may not be clear to a reader which contributions are from this paper versus other work.\n- Relatedly, much of the discussion of Inductive Biases that appear throughout the paper is of mixed relevance for this work. On the one hand, it is clear how the idea of an object-centric inductive bias helped to inform how the input to the neural network was encoded in a way that might allow the agent to apply its knowledge learned between two of the objects to a policy that allows it to manipulate all three. However, the goal condition is necessarily specific when it comes to representing which objects to which each element it refers. The structure of the goal and the semantic relations it encodes are quite specific to the particular problem at hand, and it is\n- The reward for the \"Position only\" baseline seems artificially constructed: a non-binary reward function would likely allow the system to learn more easily. As of now I am unconvinced that the authors have worked hard enough to make a fair baseline for comparison. This is particularly problematic since this baseline is a key motivator for the existence of the proposed semantic goal representation.\n- The paper overall is quite well written, despite relegating too much information to the abstracts.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "v9rKISMsML", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces DECSTR, which is an agent having a high-level representation of spatial relations between objects. DECSTR is a learning architecture that discovers and masters all reachable configurations from a set of relational spatial primitives. They demonstrated the characteristics in a proof-of-concept setup.\n\nIn the introduction, the inspiration obtained from developmental psychology is described. Motivation and background are broadly introduced. A wide range of related works are introduced in section 2.\nThe motivation and target of this paper are ambitious and important.\n\nHowever, from the \"methods\" part, i.e., section 3, this paper is hard to follow. \nThe supplementary material helps to understand. However, I believe some of the informative and detailed information in the supplementary material should come to the main manuscript.\n\nThe proposed method, i.e., DECSTR, is comprised of many components. Therefore, the main contribution is also not clear. # What is the main argument of the paper?\nExperimental conditions are also hard to follow.\n\nIn evaluation, Figure 1 shows ablation studies alone, i.e., comparison with the variants of DECSTR.\nTherefore, the contribution of the paper is hard to grasp.\n\nWe can understand what kind of task is achieved in this paper.\nCurrently, the paper somehow seems to be a demonstration of DECSTR. \nIn this sense, if the authors state research questions, challenges, and contributions of this paper more clearly, that will make this paper more impactful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y4HrV3QfDJY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe DECSTR system's intrinsic motivations may be applicable to other application domains, depending on how objects and relations are enumerated. This potential is not explored beyond the toy environment presented. The learning methods (especially inductive biases) are hand-crafted based on human-level knowledge about semantic predicates, but only two (\"above\" and \"close\") are demonstrated. Without demonstrating the system on any other configuration or world, it's difficult to tell whether it's able to solve only the problem it's been crafted to solve in this specific environment.\n\nQuestions:\n\n3.1 \"in principle ... could use any other combination of binary predicates and could be extended to use n-ary predicates\" this claim is not demonstrated in the paper, and in 3.2 the inductive biases seem bespoke crafted for binary predicate 'above' which has particular symmetry. Would similar careful design of inductive biases be necessary and possible for n-ary predicates that do not demonstrate these as easily (e.g., \"topmost\")? What about predicates that involve an unspecified number of discrete arguments, like \"base\" -> holding up an indefinite N of other objects/structures in \"use the green block as the base\".\n\n3.4 \"or is union\" this doesn't generally hold for natural language. A statement like \"put the red block or the green block above the yellow block\" does not mean to put both red and green (union of goals) above yellow. Typically langauge \"or\" is \"xor\"; is the notion of \"or\" here not given in language or not meant to represent human language?\n\n\nAreas for Improvement:\n\n5 \"a learning architecture that discovers and masters all reachable configurations from a set of relational primitives\" this is literally true but only demonstrated on 'a' single set of relational primitives, so it feels like overclaiming.\n\n\nNits:\n- double citation for Mandler, 2012 in intro in adjacent sentences can be condensed to once\n- footnotes on other side of period\n- \"Besides\" in \"Blocks Manipulation\" seems a bit off-sounding; maybe \"In addition,\"?\n- typo section 3 \"based o abstract\"\n- Typo section 5 backwards quotes \"overlapping waves\" LHS.\n- \"Caregiver\" in section 5 is an unintroduced role. The rest of the paper does not frame DECSTR or the oracle generator this way.\n- Ending the paper with \"etc.\" feels weird/informal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4rmRMNBJjDf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper proposes DECSTR, a goal-driven RL framework where the goal is represented as a binary vector that encodes the semantic relationships between objects. The state is assumed to contain disentangled features for each of the objects (and other features relating to the agent’s end-effectors). The architecture is based on Deep Sets (Zaher et al., 2017), which allows the pairs of the objects to be encoded with a shared network. The paper also introduces a curriculum learning strategy similar to CURIOUS (Colas et al., 2019), which relies on metrics such as competence and learning progress (LP) in order to select goals to pursue during an episode. One key difference is that unlike CURIOUS which uses expert-defined “goal buckets”, DECSTR groups the goals based on recency of discovery. Once trained to be able to behave with respect to these semantic relationship goals, the second phase is language grounding. They learn a module (implemented as C-VAE) that converts from natural language text to the semantic configuration goal space. Experiments were conducted in the Fetch Manipulate robotic arm environment and compared with ablations of DECSTR without some of its components, demonstrating strong performance and generalization to various types of language instructions. \n\n**Pros**:\n- The paper is well-motivated, citing literature from several fields.\n- The sum is greater than its parts: many components in DECSTR are based on existing works (e.g. Deep Sets, C-VAE, using LP for intrinsically motivated goals, etc.), but empirically they have shown through ablations that all of their components were necessary for the agent to solve the Fetch Manipulation task successfully. \n- The experiment sections are fairly thorough, with ablations on the components of their methods (as said above), and various kinds of language command generalization evaluations (in a similar style to IMAGINE (Colas et al., 2020). \n- The interpretability of the semantic goal space aspect is interesting. And being able to have the agent explicitly maps from the natural language text to the semantic goal space also helps us debug/understand what the agent is thinking at inference time\n\n**Cons**:\n- Part of the thesis is that decoupling of sensorimotor learning from language acquisition is advantageous to an end-to-end language to sensorimotor learning. I have concerns/clarification about some of the baselines, which might not have been a fair comparison with DECSTR (see question 1 & 2 below)\n- Some parts of the method are unclear/vague without reading the appendix section to get the full detail. I understand that is due to the space limitation issue and because there are so many components to DECSTR. (see question 3)\n\n**Recommendation**:\n \nOverall, I vote for marginally below acceptance threshold in the current form. As mentioned in the strengths section, I do like the motivation of the paper and the strong performance of the method. But I am also suspicious of the poor performance of the baselines (e.g. Figure 1c), which may be due to not having HER, instead of their proposed contributions. It would be good if the authors can clarify that concern. \n\n**Question**:\n1. In Figure 1c, for the Language Goals baseline, was HER applied to the Language Goals in this case (i.e. similar to ACTRCE (Chan et al., 2019), IMAGINE (Colas et al., 2020)? Similarly, was HER applied to the Position-Goals baseline? If not, then it is possible the difference in performance between DECSTR and these baselines may be due more to HER than due to the difference in goal representation. \n2. Would it be possible to train Phase 1 and Phase 2 together or in an end-to-end fashion? This would provide a ‘coupled’ version that is different from any of the baselines studied in the paper because it still uses the semantic configuration as the intermediate goal representation while having joint training of the language representation and the sensorimotor. If this baseline struggles to learn (possibly due to difficult optimization/local minimas), then this will help further strengthen the thesis of the importance of decoupling the learning process into two distinct phases. \n3. Section 3.2: the main text and appendix C.2 was not very clear about the second inductive bias for the symmetry of the behavior required to achieve $above(o_i, o_j)$ and $above(o_j, o_i)$. Are you saying, for example, if we are trying to have object 1 above object 2, then we specify the goal in the form $g_1$, while if we want object 2 above object 1, then we specify the goal in the form $g_2$?  \n\n**Minor comments**:\n* When using double quotes in latex, use backticks for the opening quote.\n\n**After rebuttal responses**: \n\nI have read the authors’ updated draft and response to my concerns, as well as the other reviews. The updated paper provides a clearer framing and some missing baselines have also been included. I raised my evaluation to a weak acceptance for the paper. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new approach to grounding language-based RL tasks via an intermediate semantic representation, in an architecture called language-goal-behavior (LGB).  The architecture permits learning a mapping from internal goals to behavior  (GB) separately from learning a mapping from language to internal goals (LG), and prior to flexibly combining all three (LGB).  The architecture is studied in a specific implementation called DECSTR.  The architecture has multiple desired attributes including support for intrinsic motivation, decoupling skill acquisition from language grounding, and strategy switching.  The experiments demonstrate the utility of different components in the architecture with a variety of ablation results.

The reviews initially found the paper to be poorly organized with required content described only in the appendix (R1, R2, R4), with unclear main contributions (R1, R2, R4), and with results restricted to demonstrations (R3).  Despite these reservations, the reviewers found the content to be potentially relevant though narrow in scope.

The authors substantially revised the paper. They improved its organization, clarified contributions, separated the architecture from the specific examples, and improved the experimental baselines.  After reading the revised paper, the reviewers agreed that the paper's organization and insights were improved, making the new paper's contribution and insight clear.  The experimental baselines were also improved, providing more support for the potential utility of the proposed method.

Three reviewers indicate to accept this paper for its contribution of a novel approach to grounding language and behavior with an intermediate semantic representation. No substantial concerns were raised on the content of the revised paper. The paper is therefore accepted.
Decision: Accept (Poster)