Self-supervised Reinforcement Learning with Independently Controllable Subgoals
Authors: Andrii Zadaianchuk
Georg Martius
Fanny Yang
Venue: CoRL2021 Poster (2021)
Keywords: object-centric representations
relations
self-supervised reinforcement learning

Abstract: To successfully tackle challenging manipulation tasks, autonomous agents must learn a diverse set of skills and how to combine them.
Recently, self-supervised agents that set their own abstract goals by exploiting the discovered structure in the environment were shown to perform well on many different tasks.
In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. 
However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environments.
We propose a novel self-supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state. In addition, the estimated relations between objects can be used to decompose a complex goal into a compatible sequence of subgoals.
We show that, by using this framework, an agent can efficiently and automatically learn manipulation tasks in multi-object environments with different relations between objects. 

Self-supervised Reinforcement Learning with Independently Controllable Subgoals
Andrii Zadaianchuk 1:2, Georg Martius Fanny Yang" Max Planck Institute for Intelligent Systems, Tübingen, Germany 2 Department of Computer Science, ETH Zurich andrii hanrraaaaaaaanccuuuuuuubeigeg mpg. de
Abstract: To successfully tackle challenging manipulation tasks, autonomous agents must learn diverse set of skills and how combine them. Recently, self-supervised agents that set their own abstract goals by exploiting the discov- ered structure in the environment were shown perform well on many different tasks. In particular, some of them were applied to learn basic manipulation skills in compositional multi-object environments. However, these methods learn skills without taking the dependencies between objects into account. Thus, the learned skills are difficult to combine in realistic environments. We propose novel self- supervised agent that estimates relations between environment components and uses them to independently control different parts of the environment state. ad- dition, the estimated relations between objects can be used to decompose com- plex goal into a compatible sequence subgoals. We show that, by using this framework, an agent can efficiently and automatically learn manipulation tasks in multi-object environments with different relations between objects.
Keywords: object-centric representations, relations, self-supervised reinforce- ment learning
Introduction
Autonomous agents that need to solve manipulation tasks in environments with many objects have to master a variety of skills. In addition, such agents should be able to properly combine these skills to solve complex tasks. In modular environments, the agent must explore many different ways how it can control the environment [1]. Self-supervised agents that imagine their own goals can automate this process, and learn many skills without external reward signals [1, 2, 3, 4, 5, 6, 7]. One of the main challenges for goal-based autonomous agents is the choice of a suitable goal space and the corresponding reward function [8]. As this choice determines the difficulty of the learning, it is crucial to exploit all available structure in the environment state for construction of the goal space.
One natural way represent the state in modular environments is to use object-centric repre- sentation: the environment state is represented as a set of components, with each component corre- sponding to the state of individual object [9, 10, 11]. Such representations can be learned in an unsupervised fashion from high-dimensional observations such as images [12, 13, 14, 9, 15, 16, 17]. take high-
be readily extended autonomous dimensional data as input. A simple approach to use object-centric representations learning is to first learn how control each object individually (using the objects' representations as subgoals), and then combine learned skills to control multiple objects [10]. However, in an envi- ronment where different objects interact with each other, this method might learn an incompatible sequence of skills, i.e. achieving one of the subgoals can destroy another previously achieved sub- goal. For example, moving one object from stack of objects may change the position of the others.
Therefore, methods that object-centric representations
One line of work that aims at learning sequences of skills that are compatible is Hicrarchical Rein- forcement Learning (HRL) [18, 19, 20). In principle, hierarchical agents should be able to transform a task into a sequence of subtasks that they solve sequentially. However, to date, existing hierarchi- cal agents have mostly been applied to leam navigation reaching tasks where learned skills do
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Interaction graph
Observed environment dynamics
Random agent. TT0
x;
nnd
d
1 abj
Agent with compositional skills,
Selectivity reward, Tael
Env
Am
Soft Actor-Critic
Soval,
Goal-conditioned
policy, mg
Self-supervised training
Independently controllable subgoals
Figure 1: Our SRICS method. First, the interaction graph is inferred from observed environment dynamics containing links from cause affected entity. This gives rise subspaces that can be independently controlled, corresponding to subgoals g'. Next, the subgoals g' are used to construct a selectivity reward signal r'st- The selectivity reward I'st incentivizes the agent to only control the main entity i towards g within each subgoal g* without affecting entities outside the subgoal. SRICS learns to solve an external goal gooa by decomposing it into an ordered list of subgoals g' and solving each using SAC [24] with goal-conditioned policy πe. As result, the agent attempts to solve all the discovered subgoals one-by-one, without destroying previously solved subgoals.
interact with each other. is unclear how sensitive hierarchical agents are to possible interac- tions between learned skills. In this paper, we investigate another approach by reformulating the agent's subtasks and the corresponding reward signals. Similar to Thomas et al. [21], we train an agent such that it is motivated to control particular component of the environment state represen- tation while minimally affecting other components. Such an agent can learn to control components independently from other components, thus making the learned skills compatible with each other.
As the environment state representation is necessarily disentangled in Thomas al. [21], our method should additionally account for possible relations between components. We propose a novel selectivity reward signal that uses an interaction graph determine set of components that can be selectively controlled without interacting with the remaining scene. The interaction graph can be inferred from observed objects dynamics collected by random policy without supervision [22, 23]. Thus, we combine learning of such interaction graphs with goal-conditional reinforcement learn- ing (RL) method that operates on object-centric representations [10] and uses the selectivity reward signal. During training (schematically depicted in Fig. I), our SRICS agent (for Self-supervised Relational RL with Independently Controllable Subgoals) learns how to efficiently achieve differ- ent subgoals (and control the corresponding subspaces) while being incentivized to minimize its effects on other parts of the environment.
Our main contributions are as follows:
We show that the global interaction graph be estimated from data using recurrent graph neural network (GNN) dynamical model combined with a sparsity prior.
We propose a goal-directed selectivity reward function that allows an agent to learn how to control environment components independently from one another.
We develop SRICS, an algorithm that uses the inferred interaction graph to learn simple and independently controllable subtasks and decompose complex goal into a compatible sequence of subgoals.

---

2 Modular Goal-conditional Reinforcement Learning
We are interested in an agent that can solve multiple tasks in an environment. In particular, we consider goal-based task encodings where each task corresponds to environment state the agent has to reach, denoted as the goal state g. The task is then given to the agent by conditioning its policy ma. St, g) the goal g, and the agent's objective is to maximize the expected goal-conditional return:
Σ aine ( (r(((+1))) 8)]
where d is an unknown dynamics distribution, Prr is the state marginal distribution induced by the agent's policy and P is some distribution over the space of goals & that the agent receives for training. A common approach to define the reward function in this setting is the negative distance of the current state the goal: r(s. g) - ||s gll- In general, however, the goal space does not need to be equal to the state space S, but can be any task embedding space with potentially different dimensionality [1, 25]. As some tasks cannot be expressed as desired regions of the state space, the goal g can parameterize a more general objective r(s, g) that the agent should maximize. Many environments are modular, in the sense that an agent's overall goal (e.g. manipulating many objects) can be decomposed into different subgoals (e.g. manipulating individual objects) that can be sequentially achieved.
2.1 Object-Centric Representations
We use object-centric representations for the state space S. That is, the state space S is direct product of all object subspaces, S= x. .X.. where each S corresponds to the state of an entity. The state of entity is encoded by its position s..beere and an identifier gJ.what, The semanties behind an entity are unknown to the agent, i.e. the state of both the agent and the objects to manipulate are encoded identically. The agent has information about which objects are controllable and how they are related. Object-centric representations could be learned an unsupervised way from sequential image data [9, 12, 13] and learning them is orthogonal line of research. In this work, we focus on using object-centric representations for decomposing the goal to subgoals that are compatible with each other and can be achieved sequentially to solve the original goal.
The choice of the goal space plays a crucial role in determining the difficulty of the learning task. the environment state consists of independent parts, it is easiest to learn to control these components independently [10]. However, in the case of interactions between these components, learning to achieve subgoals in such environments and combining learned skills could be harmful to achieving the original compositional goal. For example, assume that the subgoals consist of moving objects to different positions. By solving a single subgoal without taking the other subgoals into consideration, the agent might unintentionally rearrange objects from previous subtasks, resulting in an overall deterioration instead of improvement. In the next section, present method that accounts for such dependencies by learning an interaction graph to decompose a goal into independently controllable subgoals and introduce a corresponding reward function.
3 Self-Supervised Relational RL with Independently Controllable Subgoals
In the setting we consider, at the training stage, the agent only receives a single compositional goal from the environment. The agent could try to solve the goal using the usual negative distance to the goal as a reward signal. However, achieving the compositional goal is quite a complex task by itself. This challenge can be addressed by discovering simple skills and combining them to solve the compositional goal. To achieve this, the agent needs to rely on self-supervision in the form of splitting the goal into subgoals and internally constructing the reward signal connected to each subgoal.
The agent uses data collected from the environment to discover how different parts of the environ- ment are related, including the agent itself, and then uses the discovered relations for the construction of subtasks that are solvable and can be easily combined. First, we describe how to use object-centric representations to estimate a graph of relations between objects, and then show how to utilize the learned graph during agent training, and for goal decomposition during evaluation.
3

---

(a) Hidden state dynamics
(b) State dynamics
A
Aiat
X;
A-iné
Pact
Jart
GRU
h241
A
dpesä
a
F+1
Aurt
A
duxx
X
GRU
+++
A/
dgred
F+1
dact
dent
Σ
GRU
A2
dgred
state prediction
combined effect
interaction effect
action effect
The dynamical model. For a given object j. the function d computes each of the other objects' effect on the object j using the hidden states h2 The effects from all the other objects are aggregated the interaction effect vector h?.".. Next, the function daar computes the action's 'ss effect h/ on the object j. Both effects are combined in the GRU. Finally, object's ' state estimation $/+1 estimated from the hidden state h14 using the prediction function dgred-
Figure? 2:
3.1 Estimation of the Latent Interaction Graph with a GNN Dynamical Model
Relational information in the environment can help the autonomous agent gain control over dif- ferent parts of the environment. For example, if some parts of the environment cannot be affected, the agent will be more efficient by not trying to control them. Recently, several methods to estimate this relational information in unsupervised way were proposed [23, 26, 27, 28, 29]. Most of them assume that the relations are static [23, 28, 30]. As this not the case many robotic manipulation applications, propose to similar approach suitable for constantly changing relations. For this, we use a graph neural network (GNN) [22, 31] to model the forward dynamics of the objects.
Because states could be non-Markovian, we use recurrent dynamical model. Specifically, we incorporate recurrence the GNN model by adding Gated recurrent unit (GRU) [32] to the GNN message passing operation (see Fig. 2a). We use the functions dim and d to model the object-object interaction effect and the action's 's effect, respectively. Next, both effects are combined in the GRU. More formally:
h jiin Σ dimi (ii hn). 1//200 daci (s), h GRU ([hg (gF-)
h p.at hn).
(1)
where h j.int and hg."ct are vectors representing interaction and action effects, whereas hy is the hidden state for object time step t.
To model dynamics with sparse interactions between objects, we model dia as the product of an interaction weight w (0, 1} and an interaction effect function dint-nt:
dm (h/; h/) = w d en (h/, h2).
The interaction weight wt represents the belief in the absence presence of the interaction between object and object j at time step f. We model the weight's distribution as
q (w (ss) softmax (dem (hi b2))
where dint-pres the interaction presence function. As we are interested the estimation of the connections that are necessary for predictions, additionally encourage the interaction weights distribution ? ss to be close to the sparsity prior Pprior In our case, the sparsity prior Pprior is the Bernoulli distribution with a large probability for zero (see App. G).
Finally, we use function dprea to predict the change in coordinates (see Fig. 2b):
whwher
[+1
st
w.werre
dpreed (n/-.)).
(4)
All functions in Eqs. 1-4 are modeled by small MLPSs with parameters Φ.

---

Now, as we defined all the parts of the GNN dynamical model, we describe how to estimate the inter- action graph using a variational approach. First, similar to [23]. we train our model by minimizing the negative ELBO loss:
K T-1
w.beree s41
w.weeee
2
(())
=
2x²
+ (g Pprior);
(5)
1=1
where gü.where is the prediction of the position of object j. o² is fixed variance parameter and DKL denotes the Kullback-Leibbe divergence. After training, we predict interaction weights Wy for each timestep independently. then we average them across the whole dataset. Next, we estimate the global interaction graph by thresholding the average interaction weights to find the most active relations. Finally, we identify which object is directly controlled by the actions by finding the node that is most correlated with the action variable ag In the graph of our running example as in Figure 1, we denote this node as "arm" since in all experiments the identified node corresponds to a simulated robot arm. We add the action node with index O and the corresponding edge to the most correlated object to the graph (see App. G for the details and the graph learning results).
3.2 Learning to Independently Control Objects using the Interaction Graph
In this section, we show how the agent can use the learned interaction graph to solve composi- tional goal go0a that consists of goals for individual objects gaodl The SRICS agent sequentially gains control over the objects without affecting the previously moved objects. To achieve this, the SRICS method first identifies a set of objects P that could be used to actively control ob- ject i by analyzing the discovered relations in the interaction graph. For each node i, we find the set p of all nodes that lie in a path from the action node 0 to object node i. These ances- tral nodes p are the objects that could be used by the agent to control object i. All the other nodes are not required and thus should not be affected during the manipulation of object
Next, introduce the reward signal that uses pi to in- centivize the agent to learn to control an object without moving others (line 8 of Alg. 1). In order to achieve this, we propose to replace the original subgoal g by a novel independently controllable subgoal g' that consists of the subgoal g and the ancestral nodes pi:
g
Action
'Pi
Ann
ang
Arm
g' ===(ssoal,5, PPt).
(6)
g
2nd
In contrast to the original notion of subgoal which only specifies a state component geoal, that the agent should reach, an independently controllable subgoal g' also in- cludes information about which objects should not be in- teracted with to reach the target state component.
obj
grd obj
AA obj
JId sbj
Interaction graph (left) and the indepen- dently controllable subgoal g¹ for ob- ject 3 (right).
We now formulate the goal-directed selectivity reward signal that explicitly incentivizes the agent to leave all objects except and Pi untouched. As opposed to the usual reward signal, it depends on the inde- pendently controllable subgoal g* and reads:
Tel (t,, g - g + (sel, St-1, St-1,, 1).
(7)
The first term is the usual goal-based negative distance to the goal, which is needed to learn directed control over object The second term includes the selectivity that we define as
|||| I1l Ejept E, (p) ||s/ - otherwise.
if subgoal is not solved:
sel; (St, St Pt)
||
(8)
The selectivity sel, incentivizes the agent to maximize its influence [33, 34, 35] on object i while having a minimal effect on objects j & p' (corresponding to non- oon-sscettall nodes in graph G) until the subgoal corresponding to the object i is solved. Selectivity reaches its maximum value of I when the agent changes only the state of the object i without affecting any objects j & Pi, In App. F, we show that selectivity naturally increases during learning to control the environment and that using as reward signal increases efficiency and stability.
5

---

3.3 SRICS Policy Architecture and Training
Similar to the SMORL agent [10], we use goal-conditioned attention policy for achieving subgoals. This kind of policy receives a set of object-centric representations as input together with the current subgoal representation. The aforementioned approach allows us to learn several different skills using only one policy. In addition, it is compatible with a different number of objects inputs, thus allowing to use the agent novel situations with a different number of objects. For more details on the coal-conditioned attention policy, we refer to App. E.
SRICS can be trained with any off-policy goal-conditioned RL algorithm. In particular, we use Soft-Actor Critic (SAC) [24] with Hindsight Experience Replay (HER) [25] as a method to improve sample efficiency. The training of SRICS presented in Alg.
3.4 Subgoal Ordering during Evaluation
After training, the agent can be applied to more complex tasks than the simple subtasks was trained During the evaluation stage (Fig. 13 in App. I), SRICS encodes the compositional goal given by the environment into a set of independently controllable subgoals. Subsequently, it orders them by the depth of the corresponding nodes in the interaction graph G. Due to this order, subgoals that have a large number of dependencies are attempted first and subgoals that have only a few dependencies, like the robotic arm itself, are attempted controllable subgoals makes them compatible with each other. For example, the agent has to first rearrange all objects that need to be manipulated and then try to "solve" the arm subgoal, without destroying the already rearranged objects. More details can be found in App. I.
the later subgoals. The order of the independently
Related Work
In self-supervised reinforcement learning, self-supervision refers to the agent constructing its own goals together with the corresponding reward signal and using them to learn to solve self-proposed goals [1, 29, 36, 37, 2. 3, 6, 38, 39, 40, 41, 42, 4, 10]. Self-supervised agents can acquire diverse general-purpose purpose robotic skills. In the case of complex tasks, it is often beneficial to discover simpler subgoals and learn to solve them [20]. From this point of view, recent hierarchical RL (HRL) agents [20, 19, 43, 44, 18, 45] that try to solve external tasks by proposing several levels of internal subgoals also self-super agents.
set of general-pe
Levy et al. [20]. Nachum et al. [19] and Wang et al. [43] propose to learn several goal-condi policies. In the HIRO agent [19]. lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. In contrast, the HAC agent [20] trains each level of the hierarchy independently of the lower levels. The I'HRL agent [43] additionally allows bi-directional communication among HRL levels and influence-basee training more stable and efficient. As such agents need to discover all the structure in the environ- ment while learning on several levels, such approaches struggle to solve complex tasks in modular environments [46]. Next, we review agents operating in environments where some structure is given.
exploration to make
The SMORL agent [10] exploits learned object-centric representations for gaining control over dif- ferent objects in sulf-super way and combines the learned skills for solving more complex compositional tasks. However, Zadaianchuk et al. [10] assume independence of different objects, restricting the use of the SMORL agent to settings where objects almost do not interact with each other. CURIOUS [8] and CWYC [29] exploit the modular structure of the goal space for efficient exploration in given goal space. Colas et al. [8] use policy that obtains the goal module identifier together with the goal value. Blaes et al. [29] also learn a relational graph between tasks. Both agents use given modular structure for learning curriculum [36], however, discovered subtasks are evaluated independently.
In realistic applications, autonomous agents usually do not have any well-structured representation. Nevertheless, agents can potentially infer it from data. We cover several directions that could be useful for such structure discovery. The first line of works [13, 12, 9, 14, 16] learns object-centre representations from images or videos. Such representations could be potentially used in combi- nation with the SRICS agent. The second line [23, 27, 26, 28, 30] studies how object relations can be discovered from data. The improvements in both of these lines could lead to more general self-supervised agents that use a discovered structure for the generation of goals.

---

Rearrange objects
Rearrange with 4 objocss
Relational Rearrange objects
0.23
0.22
0.2
0.30
0.3
Avg. dist.
0.173
0.1
0.3
0.5
0.123
0.14
0.13
0.73
0,72
0.30
0.30
Training examples
X10
Training examples
X10
Training examples
SRICS
SMORL
SAC+HER
HAC
Figure 3: Average distance of objects and arm to the goal positions, comparing SRICS to SMORL, SAC+HER and HAC baselines. For all the experiments, results are averaged over 5 random seeds, shaded regions indicate one standard deviation.
Experimental Results
In this section, we present our experiments that address the following questions:
How does SRICS perform compared to prior goal-conditioned RI. methods on multi-object continuous control manipulation tasks?
What is the performance gain obtained from the goal-directed selectivity reward and subgoal ordering during evaluation?
How does our agent perform in an environment with an unseen combination of objects?
We run SRICS and the baseline algorithms in the Multi- Mujti-Object Rearrange from Zadaianchuk et al. [10] and the novel Multi-Object Relational Rearrange environments. The latter environment incor- porates additional physical connections between objects such as spring connections. Both environ- ments are based on the multiworld package for continuous control tasks introduced by Nair et al. [2] and MuJoCo [47] realistic simulator. They contain 7-DoF Sawyer arm where the agent needs to manipulate a variable number of pucks on a table. the first environment, the task is to rearrange the objects from random starting positions random target positions. In the second envi- ronment, we add a spring connection between some the objects and constrain other objects to be static (see App. C). This makes the resulting interaction graph more challenging and thus provides additional insights on the sensitivity of the agent to different interactions between objects. For both environments, measure the performance of the algorithms as the average distance of all objects (including the robotic arm) to their goal positions (computed on the last step of the episode).
5.1 Comparative Analysis
Relational Rearrange with 4 objects
As manipulation tasks in compositional environ- ments can be approached from different perspec- tives, we provide a comparison with a state-of-the- art method from each perspective. In terms of prob-
Subtask success rate
SRICS
SMORL
lem assumptions, our work is closest to that of object-centric representa-
0.4
SMORL [10] which
tions for subgoals and reward construction. In con-
0.2
trast to SRICS, SMORL executes subgoals in ran-
0.0
dom order and thus can potentially destroy previ-
Object
ously solved subgoals. In addition, the SMORL agent does not have the incentive to influence the subgoal object during training. Another approach to learn goal-conditioned policy with coherent be- havior is using Soft Actor-Critic (SAC) [24] with Hindsight Experience Replay (HER) [25] relabeling. This method tries to achieve the overall goal without splitting it into subgoals. Finally, we consider the that tries to solve compositional tasks on several levels and is state of the art on several continuous control tasks.
Figure 4: Subtask success rate for SRICS and SMORL for each subtask individually during evaluation in the Relational Rear- range environment. Both methods solve Arm reaching subgoal, whereas on other sub- tasks SRICS performs better than SMORL.
Hierarchical Actor-Critic (HAC) [20] method

---

We show the results in Fig. 3 and Fig. 4. The performance of SRICS is significantly better than all other algorithms in both enviromments. SMORL is able to partially rearrange pucks on table in the simpler Multi- Object Rearrange environment. However, its random subgoals ordering is inefficient for arranging all the objects including the arm. In addition, even when evaluating only based on the puck subtasks (see App. H), SRICS outperforms SMORL, which further demonstrates the benefits of using a goal-directed selectivity reward signal. Moreover, in the more complex Multi-Object Relational Rearrange environment, the gap between SRICS's and SMORL's performance is even larger. Furthermore, in all environments SAC is only able to solve the Arm subtasks, whereas HAC performance is close to that of a random agent. We present further comparison in more challenging environments with 6 different objects and velocity- -ased state representations in App. D.
5.2 Ablative Analysis
Here, we study the importance of different ingredients of our method for the overall performance of the agent. First, we ablate the selectivity term in our reward signal, using only the negative distance between the object and the desired position ing of subgoals described in Sec. 3.4, using instead a random ordering of all subgoals. The results of the ablations are presented in Fig. 5. Both ablations significantly deteriorate the performance of SRICS, showing the importance of both the goal-directed selectivity reward signal and the correct ordering in the goal decomposition for object manipulation multi-object environments.
reward signal. We then additionally ablate the order-
Rearrange with objects
Rearrange with objects
0.235
0.33
0.30
Avg. dist.
0.133
0.19
0.16
0.125
0.100
0475
0.10
Training examples
Training examples
SRICS
w/o Selectivity
w/o Selectivity and Ordering
Figure 5: Average distance of objects and arm to the goal positions, comparing our method and two ablated variants on and 4 objects Rearrange environments.
5.3 Generalization to Unseen Object Combinations
As SRICS can be used with different sets of objects inputs, we investigate its performance seen combinations of objects. We train SRICS on the Multi-Object Rearrange environment, where 4 different combinations of 3 objects are presented. We leave out one of the combinations for evalu- ation and use the other 3 combinations for training. The performance on this modified environment indistinguishable from SRICS's performance on Multi Multi-Objectt Rearrange with 3 objects, showing that SRICS can operate on novel combinations of objects (details in App. B).
Conclusion and Future Work
In this work, we introduce SRICS, self-supervised RL method that learns the relational structure of the environment and exploits this structure to learn compatible sequence of skills to solve a dif- ficult compositional goal. In a range of experiments in multi-object environments with robotic arm manipulation tasks, we demonstrate that SRICS is effective at discovering the most active dynamic relations between objects and can successfully rearrange multiple objects even in the presence of object interactions.
There are several interesting directions for future work. First, one can extend SRICS to image-based object-centric representations, making it more applicable to realistic robotic settings where only nigh-dimensional sensory information is provided as input to the agent. Moreover, we expect that SRICS can be combined with different modular curriculum learning and exploration strategies [8, 29]. Finally, we expect that active training of the dynamic interaction graph (i.e. when the data for training collected by the agent that actively explores the environment) could further improve the discovery of important structures in the environment.
8

---

Acknowledgments
Andrii Zadaianchuk is supported by the Max Planck ETH Center for Learning Systems. We knowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S18039B).
References
[I] C. Colas, T. Karch, O. Sigaud, and P.-YY Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: short survey. arXiv preprint arXiv:2012.09830, 2021.
[2] A. Nair, V. Pong, M. Dalal, Bahl, Lin, and Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems (NIPS), 2018.
[3] V. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self- supervised reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
[4] A. Nair, S. Bahl, A. Khazatsky, V. Pong, G. Berseth, and S. Levine. Contextual imagined goals for self-supervised robotic learning. In Conference on Robot Learning (CoRL), 2019..
[5] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference Machine Learning (ICML), 2018.
[6] A. Aubret, L. matignon, and S. Hassas. Distop: Discovering topological representation to learn diverse and rewarding skills. arXiv preprint arXiv:2106.0388: 03853, 2021.
[7] A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud. Grounding language to autonomously-acquired skills via goal generation. In International Conference on Learning Representations (ICLR), 2021.
[8] C. Colas, P. Oudeyer, O. Sigaud, P. Fournier, and M. Chetouani. CURIOUS: intrinsically motivated modular multi-god goali reinforcement learning. In International Conference Machine Learning (ICML), 2019.
[9] R. Veerapaneni, J. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. B. Tenenbaum, and S. Levine. Entity abstraction in visual model- reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
[10] A. Zadaianchuk. M. Seitzer, and G. Martius. Self-supervised visual reinforcement learning with object-centric representations. International Conference on Learning Representations (ICLR), 2021.
[11] C. Devin, Abbeel, T. Darrell, S. Levine. Deep object-centric representations for general- izable robot learning. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[12] Y. Wu, O. P. Jones, M. Engeleke, and I. Posner. APEX: Unsupervised, object-centric scene segmentation and tracking for robot manipulation. arXiv preprint 20XSiv2205.. /4895, 2021.
[13] Jiang, S. Janghorbani, G. D. Melo, and S. Ahn. SCALOR: Generative world models with scalable object representations. In International Conference Learning Representations (ICLR), 2020.
[14] Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, Uszkoreit, A. Dosovitskiy, and T. Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NIPS). 2020.
[15] K. Greff, R. L. Kaufman, R. Kabra, N. Watters, C. Burgess, D. Zoran, L. Matthey, M. Botvinick, and Lerehner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML), 2019.

---

[16] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Ler- chner. MONet: Unsupervised scene decomposition and representation. arXiv preprint UrXiv:1901 11390, 2019.
[17] K. Greff, S. Steenkiste, and Schmidhuber. Neural expectation maximization. In Ad- vances in Neural Information Processing Systems (NIPS), 2017.
[18] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference Machine Learning (ICML), 2017.
[19] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficien hierarchical reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), 2018.
[20] A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Represenlations (ICLR), 2019.
[21] V. Thomas, E. Bengio, W. Fedus, Pondard, P. Beaudoin, H. Larochelle, J. Pineau, D. Precup, and Y. Bengio. Disentangling the independently controllable factors of variation by interacting with the world. arXiv preprint I 1802. 802.09444 2018.
[22] P. W. Battaglia, R. Pascanu, M. Lai, D. J. Rezende, and K. Kavukcuoglu. Interaction net- works for learning about objects, relations and physics. In Advances in Neural Information Processing Systems (NIPS), 2016.
[23] N. Kipf, E. Fetaya, K. Wang, M. Welling, and R. S. Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning (ICML), 2018.
[24] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018.
[25] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems (NIPS), 2017.
[26] S. van Steenkiste, M. Chang, K. Greff, and J. Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International Con- ference on Learning Representations (ICLR), 2018.
[27] T. N. Kipf, E. der Pol, and M. Welling, Contrastive learning of structured world models. In International Conference on Learning Representations (ICLR), 2020.
[28] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg. Causal discovery in physical systems from videos. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[29] S. Blaes, M. V. Pogancic, J. Zhu, and G. Martius. Control what you can:
Intrinsically motivated task losk-planning agent. In Advances in Neural Information Processing Systems (NIPS), 2019.
[30] S. Löwe, D. Madras, R. Zemel, and M. Welling. Amortized causal discovery: Learning to infer causal graphs from time-series data. arXiv preprint arXiv:2006. 10833, 2020.
[31] Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR), 2016
[32] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, 2014.
[33] M. Seitzer, B. Schölkopf, and G. Martius. Causal influence detection for improving efficiency in reinforcement learning. arXiv preprint arXiv:2/06.03443, 2021.
[34] R. Zhao, Y. Gao, Abbeel, V. Tresp, and W. Xu. Mutual information state intrinsic control. arXiv preprint arXiv: 103.00007,,022022
10

---

[35] A. Klyubin, D. Polani, and C. Nehaniv. Empowerment a universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, 2005.
[36] Forestier, R. Portelas, Y. Mollard, and P.-.. Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning. arXiv preprint arXiv: 2017.
[37] C. Colas, T. Karch, N. Lair, J.-M. Dussoux, C. Moulin-Frieer F. P. Dominey, and P.-Y. Oudeyer. Language as a cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[38] A. Baranes and P.-Y. Oudeyer. Active learning inverse models with intrinsically motivated goal exploration in robots. Robotics Auton. Syst., 2013.
[39] A. Péré, S. Forestier, O. Sigaud, and P. Oudeyer. Unsupervised learning of goal spaces for intrinsically motivated goal exploration. In International Conference on Learning Representa- fions (ICLR), 2018.
[40] K. Hausman, J. T. Springenberg. Z. Wang, N. Heess, and M. A. Riedmiller. Learning embedding space for transferable robot skills. In International Conference on Learning Rep- resentations (ICLR), 2018.
[41] C. Lynch, M. Khansari, T. Xiao, V. Kumar, Tompson, S. Levine, and Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
[42] Y. Wang, G. N. Narasimhan, X. Lin, B. Okorn, and D. Held. ROLL: Visual self-supervised
reinforcement learning with object reasoning. In Conference on Robot Learning (CoRL), 2020.
[43] R. Wang, Yu, B. An, and Z. Rabinovich. 12hrl: Interactive influence-based hierarchical reinforcement learning. In Proceedings of the 29th International Joint Conference on Artificial Intelligence. IJCAI, Yokohama, Japan, pages 3131-3138, 2020.
[44] S. Li, L. Zheng. Wang, and C. Zhang. Learning subgoal representations with slow dy-
namics. In International Conference Learning Representations, 2021. URL https: //openrview net. watt/ooumm/aii www.ord wdw..wwsswww..........
[45] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
[46] Z. Dwiel, M. Candadai, M. Phielipp, and Bansal. Hierarchical policy learning sensitive to goal space design. arXiv preprint arXiv: 190.013795000337,20000
[47] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.


Reviews:
[{"review_id": "heZUhA09Q76", "reviewer": "Reviewer_uiwp", "summary": "This paper presents an object-centric self-supervised reinforcement learning (RL) method in multi-goal RL settings. The major problem in prior works is that they assume each object in the environment is independently controllable without interference, which prevents the skills from being combined to solve compositional goals in realistic environments. To lift this limitation, the authors propose to learn the interaction graph between objects by recurrent graph neural network (GNN). Based on this graph, they derive a sequence of subgoals that can be controlled independently without interference, for the agent to execute. Also, they add selectivity rewards that encourage the agent to maximize the influence on the goal object while minimizing the interference to the other objects. The experimental results show that the proposed method outperforms several baselines including the recent object-centric RL and typical RL methods in multi-goal RL settings. Furthermore, the paper shows that the proposed method can generalize to unseen compositional goals.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The insight in this paper is important. Typical multi-goal RL and hierarchical RL approaches overlook the potential compositional structure and interference between objects, which limits the generalization and data efficiency. The learned interaction graph is useful for decomposing a compositional goal into several pieces of independently controllable subgoals for expediting the learning process.\n- The selectivity reward is an interesting design from the self-supervised RL point of view.\nCons:\n- Ordering the subgoals by the interaction graph is a sound approach to derive a sequence of independent subgoals.\n\nCons:\n- The major issue of this paper is notation. There are too many notations in this paper without a systematically or smooth introduction to them. For example, in Section 3.1, there are many similar notations (e.g., d_oo, d_o, d_pred, d_rel-eff), but it's hard to see their relation unless you read this section multiple times. I would suggest either having a list of notations with brief explanation or plotting a figure to depict their relationship. I believe that the readers can digest these materials better.\n\n- Lack of intuitive explanation or high-level overview before explaining the details. For example, I have no idea why do you need to estimate set $P^i$ in all paths from the action variable to object $i$? What is action variable to object $i$? Besides, I'm confused about the difference between $g^i$ and $(s^{goal,i}, P^i)$. I agree $s^{goal,i}$ is a subgoal, but why do we need $P^i$ for defining the \"independently controllable subgoal\"?\n\n- Lack of baseline explanation. As SMORL is an important baseline, I would like to know more about how is SMORL different from the proposed method? For now, the paper just said SMORL uses object-centric representations and reward constructions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed idea is interesting and technically sound. Though the clarity and readability need lots of works, I think this idea is worth presenting at the conference. I would adjust my ratings if the author can improve their writing.\n\n---\n\nPost rebuttal:\n\nThe author's response addressed my questions. I've raised the score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- How is selectivity compared with empowerment rewards? \nForrester, Russ. \"Empowerment: Rejuvenating a potent idea.\" Academy of Management Perspectives 14.3 (2000): 67-80.\n\n- In Equation 2, what's the purpose of $p(. | s_t, a_t) = \\mathcal{N}(s^{j, where}_{t+1}, \\sigma^2I)$? I didn't see it in other parts.\n\n- In Equation 5, $s^{j, where}_{t}$ should be $s^{j, where}_{t+1}$?\n\n- What is the goal space in all experiments? Is it a joint space of all objects? I suppose the baselines should use the joint space; otherwise, they cannot achieve the compositional goal. What I mean by joint space is a vector that concatenates all objects' information (e.g., positions) to a 1-D vector. I ask this question because the proposed method seems to be able to use single object information as the goal representation because the agent just needs to know a subgoal that corresponds to a single object. For a fair comparison and comprehensive study, one should use the same goal space or test both goal spaces."}}, {"review_id": "ZVjJ-K6bsH", "reviewer": "Reviewer_BPut", "summary": "In this paper, a new representation learning that considers the relations between environmental objects is proposed, then a relation graph of objects is used to estimate the dependencies. Finally, the agent selects subgoals and learns to achieve them and also minimizes the influence on other objects. Experiments on locomotion tasks show the proposed method achieves higher performance compared with several previous HRL methods and representation learning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe relations of objects have been investigated in MBRL, object-oriented RL, but from the perspective of HRL, it seems to have few works currently, so this paper is interesting and novel to some extent.\n\nExperiments show some promising results.\n\nWeaknesses:\n\nThe authors claim previous works' strong assumption of fully independent objects, while the experiments do not well support this claim.\n\nThe literature review covers many related works, but some recent works are missing, such as [1-3], where [1,2] performs better than baselines in this work.\n\nSome formulations are not explained well. Such that the function d_{rel-eff} and d_{rel-press}, what objective ensures these functions have the effect?\n\nI am not sure whether this kind of interaction graph restricts the scenarios to locomotion tasks. Since many atari games, some objects are not task-irrelated, so learning a subgoal as well as minimizing the effects on other objects seems not that important.\n\n1. I2HRL: Interactive Influence-based Hierarchical Reinforcement Learning.\n2. Learning Subgoal Representations with Slow Dynamics.\n3. DisTop: Discovering a Topological representation to learn diverse and rewarding skills", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on the review above, I give a weak accept to this paper, I think some improvements should be further made, such as the clarity of some formulations, and more analysis to support the claim and motivation, more recent baseline comparisons.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the weaknesses.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "UY-pVu6jFwo", "reviewer": "Reviewer_h8jP", "summary": "The paper presents an approach that can utilize object-centric representations for solving complex goals by decomposing them into sub-goals. In particular, first the dynamics of an environment are captured in an interaction graph between different objects (using their object-centric representation such as their identity and positions). After that, various sub-graphs of the interaction graph are considered to be sub-goals. The sub-goals are ordered by the depth of their sub-graph to ensure that objects that have a lot of dependencies are attempted first, such that the corresponding sub-goals are not destroyed while attempting other sub-goals in the future. In addition, the robot is incentivized to only manipulate objects within the current sub-goal through a selectivity reward. Recurrent Graphical Neural Networks (GNN) with GRUs are used to represent the interaction graph. The goal-conditioned policy used to reach the sub-goals is trained using SAC. The method is evaluated on a simulated multi-object environment with up to 4 objects and some objects having relations between each other. It is shown to outperform a previous method SMORL, which also uses object-centric representations, a goal-conditioned baseline based on SAC with hindsight experience replay (HER), and a hierarchical actor-critic baseline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Decomposing difficult problems into sub-goals is important for scaling up robot learning. Using object-centric representations for representing sub-goals enables a significant reduction of the problem dimensionality. \n- Encouraging interaction of objects only within sub-goals and ordering sub-goals by the depth of their dependency graph is important for a structured exploration without undoing previously achieved sub-goals.\n- The paper is well-written and easy to understand and follow.\n- The method is shown to outperform baselines in a simulated multi-object environment.\n\nWeaknesses:\n- Right now experimental evaluation only includes manipulation of a small number of simple objects. It would be interesting to see more experiments on more complicated objects and higher-dimensional environments.\n- Presented method requires collecting robot interactions in order to pre-train the object interaction graph. It would be useful to know the amount of data used for pre-training to better compare it to other methods, as other methods such as SAC with HER do not require pre-training.\n- The objects in the experiments are represented by their identity and position in the environment. It would be interesting to see how the method works with more examples of object representations, for example the ones that include velocity for more dynamic tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presented method is a great step towards a better structured learning of sub-goals. However, the scope of experiments would need to be extended in order to better understand the impact of the method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to \"Strengths And Weaknesses\" for a detailed list of issues.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are generally positive about the idea and contribution, and agree that it is important to scaling up robot learning. Post-rebuttal, they are all weakly positive. 
Decision: Accept (Poster)