Regularity as Intrinsic Reward for Free Play
Authors: Cansu Sancaktar
Justus Piater
Georg Martius
Venue: NeurIPS 2023 poster (2023)
Keywords: Intrinsic Motivation
Reinforcement Learning
Model-based Planning
Regularity
Manipulation
Zero-shot Generalization
Unsupervised Exploration
TL;DR: We propose regularity as an intrinsic reward signal, to not bias agents towards chaos by naive novelty seeking objectives and instead favor balance and alignment.

Abstract: We propose regularity as a novel reward signal for intrinsically-motivated reinforcement learning. Taking inspiration from child development, we postulate that striving for structure and order helps guide exploration towards a subspace of tasks that are not favored by naive uncertainty-based intrinsic rewards. Our generalized formulation of Regularity as Intrinsic Reward (RaIR) allows us to operationalize it within model-based reinforcement learning. In a synthetic environment, we showcase the plethora of structured patterns that can emerge from pursuing this regularity objective. We also demonstrate the strength of our method in a multi-object robotic manipulation environment. We incorporate RaIR into free play and use it to complement the model’s epistemic uncertainty as an intrinsic reward. Doing so, we witness the autonomous construction of towers and other regular structures during free play, which leads to a substantial improvement in zero-shot downstream task performance on assembly tasks.

Regularity as Intrinsic Reward for Free Play
Cansu Sancaktar'
Justus Piater University of Innsbruck
Georg Martius 1.3
Maa Planck Institute for Intelligent Systems, Germany
@University of Tübingen Germany
Austria
cansu. waaggasssnnctttttttebmnen.....
Abstract
We propose regularity as a novel reward signal for intrinsically-motisee rein- forcement learning. Taking inspiration from child development, we postulate that striving for structure and order helps guide exploration towards subspace of tasks that are favored by naive uncertainty- -based intrinsic rewards. Our generalized formulation of Regularity Intrinsic Reward (RaIR) allows us to operationalize it within model-based reinforcement learning. showcase the plethora of structured patterns that can emerge from pursuing this regularity objective. We also demonstrate the strength our method in a multi- object robotic manipulation environment. We incorporate RalR into free play and use to complement
synthetic environment, we
model's epistemic uncertainty as an intrinsic reward. Doing so, we witness the autonomous construction of towers and other regular structures during free play, which leads to substantial improvement in zero-shot downstream task performance assembly tasks. Code and videos available at https://siten. http:/oossss.gg.e....e/ooooooo.. com/ci////////e www.///ieeaaarrrrr./.... project..
Figure 1: Regularity as intrinsic reward yields ordered and symmetric patterns. In SHAPEGRID- WORLD (top row) and CONSTRUCTION (bottom row), we showease the generated constellations when maximizing our proposed regularity reward RaIR with ground truth (GT) models.
Introduction
Regularity, and symmetry as specific form of regularity, are ubiquitous in nature as well as in our manufactured world. The ability detect regularity helps identify essential structures. minimizing redundancy and allowing for efficient interaction with the world [1]. Not only do we encounter symmetries arts, design, and architecture, but our preference also showcases itself in play behavior. Adults and children have both been observed to prefer symmetry in visual perception, where symmetric patterns are more easily detected, memorized, and copied [2, 3]. Several works in developmental psychology show that regular patterns and symmetries are actively sought out during free play in children as well as adults [4-6].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Considering this in the context of a child's developmental cycle is intriguing. Studies show that children at the age of 2 exhibit a shift in their exploratory behavior. They progress from engaging in random actions on objects and unstable arrangements to purposefully engaging in functional activities and intentionally constructing stable configurations [7, 8, 5]. Bailey [4] reports that by 5 years of age, children build more structured arrangements out of blocks that exhibit alignment, balance, and examples of symmetries [5].
Despite the dominance of regularity in our perceptual systems and our preference for balance and stability during play, these principles are not yet well investigated within aerinsically-motivvte reinforcement learning (RL). One prominent intrinsic reward definition is novelty, i.e. the agent is incentivized visit areas of the state space with high expected information gain [9-12]. However, one fundamental problem with plain novelty-seeking objectives is that the search space is often unconstrained and too large. As an agent only has limited resources to allocate during play time, injecting appropriate inductive biases is crucial for sample efficiency, good coverage during explo- ration, and emergence of diverse behaviors. As proposed by Sancaktar al. [12], using structured world models to inject a relational bias into exploration, yields more object and interaction-related novelty signals. However, which types of information to prioritize are not explicitly encoded in any of these methods. The direction of exploration is often determined by the inherent biases in the practical methods deployed. With imperfect world models that have a limited learning capacity and mnite-horizon planning, novelty-seeking methods are observed to prefer "chaotic" dynamics, where small perturbations lead to diverging trajectories, such as throwing. flipping, and poking objects. This in turn means that behaviors focusing on alignment, balance, and stability are overlooked. Not only are these behaviors relevant, as shown in developmental psychology. they also enable expanding and diversifying the types of behavior uncovered during exploration. As the behaviors observed during exploration are highly relevant for being able to solve downstream tasks, a favoring exploration will make it hard solve assembly tasks, such as stacking. Indeed, successfully solving assembly tasks with more than 2 objects has been challenge for intrinsically-motivvaee reinforcement learning.
We pose the question: how can we define an intrinsic reward signal such that RL agents prefer structured and regular patterns? We propose RaIR: Regularity as Intrinsic Reward, which aims to achieve highly ordered states. Mathematicallyy, of a suitable state description. Entropy and symmetries have been linked before [13, 14], however, we follow general notion of regularity, i.e. where patterns reoceur and thus their description exhibits high redundancy low entropy. In this sense, symmetries are a consequence of being ordered [15]. Regularity also means that the description is compressible, which is an alternative formulation. As argued by Schmidhuber [16], aiming for compression-progres, is a formidable curiosity signal, however, it currently unclear how efficiently predict and optimize for it.
operationalize this idea using entropy minimization
After studying the design choices in the mathematical formulation of regularity and the relation to symmetry operations, we set out to evaluate our regularity measure in the context of model-based reinforcement learning/planning, as it allows for highly sample-efficien complex tasks zero-shot, as shown in Sancaktar et al. [12]. To get clear understanding of RaIR, first investigate the generated structures when directly planning to optimize it using the ground truth system dynamics. A plethora of patterns emerge that are highly regular, as illustrated in Fig. 1.
exploration and solving
Our ultimate goal is, however, to inject the proposed regularity objective into a free-play phase, where a robot can explore its capabilities in task-free setting. During this free play, the dynamics model is learned on- on-the-go. We build on CEE-US [12], a free-play method that uses an ensemble of graph neural networks as a structured world model and the model's epistemic uncertainty as the only intrinsic reward. The epistemic uncertainty is estimated by the ensemble disagreement and acts effective novelty-seeking signal. We obtain structure-seeking free play by combining the conventional novelty-secking objective with RaIR.
Our goal is to operationalize regularity, which is a well-established concept in developmental psychology, within intrinsically motivated RL. Furthermore, we showcase that biasing information- search towards regularity with RaIR indeed leads to the construction of diverse regular structures during play and significantly improves zero-shot performance downstream tasks that also favor regularity, most notably assembly tasks. Besides conceptual work on compression [16, 17], to our knowledge, we are the first to investigate regularity as an intrinsic reward signal, bridging the gap between the diversity of behaviors observed in children's 's free play and what can achieve with artificial agents.
2

---

Maximize Regularity Epistemic Uncertainty intriinsic reward
Ensemble of Models
O
Relation Matrix
Generate data
Update agent's madel
Occurrence
Plans in imeginetion horizon M
4(x:
RaiR Entropy -H((ss. 51)
RaIR( (M RaIR(/
RaIR(
5g
(a) RaIR CEE-US: Regularity with Epistemicc Uncertainty Structured World Models
(b) RaIR compulation
Figure Regularity intrinsic reward during free play. (a) RaIR + CEE-US uses model-based planning to optimize H timesteps into the future for the combination of RalR (Eq. 2) and epistemic uncertainty (ensemble disagreement of world models). (b) Here, for RalR we use the absolute difference vector between objects: @(8t, 8j) = {(I $j,= 11. [ $jy 1))
Method
First, we introduce our intrinsic reward definition for regularity. Then, we present its practical imple- mentation and explain how combine this regularity objective into model learning within free play.
2.1 Preliminaries
In this work, we consider environments that can be described by fully observable Markov Decision Process (MDP), given by the tuple fag, rer). with the state-space S E RT* the action-space A E R"* the transition kernel f S x A - S, and the reward function r. Importantly, we consider the state-space to factorized into the different entities, e.g. S (Swy)) N Syobat for the state space of a robotic agent and N objects. We use model-based reinforcement learning, where data from interactions with the environment is used to learn model f of the MDP dynamics [18]. Using this model, we consider finite-horizon (H) optimization/planiing for undiscounted cumulative reward:
H-1
a7 arg ITIXX
Σ h=0
r88thh, Ot+h: 81+h+1),
S.
where 86+& are imagined states visited by rolling out the actions using f. which is assumed to be deterministic. The optimization of Eq. is done with the improved Cross-Entropy Method (iCEM) [19] in a model predictive control (MPC) loop, i.c. re-planning after every step in the environment, Although this is not solving the full reinforcement learning problem (infinite horizon and stochastic environments), it is very powerful in optimizing for tasks on-the-fly fly and is thus suitable for optimizing changing exploration targets and solving downstream tasks zero-shot.
2.2 Regularity as Intrinsic Reward
Quite generally, regularity refers the situation in which certain patterns reoccur. Thus, formalize regularity as the redundancy in the description of the situation, to measure the degree of sub-structure recurrence. A decisive question is: which description should use? Naturally, there is certain freedom in this choice, as there are many different coordinate frames. For instance, we could consider the list of absolute object positions rather relative representation of the scene.
To formalize, we define a mapping + S - {X}+ from state to a multiset (x)+ + of symbols (e.g. coordinates). A multiset is a set where elements can occur multiple times, e.g. {a, a, b)b This multiset can equivalently be described by a tuple (), where X is the of the unique elements, and X + z is function assigning the multiplicity, i.c. the number of occurrences m(x) for

---

the elements z E X. For the previous example, we get (fa, b), {a 2, b. 1)). Given the multiset (X,m) {x}*, we define the discrete empirical distribution. also referred as a histogram, by the relative frequency of occurrence p(a)) = m(x)/ Erex m(x') for x E X.
We define the regularity reward metric using (negative) Shannon entropy [20] of this distribution as:
m(x)
TRAIR (s) = -H(((s))
Σ
log
with (X, m)) = ((s): p(z)
(2)
m(x²)
ZEX
x'X
We will now discuss concrete cases for the mapping b i.e. how to describe particular state.
Direct RaIR. In the simplest case, we describe the state s directly by the properties of each of the entities. For that, we define the function @ Sabj + {X}+, that maps each entity to set of symbols and obtain (s) = / ( as a union of all symbols. The symbols can be, for instance, discretized coordinates, colors, or other properties of the entities.
Let us consider the example where d is extracting the object's Cartesian &: and y coordinates in rounded manner d(s) [[sx), [sy as shown in Fig. 3. The most irregular configu- 13
{6, (6)1
occurrence
ration would be when no two objects share the same rounded value in and y. The object configuration becomes more and more regular the more objects share the same [<r] and Ly/ coordinates. The most regular configuration is if all objects are in the same place. Note that this choice favors an aligned configuration, and not invariant under global rotations.
10 16
(() (0. 10, 6, 10,0, 0,,
Figure 3: Illustration of direct
RalR for Φ =[1=], Lv1}.
Relational RaIR of order k. Our framework for regularity quantification can easily be extended to a relational perspective, where we don't compute the entropy over aspects of individual entity properties, but instead on their pairwise or higher-order relations. This means that for a k-order regularity measure, we are interested in tuples of l entities. Thus, the mapping function á no longer takes single entities as input, but instead operates on k-tuples:
(Sob)) j {x}-
(3)
é is a function that describes some relations between the k input entities by a set of symbols. For k-order regularity, the multiset o, over which compute the entropy, is now given by with P =
(A)
(
merged from all k- k-permutations of the N entities, denoted P((1, N), k). In the of permutation invariant ø, Eq. 4 regards only the combinations C( NJ. k)). Note that direct RaIR is equivalent to the relational RaIR of order Given the mapping 4 the RaIR measure is computed as before with Eq. 2.
Depending the order & and the function di we can select which regularities are going to be favored. Let us consider the example of a pairwise relational RaIR (k = 2), where & computes the relative po- sitions: o( $y) {[st- s, }. and rounding is performed elementwise. Whenever two entities have the same relative position to each other, the redundancy is detected. For = our regularity measure would be able pick up sub-patterns composed of three objects, such triangles and so forth.
As we are interested in physical interactions of the robot with objects and objects with objects, we choose RaIR of order h = 2 and explore various é functions.
2.2.1 Properties of RaIR with Pairwise Relations and Practical Implementation
For simplicity, we are considering the following that é maps to a single symbol. Then for pairwise relationships (k = 2), RalR can be implemented using relation matrix F E X N The entries Fi₃ are given by ((s;, 8;) with 8;, 8, E Sunj- After constructing the relation matrix, we need the histogram of occurrences of unique values in this matrix to compute the entropy (Eq. 2). For continuous state spaces, the mapping function needs to implement a discretization step, which we implement by a binning of size b. For simplicity of notation, we reuse the rounding notation [.] for this discretization

---

Table 1: Properties RaIR with different & regarding symmetry operations. The first block indicates to which operations RaIR is invariant, ignoring rounding (a.a.: axes aligned). The second block assesses whether a pattern, where the given symmetry operation maps several entities to another, has increased regularity. Rounding and absolute value are elementwise. Distance d is also rounded.
invariant?
favored / increases RaIR?
symmetry operation
direct rel. pos |rel. pos]
distance d(s, 53)
direct rel. pos |rel. pos| distance
[s:] [s
s;]
[[8 -
s,
[sc]
[8
s,
[1: s,1l
d(s; 81) Sum
translation
X
X
translation a.a.
/
/
/
rotation
X
X
rotation 90"
X
X
reflection
X
x
X
X
x
/
reflection a.a.
X
/
glide refl.
x
X
X(1)
x(1)
glide refl. a.a.
This is for one glide refl. operation. RaIR is increased for 2 glide refl. composition as it collapses onto transl.
step. This bin size b determines the precision of the measured regularity. In practice, we do not apply 6 on the full entity state space, but on a subspace that contains e.g. the x-y((2) positions.
To understand the properties of our regularity measure for different e we present in Table 1 a catego- rization using the known symmetry operations in 2D and the following á (applied to x-y positions): di-
rect ((s;) = Is (see previous section), relative position (difference vector) ((s;, sy) = [8; 1 sj]. ab- sy II. and Euclidean distance @ 8,) = [||s;
solute value of the relative position' á(s;, 8,) = -=
sy [1. Figure 2b illustrates the RaIR computation using the absolute value of the relative position.
In Table 1, we first consider whether the measure is invariant under symmetry operations. That means if the value of RalR stays unchanged when the entire configuration is transformed. We find that both Euclidean distance and relative position are invariant to all symmetry operations. The second and possibly more important question is whether a configuration with sub-structures that symmetry has a higher regularity value than without, i.e. will patterns with these symmetries be favored. We find that Euclidean distance favors all symmetries, followed by absolute value of the relative position. A checkmark in this part of the table means that the more entities can be mapped each other with the same transformation, the higher RaIR. Although the Euclidean distance seems favorable, we find that it mostly clumps entities and creates fewer alignments. To get a sense of the patterns scoring high in the regularity measure, Fig. showcases situations that emerge when RaIR with absolute value of relative position is optimized (details below).
2.3 Regularity in Free Play
Our goal is explicitly put the bias of regularity into free play via RaIR, in Fig. 2a. What want to achieve is not just that the agent creates regularity, but that it gathers valuable experience in creating regularity. This idcally leads to directing exploration towards patterns/arrangeemnnts that are novel.
We propose RaIR augment plain novelty-seeking intrinsic rewards, in this work specifically ensemble disagreement [10]. We choose ensemble disagreement because 1) we need reward definition that allows predict future novelty, such that we use it inside model-based planning (this constraint makes methods relying on retrospective novelty such as Intrinsic Curiosity Module (ICM) [9] ineligible), and we want to use the models learned during free play for zero-shot downstream task generalization via planning in a follow-up extrinsic phase. It has been shown in previous works that guiding exploration by the model's own epistemic uncertainty, approximated
'The rounding and the absolute value functions applied coordinate-wise.
5

---

via ensemble disagreement, leads to learning more robust world models compared to e.g. Random Network Distillation (RND) [21] (Suppl. J.1), resulting in improved zero-shot downstream task performance [12]. That is why we choose ensemble disagreement to compute expected future novelty,
We train an ensemble of world models {(fem) ) I. where M denotes the ensemble size, The model's epistemic uncertainty is approximated by the disagreement of the ensemble members' predictions. The disagreement reward is given by the trace of the covariance matrix [12]:
"Dis tr Cov (1 fem (St, az) m ,,..., M))).
(5)
We incorporate our regularity objective into free play by using a linear combination of RaIR and ensemble disagreement. Overall, we have the intrinsic reward:
lintrinsic
TRaIR A rDis.
(6)
where A controls the trade-off between regularity and pure epistemic uncertainty.
Model-based Planning with Structured World Models To optimize the reward function on-the- fly, we use model-based planning using zero-order trajectory optimization, as introduced in Sec. 2.1. Concretely, use CEE-US [12]. which combines structured world models and epistemic uncertainty (Eq. 5) as intrinsic reward. The structured world models are ensembles of meessage-passing Graph Neural Networks (GNNs) [22]. where each object corresponds to a node in the graph. The node attributes {st, E Sayj = 1....,NI .N} are the object features such as position, orientation, and velocity time step t. The state representation of the actuated agent Sobbtt E Syobor similarly contains position and velocity information about the robot. We treat the robot as a global node in the graph [12]. We refer to the combination of RaIR with ensemble disagreement, medium--hoiizon planning (20-30 time steps), and structured world models as RaIR + CEE-US.
3 Experiments
We evaluate RaIR in the two environments shown Fig. 1.
ShapeGridWorld is a grid environment, where each circle represents an entity/agent that is controlled separately in x-y directions. Entities are controlled one at a time. Starting from timestep = 0, the entity with = 1 is actuated for T timesteps, where T' is the entity persistency. Then, at = T, actua- tion switches over to entity i and we keep iterating over the entities in this fashion. Each circle treated as an entity for the regularity computation with 2D-entity state space Sabj with a-y positions.
Fetch Pick & Place Construction is an extension of the Fetch Pick & Place environment [23] to more cubes [24] and large table [12]. An end-effector-comtrollde robot arm is used to manipulate blocks. The robot state Spobr R10 contains the end-effector position and velocity, and the gripper's state (open/close) and velocity. Each object's 's state Sunj E IR12 is given by its pose and velocities. For free play, we use 6 objects and consider several downstream tasks with varying object numbers.
3.1 Emerging Patterns in SHAPEGRIDWORLJ, and CONSTRUCTION with RaiR
To get a sense of what kinds of patterns emerge following our regularity objective with RaIR, we do planning using ground truth (GT) models, i.c. with access the true simulator itself for planning. We perform these experiments to showcase that we can indeed get regular constellations with our proposed formulation. Since we can perform multi-horizon planning without any accumulating model errors using ground truth models, we can better investigate the global/local optima of our regularity reward. Note that as we are using zero-order trajectory optimizer with a limited sample budget and mnite-horizon planning, we don't 't necessarily converge to the global optima. We use ((s,, 8,) = {[[[s st.. $j.= 1I. || + for RaIR in both environments. The emerging patterns are shown in Fig. 1.
In the 2D SHAPEGRIDW WORLD environment, indeed observe that regular patterns with transla- tional, reflectional (axis-aligned), glide-reflectional (axis-aligned), and rotational symmetries emerge. Regularity not restricted to spatial relations and can be applied to any set of symbols. To showcase this, we apply RalR to colored SHAPEGGIDWORLD where color is part of Sayy. Generated patterns are not just regular in r-y but also in color, as shown in Fig. S7 and the top right examples in Fig. 1.

---

For CONSTRUCTION, we also observe complex constellations with regularities, even stacks of all 6 objects. Since we are computing RaIR on the x-y positions, a stack of 6 is the global optimum. The optimization of RalR for this case is shown in Fig. 4. Note that stacking itself is a very challenging task, and was so far only reliably achievable with reward shaping or tailored learning curricula [24]. The fact that these constellations appear naturally from our regularity objective, achievable with a planning horizon of 30 timesteps, is by itself remarkable.
Timessep
Figure 4: RaIR throughout a rollout start- ing from random initial configuration when optimizing only for regularity with the GT model.
Additional example patterns generated CONSTRUCTION with RaIR on the x-y-2 positions can be found in the Suppl. A. In that case, a horizontal line on the ground and a vertical line into air, i.e. a stack, are numerically equivalent with respect to RalR. Choosing to operate on the z-y-subspacc injecting the direction of gravity and provides bias towards vertical alignments. We also apply RaIR to custom CONSTRUCTION environment with different shapes and masses (cubes, columns, balls and flat blocks) and once again observe regular arrangements, as in Fig. S8. More details in Suppl. F,
3.2 Free Play with RaIR in CONSTRUCTION
We perform free play in CONSTRUCTION, i.c. only optimize for intrinsic rewards, where we learn models on-the-go. During free play, we start with randomly initialized models and an empty replay buffer. Each iteration of free play consists of data collection with environment interactions (via online planning), and then model training the collected data so far (offline).
In each iteration of free play, we collect 2000 samples (20 rollouts with 100 timesteps each) and add them to the replay buffer. During the online planning part for data collection, we only perform infer- ence with the models and no training performed. Afterwards, we train the model for fixed number of epochs on the replay buffer. We then continue with data collection in the next free play iteration. More details can be found in Suppl. H. For this intrinsic phase, we combine our regularity objective with ensemble disagreement as per Eq. 6. The goal is to bias exploration and the search for information gain towards regular structures, corresponding to the optima that emerge with ground truth models, as shown in Fig. 1. We also show results for the baselines RND [21] and Disagreement (Dis) [10], using the same model-based planning backbone as CEE-US (see Suppl. J). The Dis baseline also uses ensem- ble disagreement as intrinsic reward, however unlike CEE-US, only plans for one step into the future.
we analyze the quality of data generated during free play, in terms of observed interactions, for RaIR + CEE-US with the augmentation weight X = 0.1, pure RaIR run with no information-gal -aain component in the intrinsic reward (A = 0), CEF-US, as well as RND and Disagreement. For pure RalR, we observe a decrease in the generated interactions. This has two reasons: 1) RaIR only aims to generate structure and the exploration problem is not solved, 2) once the controller finds a plan that leads to optimum, even if it is local, there is no incentive to destroy
In Figure 5,
RaIR
RaIR CEE-US
CEE-US
RND
Dis
50%
50%
30%
30%
relative time
25%
25%
15%
0%
056
50 100 150 200 250
300
0
50 100 150 200 250 300
0
50 100 150 200 250 300
50 100 150 200 250 300
Free Play Iteration
Free Play iteration
Free Play iteration
Free Play Iteration
(a) 1 object moves
(b) 2 more objects move
(c) object(s) in air
(d) object(s) flipped
Figure 5: Comparison of interactions during free play in CONSTRUCTION when combining ensemble disagreement with RaIR (with x= 0.1) compared to CEE-US, pure RaIR, RND and Dis. These metrics count the relative amount of timesteps the agent performs certain types of interactions in the 2K transitions collected at each free play iteration.
(a) I object moves checks the amount of time the agent spends moving only one object. (b) 2 or more objects move checks if least 2 objects are (c) Object(s) in air means one or more objects are in air (including being held in air by the agent or being on top of another block). (d) Object(s) (s) flipped checks for angular velocities above threshold for one or more objects, i.e. if they are rolled/flipped. We used 5 independent seeds.
moving at the same time.
7

---

Iteration 251
Iteration 255
lteration 259
Iteration 275
Iteration 284
Iteration 286
Iteration 296
Figure 6: Snapshots from free play with RaIR + CEE-US. We showcase snapshots of highest RaIR values, equivalent to lowest entropy, from exemplary rollouts at different iterations of free play. Following the regularity objective, stacks and alignments are generated.
it, unless a plan that results in better regularity can found within the planning horizon. There no discrimination between "boring" and "interesting" patterns with respect to the model's current capabilities. This in turn means that the robot creates e.g. a (spaced) line, which a local optimum for RaIR, and then spends the rest of the episode, not touching any objects to keep the created alignment intact. With the injection of some disagreement RaIR + CEE-US, we observe improved interaction metrics throughout free play in terms of 2 more object interactions and objects being in the air (either being lifted by the robot or being stacked on top of another block). In practice, since the ensemble of models tends to hallucinate due imperfect predictions, even for pure RaIR we observe dynamic pattern generations, as reflected in the interaction metrics (more details in Suppl. C). For the plain disagreement case with CEE-US, more flipping behavior, and less air time are observed during free play, since the agent favors chaos. We also observe that the baselines RND and Disagreement (planning horizon 1) produce less interaction-rich data during free play. Especially for Disagreement, this further shows that planning for future novelty is an essential component for free play.
Another reason why disagreement in RalR CEE-US US is helpful is due to the step-wise landscape of RaIR as shown in Fig. 4. Here, combin- ing RaIR with ensemble disagreement effectively helps smoothen this reward function, making easier to find plans with improvements in regularity with imperfect world models.
RaiR
-200
-22
-2.5
200 300
In Fig. 7, we report the highest achieved RaiR value in the collected rollouts throughout free play. We observe that pure RaiR and RaIR + CEE-US indeed find more regular structures during play compared to the baselines. Some snapshots of regular structures observed during a RaIR + CEE-US free play run are illustrated in Fig. 6. Results for ((s.,))) = |ss sy] can be found in Suppl. B.
Free Play Iteration
Figure 7: Highest RaIR value throughout free play for pure RaIR, RaIR + CEE-US, CEE-US, RND and Dis.
3.3 Zero-shot Generalization to Assembly Downstream Tasks with RaIR in CONSTRUCTION
After the fully-intrinsic free-play phase, we evaluate zero-shot generalization performance on down- stream tasks, where we perform model- based planning with the learned world models. Note that now instead of optimizing for intrinsic rewards, we are optimizing for extrinsic reward functions Tusk given by the environment (Suppl. 1.4.1). ).. In Fig. 8, we present the evolution of success rates of models checkpointed throughout free play on the following assembly tasks: singletower with 3 objects, multitowers with 2 objects each, pyramid with 5 and 6 objects.
The combination RaIR CEE-US yields significant improvements in the success rates of assembly tasks, as shown in Fig. and Table 2. RaIR alone, outperforms CEE-US in assembly tasks. As we biasing exploration towards regularity, we see a decrease in more chaotic interactions during play time, which is correlated with a decrease in performance for the more chaotic throwing and flipping tasks. For the generic Pick & Place task, we observe comparable performance between RaIR + CEE-US and CEE-US. The decrease in performance for RaIR in non-assembly tasks shows the importance of an information-gaae -aaia objective. The baselines RND and Disagreement exhibit very poor performance in the assembly tasks. In the other tasks, RND and CEE-US are comparable (bold numbers show statistical indistinguishabiliy from best with p > 0.05) This showcases that guiding free play by the model's 's epistemic uncertainty as in the case with ensemble disagreement, helps learn robust and capable world models. The decrease in the zero-shot performance for Disagreement further proves the importance of planning for future novelty during free play. We also run free play in the custom CONSTRUCTION environment for RaIR + CEE-US and CEE-US and observe improved shot downstream performance in assembly tasks with RaIR + CEE-US US (see Suppl. F.2).
8

---

Table 2: Zero-shot downstream task performance of RaIR+ CEE-US, RaIR, CEE-US, RND and Dis for assembly tasks as well as the generic pick & place task and the more thaos-oriented throwing and flipping. Results shown for five independent seeds. In the bottom row, we report the success rates achieved via planning with ground truth models. This is to provide a baseline for how hard the task is to solve with finite-horizon planning and potentially suboptimally designed task rewards.
Singletower 3
Multitower 2+2
Pyramid 5
Pyramid
Pick&Place
Throw
Flip
RaIR CEE-US
0.75 0.07
0.77 +: 0.06
0.18
0.90
0.32 0.02
0.63 0.08
RaIR
0.61 = 0.03
0.62 = 0.03
0.25 + 0.05
0.10 0.02
0.05
0.21 0.01
0.65 0.1
CEE-US
0.40 = 0.12
0.52 = 0.05
0,14 0.09
0.02
0.002
0.49=0.00
0.73 0.1
+ 0.07
= 0.12
0.02 0.02
0.0
0.91 0.01
0.42 0.07
0.82 0.1
Dis
0.0 + 0.0
0.01 + 0.01
0.0 0.0
0.0 0.0
0.89 0.01
0.30 0.04
0.69 0.09
GT
0.99
0.97
0.82
0.81
0.99
0.97
1.0
Singletower
Multitwwr 2+2
Pyramid
Pyramid
60%
25%
success
5.94
50%
30%
12%
0%6
0%
0%
100
200
300
100
200
300
0
100
200
300
100
300
300
Free Play Iteration
Free iteration
Free Play Iteration
Free Play Iteration
RaIR
RaIR+ CEE-US
CEB-US
RND
Dis
Figure 8: Success rates for zero-shot downstream task generalization for assembly tasks in CONSTRUCTION for model checkpoints over the course of free play. We compare RaIR CEE-US (A= 0.1) and RaIR with CEB-US, RND and Dis. We used five independent seeds.
3.4 Re-creating existing structures with RalR
We test whether we can re-create existing arrangements in the environment with RaIR. If there are regularities / sub-structures already present in the environment, then completing or re-creating these patterns naturally becomes an optimum for RaIR, as repeating this pattern introduces redundancy. We initialize pyramids, single- and multitowers out of the robot's 's manipulability range in CONSTRUCTION, We then plan using iCEM maximize RaIR with GT models. Doing so, the agent manages to re-create the existing structures the environment with the blocks it has within reach. Fig. this is showcased for pyramid with objects, where in 15 rollouts a pyramid is recreated in 73% of the cases. Without need to define any explicit reward functions, we can simply use our regularity objective to mimic existing ordered constellations. More details can be found in Suppl. D.
(a) t=)=
(b) f = 200
Figure 9: A pyramid initialized out of the robot's reach re-created by optimizing for RaIR.
RaIR in Locomotion Environments
The only requirement to incorporate RaIR in a given environment is an entity-based view of the world. We can easily apply this principle to locomotion environments, where we treat each joint as an entity. In Figure 10, we showcase the generated poses in the DeepMind Control Suite environments Quadruped and Walker, when we optimize for regularity using ground truth models. Here, regularity is computed over the z-y-2 positions of the knees and toes of the quadruped. For walker, we take the positions the feet, legs and the torso. These poses also heavily overlap with the goal poses proposed in the Robo Yoga benchmark [25] (see Fig. S11), which further supports our hypothesis that regularities and our preference for them are ubiquitious. We also apply free play with learned models in the Quadruped environment. More details and results be found in Suppl. G.

---

Quadruped
Walker
Figure 10: RaIR in Quadruped and Walker environments with GT models. We show generated
poses when maximizing for regularity over the positions of different joints (e.g. knees, toes).
Related Work
Intrinsic motivation in RL uses minimizing novelty/surprise to dissolve cognitive disequilibria as a prominent intrinsic reward signal definition [9, 26-31]. As featured in this work, using the disagreement of ensemble of world models as an estimate of expected information gain is widely-used metric as it allows planning into the future [10-12]. Other prominent intrinsic rewards deployed in RL include learning progress [26, 30, 32], empowerment [33, 34] and maximizing for state space coverage with count-based methods [35, 36] and RND [21]. Another sub-category would be goal-conditioned unsupervised exploration methods combined with e.g. ensemble disagreement [25, 37] or asymmetric self-play [38]. In competence-based intrinsic motivation, unsupervised skill discovery methods aim learn policies conditioned on latent skill variable [39, 40]. Berseth et al. [41] propose surprise minimization as intrinsic reward to seek familiar states in unstable environments with an active source of entropy. Note that this differs from our work, as our notion of regularity is decoupled from surprise: RaIR aims to get to a state that is regular in itself.
Compression and more specifically compression progress have been postulated as driving forces in human curiosity by Schmidhuber [16]. However, the focus has been on temporal compression, where it argued that short and simple explanations of the past make long-horizon planning easier. In our work, we don't focus on compression in the temporal dimension, i.c. sequences of states, Instead, we perform compression as entropy minimization (in the relational case, equivalent to lossy compression) at a given timestep f, where we are interested in the relational redundancies in the current scene. More details on connections to compression can be found in Appendix. L.
Assembly Tasks in RL with 3+ objects pose an open challenge, where most methods achieve stacking via tailored learning curricula with more than 20 million environment steps [24, 42], expert demonstrations [43], also together with high-level actions [44]. Hu et al. [37] manage to solve 3-object stacking in an unsupervised setting with goal-conditioned RL (GCRL), using very similar robotic setup to ours, but only with 30% success rate. More discussion on GCRL in Suppl. K.
6 Discussion
Although the search for regularity and symmetry has been studied extensively in developmental psychology, these concepts haven't been featured within reinforcement learning yet. In this work, propose a mathematical formulation of regularity as an intrinsic reward signal and operationalize within model-based RL. We show that with our formulation of regularity, we indeed manage to create regular and symmetric patterns in 2D grid environment as well as in challenging compositional object manipulation environment. We also provide insights into the different components of RaIR and deepen the understanding of the types of regularities emerging from using different mappings ø. In the second part of the work, we incorporate RaIR within free play. Here, our goal is to bias information-search during exploration towards regularity. We provide a proof-of-concept that augmenting epistemic uncertainty-basee
intrinsic rewards with RaIR helps exploration for symmetric and ordered arrangements. Finally, we also show that our regularity objective can simply be used imitate existing regularities in the environment.
Limitations and future work: As we use finitehhoizzo planning, we don't necessarily converge to global optima. This can both be seen as a limitation and a feature, as naturally allows us to obtain different levels of regularity in the generated patterns. Currently, we are restricted to fully-observable MDPs We embrace object-centric representations as a suitable inductive bias in RL, where the observations per object (consisting of poses and velocities) are naturally disentangled (more discussion in Appendix. M). We also assume that this state space interpretable such that we take, for instance, only the positions and color. The representational space, in which the RaIR measure computed, is specified by the designer. Exciting future work would be to learn a representation under which the human relevant structures in the real-world (c.g. towers, bridges) are indeed regular.
10

---

Acknowledgments and Disclosure of Funding
The authors thank Sebastian Blaes, Anselm Paulus, Marco Bagatella, Núria Armengol Urpí and Maximilian Seitzer for helpful discussions and for their help reviewing the manuscript. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support- ing Cansu Sancaktar. Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645, We acknowledge the financial support from German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FKZ: 011S 08039B. This work was supported by the Volkswagen Stiftung (No 98 571).
References
[1] Yanxi Liu, Hagit Hel-Or, Craig S Kaplan, Luc Van Gool, et al. Computational symmetry in computer vision and computer graphics. Foundations and Trends@ in Computer Graphics and Vision, 5(1-2):1-195, 2010.
[2] Fred Attneave. Symmetry, information, and memory for patterns. The American journal of psychology, (8(2):209-222, 1955.
[3] Marc H Bornstein, Kay Ferdinandsen, and Charles G Gross. Perception of symmetry in infancy. Developmental psychology. 17(1):82, 1981.
[4] Marjory W Bailey. scale of block constructions for young children. Child Development, 4(2): 121-139, 1933.
[5] William A Zingrone. The construction symmetry in children and adults. The Journal of genetic psychology, 175(2)911-044, 2014.
[6] Claire Golomb. The development of compositional strategies in children's drawings. Visual Arts Research, pages 42-52, 1987.
[7] Eleanor J Gibson. Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge. Annual review of psychology, 39(1):1-42) 1988.
[8] Jonas Langer. The origins of logic: One to two years. Academic Press, 1986. ISBN 978- 0124365551.
[9] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven explo- ration by self-supe prediction. International Conference on Machine Learning (ICML), 2017. URL https: http://////c/ccssss/////e mlr perressvv0/ppaa 7700paahak17a. html.
[10] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via dis- agreement. In International Conference on Machine Learning (ICML), 2019. URL https: //proceedings. mlr. press/v97/patharr html.
[11] Ramanan Sekar, Olch Rybkin, Kostas Danilidis, Picter Abbecl, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Confer- ence on Machine Learning (ICML), 2020. URL https: mlr. wwt.sprrsss/v/ve/ sekar20a. html.
[12] Cansu Sancaktar, Sebastian Blaes, and Georg Martius. Curious exploration via structured world models yields zero-shot object manipulation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[13] Shu-Kun Lin. Information and Computer Sciences, 36(3):367-376, 1996.
Correlation of entropy with similarity and symmetry. Journal of Chemical
[14] Daniel Lazarev. Information measures for entropy and symmetry. arXiv preprint 2rX.v:2211.14857 2022.
[15] Edward Bormashenko. Entropy, information, and symmetry; ordered is symmetrical, ii: system of spins in the magnetic field. Entropy, 22(2):335, 2020.

---

[16] Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty. surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In Anticipatory Behavior in Adaptive Learning Systems: From Psychological Theories to Artificial Cognitive Systems 4, pages 48-76. Springer, 2009. Longer preprint: https:/ /arxiv.....v. org/ abss/812.444
[17] Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in Psychology, 4:313, 2013.
[18] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural In- formation Processing Systems (NewrIPS), 2018. URL http:tt/////ooocciinggg neurips cc/ paper_files httpppra...aaaaaa/////////////aa httpp///pee.ee///eooo//ooooooaaa
Paper.ddf.
[19] Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning (CoRL), 2020. URL https: hwtpsstttp/eee h/tpoceedoocc......o mmr... press// v155/pin 1.55/pinneri211 htm.m.
[20] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3)3379 423, 1948.
[21] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICIL) 2019. URL htt.httts///openreviee
[22] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sunchez-Gonzaler Vinicius Zambaldi. Matcusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song. Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Re- lational inductive biases, deep learning, and graph networks. arXiv: 00288000000111. 2018. URL https://arrae /ttp:i//axivvv//// org/ abaaaaa///a 11///8806.000 1et.//.10...........
[23] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, J. Schneider, Joshua Tobin, Maciek Chociej. P. Welinder, V. Kumar, and W. Zaremba. Multi-godt reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 1802 09464, 2018. URL / htt////axvvv... org/ abs/ 0t///88020000 09464.
[24] Richard Li, Allan Jabri, Trevor Darrell, and Pulkit Agrawal. Towards practical multi-object manipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 4051-4058. IEEE, 2020. URL atttps://arxst org/ abs/ /atps//911...0 11032.
[25] Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https: /ttee:///pppeeeeee http///ppereeeeeeeeeeeeee wwtpeeewwwwwwweeeeeeeeee foatt//frommmmiie 1--
[26] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, 1991. URL https:/ //dl. acm. org/doi//g 55555/ 116517. 116542.
[27] Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, and Daniel Yamins. Active world model learning with progress curiosity. In International Conference on Machine Learning (ICML), 2020. URL https://aaxii org/
[28] Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv preprint arXiv:2 109.0009.0. 2021. URI. https:/ //aa/iv////vv../// arxiivv org/ abs//b/s// 08603.
12

---

[29] J. Storck, S. Hochreiter, and J. Schmidhuber. Reinforcement driven information acquisition in non-deterministic environments. In Proceedings of the International Conference on Artificial Neural Networks, pages 159-164, Paris, 1995. HC2 & Cie. https: //pepppe//ppppe..... .isial.eedddii.a.......ia...aiii ch/" heeeueeeeeiannneeaeneee pdf.
[30] Sebastian Blaes, Marin Vlastelica, Jia-Jie Zhu, and Georg Martius. Control What You Can Intrinsically motivated task planning agent. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL https: http:///eoocc...../////// wtw.eeediinnnnnniiiii.eeee bff977effo0a17766191006666633339 hwtp//aoob-bbbbrraeeee Abstract tttbb.ttaat..tl......
[31] Giuseppe Paolo, Alexandre Coninx, Stephane Doncieux, and Alban Laftaquière. Sparse reward exploration via novelty search and emitters. In Proceedings of the Genetic and Evolutionary Computation Conference, page 154-162, 2021. URL https: :topps://doo. http:///oooooo..////ooooooo/ 10./10..11 1144/34449669. 3459314.
[32] Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre- Oudeyer. CURIOUS: Intrinsically motivated modular multi-goal reinforcement learning. In International Conference on Machine Learning (ICML), 2019. URL https: p/wp://edioo/o/...//oooeo//a mmr. calrsp v97/colas19e html.
[33] A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: a universal agent-centric measure of control. In IEEE Congress on Evolutionary Computation, volume 1, pages 128-135 V 2005. URL https: /ttp:///eeeeeeee /tteeeeeeeeeeeeeeeeeeeeeeeeeeeee ieee. .rgg //oc/mmet//1556776
[34] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing
Systems (NeurIPS), 2015. URL https:////axx org/ abs9/509...... 08731.
[35] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[36] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurrkk and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[37] Edward S. Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration, In International Conference on Learning Representations (ICLR), 2023. URL
[38] OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder. Ruben D' Sa. Arthur Petron, Henrique Ponde de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self- play for automatic goal discovery in robotic manipulation. arXiv:2101.0488/ 2021. URL https: h //tp:i//aaiiv.aeea org/ abs /bs/ 04882.
[39] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations (ICIR), 2019. URL https:t//// httttp:://pperrvviii wtwpeerrriiee.nne/ffon///// dded/ RqFm.
[40] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- tions (ICLR), 2020. URL https: http:::ppp////eeeeeeeeee/ net wttt///ooummii-tlglllaaaa
[41] Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinchart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. {SM]irl: Surprise minimizing reinforcement learning in unstable environments. In International Conference on Learning Representations (ICLR), 2021. URL
[42] John B. Lanier, Stephen McAleer, and Pierre Baldi. Curiosity-driven multi-criteria hindsight experience replay, 2019.
13

---

[43] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 JEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
[44] Ondrej Biza, Thomas Kipf, David Klee, Robert Platt, Jan-Willem van de Meent, and Lawson LS Wong. Factored world models for zero-shot generalization robotic manipulation. arXiv preprint arXiv:2202.05533. 2022. URL htttps//taartie
org/ 22022 05333.
[45] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprini arXiv: 0011110000000 2018.
[46] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances Neural Information Processing Systems (NeurIPS), 2020.
[47] Pedro Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J. Gershman, and Joshua B. Tenenbaum. Human-level reinforcement learning through theory-based modeling, exploration, and planning. arXiv:2107,12544, 2021. URL https: //arxiv. org/ btrg/abs///n abs 2207 12544.
[48] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[49] Elizabeth S Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
[50] Elizabeth Spelke, Karen Breinlinger, Kristen Jacobson, and Ann Phillips. Gestalt relations and object perception: A developmental study. Perception, 22(12): (2:22)::488 148--1501, 1993.
[51] Benjamin Peters and Nikolaus Kriegeskorte. Capturing the objects of vision with neural networks. Nature human behaviour, 127-1144,,,, 2021.
14

---

Supplementary Material for Regularity as Intrinsic Reward for Free Play
Code and videos are available at https: wwwpiittee.gggg..o.oe.oe pooggle..oo/vveee//rrrrrrrgggsse
A Experiment Results with Ground Truth Models
A.1 Experiment Results for RaIR in CONSTRUCTION with Z-1/-2
As discussed in Sec. 3.1, in our experiments we compute RaIR on the z-y subspace of the object positions in CONSTRUCTION to inject bias towards vertical alignments. Examples of patterns generated when optimizing for RalR using
o(s sj) {(I $j 11. $j.y 11, ||,,1 $1,= $,,=11)3 11)
showcased in Fig. S1. When also include the z-positions of the objects in the RalR computation, patterns and constellations on the ground are preferred. In this case, there is no difference between a vertical line, i.e. stack. Since creating stack, however, is more
horizontal line on the ground sparse solution, in practice the zero-order trajectory optimizer converges already to regular structures on the ground and vertical constellations don't emerge. Starting from randomly initialized scene with all objects on the ground, the regularity metric for z-y-z only starts increasing when multiple objects are in the stack, which would require very long planning horizon to find this solution.
Figure S1: Emerging patterns with RaIR on the r-y-2 subspace with GT models, where we use absolute relational Φ
B Experiment Results for Relational Case without Absolute Value
We present the interaction metrics observed during free play with RaIR + CEE-US in the case of relational é with ((8,, 8,) {[s sy1} {[ $jzz [sk.y sy.,)). We find the interaction metrics to be comparable to the absolute relational case presented in the main paper with ((s;, 8,) {() $1 5 11. || s,v,11) 11)) In Fig. S2, we also include the results for RaIR + +CE- US with À = 1. In the case of the increased weighting on the ensemble disagreement term, the free-play behavior indeed collapses back onto CEE-US. This means we have more flipping behavior and less air time. In the case of RaIR + CEB-US with smaller X 0.1, we seek regular states, which include vertical alignments, such that the air time doesn't go down. Note that this case, there is still incentive to "destroy" and lean towards chaos due to the ensemble disagreement reward term, such that the constellations showcased in Fig. (snapshots for the absolute relational Φ) and Fig. S3 are not necessarily preserved.
We also evaluate the success rates for zero-shot downstream task generalization using the models trained in free-play runs with relational (R) 6 and present them in Table S1. We find the performance in this case be comparable previous work [12], we use controller mode "best", not "sum" and shorter planning horizon of 20 timesteps instead of 30. The environment also intialized with 6 objects instead of That's why the reported success rates differ from previous work. We also include the success rates for CEE-US with mode "sum". and see that using mode "best" for free play also results in improved performance in assembly tasks.
the absolute relational (AR) é case. Note that in comparison to
15

---

RaiR
RaiR + CEB-US US (A 0.1)
RaiR + CHEE-US US (A
CEE-US
30%
40%
50%
relative time
25%
50 100 150 200 250 300
50 100 150 200 250 300
100
200
200
50 100 150 200 250 200
Transitions x 2000
Transibions x 2000
Transibions 2000
Transitions x 2000
(a) 1 object moves
(b) or more objects move
(c) object(s) in air
(d) object(s) flipped
Figure S2: Comparison of interactions during free play in CONSTRUCTION when combining ensemble disagreement with RaIR for different augmentation weights À with relational Interaction metrics of free-play exploration count the relative amount of time steps spent moving object (a), moving and more objects (b), moving objects in the air (c), and flipping object(s) (d). We used independent seeds.
RO
Iteration 245
Iteration 246
Iteration 247
Ierration
Iteration 270
Iteration 275
Iteration 295
Figure S3: Snapshots from free play with RaIR CEE-US and relational di We showcase snapshots of lowest entropy from exemplary rollouts at different iterations of free play. Following the regularity objective, stacks and alignments generated. These snapshots come from with = 0.1.
Table S1: Zero-shot downstream task generalization performance RaiR CEE-US for differ- ent and À for assembly tasks well as the generic pick & place task and the more chaos-oriented throwing and flipping. Results are shown for five independent seeds. AR: Absolute relative &, R: Relative do In the row CEE-US (sum), we report success rates when we use the controller mode sum instead of best, which the default for CONSTRUCTION in this work. In the bottom row, we report the success rates achieved via planning with ground truth models. This is to provide baseline for how hard the task is to solve with finite-horizon planning and potentially suboptimally designed task rewards.
Singletower 3
Multitower 2+2
Pyramid
Pyramid
Pick&Place
Throw
Flip 4
6
CEE-US
0.03
0.47
0.17
0.38 0.02
1:
RaIR + CEE-US (AR)
0.75
0.07
0.77 + 0.06
0.49 + 0.06
0.18 + 0.04
0.90 + 0.02
0.32 + 0.02
0.63 + 0.08
(AR) (A
0.64
0.62
0.25 0.05
0.10 0.02
+ 0.05
0.21
0.65
CEE-US
0.10 - 0.12
0.32 0.05
0.11 0.09
0,02
0.01
0.90 : 0.02
0.49 0.05
0.73 0.1
CEE US (sum)
0.26 = 0.11
0.48 a 0.13
0.10 0.10
0.0 0.0
0.91 +: 0.02
0.51 +: 0.04
0.76 0.15
0.99
0.97
0.82
0.81
0.99
0.97
1.0
C Experiment Results for Free Play with pure RaIR
In this section, further discuss the zero-shot downstream task generalization performance for free play with pure RaIR and the role the information-gas gain term our intrinsic reward combination used free play, as specified in Eq. 6. As discussed Sec. 3.2, adding ensemble disagreement to our regularity objective leads to 1) more interaction-rich free play and 2) more robust world models which yield higher success rates for zero-shot downstream task generalization. For both the absolute relational case presented in Fig. 5 and the relational case in Fig. S2, RaIR with no disagreement term yields less interactions terms object(s) being moved, being in air and being flipped. This is because the exploration problem is not solved by RaIR alone. When we use ensemble disagreement as an intrinsic reward, the discovery of different types interactions is accelerated. When one of the models in the ensemble learns new type of dynamics, such as an object moving, the ensemble disagreement goes up, incentivizing the agent to repeat this behavior until is learned by all models such that disagreement goes down. In the case of RaIR, this only happens implicitly: with some
16

---

Iteration 270
Iteration 273
Iteration 276
Iteration 295
Itecation 296
Iteration 298
Iteration 299
Figure S4: Snapshots from free play with pure RaIR. We showcase snapshots of lowest entropy from exemplary rollouts at different iterations of free play. These snapshots come from a run with absolute relational ạ).
models in the ensemble learning certain type of dynamics in the environment, during planning, the models can hallucinate objects being aligned and creating regular pattern with high RaIR such that these actions are executed by the controller. These false attempts also help exploration.
As the models produce better predictions, especially after free play iteration 200, we observe that more stable patterns are generated with RaIR compared to RaIR + CEE-US (Fig. 7) and the amount of time objects are moving starts decreasing Fig. This is because in this case when the models get better and hallucinate less, there is no reason to leave local optima such as spaced line unless pattern that yields a higher regularity value can be found within the planning horizon.
The challenge of exploration with pure RaIR is also reflected in the interaction time for object(s) in air. Starting create regular patterns such stacks takes longer, exploring lift objects happens later without the disagreement reward. This also connected to the step-wise landscape of RaIR as discussed in Sec. 3.2 such that explicit exploration via ensemble disagreement is beneficial.
As showcased in Fig. S4, we still observe the stable generation of patterns such as spaced lines later on in training well, as these are local optima of RaIR. However, we start to more stacks generated in the later stages free play. In Fig. 7, the highest RaIR value achieved for the different variants are showcased throughout training. Pure RaIR, achieves slightly higher regularity then RaIR CEE-US. This is also because pure RaIR, tends to generate more regular patterns that feature all objects, i.e. all objects in-line or build square. With RaIR CEE-US, some chaotic behavior is injected to free play via ensemble disagreement, more local regularities such as a stack of 2, with the rest of the objects in disorder, are likely to emerge.
Through injecting ensemble disagreement into free play, the robustness of the learned world models is also increased as they guided by their epistemic uncertainty [12]. During free play, data is actively collected from regions where the models are uncertain, acting as their own adversary. This in turn makes the models more robust for deployment in model-based planning in the follow- up extrinsic phase, where the accuracy of model predictions is paramount for good performance. This is reflected in the downstream task performance evaluations in Fig, 8, where RaIR CEE- US consistently outperforms both RaIR and CEE-US US in the assembly tasks. Note that as regularity explored better,
explicitly favours alignments such as stacks, unlike CEE-US, these dynamics leading to higher success rates. This also showcases the importance of guiding free play towards regularity. In Fig. S5, the results for the pick & place, throwing and flipping tasks are shown. Due to the increased robustness of the model with the disagreement term, we indeed observe better performance for RaIR + CEE-US and CEE-US for the Pick & Place task compared to pure RaIR. This is also true for the throwing task. However, another contributing factor here is that models with disagreement favor more chaotic behaviors and perform more "throwing"-iid behaviors during free play. As CEE-US US has no bias towards regularity. it performs best, whereas pure RaIR performs worse than RaIR CEE-US. Interestingly, for the flipping 4 objects task we found performance for RaIR and RaIR + CEE-US to be comparable despite the significantly reduced amount of time spent flipping objects in the case of pure RaIR, as can be seen in Fig. Upon inspecting the data generated during free-play, we hypothesize this is because unlike RaIR CEE-US, which flips and rolls objects together in a chaotic way, we found RaIR to produce more isolated flipping of individual objects.
D Experiment Results for Re-creating Existing Patterns
As presented Sec. 3.4, we test whether we can re-create existing regularities in the environment by simply optimizing for RalR with iCEM, using ground truth models. As for the pyramid with
17

---

RaIR
CEB-US
CEE-US
100%
success rate
50%
50%
100
200
300
100
200
300
100
200
300
Free Play Iteration
Free Play Iteration
Free Play Iteration
(a) Pick & Place
(b) Throw
(c) Flip
Figure S5: Downstream Task Performance for Pick & Place and the more chaotic tasks of
throwing and flipping with only RaIR (A = 0), RaIR + CEE-US (A = 0.1), CEE-US, RND and
Disagreement. We use absolute relational & for RaIR computations. Results are shown for 5 independent seeds.
3 objects in Sec. 3.4, we initialize different regular structures outside of the robot's manipulability range and test whether these regular patterns he re-created, merely by maximizing for RalR. We test for the re-creation of singletower with and 4 objects, 2 towers with 2 objects each (referred to multitower 2+2), as well spaced line and rhombus with 4 objects. We test this with ground truth models for 15 independent rollouts for each structure and report the re-creation rates, Example rollouts are illustrated Fig. S6. Note that due the limited sample-budget with iCEM and the finite-horizon, we don"t 't necessarily converge to the global minima, which corresponds to the full recreation of the structure. However, in all of the tested cases, the generated structures repeat least one prominent sub-structure present in the underlying regular constellation by optimizing for RaIR.
For Singletower 3, the entire stack of gets recreated 73% of the time. A partial recreation with stack of 2 blocks is observed all but one of the remaining cases.
When a Singletower 4 is initialized outside of the robot's range, the full tower with 4 blocks gets recreated 40% of the time. In the remaining cases, either tower of 3 (33%) or towers (27%) are built.
For the challenging Multitower 2+2 case, the two towers are built, with the same distance each other as in the original pattern, 20% of the time. An example of this "complete" recreation is illustrated in Fig. S6c. Otherwise, 53% of the time stack of 2 is built (Fig. S6d) or a spaced line repeating the relative position of the two towers in the original pattern.
For the patterns the ground, namely Spaced Line and Rhombus, the recreation rates are higher since the exploration problem is less prominent. At least 75% of the original pattern re-created at each rollout, i.e. for the case of 4 objects, at least objects follow the original pattern. The complete Spaced Line is recreated 80% and the entire rhombus 73% of the test rollouts.
In these experiments, we use RaIR with o(s;, 8j) [(ILsi.. 5j 11. IL Sj. |b.|_ 1552-88,4 83,2 10) This is because the case of existing structures in the scene, we don't need/want to inject any biases, e.g. towards vertical alignments, into the optimization. As the existing pattern is outside of the manipulability range of the robot, re-creating the pattern becomes a direct global optimum for RaIR, all regularities reoccur. However, if restrict ourselves to the x-y subspace, this is no longer the case: even for a rhombus, a stack built with the blocks in-reach becomes the global optimum. This is because all the blocks in the stack then have the same z-y relation the blocks in the rhombus.
E Experiment Results for RaIR in Colored SHAPEGRIDWORID with GT Models
In colored SHAPEGRIDWORLD, Sayj includes not only the x-y positions, but also binary variable encoding the different classes of colors. For e colors, we have encoding of length ceil c). So
when we are computing relational RaIR, we also take into account the relations between colors. Take as example the case where we have two color classes blue (0) and orange (1) and use absolute relational
o: if all blue objects are in horizontal spaced line, we have repetitions of [[x + +5, 0, [r; 0, 0]
[62, 0, 0] between neighboring objects with spacing 8_. we have a line with an alternating pattern of colors in sequence, then we again have redundancies. For instance, for neighboring entities
orange-blue we have: ||r; + &z, 0, 1] [.., 0, 0]l = 18,, 0, 1]. Since we have absolute relational
é, we get the same symbol for the blue-orange neighbors [| + 8 0,0] [x,, 0, 1]I
[6,, 0, 1].
18

---

(a) Singletower
(b) Singletower 4
Start
Find
Start
End
(c) Multitower 2+2
(d) Multitower 2+2 (partial recreation)
Start
End
Start
End
(e) Spaced line 4
Rhombus 4
Figure S6: Different regular structures initialized outside of the robot's reach at the start of the episode (t 0) and re-created by optimizing for RaIR with GT models. Showcased here for the end of the episode (f = 200).
:
Hejpll
Figure S7: Including color into the regularity computations in SHDPPERRDWOOLE We show generated patterns when maximizing for RaIR, with x-y dimensions and color as an extra symbol. In this case, RaIR tries to find patterns that are not notjust spatially symmetric but also regular in color.
As result, cateeorrane-bler blue-orange-.... .. alternating line is also an optimum for RaIR. Additional patterns generated with RaIR in colored SHAPEGRID with 2 and 3 colors using GT models are shown in Fig. S7.
F Experiment Results for RaIR in Custom CONSTRUCTION
We test regularity on a variant of CONSTRUCTION with diverse object shapes and masses (more details in Appendix. I.I).
19

---

F1 RaIR with GT models
We apply RaIR eustom CONSTRUCTION with different shapes and masses, where we compute regularity between the z-y positions of the center of mass of objects. Here our aim is to show the generality RaIR and that we are constrained to identical entities. The generated constellations when optimizing for RalR using GT models are showcased in Fig. S8.
Figure S8: Applying RaIR to a custom CONSTRUCTION environment with different shapes and masses with GT models. We show generated arrangements when maximizing for RalR with x-y positions of objects (center of mass) in 2 instances of the environment: 1) with 2 cubes, 2 short columns and flat block (the left images) and 2) with 2 cubes, ball, column and flat block (the right 4 images).
F.2 RaIR in Free Play with Learned Models
We run free play in the custom CONSTRUCTION. During free play, we initialize one instance from each object type such that we have cube, 1 ball, 1 column and flat block. We then evaluate the zero-shot downstream task performance of RaIR + CEE-US and CEE-US for different stacking tasks (Fig. S9). The results are shown in Table S2, where obtain higher success rates with RaIR + CEE-US.
The free play parameters (model and controller settings) are identical to the standard CONSTRUCTION environment given in Table S6. The only difference is, we include a categorical variable to encode the different object types and append this to the observation vector for all objects. This is done in the same fashion as the static observations proposed in Sancaktar et al. [12]. For RaIR + CEE-US, we again use À 0.1.
Table S2: Zero-shot downstream task generaliza- tion performance of RaIR + CEE-US for custom CONSTRUCTION. Results are shown for five inde- pendent seeds, for models evaluated after 300 free play iterations (equivalent 2K x300 600K transi- tions).
Stack Cube + Ball
Cube Block
Stack Cube + Column Ball
Stack Cube Ball
Stack Cube
Stack Cube Column Ball
Flat Block
Figure S9: Downstream tasks for custom CONSTRUCTION.
RaIR + CEE-US
0.66 0.10
0.67 + 0.09
0.15 - 0.05
CEE-US
0.38 0.11
0.45 + 0.04
0.03
G Experiment Results for RaIR in the Quadruped and Walker Environments
We showcase that regularity also finds application in locomotion environments, where compute RaIR on the world coordinates of the robot joints. Environment details for both Quadruped and Walker can be found in Appendix. I.1.
We run free play with RaIR CEE-US (A 0.02) and CEE-US in the Quadruped environment. With RaIR + CEE-US, we indeed observe more regular poses during free play, as reflected in the RaIR values shown in Fig. S10. We evaluate the zero-shot downstream performance of the learned models for the Roboyoga poses shown Fig. $11. The success rates for the models at the end of free play (Table S3), as well as their temporal evolution (Fig. S12) are reported. We evaluate success for pose as having distance smaller than threshold of 0.4 to the goal pose for more than 10 timesteps. We only see marginal improvements with RaIR + CEE-US compared to CEE-US. We hypothesize that this is because model-based approaches can generalize very well locomo- tion environments and CEE-US also provides sufficient exploration. It
-2.
RaiR
-3.0
Free Play Iteration
Figure S10: Highest
RaIR in Quadruped free play with RaIR + CEE-US and CEE-US.
20

---

only comes down to whether the pose can be held longer stably, where in the balancing poses RaIR+ CEE-US has slight edge over the course of free play (especially for Balance Front). For any goal that is shown in Fig. S11 I and not included in Table S3 (i.e. the poses with Quadruped lying on its back), we get perfect success rates for both RaIR CEE-US and CHE-US. We didn't 't run free play in the Walker environment, but we expect model- approaches to be very strong there altogether, as the environment dynamics are very simple.
Stand Leg Up
Allack
Balance Front
Balance Back
Balance Diagonal
Lie Side
Side Legs Together
Rotated
Lie Legs Up
Stand
Rotated
Lie Back
Lic Front
Legs Up
Kneel
Side Angle
Stand Up
Bridge
Up
Stand
Arabesque
Robo Yoga benchmark proposed in Mendonca et al. [25] for Quadruped (top 2 rows) and Walker environments (bottom 2 rows).
Figure S11: Goals in the Robo
H
CEE-US
In this section, we present the details CEE-US [12], which we build upon in this work. CEE-US structured world models together with model-based planning during exploration, achieving in- creased sample-efficiency and superior downstream task performance compared to other intrinsically- motivated RL baselines. The free-play pseudocode is presented in Alg. S1. This free-play structure is used for all methods presented in our paper by swapping out the intrinsic reward term (line 5)
RaIR CEE-US
CEE-US
100%
100%
successrate
50%
50%
50%
50%
30
200
100
Free Play Itaration
Itaration
Free Play Iteration
Iteration
(a) Stand Leg Up
(b) Attack
Balance Front
Balance Back
100%
success rate
5004
50%
50%
50
180
700
100
150
50
100
700
Free Play Iteration
Free Play Iteration
Free Play Iteration
(c) Balance Diagonal
(f) Lie Side
(g) Lie Side Back
Figure S12: Downstream task performance for subset of Quadruped Robo tasks with RaIR CEE-US (A 0.02) and CEE-US, evaluated for models checkpointed throughout free play.
We use absolute relational Φ for RaIR computations. (Five independent seeds)
21

---

Table S3: Zero-shot downstream task generalization performance RaIR CEE-US and CEE-
US for the Quadruped Robo RoboYoga benchmark. Results are shown for five independent seeds, for
models evaluated after 200 free play iterations (equivalent to 2K x 200 = 400K transitions).
Stand
Attack
Balance
Balance
Balance Diagonal
Lie Side
Lic Side
Stand
Stand
Leg Up
Back
Rotated
Ralk a CEE-US
0.80 # 0.03
0.81 # 0.06
0.74 + 0.04
0.71 # 0.03
0.89 0.00 0.03
0.73 + 0.03
0.71 + 0.04
0.89 0.04
0.83 + 0.06
CEE-US
0.78 + 0.07
0.75 + 0.07
0.65 D.03
0.65 + 0.03
0.78 =178=00 0.07
0.76 + 0.05
0.73 0
0.78 + 0.07
0.80 + D.06
with only ensemble disagreement (CEE-US), combination of our regularity objective with ensemble disagreement (RaIR + CEE-US) or pure regularity (RaIR), as well as the other baselines RND and Dis (see Sec. J).
Algorithm S1 Free Play in Intrinsic Phase (taken from [12]
1: Input: (Co... >= }:
Randomly initialized ensemble of GNNs with M members, D: empty
dataset, Planner: iCEM planner with horizon H
while explore do for e
Explore with MPC and intrinsic reward
num_ cam_episoees do 4: for { = to I' do
D Plan to maximize intrinsic reward
0, f Planner(s:, ( 1 : Tintriusic)
b e.g. RaIR with disagreement Eq. 6
St+1 t env. Ewv.step($t..r 4t)
D DU ((t,, 0t, )-1)
8:
for = 1 to do
> Train models on dataset for epochs
8m t- optimize 6m using Lm on D for I,...,M
10: return {(fo. (0 1
H.1 GNN Architectural Details
dessage-passing Graph Neural Networks (GNN) are deployed as world models. The same GNN architecture is used as in CEE-US [12]. For these structured world models, we consider object- factorized state spaces with = (Sabj) ) Smoot- Each node in the GNN corresponds to an object and the robot/actuated agent is differentiated from the object nodes as global node. The concatenation of the robots's 's state sp and the action a, is represented as global context c = [sqebut ar]. We have fully-connected GNN. The node update function Gnode and the edge update function Jedge model the dynamics of the entitics/objects, and their pairwise interactions respectively. These functions are both Multilayer Perceptrons (MLP). In the following, we denote the state of the i-th object St,ob,, timestep / st for simplicity. The object node attributes in the GNN are updated
E (i.f) Gedge ([s),
st ([s),
(S1)
s+1 Gnode (1 c, aggr (c) ())
(S2)
where ......] denotes concatenation, e
(1.)) is the edge attribute between two neighboring nodes (i.j). For the inttuuttttioiiiviiaantt aggregation function given by aggr, we use the mean.
The robot state, which is treated as a global node, computed using the global aggregation of all edges with a separate global node MLP Igiobal:
grobot #++
Iglobal ([c, aggri, (ef ())
(S3)
Moreover, the GNN predicts the changes in the dynamics such that 5 = 8, + (x))
H.2 Planning Details
For planning, we use the improved Cross-Entropy Method (iCEM) [19]. The planner minimizes the cost, corresponding to negative reward c(st, ax 11+1) -r(8,, de 8t+1), where can be intrinsic rewards Finrinsic or extrinsic task rewards Tink- The extrinsic task rewards are assumed to be given by the environment.
22

---

At each timestep t in the environment, the planner samples P action sequences, each with length H, i.. the planning horizon. These actions are rolled out either in the ground truth model (perfect simu- lations) or in the imagination of a learned model (imperfect simulations), generating corresponding P state sequences with length H. In order to assign a cost to each of the P trajectories, we need to aggregate the cost over the horizon H. A typical choice here is sum, where the cost over the length of (n) (p)
the trajectory is simply summed up: cost(P) (P) Σm= c(sith: aith: s+hh+1)
However, this type of aggregation is not suitable for cases where a decrease in cost can in general be preceded by an initial increase. In these cases, using the mode best, that assigns the plan p the cost of (p)
is a better suited choice. We also empirically found this controller mode to be better at picking up sparse signals. What we mean here is that, in the example of stacking, it is hard to find sampled trajectory that stacks the objects in a stable way with a limited sample-budge as this poses an exploration challenge. However, if we manage to find an action sequence that brings the cubes on top of each other, albeit in an unstable way, favoring this solution with best and keeping this solution in the clite set is beneficial. This can be explained follows: In iCEM the K plans with the lowest assigned cost are chosen to be the elite set, which is then used to fit the sampling distribution of iCEM. As a fraction & of these elites is potentially shifted to the next internal iCEM iteration (keep_ eempp_eeless) and possibly to the next timestep (shift_ Saalft__elttte elites)) keeping these solutions that "fail" and yet bring us closer to the actual solution provides a better strategy solve tasks which pose an exploration challenge such as stacking. Here, we are also relying on the fact that we are re-running optimization every timestep in the environment with online model predictive control, such that have the opportunity to correct these initially "wrong" solutions and find their "stable" counterparts. Note that this mode of the controller a more unstable mode compared to sum. Especially with imperfect world models, where the model can hallucinate as the model errors accumulate over the planning horizon, mode best can pick up these falsely imagined future states with low cost. It also doesn't t account for the fact that the planned trajectory keeps the lowest cost over multiple timesteps, such that a trajectory where an object flies through the goal location for single timestep has the same cost as a trajectory where the object lands in the goal position and stays there. To account for this,
the "best" timestep over the planning horizon with cost = min {c(s) 444h ( $44h41)
we use (r = min {c(s{4) (+1 "1+h $ $ct++1 +
where we don't take into account the first timestep of the plan with h O. Although this is not robust solution, we found to empirically work well. Quantitatively, stacking 3 objects when planning with ground truth models yields 99% success rate for mode best, whereas only 47 % success rate for sum, using the same reward function in both cases. Even in the case of perfect dynamics predictions with GT models, this showcases the importance of the controller mode to be able solve tasks with sparse reward signals. This is further showcased in Table $1, when comparing the results for CEE-US with default best and sum.
(p) (p)
I Experiment Details
In this section, we provide experimental details and hyperparameter settings.
L1 Environment Details
SHapeGrIDWORLD This is discrete 2D grid, where each entity/agent is controlled individually in the æ-y directions. This means entities are controlled one at a time and actuation keeps iterating over the entities, where we use an object persistency of 10 timesteps. The action ag is 2 dimensional, controlling the agent in x-y directions separately and is applied on the current actuated entity in the grid. As we are operating a discrete grid, the actions are actually discrete such that the agent can move one grid cell to the left/right and up/right (if the target grid cell is not occupied) or stay at the current grid cell. In order to make this environment work with the default iCEM implementation with Gaussian sampling distribution, we perform discretization step before inputting the sampled actions to the environment. For the experiment results with GT models presented in Fig. 1, we use grid size of 25 x 25 with 16 and 32 objects. In colored SHAPEGRIDWORLL, with colors, each object is assigned one of the e available color options, and the observation for each entity also includes binary encoding for color with length ceil c).
23

---

This is a multi-object manipulation environment as an extension of the Fetch Pick & Place environment proposed in [24]. We also applied the two modifications from Sancaktar et al. [12]. 1) The table in front of the robot is replaced with large plane such that objects cannot fall off during free play, but can still be thrown/pushed outside of the robot's reach. 2) Li et al. [24], the object state also contained the object's ss position relative to the gripper which was removed, as it
CONSTRUCTION
already introduces a relational bias in the raw state representation. Details on the dimensionalities of the object and robot state spaces can be found in Table S5.
As an extension of the standard CONSTRUCTION, there are 4 different object types with different masses: cube (same as in original CONSTRUCTION) with size 5cm and mass 2 (mass in the default unit of Mujoco), flat block with size 16 x 7 x 3em and mass 1.5, column (upright block) with size 3 x 3 x 10cm and mass 1, and a ball with diameter 8cm and mass 1. We also include categorical variables for each object type as static observations. During free play, we initialize one of each object type, so we have 4 objects. Otherwise the specifications are the same as in Table S5.
Custom CONSTRUCTION
Quadruped & Walker These environments are taken from the DeepMind Control Suite [45]. To be able to perform RalR computations, we modify the observation space of each environment and append the world coordinates of the robot's joints to the observations vector. In Quadruped, we take the toe and knee positions. We used z-y-z coordinates with the GT models (results shown in Fig. 10), and z-y for free play in Quadruped with learned models. In principle, we can also include the ankle and hip joints, but we observed similar performance in this case. For Walker, we append the feet, leg and torso z-y positions.
I.2 Parameters for Ground Truth Model Experiments with RaIR
The controller parameters used when optimizing RaIR with ground truth (GT) models are given in Table S4. To compute RaIR, we perform a discretization step in the CONSTRUCTION environment as it continuous. For GT models, that produce perfect mental simulations, we can choose a small bin size of lcm. In comparison, the size of one block in the environment is 5 cm. The bin size also gives us the upper bound of the regularity precision that can be achieved during optimization, e-g. perfectly aligned stack vs. a zigzagged stack. Note however that the higher the precision is, the harder it typically gets for the controller to converge to global optima with a horizon of 30 timesteps and limited sample-budget.
As discussed in Sec. A, this is also constraint when we are computing RaIR in the x-y-2 subspace. Due to this, for the re-creation of existing patterns experiments presented in Sec. 3.4 and Sec. D, we compute RaIR with a bin size of 2.5cm and increase the number of sampled trajectories P to 5 512. Although we could further increase the bin size, we choose this value to not negatively impact the precision of the re-created structures. In custom CONSTRUCTION, we also use a bin size of 2.5em, when optimizing RaIR with GT models.
For Quadruped, we also use a bin size of 5cm. For the GT model experiments, compute RaIR over the x-y(-=) positions of the Quadruped's toe and knee joints. We didn't 't see significant qualitative differences when including the z-positions in the regularity computation. For Walker, we use bin size of lcm and compute RaIR over the x-y positions of the Walker's 's feet, legs and torso.
1.3 Free Play with Learned Models
The environment properties with the episode lengths and model training frequencies are given in Table S5, Six objects are present in CONSTRUCTION during free play. The parameters for the GNN model architecture as well as the training parameters for model learning are listed in Table S6. For the RaIR computations in free play, we use a bin size of 5cm, which is equivalent to the size of a block.
For custom CONSTRUCTION, 4 objects (one of each object type) are present during free play. The hyperparameters are the same as for standard CONSTRUCTION, and bin size of 5cm is used.
The set of the hyperparameters for the iCEM controller used in the intrinsic phase of RaIR + CEE-US, RaIR and CEE-US are the same presented in Table S4. The only difference to the GT model case
24

---

Table S4: Base settings for iCEM. These hyperparameters used when using GT models to
optimize RaIR.
General settings.
(b) Environment-specifcc settings.
Parameter
Value
SHAPEGRIDWORLD
Parameter
Value
Number of samples P
128
Horizon H
30
Number of samples P
Size of elite-set K
10
Colored-noise exponent /
3.5
CONSTRUCTION
CEM-iterations
3
Parameter
Value
Noise strength init
0.8
Momentum
0.1
Same general settings
use use_mean www_mean_acttia actions
Yes
shift_elites
Yes
Quadruped & Walker
keep_elites
Yes
Parameter
Value
Fraction of clites reused
0.3
Cost along trajectory
best
Number of samples /
64
Colored-noise exponent /
2.5
Noise strength imit
0.3
Cost along trajectory
sum
Table S5: Environment settings for CONSTRUCTION. 2000 transitions (20 episodes with 100 timesteps each) are generated within one training iteration of free play.
CONSTRUCTION
Parameter
Value
Episode Length
100
Train Model Every
20 Episodes
Action Dim.
Robot/Agent State Dim.
10
Object Dynamic State Dim.
Num. of Objects During Free Play
6
Table S6: Settings for GNN model training in intrinsic phase of RaIR CEE-US and CEE-US. (Same
as in [12]) These settings are used for both CONSTRUCTION and Custom CONSTRUCTION free play runs.
Parameter
Value
Network Size of Gnode
2 x 128
Network Size of Jedge
2 x 128
Network Size of Iglobal
2 128
Activation function
ReLU
Layer Normalization
Yes
Number of Message-Passing
1
Ensemble Size
5
Optimizer Batch Size
ADAM
125
Epochs
25
Learning Rate
10 10-5
Weight Decay
0.001
Weight Initialization
Truncated Normal
Normalize Input
Yes
Normalize Output
Yes
Predict Delta
Yes
25

---

Table S7: Settings for the MLP model training in intrinsic phase of RalR + CEB-US and CEE-US in the Quadruped free play runs.
Parameter
Value
Episode Length
100
Train Model Every
20 Episodes
Network Size
3 x 600
Activation function
SiLU
Ensemble Size
Optimizer
ADAM
Batch Size
128
Epochs
25
Learning Rate
0.0001
Weight decay
0.0001
Weight Initialization
Truncated Normal
Normalize Input
Yes
Normalize Output
Yes
Predict Delta
Yes
is, we use a planning horizon of 20 timesteps for free play. We also use the exact same settings for the RND baseline. For Disagreement, we have a planning horizon of during free play.
For the Quadruped environment, hyperparameters are reported in Table S7. The iCEM controller parameters used in free play for Quadruped are the same the ones for GT models in Table S4, except for the noise strength Tinit 0.5.
an MLP ensemble instead of GNN ensemble. The
1.4 Extrinsic Phase: Zero-shot Downstream Task Generalization
In this section, we provide details the extrinsic phase following free play, where the learned GNN ensemble used to solve downstream tasks zero-shot via model-based planning.
L4.1 Details on Downstream Tasks and Reward Functions
The reward functions for all the CONSTRUCTION downstream tasks are computed as specified in Sancaktar et al. [12], where for all the assembly tasks we use the same structure as in the stacking reward. However, we do one modification the original reward computation in the assembly tasks. The assembly task reward is sparse incremental with reward shaping, where the reward also contains the distance between the gripper and the position of the next block to stacked. We modify how the next block ID is computed in the original implementation from Sancaktar et al. [12]. Instead of naively checking the number of unsolved objects to obtain the next block ID irrespective of order, we determine the next block to be the next unsolved block in the order. We found this modification to be important especially for the Pyramid tasks, where the sub-optimal next block computation might lead to the agent receiving reward to be close to the wrong block, in the case the robot places blocks with next_block
to their goal locations with just the sparse reward.
For the Robo Yoga experiments, the reward function is the same as proposed in [25] (taken from their code in https: / thub. com/ http//o/ykkeeeeeeee Meeiiinneeea-aee benchmark). is a dense reward as the shortest angle distance to the goal pose.
1.4.2 Planning Details for Downstream Tasks
The controller settings for the different downstream tasks are shown in Table S8, which are the same settings used in [12].
26

---

Table S8: Settings for the iCEM controller used for zero-shot generalization in the extrinsic phase of RaIR + ICEEUUS and CEE-US. The settings not specified here are the same as the general settings given Table $4. The settings are exactly the same as [12]. Task
Controller Parameters leteemean___ciiiie
Horizon
Colored-noise exponent
Noise strength
Cost Along Trajectory
dimi
CONSTRUCTION Stacking CONSTRUCTION-PIE Place
30
3.5
No
0.5
30
3.5
Yes
0.5
beat
CONSTRUCTION-Thooee Throwing
35
2.0
Yes
0.5
eun
CONSTRUCTION-PippinN
30
3.5
No
0.5
sum
Table S9: Settings for the RND predictor fo and target networks.
Parameter
Value
Train Predictor Every
2K Transitions
Network Size Embedding Dim k
256
128
Activation function
ReLU
Optimizer
ADAM
Batch Size
256
Epochs
10
Learning Rate
10
Weight Initialization
Orthogonal
Normalize Input
Yes
J Details on Baselines
J.1 Random Network Distillation
Random network distillation (RND) [21] has two networks: a predictor network and a randomly- initialized and fixed target network. The predictor network fe S RR is trained on the data collected by the agent match the outputs of the target network f S R² where k is the embedding dimension. The error between the predictor and target networks for the state 8t is used as the intrinsic reward signal:
Tiatrinsic
(1)o(se) f
(S4)
where the idea is that this error will be higher for novel states. RND can be seen extension of count-based methods to continuous domains. The parameters used for the RND module is given in Table S9.
Unlike prior work [21], we don't 't use RND with an exploration policy. Instead, we have a custom im- plementation using the same model-based planning backbone for RND as CEE-US, RaIR + CEE-US and RaIR. By using RND with iCEM instead of an exploration policy, we gain sample-efficiency, as also showcased in [12]. Note that although RND intrinsic reward itself is detached from the model and does not require an ensemble, we still deploy an ensemble of GNNs in this case. In previous work [12], single GNN model was used. In order to compute the RND intrinsic reward, we take the mean predictions the ensembles. We do this because ensembles have been shown to be more robust models [18], such that there is no inherent disadvantage to the RND baseline.
J.2 Disagreement
The Disagreement baseline [10] uses the same intrinsic reward metric CEE-US, given in equation (Eq. 5). We implement it with the same basel planning backbone in CEE-US, replacing exploration policy in [10]. The only difference to CEE-US is that the planning horizon during free play is only 1-step. The model and iCEM controller settings are otherwise the same as for CEE-US, RaIR+ + CEE-US and RaIR. Essentially, the comparison between this baseline and CEE-US showcases the importance of multi-step planning for future novelty. Note that the planning horizon
27

---

is only 1 for the free play phase. For fair comparison, we use the same horizon when solving the downstream tasks for all other methods.
K Discussion on Goal-conditioned RL for Unsupervised Exploration
In our setup with model-based planning for unsupervised RI., the focus is on distilling knowledge into world model that is robust enough such that we can plan for diverse downstream tasks afterwards. In the case of the goal-conditioned RL literature, the focus is on distilling this knowledge into a goal-conditioned policy, as in [25, 37, 38].
Although the exploration itself is unsupervised and guided by intrinsic rewards, the moment we solve a downstream task we need goal specification from the environment. Whether you use cond-condi policy to solve it or as in our case, use this goal to compute a reward function, is a separate distinction.
In goal-conditioned RL, you also need access to rewards at training time: the goals are used to compute this reward signal that is used to train the policy and value function. In a way, GCRL is supervised in terms of how you specify your goals (which goal space you use) and the rewards with respect to these goals. We consider the reward function in GCRL to be actually known to the agent. The nice part in GCRL is, you can put very general supervision with minimal assumptions, that don't 't' require much knowledge of environment dynamics. However, doing so, GCRL is restricting the class of tasks you can solve, namely to tasks that can be described as a goal. But in a wide range of environments, it is hard to specify tasks goal: for example running as fast possible in Halfchectah.
Our goal is to learn a robust world model that we can use for planning in later extrinsic phase. In our setup, during the intrinsic phase we make similar assumptions as Plan2Explore [11]. For the extrinsic phase, we need some specification of the task. For CEE-US, reward function is provided test time, whereas for GCRL is a goal (which fact also implies a reward function).
In model-based planning, since we have complete freedom of choice for the reward function, we can specify any task that can be solved in an MDP setting. Note that can use our setup also optimize a goal-conditioned sparse reward. The challenge for us is the limitedness of the zero-order trajectory optimizer: solving long horizon tasks with just sparse rewards becomes challenging, especially with imperfect world models. That's why in practice, we fall back to dense rewards for planning.
L Connections between Compression and RaIR
Our regularity objective, that seeks out low-entropy states with high redundancy, shares close ties with compression, and specifically with lossless compression using entropy coding.
We implemented version of our regularity idea using the lossless compression algorithm bzip2 corresponding to the direct version of RaIR with order k = 1 (Sec. 2.2). In this case, we describe the state s directly by the properties of each of the entities and the function Φ Sanj -> (X}+ that maps each entity to set of symbols and obtain (ss) = W 1 o ((aabj,)) as a union of all symbols for N objects. Instead of computing the entropy for the frequencies of occurrence in the resulting multiset of symbols like in RaIR, we instead transform these symbols into bytes and compress them with bzip2. We then define the intrinsic reward for compression as the negative length of the compressed ByteString such that:
I'coppression = -em((taip..commppesttt
compress (( ((s.))))) .obytes
(S5)
We also managed create regular shapes and patterns when optimizing for To 'compression via planning with ground truth models and also for free play with learned models. The reason we chose not to pursue this direction was because 1) Jossless compression algorithms like bzip2 don't 't perform as well on short ByteStrings, which is the case for us, as e.g. in CONSTRUCTION, we compress only 6 objects with their corresponding x-y positions, 2) artifacts are introducted to the regularitycompprssion metric by the transformation into bytes, where certain symbols become more compressible than others in this representation without any added regularity. As a result, we preferred our formulation with RaIR as it provides better control over the generated patterns and structures.
28

---

M Discussion on Object-centric Representations
We want to further discuss our reliance on object-centric more broadly entity-centric representations of the world for our regularity computations. We view the world collection of entities and their interactions. This a very general assumption, and one that is commonly used for its potential to improve sample efficiency and generalization of learning algorithms in wide range of domains [46-48]. The principles underlying entity segmentation and their importance for our perception are also studied in cognitive science [49-51].
Entity itself is very abstract and fluid concept. For humans, it dynamically changes depending on the granularity of the control problem we want to solve. For example, if we want to move plate of cookies from the counter to the table, each cookie on the plate is not treated as a separate entity, instead we view the whole plate as one, If we later want to cat the cookies, then the entity- view changes: now each cookie its own entity. (Example adapted from a private discussion with Maximilian Seitzer.)
Deciding which level of abstraction to in which control setting is non-trivial and a research direction which we find very exciting. In object-manipulation environments that are currently encountered in RL (with hard rigid bodies and non-composite structures), we don't find much ambiguity on entity identifications, hence our statement in Sec. 6 that observations per object in these cases are naturally disentangled. However, in open-ended real world scenarios it indeed needed to dynamically choose the right level of abstraction in the perception of scene and identify the necessary set of entities for the control problem at hand. Using RaIR for real-world e.g. cluttered scene would be very challenging without closing this action-peree perception loop. With orthogonal research in visual perception, we are hopeful that this will be possible and we see these synergies as exciting future work.
N
Code and Compute
Code available on the project webpage https://sites. www.sittss.gggleee.eoeeooo aoglle.com.. httpviiewwaarrreeeeeeeeeee We run the ground truth model experiments CPUs. As we using the true environment simulator as a model, each imagination step in the planning horizon takes as long as an environment step. We parallelize the ground truth models on 16 virtual cores The controller frequency in this case is ca. 0.25 Hz, for the settings given in Table S4. For the free-play phase, we have a fixed number of transitions collected at each training iteration, which get added to the replay buffer. After the data collection, the model is trained on the whole replay buffer for 25 epochs. Since the buffer size increases each training iteration with newly collected data, for this fixed number of epochs, the corresponding number of model training updates and thus also the runtime of the iteration, increase throughout free play. For RaIR CEE-US, the full free-play (300 training iterations) in CONSTRUCTION with 6 objects, where overall 600K data points are collected, takes roughly 87 hours using a single GPU (NVIDIA GeForce RTX 3060) and 6 cores on an AMD Ryzen 9 5900X Processor. The majority of this time is spent on the model training after data collection. The controller frequency for the collected rollouts with RaIR and the epistemic uncertainty calculations using GNN ensemble is ca. 5 Hz.
29


Reviews:
[{"review_id": "RHi1oZPAM9", "reviewer": "Reviewer_CLep", "summary": "Inspired by the fact that humans prefer symmetric patters, the author proposed to use regularity as intrinsic reward to encourage an agent to discover interesting environmental patterns. One example of regularity is to put boxes in a symmetric pattern. The results showed that this kind of intrinsic reward allows an RL agent to generate more interesting patterns in the absence of task rewards and also allows for better world model learning for zero-shot generalization.", "questions": "(See weakness)", "limitations": "Yes, the limitation is mentioned.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is novel and intuitive.\n- The experiments are comprehensive, including both analysis of the emergent behaviors and the performance of applying to downstream tasks.", "weaknesses": "- Lack of baselines: For section 3.3, I believe RND and ICM can be a reasonable baselines to strengthen the significance of the proposed method.  It seems to me that the goal of section 3.3 is to showcase the practical benefit of RaIR in solving tasks (maximizing extrinsic rewards). To show practical benefit, it would be necessary to answer why the proposed method is a better choice than prior works. Comparison with RND and ICM would answer this question.\n- Regularity considered in this paper are not well motivated. I agree the motivation of using regularity as intrinsic rewards is clear, but the specific design of each type of regularity in Table 1 are unclear to me. For example, why we consider these symmetry operations and why design distance in these ways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and feedback. Our response follows:\n\n# Baselines: \n\nIn the newly uploaded pdf, we have included experiments with RND. Here we run RND with the same model-based planning backbone (GNN ensemble together with iCEM as in CEE-US) and only alter the intrinsic reward used for planning. (Note that with a version of RND that trains an exploration policy instead of using a planner, meaningful exploration within the budget of 600K interactions is not observed as also reported in [1]). \n\nICM, on the other hand, computes the intrinsic reward only retrospectively: you need to have visited and observed the actual next state to be able to compute the ICM reward (the error between the model’s prediction and the actual observation). This means that ICM cannot be used in look-ahead planning methods. That’s why we have instead implemented a one-step disagreement version, which is essentially Disagreement from Pathak et.al, 2019 [2]. Here we also used a GNN ensemble backbone with iCEM. (the ICM+exploration policy combination has been shown to not work within the budget of 600K interactions in [1])\n\nSince the experiments are still running, we have included results from the first 180 iterations for our RND variant and 200 iterations of free play for Disagreement. We will update the results once the experiments finish.\nWe show that so far both baselines fail to solve the challenging stacking 3 objects task (we plot 3 seeds), in contrast to RaIR+CEE-US and CEE-US with the same amount of free play. Only one seed for RND solves the stacking task with 3% success rate at iteration 180. \n\nNote that we propose RaIR as an additional reward signal to drive exploration towards regularities. As such, RaIR in principle can be used to augment different intrinsic reward functions. In our paper, we illustrated this on the example of CEE-US+RaIR as CEE-US was shown to beat the other baselines (both policy and planning based) in [1]. \n\n# Motivation for regularity\nWe compute regularity as entropy of the histogram of occurrences for symbols in a multiset. The main question is: how do we obtain these symbols? The general problem is that there is no “ground truth” regularity. In our work, we intended to open the box of regularity that is otherwise not comprehensible: why certain representations lead to certain patterns. Symmetries, as well-characterized regularities, are one way to assess the properties of these representations. This is why we discussed the choice of different mapping functions $\\phi$ in terms of their invariances under known symmetry operations. This was intended to be used as an intuitive guide to choose the different representations. For example: since relational RaIR, as opposed to absolute relational RaIR, does not prefer reflections, mostly patterns with translational symmetries are generated, e.g. lines in ShapeGridWorld. \n\nKnowing the properties of these functions and its implications allows us to make design choices to inject certain biases. Similarly, for the recreation of existing patterns we cannot use direct RaIR since it is not invariant to translations. Although perhaps trivial at first glance, we wanted to formalize these properties to provide deeper insight into our regularity reward and its control knobs.\n\n[1] Sancaktar, et al (2022). Curious exploration via structured world models yields zero-shot object manipulation. NeurIPS\n\n[2] Pathak, et al (2019). Self-supervised Exploration via Disagreement, ICML\n\n"}, {"review_id": "A7tBew4oNh", "reviewer": "Reviewer_GveK", "summary": "The paper draws inspiration from a child’s development cycle and proposes the use of regularity as an intrinsic reward to help guide exploration in RL. It shows that existing metrics of state novelty can be combined with the proposed regularity metric to enable construction of regular structure during free play in a sample efficient manner. The proposed method is validated on experiments from two different tasks.", "questions": "It would be great if the authors could address the comments mentioned in the “Weaknesses” section.", "limitations": "The paper does have a limitations section. The authors must also acknowledge the following from the “Weaknesses” section - \n\n```This state representation might make it difficult to apply the method to the real world where privileged state information is not available. I would be curious to hear the author’s thoughts on this. The limitations section mentions that the same operation could be done on a latent space but its unclear if the RaIR would be applicable to such a latent space (which might not be as good as the privileged information) based state representation.```", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting and novel formulation of using regularity, in the form of symmetry, as an intrinsic reward to guide exploration during RL.\n- The paper builds on an existing novelty-based approach that uses the epistemic uncertainty obtained from an ensemble of models to guide exploration. The authors show that RaIR can be combined with such novelty-based rewards to further improve exploration in tasks that demand regularity. However, the authors acknowledge that for tasks that are chaotic like throwing and flipping, imposing such regularity hurts the performance (Table 2).\n- The paper shows impressive results for stacking towers in various configurations solely from intrinsic rewards while prior work achieves this using carefully constructed reward functions.\n- The paper demonstrates the effectiveness of pretraining with the regularity-based intrinsic reward in performing zero-shot generalization to downstream tasks (Sec. 3.3).\n- The authors also show a way of prompting (Sec. 3.4)  the agent to build certain structures by incentivizing it to raise regularity by building the same structure as the prompt.\n", "weaknesses": "- The authors organize the state in the form of graph nodes with each node representing a specific feature of an object or the agent. I had a few questions about this - \n  - How did the authors decide on this specific state representation? Does it have any advantages that a particularly attractive?\n  - It would be great if the authors could clarify how the graph is built (is it a fully-connected graph with all nodes)?\n   - The current form of state representation results in the authors using GNNs. However, the same representation can be dealt with using transformers. It would be interesting to add a transformer variant of RaIR.\n  - This state representation might make it difficult to apply the method to the real world where privileged state information is not available. I would be curious to hear the author’s thoughts on this. The limitations section mentions that the same operation could be done on a latent space but it's unclear if the RaIR would be applicable to such a latent space (which might not be as good as the privileged information) based state representation.\n- Since the method uses iCEM for policy learning, it seems like such a method of optimizing the policy for each new configuration won’t scale with reinforcement learning. It would be great if the authors could comment on this.\n- How many training steps does it take to train the transition model? This would throw some light on the sample-efficiency of the method.\nWhat is “relative time” in Figure 5?\n- In Sec. 3.4, why does the agent not arrange everything in a single line to maximize regularity? Is it because of the finite horizon optimization (meaning that tasks that need more time steps would be preferred less as compared to tasks giving higher immediate rewards)?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and address their questions and comments in the following. \n\n# Clarification on the Graph Architecture\n\nWe use a fully-connected Graph Neural Network (GNN). \n\n*How did the authors decide on this specific state representation? Does it have any advantages that a particularly attractive?*\n\nThe GNN requires an object-factorized state representation. Note that in this case we take the existing observation in the Construction environment (which is an extension of the original Fetch Pick & Place environment, and has been introduced in [1]). In the original observation, the robot observations (end effector position and velocity, gripper state) and the individual object observations (position, orientation as Euler angles, linear and angular velocities) are concatenated to form the environment observation. We don’t modify the existing state representation from the environment and only factorize this observation vector into its components, which is the format needed for the GNN. The RaIR computation is decoupled from the GNN or any model: it can be done on actual state observation or model predictions. For RaIR, we also use an object-factorized representation, but we don’t feed in the whole state and instead a subspace with x-y-(z) positions. \n\n# Transformers\n\nIndeed, GNNs with attention mechanisms for neighborhood aggregation are shown to be equivalent to transformers with a few modifications. We are planning to test with Transformers as world model backbones in future work as well. It is important to highlight that we expect intrinsic rewards for regularity via RaIR to lead to more capable world models in assembly tasks or in general tasks favoring regularity, irrespective of the world model backbone that is used. \n\n# Policy learning with RaIR\nAs RaIR is stationary/Markovian, it can be used as a reward signal to learn a policy via RL.\nNote that for this case, we wouldn’t be training a new policy for each configuration, but we would simply train one policy with RaIR as reward. When we roll out the learnt policy, with the stochasticity of the policy and the different start states in the environment, we would expect different configurations to emerge (i.e. converge to different local minima of RaIR). As such, we don’t see a reason why RaIR wouldn’t scale with RL.\n\n*In Sec. 3.4, why does the agent not arrange everything in a single line to maximize regularity?* \n\nYes, indeed due to finite-horizon planning as well as the limited sampling budget, we don’t necessarily converge to global minima, which is a full tower for x-y compression and a line (horizontal line or tower) for x-y-z. The planner can find solutions that are reachable within the planning horizon from  the current configuration. For certain starting configurations only local optima are found during an episode. \n\n# Real-world application\n*This state representation might make it difficult to apply the method to the real world where privileged state information is not available.*\n\nWe refer the reviewer to our general response, where we discuss this at length. \n\n# Details on Experiment Pipeline\n*How many training steps does it take to train the transition model? This would throw some light on the sample-efficiency of the method. What is “relative time” in Figure 5?*\n\nThis is indeed a strong point of our method, which we will explain better in the paper. We refer the reviewer to the general response, where we explain the experiment pipeline and the metrics in detail for further clarification. Here we want to highlight that during the whole free play time, only 600K environment steps are performed (corresponding to about 6.5h of real-world interaction), which underpins the high sample-efficiency.\n\n[1] Li et al (2020), Towards Practical Multi-object Manipulation using Relational Reinforcement Learning. ICRA. "}, {"review_id": "gIOmkEOZDw", "reviewer": "Reviewer_fSr3", "summary": "This paper presents a new approach that utilizes a regularity as an intrinsic reward, which aims to regularize, or constrain, the search space of exploration towards state regions where the entities are more likely to have structured patterns. To this end, the paper first formulates how to define regularity as an intrinsic reward and investigates several candidates that can be used as a regularity. And this intrinsic reward is combined with the disagreement-based intrinsic rewards to guide the exploration. Experiments show that optimizing this intrinsic reward (by using the ground-truth models) indeed leads to behaviors that induces structured patterns in the environments, and further show that this can be useful in case everything is learned end-to-end. ", "questions": "I don't have further question other than the ones described in the Weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting formulation based on a good intuition. I enjoyed reading the paper.\n- Clear writing helpful for understanding the method. This should be highlighted as understanding the concept would have been difficult without the clear writing.\n- Experiments are conducted to support the claims by using ground-truth models and learned models. It starts from showing the potential benefit of using the proposed reward by using the ground-truth model, then shows that it can be useful with learned models, and finally shows that it can be useful for improving the performance on downstream tasks.", "weaknesses": "- In contrast to clear writing until the Method section, Experiments section is a bit difficult to parse, especially it is difficult to understand the metrics used in Section 3.2. Even though they are explained in the text, further emphasis and clear description on the meaning of metrics (e.g., what relative time means, how to interpret objects in the air, flipped) could be helpful.\n- As the authors already highlighted in the Limitation section, this method severely depends on the accessiblity of ground-truth internal states of all the entities in the environment, which is not feasible in practice. But this is understandable as learning such information is not the main focus of the paper.\n- Another limitation of this paper is the difficulty of balancing (i) the chaotic exploration from novelty-seeking intrinsic rewards. (in this paper, disagreement-based reward) and (ii) regularity-seeking intrinsic rewards. Of course there could be cases when injecting our prior of prefereing the structuredness can be helpful, but sometimes it's not, as can be seen in Throw and Flip experiments in Table 2. Would there be a way to automatically balance these two rewards, or in general, how can we control the magnitude of injecting our prior to the exploration process?\n- Experiments are a bit limited as only specific type of structuredness is investigated (for simplicity).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and feedback.\n\n# Generalizability of RaIR\n\nWe have now added new experiments where we showcase RaIR in different scenarios. In order to address the concern of focusing on one specific type of structuredness, we have tested a variant of ShapeGridWorld, where we added color as part of the object representation to be compressed for regularity. In this case, we not only have x-y positions of the circles as symbols to be compressed, but also have a color encoding (as a one-hot encoding), such that RaIR aims to generate regularities not just spatially but also in the arrangement of colors. Examples can be seen in the uploaded rebuttal pdf with 2 and 3 colors (Fig. D1). \n\nWe have also optimized for RaIR in the DeepMind Control Suite environments: Quadruped and Walker, where we were able to obtain some interesting regular poses via optimizing for RaIR with ground truth models. Note that these poses mostly coincide with the goals in the Roboyoga benchmark [1]. This adds to our point that seeking regularity is reflected in human designers’ idea of interesting goals and is a very fundamental bias that pops up in various domains.\n\n# Balancing Regularity and Chaos\n\nAs is the case with any injection of bias, we enter a trade-off where we have some advantages and some disadvantages. \n\nAs the reviewer also hinted at, if we trust the fact that we want structured behavior in, we accept to sacrifice some more “chaos”-preferring behavior. This can in fact be seen as a resource-allocation problem. If the agent only has limited play time, what should it focus on to best prepare for future tasks? If we have have unlimited resources we can train for these two rewards in e.g. an alternating fashion. \n\nHowever, in our case, these two forces complement each other as discussed in Sec 3.2 in the main paper. Ensemble disagreement comes with the added problem of chaos, but it also helps the agent focus on more interesting and challenging patterns during free play, instead of generating predictable and boring patterns and helps finding high RaIR patterns that are sparse solutions. \n\nThat is why in this work, we investigated a linear combination of these two reward terms, with a hyperparameter $\\lambda$. In the rebuttal pdf in Figure D6, we have included the downstream task performance for the chosen lambda value for RaIR + CEE-US ($\\lambda=0.1$), RaIR + CEE-US with smaller lambda ($\\lambda=0.01$), pure RaIR ($\\lambda=0$) and pure CEE-US. We can see that the behavior towards regularity vs. chaos can be controlled by this hyperparameter.\n\nTuning this hyperparameter is not critical and a simple heuristics can be employed that brings the ensemble disagreement and the RaIR improvements values on a similar scale. The RaIR improvement steps can also be computed analytically, e.g. by considering the full entropic state to the one with two 2 entries in alignment. \n\nAs ensemble disagreement goes down with more training throughout free play, one could explore options to schedule $\\lambda$ accordingly. We find studying different combination strategies to be an exciting direction and leave it to future work.\n\nRegarding the questions about the experiment descriptions and a more detailed discussion on concerns about ground truth internal states, we refer the reviewer to the general response. \n\n[1] Mendonca et al (2021), Discovering and Achieving Goals via World Models. NeurIPS \n"}, {"review_id": "qpEgGhmPoB", "reviewer": "Reviewer_fHtQ", "summary": "This work proposes an exploration approach that prioritizes \"regularity\" in the exploration behavior, in contrast to typical novelty-seeking exploration. The objective is defined as the minimization of entropy over user-defined, object-centric features. The method is primarily evaluated on block rearrangement in the robotic manipulation setting.\n\n---\n**8/22/23 Update after author response**\n\nMy major concern with this paper was the generality of RaIR. In my opinion, there are two ways to create a good exploration method. The first is to come up with a general method with little-to-no assumptions that works across a variety of tasks. The second is to create a specialized method, be upfront about the assumptions made, and get very good performance by leveraging those assumptions. \n\nBefore the rebuttal, RaIR was introduced as a general exploration approach, yet its experiments were limited to stacking objects of identical shapes and sizes. While stacking is a very hard exploration challenge, I expected to see environments beyond object manipulation like locomotion, video games, etc. Furthermore, RaIR does depend on the key assumption of object-centric state representation, which it uses in both its objective and dynamics model. \n\nThe authors provided some followup experiments in the rebuttal. They evaluated their method on a more generalized version of their construction environment, as well as on a Quadruped locomotion environment. They also provided some additional new environments (RoboDesk, Walker),  but did not run exploration experiments on them, only sanity checks. \n\nSo to conclude:\n\n**My concern on generality of the method:** With the new exploration experiments on generalized construction and quadruped, the authors have shown that RaIR can handle diverse objects, and can work in a single non-manipulation task. Therefore my concern is somewhat alleviated. I would recommend the authors to show more diverse environments in general though, to remove any remaining doubt from future readers that RaIR is not engineered towards a particular type of task.  \n\n**My other concerns about object-centric assumption, baselines, related work, novelty, etc:** The authors have done a good job on addressing my concerns here. I hope these discussions will go into the next version of the paper, and influence how RaIR is presented.\n\nAs of such, I will raise my score to 5. The authors have passed the threshold for addressing my main concern on generality, given the limited rebuttal time.\n\nFor future improvements, I would recommend the following:\n1) Continue adding more diverse environments to show RaIR is a general exploration method.\n2) Study the assumption of object-centric representation a bit more - what happens if we use RaIR with an unstructured dynamics model? Will the regularity objective still work?", "questions": "What is the scope of RaIR? I can see two ways to present RaIR. \n\nOne way, and the way the paper is written, is to promise a broad scope. However, can RaIR be applied to more complex and realistic settings both in the manipulation domain and beyond it?  I would suggest for experiments that show RaIR can be applied to other domains, and in more complicated scenarios.\n\nThe other way, is to narrow down the scope and claims of RaIR to an intrinsic reward for object manipulation, which itself is a very challenging and interesting domain. I would like to see experiments in more realistic manipulation setups (different shapes, sizes, masses, etc.), as mentioned above.", "limitations": "The authors did address a main limitation, which is the assumption to an object centric, interpretable state space. \nAnother limitation not mentioned is the need to define task-specific features for RaIR.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The high level idea of using regularity to focus exploration is well motivated, and addresses the weaknesses of the more common novelty-seeking exploration methods. \n- This method explores well in block stacking / rearrangement, a hard exploration problem. The experimental section analyzes the block stacking task extensively.\n- The manuscript is generally well written.", "weaknesses": "My main concern is over the scope and generality of this method. While the introduction motivates regularity as a general-purpose exploration objective, the actual implementation and assumptions made to implement the objective restricts it to a very particular set of tasks - manipulation of identical and well characterized objects.  I would have expected, from the introduction, a method that could apply to all sorts of domains (e.g. robotic locomotion, video games) instead of only object manipulation tasks.\n\nIn RaIR's environments, all objects are the same shape and have the same physics. For the construction environment, the authors only apply RaIR to the x-y state space to bias towards vertical alignments. However, this x-y assumption would break if the objects were not identical, since the vertical alignment would now be object dependent. \n\nAnother major limitation (which I appreciate the authors mentioning in Sec. 5)  seems to be the assumption of a known state space and dependence on user features. RaIR seems to hinge on having a state space that is interpretable, so that the user can provide user-defined features to bias the exploration.  I am concerned about the scalability of providing user defined features, particularly if the state space becomes more complex, or uninterpretable (e.g. pretrained ImageNet representations). However, if RaIR claims to be a general exploration method, it should address such problems. \n\nFinally, the experiment section is lacking baselines. There is only 1 novelty-seeking baseline that the authors compare against, which is by design going to be suboptimal in assembly tasks. I would like to see how RaIR compares to an exploration baseline that actually seeks regularity in its objective as well. I believe SMiRL (1) to be a viable baseline, as it also seeks to minimize surprise. I would also like to see some baselines from the unsupervised goal-conditioned RL literature, as those methods also do stacking. \n\n## Moderate concerns:\n- Claims of Novelty - The authors claim in the first line of the abstract that they propose regularity as a novel reward signal for intrinsic motivation. However, the high level idea of regularity / stability / niche-seeking has been explored as an intrinsic motivation signal for RL(1), and more generally in the active inference literature (2). I would like to authors to revise their claims of novelty, and include the relevant literature.\n- Related work is rather short. Only a small paragraph is dedicated to intrinsic motivation. A rather long paragraph is dedicated to Compression, which is not necessary.  I would recommend the following changes.\n\t- Expand on intrinsic motivation in RL, particularly methods that also do manipulation tasks and how RaIR compares to them. Mention (1,2) as well, as they also explicitly mention concepts similar to regularity (stability, niche-seeking). Another omission is the unsupervised skill discovery literature, where some works (3) also show manipulation tasks.  \n\t- Paragraph on compression is unnecessarily long and detailed - it is sufficient to briefly mention compression progress and move the rest to the appendix if necessary. That space should be used mainly on Intrinsic Motivation.\n\n## Minor concerns:\n- Line 84: what is $n_a, n_s$?\n- Line 124: I found the \"Relational RaIR of order K\" to be a bit abstract, and I needed to read it a few times over to fully understand it. Having a figure like Figure 3 for this section would have been very helpful.\n- Line 220: Please point to a figure.\n- Recreation experiment - while the recreation behavior is interesting, the authors do not include any practical applications or downstream implications for this phenomenon. I would prefer this section be moved to the appendix and the space be used for other things (e.g. experiment in new domain) or extended related work.\n\t\n1. Berseth et al, SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments, ICLR 2021\n2. Friston et al, Active inference and learning. Neuroscience & Biobehavioral Reviews 2016\n3. Zhao et al,  Mutual Information State Intrinsic Control, ICLR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback on our work. We address their concerns in the following. \n\n# Generality of RaIR \n\nWe have provided new experiments in new domains in the rebuttal pdf (Fig. D2-D4). As the reviewer suggested we have included RaIR generated arrangements in locomotion domains (Quadruped and Walker), Construction with custom shapes and RoboDesk. \n\nFor Quadruped and Walker, we take the hips, ankles, toes etc. to be the entities over which we compute RaIR. We see the emergence of interesting regular poses, that also happen to coincide with goal poses proposed in the Roboyoga benchmark [1].\n\nFor Custom Construction with shapes, we include not just cubes (mass(m) = 2), but also a flat block (m=1.5), a short column (m=1) and a ball (m=1). In this case, we compute RaIR again on the objects’ x-y positions (center of mass (CoM)). We show that we are still able to generate stacks and regular constellations. We would also argue that there is no limitation within RaIR that would cause it to not work for objects with different shapes. If we have very complicated geometries, e.g. composite shapes, we might want to represent each entity not just with their CoM but instead add new keypoints. \nWe also showcase RaIR in Robodesk where the entities are very varied: drawer, slide, buttons and two different blocks and a ball. Here, with regularity, we observe the agent using the ball or the block to press buttons and stacking objects together. We hope that the new experiments better illustrate the generality of RaIR as a concept. \n\n*RaIR seems to hinge on having a state space that is interpretable*\n\nWe refer the reviewer to our general response, where we discuss this at length. We want to reiterate here shortly that we use proprioceptive information for objects (positions, orientation and velocities), which itself is not user-defined and is specified by the environment. We only consider access to the knowledge which subspace corresponds to object positions. We find the inference of object positions from images to be an orthogonal research problem with ongoing works showing promising results. Note that using ImageNet features alone does not help solve the control problem, as the focus there is on semantic understanding for visual learning tasks [2]. In this case, additions are needed to align the representation and actions, i.e. control over the environment. One could create regularity spatially with the added constraint of compressing ImageNet features (e.g. object labels), similar to the color experiment we show in Fig D1. \n\n# Novelty of RaIR\n\nThere is a fundamental difference between our notion of regularity and niche-seeking proposed in SMiRL and active inference literature: in all the environments showcased in SMiRL the environment itself is dynamic (or unstable) and the agent wants to reach stable and familiar states. For example, not fall down the cliff, avoid enemies that are shooting at you. Our notion of regularity is completely decoupled from surprise and is in fact closely related to compression: we want to find a state description of the current scene that is regular and compressible within itself. \n\nMethods such as SMiRL require an active source of entropy. In an environment where the environment itself is static unless you act on it, which is the case in all environments we consider, we strongly believe SMiRL wouldn’t work due to the Dark Room problem, and the agent would avoid doing anything. This is also what happens when you penalize novelty in object-manipulation environments: the agent doesn’t touch any objects to make sure everything is predictable.\n\nWe will revise the related work section to make these distinctions clearer as suggested by the reviewer, and extend the related work discussion to include skill literature as well.\n\n# Baselines \n\nWe have included new baselines in the rebuttal pdf, most notably RND (see general response for more details). RND aims for state space coverage and can be seen as an extension of count-based metrics to continuous domains, thus has no preference for chaos. We showcase that it performs worse than both CEE-US and our regularity augmented version RaIR + CEE-US on the Singletower 3 task. \n\nIt is important to highlight here that the main goal of our work is to introduce RaIR as an additional reward signal to drive exploration towards regularities. As such, RaIR in principle can be used to augment different intrinsic reward functions. In our paper, we illustrated this on the example of CEE-US+RaIR as CEE-US was shown to beat the other baselines (both policy and planning based) in [2].\nNote that in unsupervised goal-conditioned RL paradigms, the driving exploration force is still commonly ensemble disagreement, as in LEXA[1] and PEG[4]. In Sec. 4, we did mention PEG and its stacking performance in a similar environment. In principle, RaIR could also be plugged into these methods as a goal-picking strategy, augmenting ensemble disagreement.\n\n# Addressing minor concerns\n- $n_a$ is the action space size and $n_s$ is the state space size.\n- Abstract order-k: We indeed aimed to generalize RaIR and abstract away from the mapping functions such as distance or difference (with k=2) to functions that operate over k-tuples corresponding to k objects. In Figure 2b, you can find an illustration for k = 2 that we think is useful to clarify k>2. \n- Line 220: We will point to Figure 1 as the reviewer suggested.\n- Recreation experiment: Our goal with these experiments was to showcase that we have a way to automatically pick-up regularities in the environment as you observe them. \n\n[1] Mendonca et al (2021), Discovering and Achieving Goals via World Models. NeurIPS \n\n[2] Sharma et al (2023),  Lossless Adaptation of Pretrained Vision Models for Robotic Manipulation. ICLR\n\n[3] Sancaktar, et al (2022). Curious exploration via structured world models yields zero-shot object manipulation. NeurIPS\n\n[4] Hu, et al (2023). Planning Goals for Exploration."}]

Author rebuttal:
We thank all reviewers for their constructive feedback and appreciate that they found our method to be "novel and intuitive”, the experiments “comprehensive” with “impressive results for stacking towers in various configurations solely from intrinsic rewards” and the paper “generally well written”. 

# Summary 
The rebuttal addresses:

**1.** Generality: We added applications of RaIR to new environments (see pdf)
- 2 locomotion environments: Quadruped and Walker
- RoboDesk that contains entities with diverse geometries (drawer, button, blocks etc.)

and modifications of the existing environments:

- Custom Construction with diverse shapes (flat block, short column and ball), where the shapes also differ in their masses.
- ShapeGridWorld with color, where the objects’ colors are included as part of the object feature for regularity.

We showcase the generality of RaIR and its applicability to diverse environments (here using ground truth GT models). 

**2.**  Baselines: As suggested, we added two baselines, RND and a type of ICM/ 1-step prediction error, that get outperformed by our method, see Fig D5 

**3**  Addressed questions of novelty and object-centric representations

# Baselines 
We run two new intrinsic motivation baselines: RND [1] and one-step Disagreement [2] (as a look-ahead alternative to ICM [3] suggested by Reviewer CLep). We include the results on the *Singletower 3* and *Pick & Place 6* tasks. The experiments are still running, thus we include partial results. The baselines solve the pick & place task but completely fail at the more challenging stacking task. Additionally: attempting the 3 stack, RND manages to get to a stack of 2 objects only with $0.39 \pm 0.24$ success rate; one-step disagreement only gets a stack of 2 with  $0.19 \pm 0.02$ success. 

# Clarification on object-centric representations

Let us clarify the concerns regarding object-centric representations and interpretable state spaces that were raised by Reviewers fSr3, fHtQ, GveK:

We embrace object-centric representations as a suitable inductive bias in RL, where the observations per object (consisting of poses and velocities) are naturally disentangled. So we don’t see this to be limited. The wording in the paper was perhaps indeed misleading: access to object-centric proprioceptive information is not necessarily privileged or unattainable itself. It is indeed very common in robotics to do pose estimation on objects for control. Approaches such as [5] do object pose estimations from images, and YOLO or even unsupervised object-discovery methods such as Slot Attention have been used to infer object pose, or at least object positions [4]. The assumption that this object-factorized state representation is interpretable is not far-fetched. 

The computation of RaIR itself is very generalized. The main design choice goes into which symbols we use to describe the scene and to construct the multiset. In our work, we take a shortcut of putting the human bias in as “compressing positions is a good idea”. But this can be extended: see for instance our new experiments where we put in color as an additional component for regularity (Fig. D1). 

Applying RaIR directly to latent representations that are not inherently disentangled presents a challenge: developing a representation mirroring human-relevant structure and regularities. Here, examples of significant regular situations of interest could come in to learn a tokenizable representation for RaIR. This resembles real-world learning, where exposure to regular structures (e.g., towers, bridges) leads us to replicate these patterns while interacting with blocks. However, as Reviewer GveK noted, RaIR computation here faces practical complexities. We will incorporate this discussion per the suggestion in our revised paper.

# Details on Experiment Pipeline

We agree with reviewers GveK and fSr3 that details regarding the terminology and metrics used need better explanation. We provide these here and in the revised paper: 
During free play, we start with randomly initialized models and an empty replay buffer. Each iteration of free play (referred to as training iteration in the downstream task success plots) consists of data collection with environment interactions (via online planning), and then model training on the collected data so far (offline). 

In each iteration of free play, we collect 2000 samples (20 rollouts with 100 timesteps each) and add them to the replay buffer. During the online planning part for data collection, we only perform inference with the models and no training is performed. Afterwards, we train the model for 25 epochs on the replay buffer. We then continue with data collection in the next free play iteration (see Alg. S1 in appendix). 

The relative time is the percentage of timesteps in the collected 2000 samples per free play iteration the agent performs certain types of interactions. So 50% metric for *one object moved* means an object was moved in 1000 timesteps in that free play iteration. *Two or more objects moved* checks if at least 2 objects are moving at the same time. *Object(s) in air* means one or more objects are in air (including being held in air by the agent or being on top of another block). *Object(s) flipped* checks for angular velocities above a threshold for one or more objects, i.e. they are rolled/flipped.
Overall, during the whole free play time, only 600K environment steps are performed, which marks the high sample-efficiency of our method.

[1] Burda et al (2019). Exploration by Random Network Distillation, ICLR

[2] Pathak et al (2019). Self-supervised Exploration via Disagreement, ICML

[3] Pathak et al (2017). Curiosity-driven Exploration by Self-supervised Prediction, ICML

[4] Heravi et al (2023). Visuomotor control in multi-object scenes using object-aware representations, ICRA

[5] Zhang et al (2023). Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild, ICLR

Meta-review: Exploration is an important problem and the authors suggest a somewhat interesting and novel take on it -- instead of incentivizing novelty, incentivize the agent to form regular structures. All reviewers recommend the paper be accepted, and I concur. Reviewers had concerns about the scope of where the method can apply and I commend the authors for doing a good job of addressing the questions in the rebuttal period. 

However, measuring regularity needs strong assumption on the observation space which restricts the applicability of the method. In the rebuttal phase there was ample discussion between the reviewers and the authors and its well summarized here: https://openreview.net/forum?id=BHHrX3CRE1&noteId=033rWTzPdV -- I recommend authors to include additional results and the suggestions here alongwith other suggestions made by the reviewers in the final version of the paper.
Decision: Accept (poster)