Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
Authors: Sumedh Anand Sontakke
Arash Mehrjou
Theofanis Karaletsos
Laurent Itti
Bernhard Schölkopf
Venue:  (2021)
Keywords: Causal Representation Learning
Unsupervised/Self-Supervised Reinforcement Learning

Abstract: Humans show an innate ability to learn the regularities of the world through interaction. By performing experiments in our environment, we are able to discern the causal factors of variation and infer how they affect the dynamics of our world. Analogously, here we attempt to equip reinforcement learning agents with the ability to perform experiments that facilitate a categorization of the rolled-out trajectories, and to subsequently infer the causal factors of the environment in a hierarchical manner. We introduce a novel intrinsic reward, called causal curiosity, and show that it allows our agents to learn optimal sequences of actions, and to discover causal factors in the dynamics. The learned behavior allows the agent to infer a binary quantized representation for the ground-truth causal factors in every environment. Additionally, we find that these experimental behaviors are semantically meaningful (e.g., to differentiate between heavy and light blocks, our agents learn to lift them), and are learnt in a self-supervised manner with approximately 2.5 times less data than conventional supervised planners. We show that these behaviors can be re-purposed and fine-tuned (e.g., from lifting to pushing or other downstream tasks). Finally, we show that the knowledge of causal factor representations aids zero-shot learning for more complex tasks.

Under review as a conference paper at ICLR 2021
CAUSAL CURIOSITY: RL AGENTS DISCOVERING SELF-SUPERVISER EXPERIMENTS FOR CAUSAL REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Humans show an innate ability to learn the regularities of the world through interaction. By performing experiments in environment, we able to discern the causal factors of variation and infer how they affect the dynamics of our world. Analogously, here attempt equip reinforcement learning agents with the ability to perform experiments that facilitate a categorization of the rolled-out trajectories, and subsequently infer the causal factors of the environment in a hierarchical manner. We introduce novel intrinsic reward, called causal curiosity, and show that it allows our agents to learn optimal sequences of actions, and discover causal factors in the dynamics. The learned behavior allows the agent infer binary quantized representation for the ground-trute causal factors in every environment. Additionally, we find that these experimental behaviors are semantically meaningful (e.g., to differentiate between heavy and light blocks, our agents learn to lift them), and are learnt in a self-supervised manner with approximately 2.5 times less data than conventional supervised planners. We show that these behaviors can be re-purposed and fine-tuned (e.g., from lifting to pushing or other downstream tasks). Finally, we show that the knowledge of causal factor representations aids zero-shot learning for more complex tasks.
INTRODUCTION
Discovering causation in environments an agent might encounter remains an open and challenging
problem for causal reinforcement learning (Schölkopf 2015), Bengio et al. (2013), Schölkopf
(2019)). Most approaches take the form of BAMDPs (Bayes Adaptive Markov Decision Processes) ZZintgraf al.] (2019) or Hi-Param MDP (Hidden Parameter MDPs) Velezi & Konidaris 2016) Yao et al. (2018) Killian et al.) (2017); Perez et al. (2020) which condition the transition pl M88+118 St, 0,; H) and/or reward function l8(r++1 [8t, Ge 8 H) of each environment hidden parameters (also referred to as causal factors in some of the above studies). Let S. E A. R, H E H. where S, A. R. and H. are the set of states, actions, rewards and feasible hidden parameters. In the physical world and in the case of mechanical systems, examples of the parameter hy E H include gravity, coefficients of friction, masses and sizes of objects. Typically, is treated as latent variable for which an embedding is learned during training, using variational methods (Kingma al. (2014); y:::1::11 lsse et al. (201)). Let 50:7 be the entire state trajectory length T. Similarly, corr is the sequence of actions applied during that trajectory by the agent that results in 80.7- In an environment parameterized by these causal factors, these latent variable approaches define a probability distribution over the entire sequence of (rewards, states, actions) conditioned latent as p(ro:T, So.T, Qg:r-1; 2);2 that factorizes as
T-1
II
pr on, S141, 2)p($4411))))) /51, ae, =)p(arlst,2)
due the Markov assumption. At test time, the agent infers the causal factor associated with its environment by observing the trajectories produced by its initial actions that can be issued by any policy such model-based reinforcement learning.
In practice, however, discovering causal factors in a physical environment is prone to various challenges that caused by the disjointed nature of the inffuence of these factors on the produced

---

Under review as a conference paper at ICLR 2021
trajectories. More specifically, at each time step, the transition function is affected by a subset of global causal factors. This subset is implicitly defined on the basis of the current state and the action taken. For example, if body in environment loses contact with the ground, the coefficient of friction between the body and the ground no longer affects the outcome of any action that is taken. Likewise, the outcome of an upward force applied by the agent body the ground is unaffected by the friction coefficient. We can therefore take advantage of this natural discontinuity to discern causal factors.
Without knowledge of how independent causal mechanisms affect the outcome of a particular action in a given state an environment, it becomes impossible for the agent to conclude where the variation it encountered came from. Unsurprisingly, Hi-Param and BAMDP approaches fail to learn a disentangled embedding for the causal factors, making their behaviors uninterpretable (Perez et al.. (2020)). For example, if, in an environment, a body remains stationary under particular force, the Hi-Param or BAMDP agent may apply higher force to achieve its goal of perhaps moving the body, but will be unable to conclude whether the "un-movability" was caused by high friction high mass of the body, Additionally, these approaches require human-superrised reward engineering, making difficult to apply them outside of the simulated environments they are tested in.
Our goal is, instead of focusing on maximizing reward for some particular task, to allow agents to discover causal processes through exploratory interaction. During training, our agents discover self-supervised experimental behaviors which they apply to a set of training environments. These behaviors allow them to learn about the various causal mechanisms that govern the transitions in each environment. During inference in a novel environment, they perform these discovered behaviors sequentially and use the outcome of each behavior to infer the embedding for a single causal factor (Figure 1).
The main challenge while learning a disentangled representation for the causal factors of the world is that several causal factors may affect the outcome of behaviors in each environment. For example, when pushing a body on the ground, the outcome, i.e.. whether the body moves, or how far the body is pushed, depends on several factors, e.g., mass, shape and size, frictional coefficients, etc. However, if, instead of pushing on the ground, the agent executes a perfect grasp-and- behavior, only mass will affect whether the body is lifted off the ground or not.
Thus, it is clear that not all experimental behaviors are created equal and that the outcomes of some behaviors are caused by fewer causal factors than others. Our agents learn these behaviors without supervision using causal curiosity, an intrinsic reward. The outcome of a single such experimental behavior is then used to infer binary quantized embedding describing the single isolated causal factor. Even though causal factors of variation in physical world are easily identifiable to humans, a concrete definition is required back up our proposed method. We conjecture that the causality of factor of variation depends on the available actions to the agent. If the set of actions that an agent can take very limited, there no way for it discern diverse of causal factors in the environment.
Definition 1 (Causal factors). Consider the POMDP (O, S. A. p. r) with observation space O, state space S, action space A, the transition function p. and the reward function r. Let Ou:T E OT denotes trajectory of observations and T be the length of such trajectories. Let d(.,-) O x OT + R, be a distance function defined on the space of trajectories of length T. The set H = (h1,12,,-.. ,,,,..,,haa is called a set of eccaual factors if for every hj H, there exists a unique sequence of actions og:r that clusters the state trajectories into two sets S and S' such that
min{ (d(oww, ( ourr E 0, ObTT E0'}>c
(2)
and that hy is the cause of the trajectory states obtained i.e.,
(ooo k), oo.T) + plooor|do(h, = k'), TD.T) Vk K
(3)
Intuitively, factor of variation affecting a set of environments is called causal if there exists sequence of actions available to the agent where the resultant trajectories are clustered into two or more sets (for simplicity here we assume binary clusters). This is analogous the human ability to conclude whether objects are heavy or light, big or small. For a gentle introduction to the intuition about this definition, we refer the reader to Appendix [
According to Def. [ a causal factor is a parameter in the environment whose value, when intervened on (i.e. varied) over set of values, results in trajectories of states that are divisible into disjoint
2

---

Under review as a conference paper at ICLR 2021
Temporal Clustering
Causal Inference Module
Enviromment
Reset
Environment
Zox
Policy Network
Self-supervised
Downstream lask
Seirt
Experiment Planner
Maximizing external reward
Exploration Phase
Figure Overview of Inference. The exploration loop produces series of K experiments allowing the agent infer the representations for K causal factors. After exploration, the agent utilizes the acquired knowledge for downstream tasks.
clusters under a particular sequence of actions. These clusters represent the quantized values of the causal factor. For example, mass, which is causal factor of body, under action sequence of clusters, liftable (low mass) and not liftable (high mass).
a grasping and lifting motion, results However, such an action sequence is not known in advance. Therefore, discovering causal factor in the environment boils down to finding sequence of actions that makes the effect of that factor prominent by producing clustered trajectories for different values of that environmental factor.
Using the above, we propose an intrinsic reward, which allows our agents to discover experimental behaviors which resulting in high sample efficiency. Our work, therefore, forms an important link between structured representation learning and skill discovery, largely disjoint fields in RL, which stand benefit from each other.
semantically meaningful and be used re-train for downstream tasks,
The contributions of the work follows:
We equip agents with the ability to perform experiments and behave meaningfully in a set of environments in unsupervised manner. These behaviors expose obfuscate specific independent causal mechanisms that occur in the world of the agent, allowing the agent to learn about each in the absence of the others, an important human behavioral trait.
We introduce an intrinsic reward, causal curiosity, which allows our agents to discover these behaviors without human- engiineeed rewards. The outcomes of the experiments are used to learn disentangled quantized binary representation for the causal factors of the environment, analogous to the human ability conclude whether objects are light/heavy, big/small etc.
Through extensive experiments, we conclude that knowledge of the causal factors aids sample efficiency in two ways first, that the knowledge of the causal factors aids transfer learning across multiple environments, and, second, that the experimental behaviors acquired can be repurposed for downstream tasks.
METHOD
Consider set of N environments & with e(0) where el)) denotes the 7th environment.
The letter H is overloaded. While H is a set of global causal factors (as defined in Def. I such that hy E H, each causal factor hy is itself a random variable which assumes a particular value for every instantiation of an environment. Thus every environment e(1) is represented by a set of causal factors (i)
(A) Vj}. vj}. For each environment ! (2 (z) (1)).. (k represents the disentangled embedding
vector, such that 2 encodes ())
3

---

(1) 0:7 Under review as a conference paper at ICLR 2021
Algorithm 1 Training Scheme
1: Initialize j j=0
2: Initialize training environment set Envs
3: for iteration to M do
D Experiment Planner Training Loop
4: Sample experimental behavior do:T CEM()
5: for 1t env in Erws do
6:
Apply 40:r to env
7:
Collect 541
8:
Reset env
9:
Calculate (L(S M) given that M is bimodal clustering model
Calculate Curiosity
10:
Update CEM() distribution with highest reward trajectories
11:
Use learnt 9ax (2)
(z|S) SS) for cluster assignment of each env in Enes i.e. 2 gar (2|S(0)
Update j j
Repeat from step 2, first setting Enes = {e!() (6) 2j-1 = 0} and then, setting
13:
Enos = {e( 2j-1
(i)
2.1 TRAINING THE EXPERIMENT PLANNER
To learn about causal processes through interaction, the agent must produce a sequence of actions 40:7-1 that we call experimental behavior, which, when applied to environment e(t) E E, produces a sequence of observations (state) s() = 100
(i) Oj ..of)) which is then used to infer the value of the
embedding for a single causal factor (())"
We motivate this using model selection criterion. Normally in model selection applications. the observations are fixed and the goal is find a model M that closest to reality, as represented by:
M arg min ((M)) L(S
(4)
M
where L(-) is the description length. However, here, the situation is reversed. A simple bi-modal clustering model is fixed, motivated by Definition] [ Then, the agent motivated to produce actions that result in observations that are best explained by this model. These discovered action sequences are the experimental behaviors we desire.
Go.r = arg min ((M) + L(S)M))
(5)
00:T
where each observed trajectory = S(aor)) ) is function the action sequence. As mentioned carlier, the model is fixed in this formulation; hence, the first term L(M) is constant and not function of the actions. - (S M) that is fed back to the RL. agent as a reward function to maximize. We regard this reward function causal curiosity.
Note that since each causal factor has its own independent causal mechanism that causes S, the MDL of 5 will be higher if multiple causal factors cause S. On the contrary, if the agent produces actions which result in an S that is easily explained by a low-capacity bi-modal model M, then will imply that S is caused by fewer causal factors. Consequently, the causal curiosity reward for such an action sequence, L M). will be high. Therefore, causal curiosity favors experimental behaviors that result in observations caused by few causal factors thereby allowing us use S to infer a representation for single causal factor. For details, please refer Appendix A
2.2 CAUSAL INFERENCE MODULE
By maximizing the causal curiosity reward it possible to achieve behaviors which result in trajectories of states only caused by a single hidden parameter. However, wish to the outcome of performing these experimental behaviors each environment to infer a representation for the causal factor isolated by the experiment in question.
We achieve this through cluster membership. After training the Model Predictive Control Planner (Camacho & Albal (2033). sample from action sequence dp:r and apply it each of the
4

---

Under review as a conference paper at ICLR 2021
training environments. The learnt clustering model M is then used to infer representation for each environment using the collected outcome 8 obtained by applying do:r to each environment.
qar (2 ((()
This corresponds to Step 11 of Algorithm The representation learnt is binary in nature corre- sponding to the quantization the continuous spectrum of values a causal factor takes in the training set into high and low values. Note however that a binary quantized embedding is not necessary part of our method. A dense embedding may alternatively be learnt here similar to (Perez et al. (2020); Zintgraf et al. (2019) using approximate variational inference. However, performing interventions on a dense embedding (Section?) 2.3 increases the computational complexity exponentially. Balancing space and time complexity, we report results using the quantized binary form of Equation (6). We discuss the implications of increasing the complexity of 2) in the discussion.
2.3 INTERVENTIONS ON BELIEFS
Having learnt about the effects of a single causal factor of the environment we wish to learn such experimental behaviors for each of the remaining hidden parameters that may vary in an environment. To achieve this, in an ideal setting, the agent would require access to the generative mechanism the environments it encounters. Ideally, it would hold the values of the causal factor already learnt about constant i.e. do(h, = constant). and intervene over (vary the value of) another causal factor over a set of values K i.c. do(h, k) such that & E K. For example, if human scientist were to study the effects of a causal factor, say mass of a body, she would hold the values of all causal factors constant, (interact with cubes of the same size and external texture) and vary only mass to see how it affects the outcome of specific behaviors she applies to each body.
However, in the real-world the agent does not have access to the generative mechanism of the environments it encounters, but merely has the ability to act in them. Thus, it can intervene on the representations of causal factor of the environment i.e. do(z; = constan!). For, example having learnt about gravity, the agent picks all environments it believes have low gravity, and uses them to learn about separate causal factor say, friction.
This corresponds to Step 13 of Algorithm D. Thus, to learn about the jth causal factor, we repeat steps onwards on each of the clusters obtained for the j 1th
Enus = {e(() - = k), {0.1)
This process continues in the form of a tree (Figure 4 where for each cluster environments, a new experiment learns to split the cluster into 2 sub-clusters depending on the value of another hidden parameter. At level n, the agent produces 2" experiments and inference models, having already intervened the binary quantized representations of causal factors.
RELATED WORK
Doshi- Velez & Konidaris 2016) define class Markov Decision Processes where transition probabil- ities p(8++ St, ag; () depend on a hidden parameter 0, whose value is not observed, but its effects felt. Killian et al. (207)) and Yao et al. (2018) utilize these Hidden Parameter MDPs (Markov Decision Processes) to enable efficient policy transfer, assuming that transition probabilities across states a function of hidden parameters. |errez et al. (2020) relax this assumption. allowing both transition probabilities and reward functions to be functions of hidden parameters. Zintgraf et al. (2019) approach the problem from a Bayes-optimal policy standpoint, defining transition probabilities and reward functions to be dependent on a hidden parameter characteristic of the MDP in considera- tion. We utilize this setup define causal factors.
Substantial attempts have been made at unsupervised disentanglement, most notably, the 3-VAE Higgins et al. Burgess et al. (2018), where a combination of factored priors and the information bottleneck force disentangled representations. Kim & Mnih (2018) enforce explicit factorization of the prior without compromising on the mutual information between the data and latent variables, a shortcoming of the VAE. Chen et al. (2018) factor the KL divergence into more explicit form, highlighting an improved objective function and a classifier-ad classifier-agnostic disentanglement metric. Lo catello et al. (0088) show theoretically that unsupervised disentanglement (in the absence of inductive
5

---

Under review as a conference paper at ICLR 2021
biases) is impossible and highly unstable, susceptible to random seed values. They follow this up with with|Locatteloo et al. (2020) where they show, both theoretically and experimentally. that pair-wise images provide sufficient inductive bias to disentangle causal factors of variation. However, these works have been applied to supervised learning problems whereas we attempt to disentangle the effects of hidden variables in dynamical environments, relatively untouched question.
Curiosity for robotics is not a new area of research. (Schiichuber (2006). Ngo et al. (2012), Pathak al. (2017) describe curiosity as the motivation behind the behavior of an agent in an environment for which the outcome is unpredictable, i.e., an intrinsic reward that motivates the agent to explore the
unseen portions of the state space (and subsequent transitions). While causal curiosity is an intrinsic
reward, it differs from these traditional definitions of curiosity in that it motivates the agent to produce structure in the outcome of its behavior.
4 EXPERIMENTS
Our work has 2 main thrusts- - the discovered experimental behaviors and the representations ob- tained from the outcome of the behaviors in environments. The experimental behaviors tied to contributions and 2 in the Introduction. The causal factors allow us to achieve contribution 3 in the Introduction. We visualize these learnt behaviors and verify that they are indeed semantically meaningful and interpretable. We quantify the utility of the learned behaviors by using the behaviors as pre-training for downstream task. In our experimental setup, we verify that these behaviors are indeed invariant to all other causal factors except one.
We visualize the representations obtained using these behaviors and verify that they are indeed the binary quantized representations for each of the ground truth causal factors that manipulated in our experiments. Finally, we verify that the knowledge of the representation does indeed aid transfer learning and zero-shot generalizability in downstream tasks.
Causal World We use the Causal World Simulation (Ahmed et al. (Under submission 2020)) based on the Pybullet Physics engine to test our approach. The simulator consists of a 3-fingered robot, with 3 joints on each finger. We constrain each environment to consist of a single object that the agent can interact with. The causal factors that we manipulate for each of the objects are size, shape and mass of the blocks. The simulator allows us to capture and track the positions and velocities of each of the movable objects in an environment. While, for most experiments. the 3D position and 3D pose of the blocks is used as the state at each time step, we perform ablation studies where less information provided to the agent.
4.1 VISUALIZING DISCOVERED BEHAVIORS
We would like analyze whether the discovered experimental behaviors are human interpretable, i... are the experimental behaviors discovered in each of the setups semantically meaningful? We find that our agents learn perform several useful behaviors without any supervision. For instance, to differentiate between objects with varying mass, we find that they acquire a perfect crusp-and-liff behavior with an upward force. In other random seed experiments, the agents learn to lift the blocks by using the wall of the environment for support. To differentiate between cubes and spheres, the agent discovers a pushing behavior which gently rolls the spheres along a horizontal direction. Qualitatively, we find that these behaviors are stable and predictable. See videos of discovered behaviors here (website under construction).
Concurrent with the objective they are trained on, we find that the acquired behaviors impose structure on the outcome when applied to each of the training environments. The outcome of each experimental behavior on the set of training environments results in dividing it into 2 subsets. These subsets correspond to the binary quantized values of a single factor, e.g., large or small, while being invariant to the values of other causal factors of the environments. We also perform ablation studies where instead of providing the full state vector, we provide only one coordinate (c.g., only x, y or Z coordinate of the block). We find that causal curiosity results in behaviors that differentiate the environments based on outcomes along the direction provided. For example, when only the x coordinate was provided, the agent learned to evaluate mass by applying pushing behavior along the x direction. Similarly, a lifting behavior was obtained when only the z coordinate was supplied to the curiosity module (Figure

---

Under review as a conference paper at ICLR 2021
Z cvedd block
Z coard of
Duster
250
Time
Time
Principal Components
Figure 2: Examples of discovered behaviors. The agent discovers experimental behaviors that allow it to characterize each environmental object in binary manner, e.g., heavy/light, big small, rollable/not rollable, etc. These behaviors acquired without any external supervision by maximizing the causal curiosity reward. A, B, C correspond to self-discovered toss, lift-and-spin and roll behaviors respectively. shows ablation study, where the agent only provided the z coordinate of block every environment. Fach corresponds to one environment and the Z coordinate of the block is plotted with time when the discovered behavior is applied. It learns a lifting behavior, where cluster 1 represents the heavy blocks (z coordinate does not change much) and cluster represents the light blocks (z increases as block is lifted and then falls when dropped and subsequently increases again when bounces).
4.2 UTILITY OF LEARNED BEHAVIORS FOR DOWNSTREAM TASKS
While the behaviors acquired are semantically meaningful, would like to quantify their utility as pre-training for downstream tasks. We analyze the performance on Lift ffing where the agent must grasp and lift block to a predetermined height and Travel, where the agent must impart a velocity to the block along a predetermined direction. We re-train the learnt planner using an external reward for these tasks (Curious). We implement a baseline vanilla Cross Entropy Method optimized Model Predictive Control Planner (De Boer et al. (2005)) trained using the identical reward function and compare the rewards per trajectory during training. We also run a baseline (Additive reward) which explores whether the agent recieves both the causal curiosity reward and the external reward. We find high zero-shot generalizability and quicker convergence as compared to the vanilla CEM planner (Figure ??). We find that maximizing the curiosity reward in addition to simultaneously maximizing external rewards results in suboptimal performance due to our formulation of the curiosity reward. To maximize curiosity, the agent must discover behaviors that divide environments into 2 clusters. Thus in the context of the experimental setups, this corresponds to acquiring a lifting/pushing behavior that allows the agent lift/impart horizontal velocity to blocks in half of the environments, while not being able to do so in the remaining environments. However, the explicit external reward incentivizes the agent lift/impart horizontal velocity blocks in all environments. Thus these competing objectives result in sub-par performance.
4.3 VISUALIZATION OF IIERARIIICAL BINARY LATENT SPACE
Our agents discover a disentangled latent space such that they are able to isolate the sources of causation of the variability they encounters in their environments. For every environment, they learn a disentangled embedding vector which describes each of the causal factors.
To show this, we use separate experimental setups Mass, Si zeMass and ShapeSi se where each of the causal factors are allowed to vary over a range of discrete values. During Mass, the agent is allowed access to environments with objects having the same shape (cuboids) and size but differing only in mass. During S ZiMass, the agent has access to 30 environments with cuboids having sizes and masses ranging over 6 and values respectively. Finally, during ShapeSi zeMa
7

---

Under review as a conference paper at ICLR 2021
Rewnrds
Worst Case Rewards
Discovered Expertments Pre training
Method
Time Steps train-
Vanilla CEM
ing
Vanilla CEM Plamerr
15x10* 122.24 6x10
Additive
Curlous (ours)
Training epochs for CEM Planner
Triining epochs for CRM Planner
Figure 3: Utility of discovered behaviors. We find that the behaviors discovered by the agents while optimizing causal curiosity show high zero-shot generalizability and converge to the same performance conventional planners for downstream tasks. We also analyze the worst case performance and find that the pre-training ensures better performance
random initialization. average to acquire a skill with the time steps required to learn similar behavior using external reward. We find that the unsupervised experimental behaviors are approximately 2.5 times more sample efficient. We also find that maxizing both curiosity and external reward in our experimental setups results in sub-optimal results.
table compares the time-steps of training required on an
Priscipal Compoteees of Trijectocies
Unnolll-able
(Sphenc)
Unctipp able lHtgh Mass)
Flip--ble
Lurge enorghh puah (Large Sinc)
Ton enall so push (Small Size)
Figure 4: Discovered hierarchical latent space. The agent learns experiments that differentiate the full set of blocks in ShapeSiza zeMass into hierarchical binary clusters. At each level, the environments are divided into clusters the basis of the value of single causal factor. We also show the principal components of the trajectories in top left. For brevity, full of extent the tree not depicted here. For each level hierarchy k, there are 2 number clusters.
the agent has access to 60 environments with objects having shapes, sizes and masses ranging over 6, 5, and values respectively.
During training. the agent discovers hierarchical binary latent space (Figure 4), where each level of hierarchy corresponds to a single causal factor. The binary values at each level of hierarchy correspond to the high/low values of the causal factor in question. To knowledge, the first interpretable latent space describing the various causal processes in the environment of an agent. This implies that it learns quantify each physical attribute of the blocks it encounters in completely unsupervised manner.
obtain
4.4 KNOWLEDGE OF CAUSAL FACTORS AIDS TRANSFER
Next, we test whether knowledge of the causal factors does indeed aid transfer and zero-shot general- izability. To this end, supply the representations obtained by the agent during the experimental behavior phase as input to a policy network in addition to the state the simulator, and train it for a alace-and-orient downstream task (Figure [.. We define 2 experimental setups Trans ferMass and Transfers FersizeMass. In Mass, the agent is given access to 10 environments, with 10 varying values of mass. In TransferSi zeMass, the agent is allowed access 10 environments, with
8

---

Under review as a conference paper at ICLR 2021
and 5 values of size and mass respectively. In both setups, the agent learns about the varying causal mechanisms by optimizing causal curiosity. Subsequently, using the causal representation along with the state for each environment, it is trained to maximize external reward. For details of the setup, please see Appendix B
After training, the agents are exposed set of unseen test environments, where we analyze their zero-shot generalizability. These test environments consist of unseen masses and sizes and their unseen combinations. This corresponds to "Strong Generalization" as defined by Perez et al. (2020). We report results averaged over 10 random seeds.
For each setup, we train PPO-optimized Actor-Critic Policy (refemed to as Causally-curious agent) with access to the causal representations and a 56 dimensional state vector from the environment i... a₂ [-||s;, 20:k) (thus, total of 57 dimensional input for TransferMar dimensional for weennsffriiiieeee
and a 58 Similar to [Perez. et al. (220)) we implement 2 baselines the Generalist and the Specialist. The Specialist consists an agent with identical architecture as Causally Causaoly-curious curious agent, but without access to causal representations (i.e., receives 56 dimensional state vector). It is initialized randomly and trained only on the test environments, serving as a benchmark for complexity of the test tasks. It performs poorly, indicating that the test tasks are complex. The architecture of the Generalist is identical to the Specialist. Like the Specialist, the Generalist also does not have access to the causal representations, but is trained on the same set of training environments that the Causally-curious agent is trained on. The poor performance of the generalist indicates that the tasks distribution of training and test tasks differs significantly and that memorization of behaviors does not yield good transfer. We find that causally-curious agents significantly outperform the both baselines indicating that indeed, knowledge of the causal representation does aid zero-shot generalizability.
Place and Orient Mass variies
Place-and-Oriemt: : Misa and Size Vary
Extra time steps ueedeed baseline match performance
Rowards
Rewards
Extra steps training
Generalist Mass
4.8 10 10.75x10
Generalist Mass and Size
9.8x10 40 10.9Xx00
x10 Epiaodes
X1U Egisodes
Figure 5: Knowledge of causal factors aids transfer. We find that knowledge of the causal representation allows agents generalize unseen environments with high sero-shot performance. The table depicts the extra timesteps required by the Generalist in each experimental setup to match the zero-shot performance of causally-curious agent. We find that as the number of varying causal factors increase, the difference in zero-shot performance of the Causally-curious agent and the Generalist increases, showing that the CC agents are indeed robust multiple varying causal factors.
CONCLUSION
We introduce causal curiosity, an intrinsic reward that allows agents to discover binary quantized representations for the causal factors that affect environments an RL agent may encounter. We show that optimizing causal curiosity rewards results in the agent performing self-supervised experiments. We find that these experiments happen to be semantically meaningful and can be used as pre-training for downstream tasks. While our work learns binary quantized causal representations, a dense encoding may improve the amount of encoded information about the causal mechanisms of the environments. We leave this to future work.
REFERENCES
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wüthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. Under submission 2020.

---

Under review as a conference paper at ICLR 2021
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798-184) 2013.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in B-vae. arxiv 2018. arXiv preprini arXiv: 1804.03599 2018.
Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer Science & Business Media, 2013.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. arXiv preprint /tdtv:7703.011 2017.
Veronica Czitrom. One-factor-at-a-tiee versus designed experiments. The American Statistician, 53 (2):126-131, 1999.
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Finale Doshi-Velez and George Konidaris. Hidden parameter markov decision processes: A semi- parametric regression approach for discovering latent task parametrization. ICCAI: proceedings of the conference, volume 2016, pp. 1432. NIH Public Access, 2016.
Ronald Aylmer Fisher. Design of experiments. Br Med J, 1(3923)5554-555 1936.
Charles Robert Hicks. Fundamental concepts in the design of experiments. 1964.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://github com/ hill a/saassssa stable basel lines 2018.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. arXiv preprint arXiv:1905.100 2019.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale transfer learning with hidden parameter markov decision processes. In Advances in neural information processing systems, pp. 6250-6261, 2017.
Robust and efficient
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, arXiv preprint arXiv: 1802::8800.0983 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances neural information processing systems, pp. 3581-3589, 2014.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. arXiv preprint
2018.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. arXiv preprint arXiv:2002. 02886. 2020.
10

---

Under review as a conference paper at ICLR 2021
Hung Ngo. Matthew Luciw, Alexander Forster, and Juergen Schmidhuber. Learning skills from play: artificial curiosity on a katana robot arm. In The 2012 international joint conference on neural networks (IJCNN). pp. 1-8. IEEE, 2012.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036-4044.
PMLR, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 16-17, 2017.
Christian F Perez, Felipe Petroski Such, and Theofanis Karaletsos. Generalized hidden parameter mdps: Transferable model-basee in a handful of trials. AAAI Conference On Artifical Intelligence, 2020.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference. The MIT Press, 2017.
Peter Rousseeuw. Silhouettes: graphical aid the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
Jürgen Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection Science, 1832)1173-17, 2006.
B. Schölkopf. Artificial intelligence: Learning to see and act (News & Views). Nature, 518(7540):
486 487, 2015.
Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500 2019.
Jiayu Yao, Taylor Killian, George Konidaris, and Finale Doshi- Velez. Direct policy transfer via hidden parameter markov decision processes. In LLARLA Workshop. FAIM, volume 2018, 2018.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for adaatiide deep rl via meta- meta-learning. arXiv preprint arXiv:1910.08345 2019.

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS FOR EXPERIMENT DISCOVERY
A.1 PLANNER
The Experiment Planner consisted of uniform distribution planner for a horizon of 6 control signals. The planner was trained using the Cross Entropy Method Model Predictive Control (Camacho & Alba (2013); ; De Boer et al. (2005) on the true environment. We sampled 40 plans per iteration from the distribution initialized to uniform UI (control Low, control H igh). Each of the sampled plans are applied to each of the training environments and the top 10% of the plans are used to update the distribution.
Sel of civiiooomnntsss
Model M'
Simple model encodes Eg
Experiment Plaaner
O0
L(M')
Overllow L(O M' negalive reward
Figure 6: Overview of training. The experiment planner generates a trajectory of actions which is applied to each of the environments with varying causal factors namely mass, shape and size of blocks. For each environment, an observation trajectory or state s(0) E $ is obtained. A simple model with fixed low expressive power is used to approximate the generative model for S. The "information overflow" L(S) (S) 3 is returned as negative reward forcing 8 to be caused by few causal factors.
A.2 TRAINING ENVIRONMENTS
The training environments vary in each experiment. In Section +3 we utilize 3 setups, Mass, SizeMass and ShapeSizeMass, zeMass. For Mass, we allow the agent to access 5 environments with masses varying from 0.1 kg to 0.5 kg. In SizeMass, the agent has access to 30 environments with masses varying uniformly from 0.1 to 0.5 kg and sizes from 0.05 to 0.1 meters. Finally, in ShapeSi zeMass s, the agent has access to environments, with masses varying uniformly from 0. to 0.5 kg, sizes from 0.05 to 0.1 meters and shapes either being cubes or spheres. During experiment discovery, each environment, the agent has access to the position of the block the environment along with its quaternion orientation.
The total number of causal causal factors of each environment are rather large in number due to the fact that the simulator is complex realistic physies engine. Examples of the causal factors in the environment include gravity, friction coefficients between all on interacting surfaces, shapes, sizes and masses of blocks, control signal frequencies of the environment. However, only vary during Mass during Si zeMass and 3 during Shapes: zeMass.
A.3 CURIOSITY REWARD CALCULATION
We predetermine the minimum description length of the clustering model L(M) by assuming that the observations Op.r, obtained by applying experimental behavior do:r are produced by bi-modal generator distribution, where each mode corresponds either a low or high (quantized) value of causal factor. This also ensures that L(M) is as small possible. The planner, eq. (5) solves the following optimization problem:
arg max/min (d(ou:r, (brT) ourr E 0, E O'}- max(d(our, )br) o ourr 0}- 00:7 E.AT
max{d((bbr o(r)) ob. T E O'}]
the distance function d(.,-) in the space of trajectories is set to be Soft Dynamic Time Warping (Cuturi & Blondel (217)). ). The trajectory length T is 6 control steps long. The objective is a modified version of the Silhouette Score (Rousseeuw (187)).
12

---

Under review as a conference paper at ICLR 2021
Intuitively, Objective (8) expresses the ability of low complexity model, assumed to be bi-modal, to encode the state S op:r. If multiple causal factors control S, then the Minimum Description Length of L(S) will be high. Subsequently, since M is simple model, the deviation of S from M will be high i.e. L(S will be high resulting in a low value of the optimization objective. O and O' correspond to clusters of outcomes which quantize the values of causal factor isolated by dp:r- , 06r E S correspond to trajectories of states i.. observations obtained by applying T to environments with say, low values of a causal factor while do. ooT E O' correspond to trajectories of observations i.e. state obtained by applying Go:r to environments with say, high values of the same causal factor. Objective (8) attempts to ensure that these clusters are far apart from each other and are tight i.e. a simple model M encodes S well.
We further motivate how this formulation allows disentanglement of causal factors. A central assumption is that causal factors are independent, by definition, i.e. Independent Mechanisms Assumption Peters et (201)). Consider the outcome S obtained by applying an action sequence 40:7 to a set of environments. If the action sequence do:T results in multiple causal factors affecting the outcome S. the Kolmogorov complexity of S will be high. The reason for this is that each causal factor has its own independent causal mechanism (Peters et al., (2017); Parascandolo et al. (2018)) that affects S. Thus, given this independence, the information in S will be a sum of the information "injected" into it from the multiple causes. Conversely, if the outcome S obtained by applying an action sequence 0p:r has a lower Kolmogorov Complexity, then S is caused by fewer causal factors. Causal Curiosity attempts to reduce this complexity of S. by assuming simple generative model M is sufficient to encode S. Thus for experimental behaviors which allow several causes to affect S, the "overflow" of S from M will be high and subsequently the causal curiosity reward will be low. Thus, post- most-optimization of the objective, we arrive at an action sequence that allows for disentanglement of the causal factors.
B IMPLEMENTATION DETAILS FOR TRANSFER
In Section 4.4 we show the utility of learning causal representations in separate experimental setups. During Trans fe rMass, the agent has access to 10 environments during training, with masses ranging from 0.1 to 0.5 kg. At test time, the agent is required to perform the place-and-oriem task masses 2 masses 0.7 kg and 0.75 kg. During Trans ferSi i ZeMass, the agent has access to 10 environments during training, with sizes from either 0.01 or 0.05 and masses ranging from 0.1 to 0.5 kg. At test time the agent is asked to perform the task on 2 environments with masses 0.7 kg and 0.75 kg with sizes = 0.05 m.
We find that testing with large and light blocks increase the chances of accidental goal completions. Thus, during test-time, we use environments with high masses for out-of-distribution testing. The causal representation is concatenated to the state of the environment a contextual input and supplied to a PPO-Optimized Actor- Policy. The policy network consists of 2 hidden layers with 256 and 128 units respectively. The experiments are parallelized on 10 CPUs and implemented using stable baselines (Hill et al. (2018)).
The agent receives dense reward each time step during the maximizing external reward phase (Figure [], the negative of the distance of the block from the goal position scaled by factor of 1000. The control signal was repeated 10 times to the actuators of the motors on each finger.
C IMPLEMENTATION DETAILS FOR SECTION
In section 4.2 we study how the acquired experimental behaviors obtained through Causal Curiosity can be used as pre-training for a variety of downstream tasks. The Vanilla CEM depicts the cost of training an experiment planner from scratch maximize external dense reward where the agent minimizes the distance between the position of a block in an environment from the goal in the Lifting setup and imparts velocity the block along a particular direction in the Travel setup.
R(aoT) = = Σ dist block;)
The second baseline (Additive Reward) studies the setup when the agent receives both the curiosity signal and the external reward and attempts maximize both. The agent receives access all the
13

---

Under review as a conference paper at ICLR 2021
training environments with varying causal factors and must simultancously maximize both curiosity and the task reward. The equation below shows the reward maximized for the Li sifting ing task.
R(co:T)= = ΣΣ dist(goale - blocki)) +
(10)
[min(d(on:: ooT) ou.r E 00,6 E E0'-- max(d(oo:r: (0. oo.. Ourr E EO- max[d(dgrr d do.r, Øor E O'}l
The curious agent first acquired the experimental behavior by interacting with multiple environments with varying causal factors. The lifting skill was obtained during Mass, when the agent attempted to differentiate between multiple blocks of varying mass. The curious agent trained for 600,000 time steps on the curiosity reward. The acquired behavior was then applied to the downstream lifting task and fine tuned to external rewards. The Vanilla CEM baseline had an identical structure to that of the Curious agent, and received only external reward in Equation (9) The additive agent simultaneously optimized both external reward and the curiosity reward as in Equation (TO).
D INTUITION FOR DEFINITION OF CAUSAL FACTORS
We begin with simple example of a person walking on earth. This person experiences various physical processes while interacting in her world, for example gravity, friction, wind etc. These physical processes affect the outcome of interactions of the person with her environment. For example, while jumping on earth, the human experiences gravity which affects the outcome of her jump, the fact that she falls back to the ground. Additionally, these physical processes (or causal mechanisms) are parameterized by causal factors, for example, acceleration constant due to gravity g = 9.8m s² on earth, or coefficients of friction between her feet and the ground which assume particular numerical values.
These causal factors may vary across multiple environments. For example, the person may walk on sand or on ice, surfaces with varying frictional values. Thus the outcome of running on such surfaces will vary, running on sand will require significant effort, while running ice may result in the person slipping. Thus the coefficient of friction between the person's feet and the surface she walks affects the outcome of particular behavior in said environment. In our definition, h2 causal factors such friction with some particular coefficient of friction, or gravity with accelcration constant g other. H is the global set containing all such causal factors.
Now we ask the question (which we subsequently answer), given multiple environments, how would human characterize each of them depending on the value of a causal factor? Through experimental behaviors. The human in the above example would attempt to run in each of the environments she encountered, be it sand, characterize it as slippery. she didn't, she would characterize it as non-slippery. We attempt to equip our agent with similar logic. The "sequence of actions" (ao.T) described in our paper corresponds to the human running. The state 5(0) in the environment e(t) consisting of the sequence of observations (oorT) corresponds to the outcome of running. S might belong to either of the clusters of outcomes S or S' corresponding slipping or not slipping.
ice, in mud etc. If she slipped in an environment, she would
E SCALABILITY LIMITATION
We utilize the extremely popular One-Factor-at-a-time (OFAT) general paradigm of scientific in- vestigation, as an inspiration for our method. In the case of many hundreds of causal factors, the complexity of this method will scale exponentially. However, we believe that this would indeed be the case given a human experimenter attempting to discover the causation in any system she is studying. Learning about causation is a computationally expensive affair. We point the reader towards wealth of material on the design of scientific experiments and more specifically the lack of scalability of OFAT FFisher [936); Hicks (1964); Czitrom (1999). Nevertheless, OFAT remains the de facto standard for scientific investigation.
14

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Inference Loop
Input: Unseen
Test Environment env, trained Planner and Causal Inference Module
2: Initialize causal causalRep= = []]
3: Initialize training environment set Envs
4: for k in range(K) do
5: Reset env
6: Sample experimental behavior do:r CEM(-1 causal
7: Apply ag:r to env
Exploration Phase
8: Collect S = 0p:r
9: Use learnt qar (=|S) for cluster assignment i.e. 2₁ = qm (2|8, causal
Append zạ causalRep
>> Causal Inference Module
Learn a policy conditioned on causal factors az x-110; =0:K) to maximize external reward.
15


Reviews:
[{"review_id": "WaAZgwcL8tO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### **Summary**\n\nThis paper develops an intrinsic reward to help identify factors of variation within a family of MDPs. This intrinsic reward takes a form of curiosity and is used to develop initial behaviors to identify the causes of the latent variation in the environment dynamics. The experiments are used to validate the proposed intrinsic reward across several analyses used to identify its utility and effectiveness.\n\n#### **Assessment**\n\nI found the conceptual basis and framing of the paper to be very strong. The authors have highlighted shortcomings in prior methods that learn contextual parameters over the variation in the observed dynamics of environments and proposed an improvement in a simple combinatorial setting. I do however have strong reservations about how the paper is presented and as such cannot recommend it for publication as currently written. There is very little technical content that describe the methods and how they are used to learn effective behaviors. I also do not feel that the paper properly outlines the constraining assumptions and limitations of the proposed approach (see \"Weaknesses\" for more detail). The authors make claims in the abstract (e.g. \"learnt... with approximately 2.5 times less data...\") that aren't addressed or discussed in the main paper. In some ways this paper, as currently written, doesn't feel complete.\n\n#### **Strengths**\n- The paper appropriately criticizes prior approaches for not learning a disentangled embedding over the contextual parameters of variation within an environment. While a counterargument could be made that these approaches are looking for a low-dimensional, high information content parameterization of the factors of variation, the authors correctly point out that this limits the behaviors that could be used to efficiently identify what the causal factors are. \n- The proposed solution to this shortcoming is a simple enumeration of factors of variation used to cluster trajectories with divergent behavior along the specified factor.\n- This clustering of the trajectories by action sequences is used to define an intrinsic reward that encourages a clear separation between observed behaviors.\n- The experiments identify salient factors of variation, allowing the learned policies to leverage the context in meaningful ways when operating in a variety of settings.\n\n#### **Weaknesses**\nI will highlight various weaknesses that I found while reading this paper below. Before I get to these things, I wanted to first discuss the major weakness of the paper that I touched on in my assessment of the paper. As currently written, there is surprisingly little technical information about how each of the components of Figure 1 are implemented. The most concerning omissions are about how the intrinsic reward is computed and how the interventions are chosen/estimated. While there are some limited details provided in the paragraphs following Equation 3, it is unclear how the clusters $S$, $S'$, $S''$ and $S'''$ are determined and how they are related. There is also no justification for the construction of Equation 3. Why 3 clustering terms? What would happen if you compared $S''$ and $S'''$? Is there any semantic meaning behind these set clusters? What is the significance of maximizing the inter-cluster distance while minimizing the intra-cluster distance? How does the objective 3 assist in computing the factors of variation $z_{(j)}^{(i)}$? What range of values are the $z_{(j)}^{(i)}$ constrained to be? How is the embedding vector provided to the policy network? Is it used to contextualize the observed state through concatenation or is it used to condition the intermediate feature representations internal to the network? What does the training and inference procedure look like? Is there psuedocode that can be incorporated into the main body of the text? The omission of these various technical details has severely weakened what appears to be a very good paper with informative results.\n\nA second major weakness of this paper is in how the causal factors are represented. The construction of the vector $H$ assumes some *a priori* knowledge about the known causal factors of variation. While these may be possible to enumerate and abstract in simple simulated experiments such as the robotics setting demonstrated in this paper, this will not scale to larger dimension scenarios where the number of known and unknown factors of variation will potentially be very large. Without any understanding of how the individual $z_{(j)}^{(i)}$ are computed, it appears (based on the initial paragraphs of Section 2.1) that there needs to be several independent experiments + interventions for each of the K elements in the vector $z^{(i)}$. This is a significant limitation of the proposed approach in comparison with the BAMDP and HiP-MDP approaches that attempt to generalize over a dense continuous parameterization of the factors of variation. As proposed in Perez, et al [AAAI; 2020] this continuous parameterization can be factored to account for different forms of variation while also inferring a dense embedding to contextualize the dynamics and reward functions. It could be suggested that the core contribution of this paper under review is a more concrete or principled form of disentangling the causal factors of variation, but without a more formal definition it is difficult to fully ascribe where the true contributions of this paper lie. The proposed future work of determining a dense embedding seems to be overlapping with the cited work of Locatello, et al as well as the BAMDP and HiP-MDP approaches named here. Further clarification about the assumptions and limitations of the proposed approach would greatly improve the paper. As currently written, there is very little technical context by which to evaluate the core ideas and methods proposed. From a high-level view, this paper has proposed a small contribution in a highly specific sub-category of the more generalized methods is aims to improve on.\n\n*Minor weaknesses:*\n- It's not clear that the interventions will always be maximally informative. In general, what happens if more than one causal factor of variation are tied to the action. Say, for example mass and texture (eg. heavy and smooth) may jointly make it difficult for an agent to lift the object because of the confounding factor of not being able to get a secure hold on the object? \n- The choice of horizon for the experiment planner (third sentence following Equation 3) seems oddly specific for the experimental environment and also seems out of place at that juncture of the paper.\n- How are the \"2 main thrusts\" listed at the beginning of Section 4 directly linked to the proposed contributions listed at the end of Section 1?\n- The statement made at the end of the first paragraph of Section 4 is possibly only true in the isolated environment that was used for the experiments. Being more forthcoming with this context and constraint is most accurate and intellectually honest.\n- There are no specific details provided for the tasks used in the experiments. What are the range of shapes and sizes used? How similar are the various tasks? When evaluating for strong generalization, how far OOD are the test tasks? \n- Results are said to be provided over 10 random seeds but there are no error bars or forms of variance provided in the figures.\n- The only comparisons used are ablations of the proposed approach. Why not use direct baselines with the BAMDP or HiP-MDP methods? Without such comparisons, it is difficult to believe the stated conclusions that the proposed approach has fully made a contribution. \n- The tables included in figures 3 and 5 are not explained nor are they very clear. What are the major takeaways that one gets from these?\n- It would be nice to see where within the PCA over the trajectories, the hierarchies break down within the separate clusters. \n\n#### **Additional Comments**\nI want to close my review by commending the authors. This is good work and presents an interesting initial improvement on prior methods for generalization among related MDPs. While much of the writing and analysis of the proposed approach is qualitative in nature I feel that, if properly presented, it could be a good paper that would be worthy of acceptance to this conference. There are however significant revisions needed in order for the paper to be ready for publication. The core concepts are good and the framing is correct. But the significance and applicability of the method is hard to place due to the lack of technical detail as well as the omission of any discussion about the core assumptions and limitations of the proposed approach. Having said this, I would be inclined to raise my score if the authors were able to sufficiently address the two major weaknesses that I pointed out above and provide adequate clarification over areas of the paper that I may have misunderstood and not fairly evaluated. \n\n- - -\n\n### After rebuttal and discussion period\n\nI want to state once again that it was a pleasure to iterate on this paper throughout the rebuttal period. It was gratifying to see it improve significantly. The reviewers all agree that this is a very valuable research direction and the authors have begun an extremely interesting line of inquiry. However, there is still a good deal left to be completed prior to this paper being fully suitable for publication. There are some critical areas of improvement still needed in improving the overall clarity of the proposed approach.\n\nOne piece that stood out to me when re-reading the final submission and the other discussions with the reviewers, it's left unstated with any formal language how the causal interventions are chosen. My concern along these lines are a dressing up of monte-carlo approaches of varying the separate causal factors in more sophistication than is actually present in order to make the paper seem more concrete. Not saying that this is how I read the paper, but there is some space to wonder and be concerned by this. Along these lines, the distributions $q(z|S)$ are never concretely formalized nor is the inference process fully detailed surrounding how one may infer the $z_j^{(i)}$. I could maybe guess that the $z_j^{(i)}$ could perhaps be the cluster assignment but this shouldn't be something left to speculation and guessing... A simple statement of how these distributions are parametrized or even approximated would go a long way.\n\nAs a final note, I would suggest that the authors revise their conclusion and discussion sections to incorporate the limitations inserted into the Appendix. This follows from the discussion I had with the authors about the general scoping and applicability of the proposed causal curiosity mechanism. It is implicit that the authors have robotics applications in mind yet have written in a more general purpose manner. I believe that adjusting their focus and naming the robotics-directed focus explicitly will help immensely.\n\nIn the end, I unfortunately cannot recommend this paper for publication in its current \"final\" form for this conference. I do however really look forward to the complete and fully published form after another series of revisions and refinements by the authors. Best wishes.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V2yUJWES51_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis paper describes the definition of an intrinsic reward designed to encourage an agent to behave in such a way that they can distinguish between MDPs that have different values of some hidden parameter that affects the way the MDP behaves. Proof-of-concept experiments are included to allow discussion of how providing this classification of MDPs to an agent that needs to learn how to perform some externally-designed task allows it to learn to complete the task faster and discussion of how the agent's observations affect its learning and behaviour.\n\n**Strengths and Weaknesses**\n\nThe primary problem with this paper is that the description of the proposed method is too imprecise to allow me to review the quality of the research completed. I feel that the language and structure are not at an acceptable standard for ICLR. Some acute problems are that the protocol for training as well as some key algorithmic choices are never specified (i.e., the clustering algorithm and the planning algorithm for selecting action trajectories) and the notation and language around causal factors are not clear. \n\nWhen I say that the protocol for training is not specified, I am referring to many missing elements, like how the training dataset is composed and what information the agent is given about each environment it is placed in (e.g., is the agent able to distinguish one environment from another?) The explanation of how the clustering tree is formed lacks substance: Do the human experimenters change the composition of the training data once one causal factor has been learned sufficiently well (and how?), or is the agent able to distinguish which node of the tree a given environment belongs to? For example, what does it mean to \"perform an intervention on the embedding of the causal factor isolated by the initial optimization of causal curiosity\"? This is a very important aspect of the project, so it is critical to explain in plain language what you did.\n\nFor the sake of reproducibility and understanding, it is necessary to know what algorithm was used for clustering and what algorithm was used to adapt the agent's behaviour to increase the amount of intrinsic reward received.\n\nOne strength of the paper is how the authors have included intuitive examples throughout to help explain the intentions behind their design (e.g., \"For example, if a body in an environment loses contact with the ground …\"), though in some cases the language applied to the examples makes it unclear how they should be understood. For example, when you say \"examples of the parameter $H$ include gravity, coefficients of friction…,\" it sounds like each $H$ could be any one of those examples, but from Definition 1, I suspect $H$ must be a set whose elements could include all or some of the examples. This should be clarified.\n\n**Recommendation**\n\nI am recommending that this paper be rejected because it is not written with sufficient clarity to understand the research project being reported.\n\n**Specific Examples of Lack of Clarity**\n\nDefinition 1 seems critical to the paper, and as such, I think it needs to be written very carefully and clearly. The relationship between H and its associated sequences of actions is very unclear to me. Does each $h_i$ require a different sequence of actions from the sequence associated with $h_j \\neq h_i$? (The definition doesn't seem to have that requirement.) An explanation of what it means for a sequence of actions to cluster state trajectories also seems to be needed. Later in the paper, Definition 1 is described using the phrase \"when intervened on over a set of values\" and I'm really not sure what this means practically.\n\nAs a reader, I need help with understanding your choice of notation. For example, I intuitively suspect that you as authors consider $H$ (pp. 1-2) and $H^{(i)}$ (p. 3) to be related by virtue of their very similar notation, but I don't know how I am supposed to connect them in my head, other than I know that both are made up of causal factors but one is a set and the other is a vector.\n\nThe connection to Rousseeuw's (1987) work is confusing. Is the reward to the agent actually given by Equation (3), which appears to be a variation of silhouette width as defined by Rousseeuw? The presentation makes it sound like Equation (3) and Silhouette Score are essentially unrelated. \n\nI suggest rephrasing the introduction of notation on page 1 to reduce ambiguity. I think that $z$ is a latent representation of the environment (my guess based on notation conventions), but I initially assumed that $z$ symbolically represented an environment based on the text.\n\nIt is unclear whether you are using the term \"causal factor\" in the same way as you have noted that previous studies do (p. 1), or whether it has some other meaning in this paper.\n\nBased on notation, I think that a hyper-parameter $H \\in \\mathcal H$ (also referred to as simply a parameter) is equivalent to a hidden parameter (or causal factor) in the noted literature. This use of the term hyper-parameter is confusing and if I understand correctly, sticking with the term hidden parameter would be clearer (unless there is a reason I am missing for using hyper-parameter). \n\n**Additional Feedback (Here to help, not necessarily part of decision assessment)**\n\nIt was unclear what the set of trajectories during a rollout would look like. In other papers, trajectory sometimes refers to any observed sequence of state-action-rewards, while in others it might refer specifically to such a sequence that begins and ends at the same time as an episode. In definition 1, the phrase \"Let T be the length of the trajetcories during each rollout and $s_{0:T} \\in S^T$ denotes a trajectory\" (p. 2) makes me think there might be multiple trajectories per rollout, so I would like to be clear on how each is defined.\n\nShould $r_{0:T}$ be defined explicitly on page 1 too?\nI encourage you to be careful with your language when you define mathematical objects to share the name of a naive concept like causal factor. The way the definition is presented already suggests that this definition doesn't capture everything you would like it to capture because of simplifying to binary clusters. While tying it to the naive concept can help provide some intuition about your intentions, I could see this name causing some confusion when this idea develops further.\n\n\"trajetcories\" (p. 2)\n\n\"simplicitly\" (p. 2)\n\nI was surprised by the curly braces used for the observations (p. 3), which I think are ordered sequences rather than sets? (Typo?) \n\nThere seems to be a typo in Equation (3): I think there should be \"max\" on the second and third terms.\n\nIf you need more space to provide a summary of the algorithm followed by your system, I don't think that the connection to model selection on page 4 is clearly necessary to the paper.\n\nYou'll probably want to go through the bibliography to check for capitalization errors like \"Soft-dtw,\" \"beta-vae,\" etc.  \n\n---------------------------------------------------------------\n\n**UPDATE after author rebuttal**\n\nThe authors have made substantial improvements to the paper over the rebuttal period, but there is still work to be done to make this paper communicate as clearly as it should for publication. I share the concerns of Reviewer 1 and feel that this paper would benefit from more thought into how to present the ideas and another round of reviews.\n\n**Additional Feedback for future revision**\n\nThe preamble to Section 2 is now quite helpful, but it would be even more helpful if placed prior to Definition 1! Helping the reader to think about the elements of $H$ as random variables and to think of causal factors as determining which environment you end up in would aid understanding of Definition 1.\n\nDepending on who you want your audience to be, I suggest considering adding an explanation of the \"do\" notation. While the sub-community focused on causality may be aware of the notation, the community interested in intrinsic motivation more broadly would probably be interested in this paper but not know the notation.\n\nI think that the following sentence could be quite helpful if written in a different way: \" In our definition, $h_j$ are causal factors such friction with some particular coefficient of friction, or gravity with acceleration constant $g$ or other\" (p. 14). It would be helpful to have the variable (e.g. amount of friction) separated from its value (which might represent the coefficient of friction). I'd like the concept that causal factors are variables that could take on any of a set of values made more explicit. \n\n\"the outcome of running\" (p. 14) is slightly ambiguous (at first I read it that running was the outcome, rather than the experiment. A phrase like \"the outcome of the attempt to run\" or the outcome of the running experiment\" might be clearer.\n\nConsider using the singular they in your human example (p. 14). A decent primer: https://apastyle.apa.org/style-grammar-guidelines/grammar/singular-they\n\nAdditional typos found:\nLet $o_{0:T} \\in \\mathcal O^T$ denotes → denote (p. 2).\n$k$ is both the length of the set $H$ and the comparator with $h_j$ (p. 2) Maybe don't use $k$ and $K$ either, since it still gives me the vague sense that they might be related, but I'm pretty sure they're not. \nNeither CEM nor MDL is written out in full (p. 4 is first use).\n\"as compared to the vanilla CEM planner (Figure??).\" (p. 7)\n\"are causal factors such friction\" → such as friction (p. 14)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ER4_Qe8TI4g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors introduce *causal curiosity*, an intrinsic reward that allows an agent to discover causal factors in an environment. The authors run experiment on `Causal World Simulation`  , a 3-fingered robot manipulating a single 3D object. These experiment show that the system:\n* Learns, in a self-supervised way, interpretable behaviors corresponding to \"experiments\"\n* Helps with two downstream tasks (`lifting` and `travel`) \n* Learns an interpretable latent space that hierarchically partitions the environment in a way that isolate its causal processes\n* Generalizes over unseen variations in the environment\n\n## Analysis\n\nI struggled to understand Definition 1. \"The set $H = \\\\{h_1, h_2, . . . , h_k\\\\}$ is called a set of $\\epsilon$−causal factors if for every $h_i \\\\in H$, there exists a sequence of actions that clusters the state trajectories into two sets $S$ and $S_0$ such that...\" I can't figure out what the $h_i$ are and how \"a sequence of actions\" could exist for each of them.\n\nI found interesting that the system self-discovered toss, lift-and-spin and roll behaviors, although I was not able to understand whether these behaviors allowed the agent to disentangle a specific aspect of the environment (mass, shape, size). For example, was lift-and-sping used to disentangle the mass? A better/clearer analysis of this would be helpful.\n\nIn Section 4.2, the two proposed downstream task (`lifting` and `travel`) seem very close to the behaviors learned during the self-supervised phase. As such, I'm not surprised about the fact that you're able to learn them really quickly. However, I do not find this very convincing. I'd be more impressed if the tasks were significantly different that the ones the system discovered in a self-supervised way.\n\nIn section 4.3, I struggle to understand how the hierarchical binary latent space shown in Figure 4 is discovered. Are you training first on `Mass`, then the result on `SizeMass` and then on `ShapeSizeMass`? Is Figure 4 only reporting the results after `ShapeSizeMass`? How do you build these hierarchical binary clusters? Figure 4 shows a 2D PCA with clear clusters, are you using a clustering algorithm on the 2D PCA projection of the trajectories? Did you arrive to this hierarchy through manual analysis?\n\nOverall, this strikes me as an interesting direction to pursue, but the experiments have not fully convinced me that the proposed approach of \"causal curiosity\" (ie. finding a sequence of actions that maximizes the inter-cluster distance between trajectories and minimizes the intra-cluster distance) is the right approach. I believe experiments on environments that are not so clearly binary and hierarchical (shape / size / mass) would be needed.\n\nThat being said, I believe the paper is interesting enough to be accepted, especially if reviewers with more background on the topic judge it valuable.\n\n## Typos\n\n* I believe you are missing some $\\\\max\\\\{...\\\\}$ in equation 3.\n* Note that *the* since each causal factor\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8zElVBNSv8z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of skill discovery in settings where the data appears to be a Markov Decision Process and part of the state is unobservable. The hidden state variables are interpreted as causal factors that control important aspects of the environment  dynamics. Under this interpretation, the paper advocates for the use of a reward that encourages learned skills that exercise individual components of the hidden state. These skills are learned with a model-based RL algorithm -- one skill per causal factor -- then transferred for use in a downstream control problem that uses a different learning algorithm. The paper claims the learned skills are qualitatively meaningful, and that they enable agents to solve downstream problems without any additional training. Data used for empirical evidence comes from a simulated manipulation robot. \n\nThe paper presented several interesting ideas with the potential for high impact in robotics and reinforcement learning. I thought the paper was generally well written and the ideas well-positioned with respect to related work. I have concerns about the empirical evaluation, which I detail in several followup questions: \n\n1. Can you explain the difference(s) in how states and observations are treated in this paper?\n2. Can you verify that the structured behaviors shown in your videos are a result of the MDL reward and not the changes imposed to the observation?   \n3. Are the same behaviors realizable using the MDL reward as an additive bonus on the downstream problem?\n4. How do the hidden causal factors numerically differ in each video? \n5. How does the MDL skill differ from a skill trained from an external reward that explicitly encodes the same behavior?\n6. Can you explain the notation in Equation 3? What does the minus sign denote?\n7. Why is the causality formalism useful and necessary? \n8. Does the MDL reward structure lead to better skills than alternatives? Examples include:  a negative linear threshold of the hidden state vector, a reward of -1 for each non-zero hidden factor, the negative magnitude of the hidden factor vector, and possible competing methods found in the literature.\n9. What is the sample cost of learning an individual skill, and how does it compare to training on the downstream problem from scratch?\n10. Why are the best rewards negative and the worst positive in Figure 3?\n11. Over how many trials was data collected for the results in figures 3 and 5?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses how one can equip reinforcement learning agents with an intrinsic reward function that helps identifying factors of variation within a family of MDPs, effectively allowing agents to do experiments in the environment. This is interpreted as causal factors that control important aspects of the environment dynamics.

Although this is a very relevant topic and there was extensive discussion during the discussion phase, with reviewers acknowledging that the final version of the submitted manuscript substantially improved over the original submission, most reviewers still recommend the rejection of the paper. This is mainly due to the assessment that there are still several unclear technical aspects related to the paper. Shortly, the reviewers felt that the paper had important clarity issues, that the claims being made were imprecise, and that there was a dearth of details about the empirical results, making them not fully convincing.

I strongly recommend the authors to take the reviewers suggestions into consideration to have a much stronger submission to future venues.

Decision: Reject