CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
Authors: Ossama Ahmed
Frederik Träuble
Anirudh Goyal
Alexander Neitz
Manuel Wuthrich
Yoshua Bengio
Bernhard Schölkopf
Stefan Bauer
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
transfer learning
sim2real transfer
domain adaptation
causality
generalization
robotics

Abstract: Despite recent successes of reinforcement learning (RL), it remains a challenge for agents to transfer learned skills to related environments. To facilitate research addressing this problem, we proposeCausalWorld, a benchmark for causal structure and transfer learning in a robotic manipulation environment. The environment is a simulation of an open-source robotic platform, hence offering the possibility of sim-to-real transfer. Tasks consist of constructing 3D shapes from a set of blocks - inspired by how children learn to build complex structures. The key strength of CausalWorld is that it provides a combinatorial family of such tasks with common causal structure and underlying factors (including, e.g., robot and object masses, colors, sizes). The user (or the agent) may intervene on all causal variables, which allows  for  fine-grained  control  over  how  similar  different  tasks  (or  task  distributions) are. One can thus easily define training and evaluation distributions of a desired difficulty level,  targeting a specific form of generalization (e.g.,  only changes in appearance or object mass). Further, this common parametrization facilitates defining curricula by interpolating between an initial and a target task. While users may define their own task distributions, we present eight meaningful distributions as concrete benchmarks, ranging from simple to very challenging, all of which require long-horizon planning as well as precise low-level motor control. Finally, we provide baseline results for a subset of these tasks on distinct training curricula and corresponding evaluation protocols, verifying the feasibility of the tasks in this benchmark.

Published as a conference paper at ICLR 2021
CAUSALWORLD: A ROBOTIC MANIPULATION BENCHMARK FOR CAUSAL STRUCTURE AND TRANS- FER LEARNING
Ossama A Frederik Träuble, x2 Anirudh Goyal, Alexander Neitz, Yoshua Bengio," Bernhard Schölkopf," Stefan Bauer, 12 Manuel Wüthrich 12
ETH Zurich, Max Planck Institute for Intelligent Systems, 'Mila, University of Montreal
ABSTRACT
Despite recent successes of reinforcement learning (RL), it remains a challenge for agents to transfer learned skills to related environments. To facilitate research ad- dressing this problem, we propose CausalWorld, a benchmark for causal structure and transfer learning in a robotic manipulation environment. 'he environment is a simulation of am open-souree robotic platform, hence offering the possibility sim-to-real transfer. Tasks consist of constructing 3D shapes from set of blocks inspired by how children learn to build complex structures. The key strength Causa/World is that it provides combinatorial family of such tasks with common causal structure and underlying factors (including, e.g., robot and object masses, colors, sizes). The user (or the agent) may intervene all causal variables, which allows for fine-grained control over how similar different tasks (or task distri- butions) are. One can thus easily define training and evaluation distributions of a desired difficulty level, targeting specific form of generalization (c.g.. only changes appearance or object mass). Further, this common parametrization fa- cilitates defining curricula by interpolating between an initial and target task. While users may define their own task distributions, we present eight meaningful distributions as concrete benchmarks, ranging from simple to very challenging, all which require long-horizon planning as well as precise low-level motor control. Finally, provide baseline results for subset of these tasks on distinct training curricula and corresponding evaluation protocols, verifying the feasibility the tasks in this benchmark
INTRODUCTION
Benchmarks have played a crucial role in advancing entire research fields, for instance computer vision with the intro- duction CIFAR-10 and ImageNet (Krizhevsky et al.) 2009 2012). When it comes to the field of reinforcement learning (RL), similar breakthroughs have been achieved in domains such game playing (Mnih et al., 2013 Silver et al. 2017),
do(floor color='white', block size=0.065, ...etc)
learning motor control for high-dimensional simulated robots
(Akkaya al., 2019), multi-agent settings (Baker al. 2019
Berner et al.] 2019) and for studying transfer the context of meta-learning Yu al. 2019). Nevertheless, trained agents often fail to transfer the knowledge about the learned skills from training environment to different but related environ- ment sharing part of the underlying structure. This can be at- tributed the fact that it is quite common evaluate an agent on the training environments themselves, which leads to over- fitting on these narrowly defined environments (Whiteson et 2011). 1), or that algorithms are com-
Example of do- interventions on exposed variables in CausalWorld.
Figure 1:
"quual contribution. Correspondence to: EEqual Advising
<ossama.ahmed www.aa/meemmmmmmmlmmll..... > fredeeikk.treeule www.eaaeeeubbnneen....mm >

---

Published as a conference paper at ICLR 2021
(a) Pushing
(b) Picking
(c) Pick and Place
(d) Stacking2
(e) Stacked Blocks
(I) General
(g) CreativeStackedBlocss
(h) Towers
Figure 2: Example tasks from the task generators provided in the benchmark. The goal shape is visualized opaque red and the blocks in blue.
pared using highly engineered and biased reward functions which may result in learning suboptimal policies with respect to the desired behaviour; this is particularly evident in robotics.
In existing benchmarks Yu et al. 2019 Goyal et al. 2019a Cobbe et al.) 2018 Bellemare et al, 2013 James al. 2020) the amount of shared causal structure between the different environments is mostly unknown. For instance, in the Atari Arcade I Learning environments, it is unclear how to quantify the underlying similarities between different Atari games and we generally do not know to which degree an agent can be expected to generalize.
To overcome these limitations, ronment that we call CausalWorle ld. It features diverse set of environments that, in contrast to previous designs, share large set parameters and parts of the causal structure. Being able inter- vene on these parameters (individually collectively) permits the experimenter to evaluate agents' generalization abilities with respect different types and magnitudes of changes in the environment. These parameters can be varied gradually, which yields continuum of similar environments. This allows for fine-graines control of training and test distributions and the design of learning curricula.
introduce novel benchmark in robotic manipulation envi-
A remarkable skill that humans learn to master early on in their life is building complex struc- tures using spatial-reasone
and dexterous manipulation abilities (Casey et al.) 2008 Caldera al.) 1999 Kamii et al. 2004). Playing with toy blocks constitutes natural environment for children to develop important visual-spatial skills, helping them 'generalize" in building complex composition designs from presented imagined goal structures (Verdine al. 2017 Nath & Szües 2014 De- war 2018 Richardsonn et al. 2014). Inspired by this, Causa 1 Wor r l is designed to aid in learning the
and investigating these skills simulated robotic manipulation environment corresponding open-source TriFinger robot platform from Wüthrich et al. (2020), which can be built in the real world. Tasks are formulated as building 3D goal shapes using of available blocks by manipu- lating them as seen in Fig. [ This yields a diverse familiy of tasks, ranging from relatively simple (e.g. pushing a single object) to extremely hard (e.g. building complex structure from a large number of objects).
Causa World improves upon previous benchmarks by exposing a large set of parameters in the causal generative model of the environments, such as weight, shape and appearance of the building blocks and the robot itself. The possibility intervening on any of these propertics at any point in time allows one set up training curricula or to evaluate with respect to different parameters. Furthermore, contrast to previous benchmarks (Chevalier- Boisvert al. 2018; Cobbe al.) 2018), researchers may build their this simulator at low cost, as detailed in Wüthrich et al. (2020), and transfer their trained policies to the real world.
agent's generalization capability
real-world platform of
2

---

Published as a conference paper at ICLR 2021
Beschmark
do- interventions inteface
procedurally penented
online dis- triburion of talks
setup custtee ula
disentnagle general- intionn
reaL- world similarity
open- source
low- level control
long- term
unified
snczess.
metrie
ments
ability
ning
RLBennc
MetaWorid
x
X
Muloban
x
RebyAl
x
8
Comnun
AtariAradee
8
8
x
518
Rorld
Table I: Comparison of Causal World with RLBench (James et al. 2020), MetaWorld (Yu al,
2019
IKEA (Lee et al. 2019). BabyAl (Chevalier-Boisver et al. 20188) CoinRun (Cobbe et al.
2018
AtariArcade (Bellemare al. 2013). MuJoBan etc. (Mirza et 20220.
Finally, by releasing this benchmark hope facilitate research in causal structure learning, i.c. learning the causal graph (or certain aspects of it) as we operate in a complex real-world environment whose dynamics follow the laws of physics, which induce causal relations between the variables. Changes to the variables we expose can be considered do-interventions on the underlying structural causal model (SCM). Consequently, we believe that this benchmark offers an exciting opportunity to investigate causality and its connection to RL and robotics.
Our main contributions can be summarized as follows:
We propose CausalWorld, new benchmark comprising parametrized family of robotic manipulation environments for advancing out-of-distibbutin generalization and causal structure learning RL.
We provide a systematic way of defining curricula and disentangling generalization abilities of RL agents with respect to different changes in the environment, since allow for do- interventions to be performed on different environment variables (parameters and states) individually.
We establish baseline results for some of the available tasks under different learning algo- rithms, thus verifying the feasibility of the tasks.
We show how different learning curricula affect generalization across different axes by reporting some of the in-distribution and out-of-distribttion generalization capabilities of the trained agents.
CAUSALWORLD BENCHMARK
Here make the desiderata outlined in the introduction more precise:
The set of environments should be sufficiently diverse allow for the design of challenging transfer tasks.
2. We need be able to intervene different properties (e.g. masses, colors) individually, such that we can investigate different types of generalization.
3. It should be possible to convert any environment to any other environment by gradually changing its properties through interventions; this requirement is important for evaluating different levels of transfer and for defining curricula.
4. The environments should share some causal structure to allow algorithms to transfer the learned causal knowledge from one environment another.
5. There should be a unified measure of success, such that an objective comparison can be made between different learning algorithms.
6. The benchmark should make it easy for users to define meaningful distributions of envi- ronments for training and evaluation. In particular, should facilitate evaluation of in- distribution and out-of-distribution performance.
7. The simulated benchmark should have real-world counterpart allow for sim2real.
3

---

Published as a conference paper at ICLR 2021
In light of these desiderata, propose a setup in which a robot must build goal shapes using set of available objects. It is worth noting that similar setups were proposed previously in a less realistic setting, e.g. in (Janner et al. 2018 Bapst et al. McCathyy et al. AAkkaya et al.) 2019 Fahlman 1974 Winston 1970; Winograd| 1972). Specifically, task is formulated as follows: given a set of available objects, the agent needs build specific goal structure, see Fig. [ an example. The vast amount of possible target shapes and environment properties (e.g. mass, shape and appearance of objects and the robot itself) makes this a diverse and challenging setting to evaluate different is a simulated version (using the Bullet physics engine (Coumans et al. 2013)) of the open-source TriFinger robot platform from Wüthrich et al.] (2020). Each environment is defined by set of variables such, as gravity, floor friction, stage color, floor color, joint positions, various block parameters (e.g. size, color, mass, position, orientation), link colors, link masses and the goal shape. See Table 3 in the Appendix for a more extensive list of these variables.
generalization aspects.
BausalWorll
Desideratum 1 is satisfied since different environment properties and goal shapes give rise to very different tasks, ranging from relatively easy (e.g. re-positioning a single cube) to extremely hard (e.g. building complex structure). Desideratum 2 is satisfied because allow for arbitrary inter- ventions on these properties, hence users or agents may change parameters individually or jointly. Desideratum is satisfied because the parameters can be changed gradually. Desideratum 4 is sat- isfied because all the environments share the causal structure of the robot, and one may also use subsets of environments which share even more causal structure. We satisfy desideratum by defin- ing the measure of success for all environments as the volumetric overlap of the goal shape with available objects. Further, by splitting the set of parameters into a set A, intended for training and in-distribution evaluation, and a set B, intended for out-of -itt----dstribution evaluation, we satisfy desider- atum 6. Finally, since the TriFinger robot (Wüthrich et al. 2020) can be built in the real real-world, we satisfy desideratum 7. Desideratum 7 and are in partial conflict since sim2real is only possible for the tasks which are constrained to the variables on which the robot can physically act upon.
Task generators: To generate meaningful families of similar goal shapes, Causa Worr dd allows for defining task generators which can generate variety of different goal shapes in an environment. For instance, one task generator may generate pushing tasks, while another one may generate tower- building tasks (see Fig. 2). Fach task generator is initialized with default goal shape from its corresponding family and comes with a sampler to sample new goal shapes from the same family. Additionally, upon construction, one can specify the environments' initial state and initial goal shape structure when deviating from the default. The maximum episode time to build a given shape is number r roteerroflllll blockss 10 seconds. CausalWorld comes with eight pre- defined task generators (see Fig.
Three generators create goal shapes with a single block: Pushing with the goal shape on the floor, Picking having the goal shape defined above the floor and Pick and Place where a fixed obstacle is placed between the initial block and goal pose.
Stacking2 involves a goal shape of two stacked blocks, which can also be considered one instance of the Towers generator.
The remaining generators use variable number of blocks to generate much more complex and challenging target shapes, as detailed in the appendix: Towers, Stacked Blocks, Creative Stucked Blocky and General.
Given that building new environments using current physics simulators is often tedious, we provide a simple API for users who wish to create task generators for new challenging shape l'amilies, which may be added to Causa 1 WWorld' s task generators repository.
The robot can be chosen to operate in either joint position con- trol mode, joint torque control mode, end-effector position control mode, or the delta of each. In any of these cases, the action is 9-dimensional (one per joint). We provide two observation modes: structured and pixel. In the structured mode, the observation vector is constructed using a rule for the ordering of the relevant variables, such joint positions, joint velocities, block positions, etc. Thus, the size of the observation space depends on the number of blocks, which could potentially change with every new goal sampled, e.g. in Towers, (Creative) Stacked Blocks and General. In
Action and Observation Spaces:
4

---

Published as a conference paper at ICLR 2021
contrast, in the pixel mode, the agent receives six RGB images (hence the dimension of the observa- tion is 6 3 128 x 128). the first three images are rendered from the three cameras mounted around the TriFinger robot, and the last three images specify the goal image of the target shape rendered from the same cameras. Additionally, Causal World allows users to set up a fully customized observation space.
The reward function is defined the fractional volumetric overlap of the blocks with the goal shape, which ranges between 0 (no overlap) and 1 (complete overlap). Since this reward function is shared across all tasks, an agent that learned from some training tasks could in principle use it to solve unseen goal structures. There is also the possibility of modifying the reward function to 1) sparsify the reward further by returning a binary reward signal instead, or 2) add a dense reward function in order introduce inductive biases via domain knowledge and solution guidance. We hope that the considerable complexity and diversity of goal shapes motivate and accelerate the development of algorithms that not dependent hand-tuned reward functions.
Rewards:
Intervention Space Used
Curriculum
Evaluation Phase
Full Stare
1
Indix
Eplence Num
Intervantion Actor
Protocol
Time
Protocol
Index
Intervetton
Space
Evalutioon
Eraluation Space
Zzamiin Comizalar
Key components for generic training and evaluation of RL agents. Left: learning curriculum which is composed of various intervention actors that decide which variables to intervene on (for a valid intervention, values need to be the allowed training space (ATS)). Right: Evaluation protocols are shown which may intervene on variables at episode resets or within episodes (for valid intervention, values need to in the evaluation space (ES)). Middle: we represent the ATS and ES, where each intervention results in one point in the spaces. As shown ATS and ES may intersect, eg. if the protocols are meant to evaluate in-distribution generalization.
Figure 3:
Training and evaluation spaces: In this benchmark, a learning setting consists an allowed training space (ATS) and evaluation space (ES), both of which are subspaces of the full parameter space. During training, in the simplest setting, parameters are sampled iid from the ATS. However, unlike existing benchmarks, Causa aaaawloleee rl allows in addition for curricula within the ATS well as settings where the agent itself intervenes on the parameters within an episode (see Fig. 33. Sim- ilarly, during evaluation, parameters may be sampled iid from the evaluation space at each episode reset, or there can be interventions within an episode. Moreover, in order to retrieve the setting con- sidered in most RL benchmarks, we could set the ATS and the ES to be identical and intervene only on object and robot states (and keep other environment properties constant) at each episode reset. However, evaluate -iittii-iitiiitiion generalization, one should set the two spaces (ATS and ES) to be different; possibly even disjoint. Additionally, to evaluate robustness with respect to a specific parameter (e.g. object mass), one may define the training and evaluation spaces which only differ in that particular parameter. In order to facilitate the definition of appropriate training and evaluation settings, we pre-define two disjoint sets, A, and B., for each parameter i. Through this, one can for instance define the training space to be A x A2 and the evaluation space to be B, x B, ... to assess generalization with respect to all parameters simultaneously. Alternatively, the evaluation space could be defined as A₁ X A2 X ... B, X A1+1 X ... assess generalization with respect to parameter i only. Lastly, users may also define their own spaces which could then be integrated into the benchmark give rise new learning settings.
Intervention actors: To provide a convenient way of specifying learning curricula, we introduce intervention actors. At each time step, such an actor takes all the exposed variables of the environ- ment as inputs and may intervene on them. To encourage modularity, one may combine multiple actors in learning curriculum. This actor is defined by the episode number to start intervening, the episode number to stop intervening, the timestep within the episode it should intervene and the episode periodicity of interventions. We provide of predefined intervention actors, including
5

---

Published as a conference paper at ICLR 2021
actor which samples parameters randomly each episode reset, which corresponds domain- randomization. It is also easy to define custom intervention actors, we hope that this facilitates investigation into optimal learning curricula (see Fig.
Probing the Causal Structure in RL: The problem setting in RL is usually formulated using the language of Markov Decision Processes (MDPs) or Partially Observable Markov Decision Pro- cesses (POMDPs) (Sutton & Barto, 1999), but can be also represented by Structural Causal Models (SCMs), as shown in (Buesing et al., 2018), refer to section E in the Appendix for a detailed ex- planation. This is achieved by formulating all conditional probability distributions as deterministic functions that take independent noise variables as inputs. These independent noise variables can specify different scenarios while the deterministic functions reflect the causal mechanisms of the system (Schölkopf et al. 2021 Changes in the environment can stem from two different sources:
1. The agent may alter the state of the environment (e.g. the position of block) indirectly, through its actions (e.g. pushing the block by applying appropriate torques at the motors).
2. During the execution of a learning curriculum or an evaluation protocol, we may directly intervene on any variable of the SCM, including all the latent variables of the causal model that are not accessible the RI. agent (such as gravity, object mass or color).
(1) is the default type of admissible interventions RL benchmarks, whereas World allows for interventions of type (2) in addition. The idea is that interventions on these latent variables, e.g. during learning curriculum, will allow the agent to distinguish between spurious correlations that only present in particular setting and true causal relations that will hold across all settings (i.e. interventions). If the agent able to learn such a representation the underlying SCM structure, would expect it to perform well Dittadi et al. 202T) because the causal structure remains the same, even when the functional form of certain relations may vary (e.g. when transferring to the real rohot). Moreover, by having access to a broad range of interventions in CausalWorld it will aid the inference of the underlying SCM structure through the different causal discovery methods (see Figure for a subset of the expected SCM to be learned), which in turn addresses the lack of causal discovery benchmarks for real world challenges.
in out-of-distribution scenarios (Schölkopf et al. 2021
hope that
Finger torque
Finger position
Block postioon at time
...
Black puiiiian time tel
Figure 4: A subset of an SCM represented DAG for an environment in CausalWorld with one block on the floor. Here, we only show a subset of the causal variables affecting the block position time t+1.
RELATED WORK
Previous benchmarks for RL mostly focused on the single task learning setting such as OpenAl Gym and DM control suite (Tassa et al., 2018 Brockman et al., 2016). In contrast, recent line of work, e.g. Meta-World and RLBench (Yu et al. 2019: James et al.) 2020), aims at studying multi-task learning as well as meta-learning, Such benchmarks mostly provide non-parametrie, hand-designed task variations, is hence unclear how much structure is shared between them. For instance, it is not clear how different is to "open door" compared to "opening drawer". To address the ambiguity in the shared structure between the tasks, Caus a World was designed to allow interventions to be performed many environment variables giving rise large space of tasks with well-defined

---

Published as a conference paper at ICLR 2021
relations between them, which we believe is missing key component to address generalization in RL. A detailed comparison between CausalWorld and similar benchmarks is shown in Table !
Similar parametric formulations of different environments were used in the generalization-fot cor-itioooooo--eeee literature, which has played an important role in advancing the field (Packer et al. 20188) Rajeswaran
et al., 2017 Pinto al., 2017 Yu et al., 2017 Henderson al. 2017a| Dulac- Arnold et al.) 2020) Chevalier--oiiveer et al. 2018 In these previous works, variables were mostly changed randomly,
as opposed to the full control over the variables provided by CausalWorle
Another important open problem for the RL community is the standardization of the reported learn- ing curves and results. RL methods have been shown to be sensitive (Henderson et al. 2017b). Thus it is crucial to devise set of metrics that measure reliability of RL algorithms and ensure their reproducibility. [Can al. (2019) distinguishes between several evaluation modes like "evaluation during training" and "evaluation after learning". Osband et al. (2019) recently proposed benchmarking suite that disentangles the ability of an algorithm to deal with different types of challenges. Its main components are: enforcing specific methodology for an agent's s evaluation beyond the environment definition and isolating core capabilities with targeted 'unit tests' rather than integrating the general learning ability.
range of different factors
Moreover, causality has been historically studied from the perspective of probabilistic and causal reasoning (Pearl 2009), cognitive psychology (Griffiths & Tenenbaum] 2005), and more recently in the context of machine learning (Goyal et al.) 2019b] Schölkopt et al., 2021; Baradel et al. 2019; Bakhtin al. 2019). On the contrary, believe its link to robotics is not yet drawn systematically.
To bridge this gap, one of the main motivations of Causal World was to facilitate research in causal learning for roboties, such as observational discovery of causal effects in physical reality, counterfactual reasoning, and causal structure learning.
EXPERIMENTS
To illustrate the usage of this benchmark and to verify the feasibility of some basic tasks, we eval- uate current state-of-the-art model-free (MF-RL) algorithms on subset of the goal shape families described in Section [ and depicted in Fig. - (a) Pushing, (b) Picking, (c) Pick and Place, and (d)
Stacking2. These goal shapes reflect basic skills that required solve more complex construc- tion tasks.
Setup: The idea here is to investigate how well agent will perform on different evaluation distributions, depending on the curriculum has been trained with. We train each method under the following curricula:
Curriculum O: no environment changes; each episode is initialized from the default task lying in space A note that here the initial state never changes (i.e. no interventions).
Curriculum 1: goal shape randomization; the beginning of each episode new goal shape is sampled from space A (i.e. interventions on goal position and orientation).
Curriculum 2: full randomization w.r.t. the task variables every episode simultaneous intervention on all variables is sampled from space A (i.c. can be seen as equivalent to extreme domain randomization in one space).
The curriculum will, as expected, affect the generalization capabilities of the trained agents. With CausalWorld's formulation, these generalization capabilities can casily be disentangled and benchmarked quantitatively, as explained in Section 2] For each of the goal shape families (a. b, from Fig. 2 we train agents under the three described curricula using the following MF-RL algorithms: The original Proximal Policy Optimization (PPO) from (Schulman et al.) (017)). Soft Actor-Critic (SAC) from Haarnoja et al.] (218)) and the Twin Delayed DDPG (TD3) from Fujimoto al.) (2118). We provided these methods with hand-designed dense reward function as we did not observe any success with the sparse reward only. Each of the mentioned setups is trained for five different random seeds, resulting in 180 trained agents.
Note that each task generator suppress interventions that would yield goal shapes outside its family.
7

---

Published as a conference paper at ICLR 2021
Pushing
Picking
Pick and Place
Stacking2
1.0
curriculum curriculum
0.5
fractional sucosss
0.0
1.0
0.5
0.0
1.0
curriculum 2
0.5
0.0
107
10a
107
108
107
108
10
10a
time steps
time steps
time steps
time steps
PPO
SAC
TD3
Figure 5: Fractional success curves averaged over five random seeds for the tasks and learning algorithms specified above, under three different training curricula: (0) curriculum, (1) goal position and orientation randomization in space A every episode and (2) curriculum where we intervene on all variables in space A simultaneously every episode.
Training model-free RL methods: We report the training curves averaged over the random seeds in Fig. 5 As can be seen from these fractional success training curves, MF-RL methods are capable of solving the single block goal shapes (pushing, picking, pick and place) seen during training time given enough experience. However,
observe that none of the methods studied here managed to solve stacking two blocks. The score below 0.5 indicates that it only learns to push the lower cube into the goal shape. This shows that multi-object target shapes become nontrivial quickly and that there a need for methods making use of the modular structure of object-based environments. To no surprise, the training curriculum has a major effect learning. For example, methods rarely manage to pick up any significant success signal under extreme domain randomization as in cur- riculum even after 100 million timesteps. Note that these curves represent the scores under the conditions of the training environments. Next, we will discuss shared evaluation protocols that allow benchmark and compare agents trained under different conditions.
Benchmarking generalization capabilities along various axes: For each of the four goal shape families, define set of 12 evaluation protocols that we consider meaningful and representative for benchmarking the different algorithms. In the protocols presented here, we sample the values from a protocol -specfil- set of variables the start of each episode while keeping all other variables fixed to their default values. We evaluate each agent 200 episodes by computing the fractional success score at the last time step of each episode and reporting the mean. These evaluation protocols allow to disentangle generalization abilities, they show robustness with respect to different types of interventions, see Fig. 6 The following are some of the observations we made for pushing:
Agents that were trained on the default pushing task environment (curriculum 0) do well (as expected) on the default task (PO). Interestingly, we likewise see generalization ca- pability to initial poses from variable space A (P4). This can be explained by a substantial exploration of the block positions via manipulation during training. In contrast, that the agents exhibit weaknesses regarding goal poses (P5), they seem to overfit to their training settings in this case.
8

---

Published as a conference paper at ICLR 2021
PPO(0)
SAC(O)
fractional success
PPO(1)
SAC(1)
TD8(1)
PPO(2)
SAC(2)
TD3(2)
0.75
0.50
0.25
0.00
PO
P1
P2
P3
P4
P5
P6
P7
P8
P9
P10
P11
space
13
A
A
A
13
A
bp
8P
bp,
8P
bp, 8P.
bp.
KP:
bp,
time step
Figure 6: Evaluation scores for pushing baselines. Each protocol was evaluated for 200 episodes and each bar is averaged over five models with different random seeds. The variables listed under each protocol are sampled from the specified space at the start of every episode while all other variables remain fixed [bp block pose, bm block mass, bs block size, gP goal pose, ff floor friction].
For agents trained with goal pose randomization (curriculum 1) we see similar results as with curriculum with the difference that agents under this curriculum generalize robustly to different goal poses (P5), as one would expect.
Finally, agents that experience extreme domain randomization (curriculum 2) at training time, fail to learn any relevant skill shown by the flat training curve in Fig. An explanation for this behavior could be that the agent might need more data and optimization steps handle this much more challenging setting. Another possibility is that it may simply not be possible to find a strategy which simultaneously works for all parameters (note that the agent does not have access to the randomized parameters and hence must be robust to them). This poses an interesting question for future work.
As expected, we observe that an agent's generalization capabilities are related to the experience gathered under its training curriculum. CausalWorld allows to explore this relationship in differentiated manner, assessing which curricula lead which generalization abilitics. This will not only help uncover agent's shortcomings but may also aid in investigating novel learning curricula and approaches for robustness in RL. Lastly, we note that this benchmark comprises extremely challenging tasks that appear be out of reach of current model free methods without any additional inductive bias.
5 CONCLUSION
We have introduced a new benchmark CausalWorld, to facilitate research in causal structure and transfer learning using simulated environment of open-source robot, where learned skills could potentially be transferred to the real world. We showed how allowing for interventions on the environment's properties yields diverse familiy of tasks with natural way of defining learn- ing curricula and evaluation protocols that can disentangle different generalization capabilities. A natural extension of work is develop generalization (whether its a different subspace of the state space or a completely different task). We hope that the flexibility and modularity of Caus a loooldd will allow researchers easily define ap- propriate benchmarks of increasing difficulty as the field progresses, thereby coordinating research efforts towards ever new goals.
RL. algorithms that focuses out of distribution
ACKNOWLEDGMENTS
The authors would like to thank Felix Widmaier, Vaibhav Agrawal and Shruti Joshi for the useful discussions and for the development the TriFinger robot's simulator (Joshi et al. 2020), which served as starting point for the work presented in this paper. AG is also grateful to Alex Lamb and Rosemary Nan Ke for useful discussions. We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting FT.

---

Published as a conference paper at ICLR 2021
REFERENCES
lige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113. 2019.
Safa Alver and Doina Precup. brief look at generalization in visual meta-reinforcement learning. arXiv preprint arXiv:2006.07262 2020.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv: 19X.0::909.07528, 2019.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems, pp. 5082-5092 5093, 2019.
Victor Bapst, Alvaro Sanchez- Gonzalez, Carl Doersch, Kimberly L Stachenfeld, Pushmeet Kohli, Peter W Battaglia, and Jessica B Hamrick. Structured agents for physical construction. arXiv preprint arXiv: 0e0..0994410000 2019.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counter- factual learning of physical dynamics. arXiv preprint arXiv:1909.12000 2019.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysaw Debiak, Christy Den- nison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1XXv:1912.00680 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01000.005400 2016.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, lean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactually-guee guided policy search. arXiv preprint 111..::111110672 2018.
Yvonne M Caldera, Anne McDonald Culp, Marion O' OrBrin, Rosemarie Truglio, Mildred Alvarez, and Aletha C Huston. Children's play preferences, construction play with blocks, and visual- spatial skills: Are they related? International Journal of Behavioral Development, 23(4):855- 872, 1999.
Beth M Casey, Nicole Andrews, Holly Schindler, Joanne E Kersh, Alexandra Samper, and Juanita Copley. The development of spatial skills through interventions involving block building activi- ties. Cognition and Instruction, 26(3):269-309, 2008
Stephanie CY Chan, Sam Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Mea- suring the reliability of reinforcement learning algorithms. arXiv preprint arXiv:1912.0566h 2019.
Maxime
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning, arXiv preprint 1rXiv:1812.023441 2018.
Erwin Coumans et al. Bullet real-time physics simulation. URL http://bllulllpphysics.. org, 2013.
Gwen Dewar. The benefits of toy blocks: The science of construction play. Parentig Science, 2018
10

---

Published as a conference paper at ICLR 2021
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. International Conference Learning Representations (ICLR), 2021.
Gabriel Amnol, Nir Levine. Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learn- ing. arXiv preprint arXiv:2003. 1881, 2020.
Scott Elliott Fahlman. planning system for robot construction tasks. Artificial intelligence, 5(1): 1-49, 1974.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the in- formation bottleneck. arXiv preprint arXiv:1901.10900 2019a.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv: 1909. 10893, 2019b.
Thomas L. Griffiths and Joshua B Tenenbaum. Structure and strength in causal induction. Cognitive psychology, 51(4))334-384, 2005.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Peter Henderson, Wei-Di Chang, Florian Shkurti, Johanna Hansen, David Meger, and Gregory Dudek. Benchmark environments for multitask learning in continuous domains. arXiv preprint arXiv: 04352, 2017a.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint STtiv:179..066500 2017b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment, IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-oriented prediction and planning. arXiv preprint 18/2: 18/2. 2018.
Shruti Joshi, Felix Widmaier. Vaibhav Agrawal, and Manuel Wüthrich. http: htgggttthhb.hgg// com/ sengerrssmuller simulation 2020.
Constance Kamii, Yoko Miyakawa, and Yasuhiko Kato. The development of logico- matiemaaatheaaaical knowledge in block-building activity at ages 1-4. Journal of Research in Childhood Education, 19(1):44-57 2004.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Youngwoon Lee, Edward S. Hu, Zhengyu Yang. Alex Yin, and Joseph J. Lim. Ikea furniture assem- bly environment for long-horizon complex manipulation tasks, 2019.
Will McCarthy, David Kirsh, and Judith Fan. Learning to build physical structures better time.
11

---

Published as a conference paper at ICLR 2021
Mchdi Mirza, Andrew Jaegle, Jonathan J. Hunt, Arthur Guez, Saran Tunyasuvunakool, Alistair Muldal, Théophane Weber, Peter Karkus, Sébastien Racamière, Lars Buesing, Timothy Lillicrap, and Nicolas Heess. Physically embedded planning problems: New challenges for reinforcement learning, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:/312.56/0 2013.
Swiya Nath and Dénes Szücs, Construction play and cognitive skills associated with the develop- ment of mathematical abilities in 7-year-old children. Learning and Instruction, 32:73-80, 80, 2014.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari, Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: / 919900880366. 2019.
Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization deep reinforcement learning. arXiv preprint 1810. 2018.
Judea Pearl. Causality. Cambridge university press, 2009.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. arXiv preprint arXiv: 1ee::020300000 2017.
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, pp. 6550-6561, 2017.
Miles Richardson, Thomas E Hunt, and Cassandra Richardson. Children's construction task perfor- mance and spatial ability: Controlling task complexity and predicting mathematics performance. Perceptual and motor skills, 1193))741-757, 2014.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. arXiv preprint vrXiv:2102.11107 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126-134, 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.0000 000900 2018
Brian N Verdine, Roberta Michnick Golinkoff, Kathy Hirsh-Pasek, and Nora Newcombe. Links between spatial and mathematical skills across the preschool years. Wiley, 2017.
Shimon Whiteson, Brian Tanner, Matthew E Taylor, and Peter Stone. Protecting against evaluation overfitting in empirical reinforcement learning. In 2011 IEEE symposium adaptive dynamic programming and reinforcement learning (ADPRL), pp. 120-127. IEEE, 2011.
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Patrick H Winston. Learning structural descriptions from examples. 1970.
Manuel Wüthrich, Felix Widmaier. Felix Grimminger, Joel Akpo. Shruti Joshi, Vaibhav Agrawal, Bilal Hammoud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, et al. Trifinger: An open- source robot for learning dexterity. arXiv preprint arXivv2008.03399 2020.
12

---

Published as a conference paper at ICLR 2021
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arX1::1910.1000 2019.
Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint (rXXiv:702.024444 02453, 2017.
13

---

Published as a conference paper at ICLR 2021
7 APPENDIX
A OBSERVATIONS
Observations in CausalWorll has two modes, "structured" and "pixel".. When using "pixel" mode, 6 images are returned consisting of the current images rendered from different views on top of the TriFinger platform, showing the current state of the environment, well as the 3 equivalent goal images rendered from the same points of view, showing the goal shape that the robot have to build by the end of the episode.
(a) Current View60
(b) Goal View60
(c) Current View 120
(d) Goal View 120
(c) Current View300
(f) Goal View300
Figure Example "pixel" mode observations returned at each step of the environment.
14

---

Published as a conference paper at ICLR 2021
Joint positions
End-effector positions
Scene features
R1
R2
R3
S1
Sn
Object Features
OR
Fearures
Partial Goal
Time left
Joint velocities
Cartesian Position
Linear Velocity
id
Cartesian
Position
Bounding Box Size
Quaternion Orientation
Angular Velocity
Bounding Box Size
Quatermion
Orientation
Figure 8: Structured observation description. For the scene features, all the blocks feature vector are concatenated first. Following that the partial goals feature vector are concatenated in the same order. Lastly, if there is any obstacles/ fixed blocks, their feature vectors are concatenated at the end following the same description as the partial goal features.
B TRIFINGER PLATFORM
The robot from Wüthrich et al. 2020 shown in figure 9 is open-sourced and can be reproduced and built in any research lab; since its inexpensive (about $5000), speeding up sim2real research.
Figure 9: The TriFinger platform.
C TASK GENERATORS
1. Pushing: task where the goal is push one block towards a goal position with a specific orientation; restricted to goals on the floor level.
2. Picking: task where the goal is to pick one block towards a goal height above the center of the arena; restricted goals above the floor level.
15

---

Published as a conference paper at ICLR 2021
3. Pick And Place: task where the arena is divided by fixed long block and the goal is to pick one block from one side of the arena to a goal position with a variable orientation on the other side of the fixed block.
4. Stucking2: task where the goal is to stack two blocks above each other in specific goal position and orientation.
5. Towers:
task where the goal stack multiple n blocks above each other in a specific goal position and orientation exactly above each other creating tower of blocks.
6. Stacked Blocks: task where the goal to stack multiple n blocks above each other in an arbitrary way create stable structure. The blocks don't have be exactly above each other; making more challenging than the ordinary towers task since the its harder to come up with stable structure that covers the goal shape volume.
7. Creative Stacked Blocks: exactly the same the Stacked Blocks task except that the first and last levels of the goal are the only levels shown or "imposed" and the rest of the struc- is not explicitly specified, leaving the rest of the goal shape the imagination of the agent itself;; this is considered the most challenging since its it needs the agent to under- stand how to build stable structures and imagine what be filled in the middle connect the two levels in stable way.
8. General: the goal shape is an arbitrary shape created by initially dropping an arbitrary number of blocks from above the ground and waiting till all blocks come a rest position
where this becomes the goal shape that the agent needs to fill up afterwards.
Variable gravitylz]
Sub Variable
Space 10, -7
Space 7,
floor friction
0.3. 0.6
00.6, 0.8
stage friction
[0.3, 0.6
00.6, 0.8
stage color [rgb]
0.50
00.5, 1]
floor color [rgb]
(0.0,
[055, 1js
joint positions
[-1.57, 1122 -3.0
[[-0,99,,00s
-0.69,
1.0, 1.57,3.0
block
size
10.055, 0.075)
10.075, 0.005,0.095
block
color
[0.0.0.5]
0.., 1]
block
mass
(0.015.0.045]
00.045.0.1]
block
position (eylindrical)
[0,-7,4/2]
[0.11,-50 h/2].
0.11,,,0.155
0.15, 1,5,,,0033 0.3
goal cuboid
size
(0055,00077)
0.075,0.0955
goal cuboid
color
0.0,0.55
0055, 1
link link
color mass
00.0, 0.5]
0.5, 1]s
[0.015, 0.045]
00.045.00 0.1]
Table 2: Description of subset of the high level variables, exposed in CausalWorld, and their corresponding spaces, refers to the height of the block.
Task Generator Picking Towers
Variable goal height tower dims
Space A [0.08, 0.20]
Space B [0.20, 0.25 [0. 0. 0.12]. (0.00, 0.20, 0.20]]
[0.08, 0.08, 0.08]. [0.12, 0.12, 0.12]
Table 3: Example of task generators' specific high level variables, exposed CausalWorld and their corresponding spaces. For full list of each task generators' variables and their corresponding spaces, please refer to the documentation at (http:://ites.goggle.ooom/ gwwgle.com/vieeeeee.e.oo wwwp/v/////wwwaaa///ooooooooaaaa world/home).
16

---

Published as a conference paper at ICLR 2021
Task generators
Dense reward
Pushing Picking
-750A (o,, e) 250 (01.91)
-750 - 91. 125A (0,, -.005))
o'
Pick and Place
-750A"(o1,e) 50A' (0.. - 250( - - -11-10 - +|)
0.005/Iu* - 111111
Stacking
(000))))0.002( (3(00)))0000(7755A 750 (01,e)
250 (01, 91))
+
(0 (-70)) 750 (02, E) 250(
L 10
g2.21
1o5.-01.=012544 (02,z.81 22, 0.005| 0 o 41
Table 4: Description the dense rewards applied our experiments. The following notation was applied: v² E R joint velocities, ef R i-th end-effector positions, of R i-th block position, of E R i-th goal block position, d' (o
>; |ee o'll the distance between end-effectors and the block, A = d'((o (o. e) - d' (),e) e) the distance difference w.r.t. the previous timestep. The target height parameter for pick and place is 0.15 if block and goal are of different height. Otherwise, t is half the goal height.
17

---

Published as a conference paper at ICLR 2021
D TRAINING DETAILS
The experiments were carried out using the stable basel nns implementation of PPO, SAC and TD3. We used a layer MLP Policy [256,256] for all the policies. PPO was trained on 20 workers up to 100 million timesteps in parallel and SAC as well as TD3 were trained serially for 10 million timesteps.
PPO
SAC
TD3
discount
0.99
discount
0.95
discount
0.96
batch size
120000
entropy coeff
le-3
batch size
128
learning rate entropy coef.
2.5e-4
batch size
256
learning rate
le-4
0.01
learning rate
lc-4
buffer size
500000
value function coef.
0.5
target entropy
auto
tau
0.02
gradient clipping (max)
0.5
buffer size
1000000
n minibatches per update
40
tau
0.001
training epochs
4
Table 5: Learning algorithms hyper parameters used in the baselines experiments.
radar_ Liitar_poot___e auu.amaaaiieeellllatiiinnn cassloooaaaslalaaaaaaa world
mean last fractionall success
1PC(1)
P1
0.6
P10
0.4
0.2
P3
P4
P8
Pb
Figure 10: An example of model selection in CausalWorld by evaluating generalization across the various axes using the previously mentioned protocols. Here we compare two agents trained on different curricula using PPO.
18

---

Published as a conference paper at ICLR 2021
PPO(0)
SAC(O)
TD3(0)
fractional sucoess
1.00
PPO(1)
SAC(L)
TD3(1)
PPO(2)
8AC(2)
TD3(2)
0.75
0.50
0.25
0.00
PO
P1
P2
P33
14
175
P6
P7
P8
P9
PI0
PI1
11
-A
A
1
A
variables
ban
bm
bs
bp
6P
bp, RP
bp: 8P.
bm
bp. EP:
bm
bp, AP,
bm
step
PPO(0)
SAC(O)
TD3(0)
fractional sucotss
1.00
PPO(1)
SAC(1)
TD3(1)
PPO(2)
SAC(2)
TD3(2)
0.75
0.50
0.25
0.00
PO
PT
P2
P3
P4
175
P6
P7
P8
P9
PTO
space
13
J.
A
A
8
A
B
B
A
variables
bm
bp
bp:
8P
bp, SP.
bmm
bp, KP.
bm
bp, 8P.
bm
PPO(U)
SAC(O)
TD3())
fractional success
1.00
PPO(1)
SAC(1)
TD3(1)
PPO(2)
8AC(2)
0.75
0.50
0.25
0.00
PO
P1
P2
13
P4
125
P6
P7
P8
19
PTO
PIT
A
A.
A
B
A
B
B
A
variablee
ban
bm
bp
EED
bp,
RP
bgp SP: bmm
hpp, EP:
bp, ZP.
PPO(0)
SAC(O)
TD3(0)
fractional suceuss
1.00
PPO(1)
8AC(1)
TD3(1)
PPО(4)
SAC(2)
TD3()
0.75
0.50
0.25
Rofutee P8
*1 Pg
0.00
PO
P1
P2
P4
P5
P6
P7
P8
P10
space
A
13
A.
A
13
A.
13
A.
B
bp
hp, 81
bp,,
BPP
bp, RP. 1mmm
bp, RP.
Figure 11: Evaluation scores, for pushing, picking, pick and place and stacking2 baselines. from top to bottom respectively. Each protocol was evaluated for 200 episodes and each bar is averaged over five models with different random seeds [bp block pose, bm block mass, bs block size, gp goal pose, ff floor friction].
19

---

Published as a conference paper at ICLR 2021
E CAUSALITY IN REINFORCEMENT LEARNING
Definition 1: A partially observable Markov decision processes (POMDP) is defined by the tuple (S, A, R, S2 O 7, Po, If) with states s S, actions a E A and observations 52 determined by the state and action of the environment O(ols,a), a). T( ( 0z) is the transition probability distribution function. R(s;,a) is the reward function, y is the discount factor, po(s) is the initial state distribution at the beginning of each episode, and H is the time horizon per episode. The objective of RL algorithms to learn a policy TTa (h,) with history H, (1,.A1,.. O,)
that maximizes the discounted expected reward (Tx E" E
Definition 2: A structural causal model (SCM) M over X (X1:-., XN) is given by DAG G over nodes X, independent noise RVs U = (U1,.., UN) with distributions Pu, and functions fi..... fN such that X; filpa, U4). where pa; C are the parents of X; in G. An SCM entails a distribution P with density p over (X, ()U)).
Definition 3:
An intervention in an SCM M consists of replacing the RHS fi(pa; U2) by If (pat, U2) with i [ (1,... N} where pa! are the parents in a new DAG G². The resulting SCM is denoted with M Mdo(1) with distribution pdo(1) and density pd((!)
Representing POMDPs as SCMs: A POMDP can be presented as an SCM M by unrolling the POMDP through time by expressing all conditional distributions, e.g the transition kemel Ps.+ as deterministic functions (which reflects the causal mechanisms of the system) with independent noise variables U, such as Si+1 (St A., Us.). This is always possible using auto- regressive formalization shown in (Buesing et al.) 2018)) The distribution P over trajectories T determined by the SCM, which means running a different policy (instead of π) in the environment is an intervention by itself [(T + μ)) resulting in model distribution (==(1(==))), Given the SCM, one reason about the alternative outcomes of different actions even different environment properties (such as friction) (known as counterfactual inference) which is not possible if we only model the system a conditional distribution PoJA.
Generalizing agents: An agent should generalize well to changes in the environment if it achieves an equal performance after an intervention / is performed on the SCM M resulting in M) This can be accomplished for instance explained in (Schölkopf al. (021). by reusing the learned causal mechanisms. For example, if an intervention was performed on the colour of one of the available blocks in the environment, it won't change the underlying causal mechanisms. Thus, this sort of interventions on non causal variable shouldn't affect the resulting actions of the agent. Nevertheless, this is often not the case with RI. agents trained using visual observations. (Alver & Precup 2020) showed recently that algorithms which were designed specifically for Meta-RI. can display strong overfitting when they are evaluated on challenging visual tasks. On the other hand, an intervention
performed on a causal variable with respect to policy such as the mass of an object, the resulting action sequence will be expected to change for robust control. Additionally, if an intervention is performed on some of the goal variables, the reward function might also change and a robust controller should also be able to react to it accordingly.
20


Reviews:
[{"review_id": "YHZDS3tVq_p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a a robotic manipulation benchmark for causal structure and transfer learning in a simulation environment considering 3D shape construction tasks given a set of blocks. Baseline results using model-free algorithms are provided for chosen tasks, e.g. pushing, picking, pick&place, stacking. It is also stated that a real version of the robot can be built (as it is open-sourced) for sim2real research. The paper is clearly written, nicely structured and, presents interesting and important ideas. It exposes a large set of parameters, e.g. properties of blocks (size, mass, pose), friction, goals for generalisation evaluations. Having a real-world counterpart makes it very valuable for sim2real research. Authors provide and discuss the relevant previous work detailing how their work connects to the existing literature.  A minor comment: The particular choice of the robot can be motivated, as it  is a special design.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UXVrXE8R6XG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nMotivated by the difficulty of evaluating RL’s ability to transfer behaviors across environments, the authors propose the CausalWorld benchmark. Unlike prior benchmarks, CausalWorld exposes well-defined casual variables, in the form of task factors, and focuses on robotic manipulation of an open-source robot platform. The authors make CausalWorld easily usable for both training (in defining a learning curriculum) and evaluation (in targeting specific expected generalizations), and make it easy to extend. In their original release, the authors include eight concrete tasks to test generalization, and present baseline results on these tasks.\n\n---\n\n### Positives\n\n- The paper is strongly motivated and tackles a real and practical problem. Evaluating transfer in RL agents has been challenging, especially for robotics, and the authors’ proposed benchmark framework could be useful in addressing this.\n- The authors’ benchmark supports useful behavior both for training, in gradually varying the task distributions, and in testing, for evaluating generalization ability. Additionally, since it is tied to a real world, open source robot platform, this benchmark in the future could also be used to evaluate sim2real transfer.\n- The authors’ framework is defined in a way that seems easy-to-extend and supports multiple use cases, including custom “task generators” for defining new tasks or goals, and “intervention actors” to define a learning curriculum.\nThe paper provides relatively strong baseline experiments, with quantitative results across several model-free RL algorithms (PPO, SAC, TD3), and multiple potential curriculum techniques.\n\n---\n\n### Negatives\n\nThe current iteration of the experiments, with figures 4 and 5, are unclear and confusing. Specifically:\n\n- In figure 4, it’s unclear to me what the new experimental result is here, given that the benchmark is meant to test transfer and generalization ability, and the results presented are on training curves. It seems that the main conclusion here is that the choice of learning curriculum is important for performance, which as the authors point out is unsurprising?\n- In figure 4, curriculum 2 seems poorly motivated, in that full randomization without any curriculum at the beginning of training seems likely to fail as it does. Have the authors considered testing curriculums where the domains for the causal variables get progressively more challenging? For instance, automatic domain randomization (ADR) from “Solving rubik’s cube with a robot hand” may be a useful curriculum to compare. \n- It was challenging for me to follow Figure 5, since it was not clear what the training environments agents were being trained on, and which environments they were being evaluated under. Further, do these results for pushing hold across other tasks (picking, pick and place, stacking2)?\n- In figure 5, what is the time step reported (0 across all evaluations)?\n- Figure 5 shows that there is some generalization to tasks in space A and B, but it is unclear how A and B differ, what the variables between both are, which environments in A were trained on, and what P0-P11 are.\n\nIn addition I think some other experimental results would be helpful:\n\n- The authors could investigate a more detailed analysis on different reward structures, other progressive curriculums, and other methods that claim better generalization and transfer performance. \n- Some qualitative results exploring the relationship between the identified and proposed causal variables (potentially through the lens of the agent performance) would be helpful.\n- A qualitative experiment comparing the difference in learned behaviors between two policies and the difference in performance, to show that the performance reported by the benchmark does match intuition would be helpful.\n\nApart from the experimental results, I have some other broader (and potentially less pressing) concerns:\n\n- The authors limit their framework and results to the manipulation of simple block shapes. Manipulating non-block objects would result in more complex goals, introducing more causal variables that are potentially harder to disentangle and represent cleanly, but are still important for real-world applications. \n- The authors are motivated by facilitating research in causal structure learning, but this paper focuses almost exclusively on studying transfer learning and generalization ability. Potentially this is mitigated by benchmarking causal learning algorithms that try to directly learn the causal graph or reason between causal variables. For instance, potentially the causal graph-parameterized policy learning approach from “Causal Confusion in Imitation Learning”, or similar algorithms, would be good to include.\n\n---\n\n### Recommendation\n\nOverall, I vote for rejecting. I think the benchmark is well motivated, but not backed up with strong experimental results. The motivation for the benchmark is to show that the framework can be used to study transfer performance, but the current experimental results do not convince me that the framework makes it easy to uncover new insights in practice. One reason to potentially accept the benchmark is that it seems easy to extend, but this is also difficult to evaluate from the limited experiments presented.\n\nIf the authors were to respond to some of my comments above, by providing a better understanding of the figures and experiments (in case I am misinterpreting the current results), and by showing the utility of the benchmark, then some of my concerns would be addressed.\n\n---\n\n### Minor feedback\n\n- Which hand designed dense reward function is being used? I see this is present in the supplementary figures, I would also add a reference in the main text.\n- Which observation spaces were the figures trained/evaluated in (state or pixel)?\n\n----\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SCbQMONAnMF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new benchmark, CausalWorld, for studying generalization, transfer learning, and causal structure learning in RL and robotics. This is a hugely important problem, and I think this benchmark has some clear advantages over existing benchmarks. The benchmark consists of a simulated three finger robot over a bin containing blocks, within which there are 8 \"families\" of tasks, (a) pushing, (b) picking, (c) pick and place, (d) stacking 2 blocks, (e) stacking many blocks, (f) general rearrangement, (g) more complex multi-block stacking, and (h) building towers. More importantly, for each family of tasks, there is controllable procedural generation of goals as well as controllable factors of the environment such as object sizes, masses, frictions, colors, etc. \n\nThis enables what I think is the key contribution of this paper - a procedural way to define training/evaluation splits where each split samples from different subspaces of the above controllable factors. This provides a systematic way of defining problems which require varying degrees of generalization, measuring the difficulty of such splits, and defining curricula within each split, which is critical to developing learning algorithms which are capable of this sort of generalization. While prior work (Yu et al, James et al) have defined many robotic tasks with some shared structure, one challenge is that it is difficult to say how much generalization one can expect between any two tasks which can be quite different, a problem which this benchmark takes a step towards addressing.\n\nLike the paper mentions, prior works have also used procedural generation over similar controllable factors like this paper does. In fact most physics simulators do allow varying these parameters directly. But, a simple API with a standardized interface to define these splits, as well as common splits that are used as benchmarks is still missing, and this paper takes an important step towards that. While I don't see a link to the code, I would encourage the authors to design their API in a simple and standardized way, as that is likely what would motivate people to use CausalWorld instead of manually defining train/eval splits in their own physics simulators.\n\nThe main weakness I see of the paper is the experimental section. The authors train a few SOTA RL algorithms on 3 different train configurations of increasing randomization, and test on 12 different eval configurations. First in terms of clarity, I found Figure 5 difficult to interpret. I think it would be helpful if for each of the Eval protocols it was clearly described what was changing. In general I think the best way to present this would be to look at each pair of \"train domain, eval domain\" and the corresponding performance, with some clear description about what sort of generalization is needed. Also in terms of performance, it seems like when faced with anything more challenging than push/pick-place with limited randomization, all the SOTA algorithms fail even on the train domain (and as a result struggle on Eval domains as well). So one concern is that of the many domains presented in the benchmark, perhaps only a few are actually solvable by current RL algorithms during training. At the same time I think this indicates the challenges in learning generalizable policies, and may inspire better RL algorithms.\n\nOverall, I think this is an exciting benchmark, and would be excited to use it. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BGIPmQiy-TW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new benchmark for studying reinforcement learning and its generalization in the context of the robotic manipulation problem. To study the generalization of a learned policy, the proposed benchmark is equipped with an interface that makes intervention easy. This interface helps to define a training space and an evaluation space so that one can systemically study both in-distribution and out-of-distribution generalization of a learned policy. At the same time, the proposed benchmark simulates an open-source robot platform, which makes sim2real transfer experiments easier.\n\nStrengths:\n* This paper proposed an RL benchmark with many good properties: systematic intervention of environment distribution and potential application to sim2real transfer experiments\n* The source code of the benchmark provided with the submission is clean and well documented, so it could benefit future research based on this work.\n* Proposed evaluation protocol gives an insight on how this benchmark can be used to evaluate generalization of RL agent.\n\nWeaknesses:\n* One concern I have about this benchmark is that the training difficulty may hinder the analysis of generalization. It is expected that a rich training distribution would lead to better generalization, but it seems rich training distribution makes training difficult and results in worse performance on evaluation distribution.\n* As far as I understand, there are a few predefined tasks, and each task distribution cannot be intervened by modifying some task-relevant parameters. However, I can imagine parameterizing the tasks. For example, we can parameterize push task distribution by introducing a range of (x, y) position of block initial positions, and introducing a range of distance from initial position to goal position. If I have misunderstood the detail of the task parameterization, please elaborate on this point.\n\n\nQuestions to authors:\n* I had the impression that this paper attempts to make a connection with the research about causality. For example, the name of the benchmark is \"CausalWorld\", and the text mentions \"opportunity to investigate causality\", and \"underlying structural causal model (SCM)\". However, it is unclear to me how this benchmark can help to study causality exactly. Could you elaborate on this point?\n* It seems that space A and space B in Table 2 (Appendix C) is an arbitrary split of a range. How these ranges are determined? Is there any motivation behind this split?\n\n\nRecommendation:\nI recommend accepting this paper because it could benefit the field by helping people to easily study generalization in a complex robotic manipulation setting. To the best of my knowledge, no existing open-sourced RL environment for robotic manipulation does not support systematic intervention of the environment distribution. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: CausalWorld is a benchmark for robotic manipulation to address transfer and structural learning. The benchmark includes (i) a variety of tasks (picking, pushing, tower, etc) relating to manipulating blocks, (ii) configurable properties for environments (properties of blocks, gravity, etc), (iii) customizable learning settings involving intervention actors, which can change the environment to induce a curriculum.

The reviewers found the paper compelling and with many strengths, including ‘interesting and important ideas’ (R4), ‘simple API with a standardized interface’ for ‘procedural generation of goals’ (R5), ‘strongly motivated and tackles a real and practical problem’ (R3), and ‘benchmark with many good properties’ (R2). By and large, the reviewers agree that the paper presents an important benchmark satisfying several desiderata, which I certainly agree with. 

On the other hand, most of the reviewers (3 out of 4) also raised serious concerns, more prominently, about the experimental results and the causal inference component. For instance, R5 commented that “all the SOTA algorithms fail,” and it is hard to quantify how agents would perform well in different tasks. R3 pointed out the lack of “qualitative results exploring the relationship between the identified and proposed causal variables,” emphasizing that ‘the benchmark is well-motivated, but not backed up with strong experimental results.‘’ R2 identified the lack of clear causal component in the paper while the paper mentions “opportunity to investigate causality” and “underlying structural causal model (SCM).” All in all, these are valid concerns.

The authors' rebuttal was quite detailed, and appreciated, but left some important questions unanswered.  The first and critical issue is about the causal nature of the simulator. The simulator's name is "causalworld" and its stated goal is to provide "a benchmark for causal structure and transfer learning in a robotic manipulation environment."  Also, the first bullet in the list of contributions is: "We propose CausalWorld, a new benchmark comprising a parametrized family of robotic manipulation environments for advancing out-of-distribution generalization and causal structure learning in RL." After reading the paper, I was quite surprised to realize there is no *single* example of a causal model, in any shape or form (e.g., SCM, DAG, Physics) or a structural learning benchmark. In other words, there is a serious, somewhat nontrivial gap between the claimed contributions and what was realized in the paper. One way to address this issue would be to make the causality more explicit in the paper, for example, by sharing the underlying structural causal model, how variables form causal relationships, what causal structures are being learned, and how these learned structures compare with the ground truth. I think these would be reasonable expectations of a simulator that aims to disentangle the causal aspect of the learning process. 

The second issue is about the experimental results in terms of generalizability. The authors emphasized on different occasions that "The primary goal of this work is to provide the tools to build and evaluate generalizable agents in a more systematic fashion, rather than building generalizable agents for the tasks specified," or "the experiments is to showcase the flexibility regarding curricula and performance evaluation schemes offered with CausalWorld, rather than solving new tasks or proposing new algorithms." These responses are somewhat not satisfactory given that the goal of the paper is providing tools to build generalizable agents, while the authors seem to suggest they are not committed to actually building such agents. Specifically, the experiments did not demonstrate the simulator as a benchmark but only showcased its flexibility (i.e., offering a large number of degrees of freedom). One suggestion would be to evaluate how algorithms (agents) with varying degrees of "generalizability" power perform across tasks with various difficulty levels. As it currently stands, the tasks are too easy or too hard for the standard, uncategorized algorithms, which makes it difficult to learn any lessons from running something in the simulator. 

Lastly, I should mention that the work has a great potential to introduce causal concepts and causal reasoning to robotics, there is a natural and compelling educational component here. Still, the complete absence of *any* discussion of causality and the current literature results hurt this connection and the realization of this noble goal. I believe that after reading the paper, the regular causal inference researcher will not be able to understand what assumptions and types of challenges are entailed by this paper and robotics research. On the other hand, the robotics researcher will not be able to understand what a causal model is and the tools currently available in causal reasoning that may be able to help solve the practical challenges of robotics. In other words, this is a huge missed opportunity since there is a complementary nature of what the paper is trying to do in robotics and the results available in causal inference. I believe readers expect and would benefit from having this connection clearly articulated and realized in a more explicit fashion.

If the issues listed above are addressed, I believe the paper can be a game-changer in understanding and investigating robotics & causality.  Given the aforementioned potential and reasons, I recommend the paper's acceptance *under the assumption that* the authors will take the constructive feedback provided in this meta-review into account and revise the manuscript accordingly. 
Decision: Accept (Poster)