Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a a robotic manipulation benchmark for causal structure and transfer learning in a simulation environment considering 3D shape construction tasks given a set of blocks. Baseline results using model-free algorithms are provided for chosen tasks, e.g. pushing, picking, pick&place, stacking. It is also stated that a real version of the robot can be built (as it is open-sourced) for sim2real research. The paper is clearly written, nicely structured and, presents interesting and important ideas. It exposes a large set of parameters, e.g. properties of blocks (size, mass, pose), friction, goals for generalisation evaluations. Having a real-world counterpart makes it very valuable for sim2real research. Authors provide and discuss the relevant previous work detailing how their work connects to the existing literature.  A minor comment: The particular choice of the robot can be motivated, as it  is a special design.
 
review_title: An interesting benchmark for causal structure and transfer learning based on simulation of a manipulation environment.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a new benchmark, CausalWorld, for studying generalization, transfer learning, and causal structure learning in RL and robotics. This is a hugely important problem, and I think this benchmark has some clear advantages over existing benchmarks. The benchmark consists of a simulated three finger robot over a bin containing blocks, within which there are 8 "families" of tasks, (a) pushing, (b) picking, (c) pick and place, (d) stacking 2 blocks, (e) stacking many blocks, (f) general rearrangement, (g) more complex multi-block stacking, and (h) building towers. More importantly, for each family of tasks, there is controllable procedural generation of goals as well as controllable factors of the environment such as object sizes, masses, frictions, colors, etc. 

This enables what I think is the key contribution of this paper - a procedural way to define training/evaluation splits where each split samples from different subspaces of the above controllable factors. This provides a systematic way of defining problems which require varying degrees of generalization, measuring the difficulty of such splits, and defining curricula within each split, which is critical to developing learning algorithms which are capable of this sort of generalization. While prior work (Yu et al, James et al) have defined many robotic tasks with some shared structure, one challenge is that it is difficult to say how much generalization one can expect between any two tasks which can be quite different, a problem which this benchmark takes a step towards addressing.

Like the paper mentions, prior works have also used procedural generation over similar controllable factors like this paper does. In fact most physics simulators do allow varying these parameters directly. But, a simple API with a standardized interface to define these splits, as well as common splits that are used as benchmarks is still missing, and this paper takes an important step towards that. While I don't see a link to the code, I would encourage the authors to design their API in a simple and standardized way, as that is likely what would motivate people to use CausalWorld instead of manually defining train/eval splits in their own physics simulators.

The main weakness I see of the paper is the experimental section. The authors train a few SOTA RL algorithms on 3 different train configurations of increasing randomization, and test on 12 different eval configurations. First in terms of clarity, I found Figure 5 difficult to interpret. I think it would be helpful if for each of the Eval protocols it was clearly described what was changing. In general I think the best way to present this would be to look at each pair of "train domain, eval domain" and the corresponding performance, with some clear description about what sort of generalization is needed. Also in terms of performance, it seems like when faced with anything more challenging than push/pick-place with limited randomization, all the SOTA algorithms fail even on the train domain (and as a result struggle on Eval domains as well). So one concern is that of the many domains presented in the benchmark, perhaps only a few are actually solvable by current RL algorithms during training. At the same time I think this indicates the challenges in learning generalizable policies, and may inspire better RL algorithms.

Overall, I think this is an exciting benchmark, and would be excited to use it. 
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary

Motivated by the difficulty of evaluating RL’s ability to transfer behaviors across environments, the authors propose the CausalWorld benchmark. Unlike prior benchmarks, CausalWorld exposes well-defined casual variables, in the form of task factors, and focuses on robotic manipulation of an open-source robot platform. The authors make CausalWorld easily usable for both training (in defining a learning curriculum) and evaluation (in targeting specific expected generalizations), and make it easy to extend. In their original release, the authors include eight concrete tasks to test generalization, and present baseline results on these tasks.

---

### Positives

- The paper is strongly motivated and tackles a real and practical problem. Evaluating transfer in RL agents has been challenging, especially for robotics, and the authors’ proposed benchmark framework could be useful in addressing this.
- The authors’ benchmark supports useful behavior both for training, in gradually varying the task distributions, and in testing, for evaluating generalization ability. Additionally, since it is tied to a real world, open source robot platform, this benchmark in the future could also be used to evaluate sim2real transfer.
- The authors’ framework is defined in a way that seems easy-to-extend and supports multiple use cases, including custom “task generators” for defining new tasks or goals, and “intervention actors” to define a learning curriculum.
The paper provides relatively strong baseline experiments, with quantitative results across several model-free RL algorithms (PPO, SAC, TD3), and multiple potential curriculum techniques.

---

### Negatives

The current iteration of the experiments, with figures 4 and 5, are unclear and confusing. Specifically:

- In figure 4, it’s unclear to me what the new experimental result is here, given that the benchmark is meant to test transfer and generalization ability, and the results presented are on training curves. It seems that the main conclusion here is that the choice of learning curriculum is important for performance, which as the authors point out is unsurprising?
- In figure 4, curriculum 2 seems poorly motivated, in that full randomization without any curriculum at the beginning of training seems likely to fail as it does. Have the authors considered testing curriculums where the domains for the causal variables get progressively more challenging? For instance, automatic domain randomization (ADR) from “Solving rubik’s cube with a robot hand” may be a useful curriculum to compare. 
- It was challenging for me to follow Figure 5, since it was not clear what the training environments agents were being trained on, and which environments they were being evaluated under. Further, do these results for pushing hold across other tasks (picking, pick and place, stacking2)?
- In figure 5, what is the time step reported (0 across all evaluations)?
- Figure 5 shows that there is some generalization to tasks in space A and B, but it is unclear how A and B differ, what the variables between both are, which environments in A were trained on, and what P0-P11 are.

In addition I think some other experimental results would be helpful:

- The authors could investigate a more detailed analysis on different reward structures, other progressive curriculums, and other methods that claim better generalization and transfer performance. 
- Some qualitative results exploring the relationship between the identified and proposed causal variables (potentially through the lens of the agent performance) would be helpful.
- A qualitative experiment comparing the difference in learned behaviors between two policies and the difference in performance, to show that the performance reported by the benchmark does match intuition would be helpful.

Apart from the experimental results, I have some other broader (and potentially less pressing) concerns:

- The authors limit their framework and results to the manipulation of simple block shapes. Manipulating non-block objects would result in more complex goals, introducing more causal variables that are potentially harder to disentangle and represent cleanly, but are still important for real-world applications. 
- The authors are motivated by facilitating research in causal structure learning, but this paper focuses almost exclusively on studying transfer learning and generalization ability. Potentially this is mitigated by benchmarking causal learning algorithms that try to directly learn the causal graph or reason between causal variables. For instance, potentially the causal graph-parameterized policy learning approach from “Causal Confusion in Imitation Learning”, or similar algorithms, would be good to include.

---

### Recommendation

Overall, I vote for rejecting. I think the benchmark is well motivated, but not backed up with strong experimental results. The motivation for the benchmark is to show that the framework can be used to study transfer performance, but the current experimental results do not convince me that the framework makes it easy to uncover new insights in practice. One reason to potentially accept the benchmark is that it seems easy to extend, but this is also difficult to evaluate from the limited experiments presented.

If the authors were to respond to some of my comments above, by providing a better understanding of the figures and experiments (in case I am misinterpreting the current results), and by showing the utility of the benchmark, then some of my concerns would be addressed.

---

### Minor feedback

- Which hand designed dense reward function is being used? I see this is present in the supplementary figures, I would also add a reference in the main text.
- Which observation spaces were the figures trained/evaluated in (state or pixel)?

----

review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a new benchmark for studying reinforcement learning and its generalization in the context of the robotic manipulation problem. To study the generalization of a learned policy, the proposed benchmark is equipped with an interface that makes intervention easy. This interface helps to define a training space and an evaluation space so that one can systemically study both in-distribution and out-of-distribution generalization of a learned policy. At the same time, the proposed benchmark simulates an open-source robot platform, which makes sim2real transfer experiments easier.

Strengths:
* This paper proposed an RL benchmark with many good properties: systematic intervention of environment distribution and potential application to sim2real transfer experiments
* The source code of the benchmark provided with the submission is clean and well documented, so it could benefit future research based on this work.
* Proposed evaluation protocol gives an insight on how this benchmark can be used to evaluate generalization of RL agent.

Weaknesses:
* One concern I have about this benchmark is that the training difficulty may hinder the analysis of generalization. It is expected that a rich training distribution would lead to better generalization, but it seems rich training distribution makes training difficult and results in worse performance on evaluation distribution.
* As far as I understand, there are a few predefined tasks, and each task distribution cannot be intervened by modifying some task-relevant parameters. However, I can imagine parameterizing the tasks. For example, we can parameterize push task distribution by introducing a range of (x, y) position of block initial positions, and introducing a range of distance from initial position to goal position. If I have misunderstood the detail of the task parameterization, please elaborate on this point.


Questions to authors:
* I had the impression that this paper attempts to make a connection with the research about causality. For example, the name of the benchmark is "CausalWorld", and the text mentions "opportunity to investigate causality", and "underlying structural causal model (SCM)". However, it is unclear to me how this benchmark can help to study causality exactly. Could you elaborate on this point?
* It seems that space A and space B in Table 2 (Appendix C) is an arbitrary split of a range. How these ranges are determined? Is there any motivation behind this split?


Recommendation:
I recommend accepting this paper because it could benefit the field by helping people to easily study generalization in a complex robotic manipulation setting. To the best of my knowledge, no existing open-sourced RL environment for robotic manipulation does not support systematic intervention of the environment distribution. 




review_title: Useful benchmark for studying generalization of RL

Meta-review: CausalWorld is a benchmark for robotic manipulation to address transfer and structural learning. The benchmark includes (i) a variety of tasks (picking, pushing, tower, etc) relating to manipulating blocks, (ii) configurable properties for environments (properties of blocks, gravity, etc), (iii) customizable learning settings involving intervention actors, which can change the environment to induce a curriculum.

The reviewers found the paper compelling and with many strengths, including ‘interesting and important ideas’ (R4), ‘simple API with a standardized interface’ for ‘procedural generation of goals’ (R5), ‘strongly motivated and tackles a real and practical problem’ (R3), and ‘benchmark with many good properties’ (R2). By and large, the reviewers agree that the paper presents an important benchmark satisfying several desiderata, which I certainly agree with. 

On the other hand, most of the reviewers (3 out of 4) also raised serious concerns, more prominently, about the experimental results and the causal inference component. For instance, R5 commented that “all the SOTA algorithms fail,” and it is hard to quantify how agents would perform well in different tasks. R3 pointed out the lack of “qualitative results exploring the relationship between the identified and proposed causal variables,” emphasizing that ‘the benchmark is well-motivated, but not backed up with strong experimental results.‘’ R2 identified the lack of clear causal component in the paper while the paper mentions “opportunity to investigate causality” and “underlying structural causal model (SCM).” All in all, these are valid concerns.

The authors' rebuttal was quite detailed, and appreciated, but left some important questions unanswered.  The first and critical issue is about the causal nature of the simulator. The simulator's name is "causalworld" and its stated goal is to provide "a benchmark for causal structure and transfer learning in a robotic manipulation environment."  Also, the first bullet in the list of contributions is: "We propose CausalWorld, a new benchmark comprising a parametrized family of robotic manipulation environments for advancing out-of-distribution generalization and causal structure learning in RL." After reading the paper, I was quite surprised to realize there is no *single* example of a causal model, in any shape or form (e.g., SCM, DAG, Physics) or a structural learning benchmark. In other words, there is a serious, somewhat nontrivial gap between the claimed contributions and what was realized in the paper. One way to address this issue would be to make the causality more explicit in the paper, for example, by sharing the underlying structural causal model, how variables form causal relationships, what causal structures are being learned, and how these learned structures compare with the ground truth. I think these would be reasonable expectations of a simulator that aims to disentangle the causal aspect of the learning process. 

The second issue is about the experimental results in terms of generalizability. The authors emphasized on different occasions that "The primary goal of this work is to provide the tools to build and evaluate generalizable agents in a more systematic fashion, rather than building generalizable agents for the tasks specified," or "the experiments is to showcase the flexibility regarding curricula and performance evaluation schemes offered with CausalWorld, rather than solving new tasks or proposing new algorithms." These responses are somewhat not satisfactory given that the goal of the paper is providing tools to build generalizable agents, while the authors seem to suggest they are not committed to actually building such agents. Specifically, the experiments did not demonstrate the simulator as a benchmark but only showcased its flexibility (i.e., offering a large number of degrees of freedom). One suggestion would be to evaluate how algorithms (agents) with varying degrees of "generalizability" power perform across tasks with various difficulty levels. As it currently stands, the tasks are too easy or too hard for the standard, uncategorized algorithms, which makes it difficult to learn any lessons from running something in the simulator. 

Lastly, I should mention that the work has a great potential to introduce causal concepts and causal reasoning to robotics, there is a natural and compelling educational component here. Still, the complete absence of *any* discussion of causality and the current literature results hurt this connection and the realization of this noble goal. I believe that after reading the paper, the regular causal inference researcher will not be able to understand what assumptions and types of challenges are entailed by this paper and robotics research. On the other hand, the robotics researcher will not be able to understand what a causal model is and the tools currently available in causal reasoning that may be able to help solve the practical challenges of robotics. In other words, this is a huge missed opportunity since there is a complementary nature of what the paper is trying to do in robotics and the results available in causal inference. I believe readers expect and would benefit from having this connection clearly articulated and realized in a more explicit fashion.

If the issues listed above are addressed, I believe the paper can be a game-changer in understanding and investigating robotics & causality.  Given the aforementioned potential and reasons, I recommend the paper's acceptance *under the assumption that* the authors will take the constructive feedback provided in this meta-review into account and revise the manuscript accordingly. 
Decision: Accept (Poster)