Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a set of RL environments that are explicitly designed to study using causal learning in model-based reinforcement learning. Different from previously purposed RL environments, the environments shown in this paper concern mainly on the causal graph underlying the environment and can variate on the structure, size, edge sparsity, length of cause-effect chains. These environments provide enough complexity to investigate causal learning in model-based RL. The experiments and results aim to study what models are more suitable to leverage the inductive bias in the environments. The results shows that structured models outperform monolithic models in both dynamics learning and RL learning performance. 
strengths: This paper provides extensive experiment results on how different model structures perform on model-based reinforcement learning and it can be a nice reference for future MBRL work. It does a good job on evaluating these models on different perspectives, including model learning and downstream RL tasks learning.
weaknesses: As mentioned by the authors, I think the purposed environments can be improved by considering delayed effect, which is more common in real life. The current environments only consider effects that happen immediately after the cause. Furthermore, I think more efforts can be made on the RL side. The current work uses an algorithm that predicts the rewards of each step. Applying more advance RL algorithm can show how modern algorithms are performing with different dynamics models.
correctness: I agree with the claims made in this paper and the environments are build in a sound way.
clarity: I think the paper is well-written. However, only small amount of results are including in the paper and the authors leave most of the results to appendix. I think more results and analysis can be added to the main paper (e.g. RL performance).
relation to prior work: It is clear that this paper focus on environments that variates on causal graphs while previous works focus on variating on various axises in an MDP. 
additional feedback: 1. In the physics environment, isn't the FixedUnobserved setting observable since the order of weights can always be inferred by the shape? They are just "unobserved" by color.
2. The success rate of the chemistry environment is generally low. What is the key insight of this? 
3. The reward prediction results are missing.
4. I think this paper can be greatly improved by having more results and analysis in the main paper.
review_title: An extensive study on causal learning with model-based RL

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper proposes a suite of environments for evaluating causal discovery in deep learning, with a particular focus on applications to MBRL. These environemnts evaluate the ability of algorithms to implicitly "parse" improperly factorised observations into variable values for an underlying SCM, even in the presence of challenges like confounders. Experiments evaluate the accuracy of the internal dynamics representations learnt by various network architectures by measuring the extent to which they make it easy to predict the outcomes of interventions.
strengths: The paper works on a problem of current interest to the RL (and more generally deep learning) community. It also includes an exhaustive evaluation of a chosen set of network architectures on the proposed environments.
weaknesses: I found the paper extremely difficult to understand, and seems to be missing several pieces of information that are essential to understanding what the benchmark does and whether the results are reasonable. I've covered my main concerns in the clarity section below.
correctness: I found this difficult to evaluate due to clarity issues, so suspending judgement on this for now.
clarity: Here are some of the higher-priority clarity issues that made it hard for me to evaluate this paper:

[C1] What are the observation spaces for the two environments? I gather they use image-based observation spaces because there is mention of convolutional networks, but how are the images produced? Are they the same as the pictures in Figures 3/4/etc.? If so, how big are the pictures when fed into the network? The precise structure of the images is important, since it ties into the motivation in the introduction about not having access to an appropriate factorisation of the SCM when doing learning. If it's trivial to parse images into an appropriate factorisation with, e.g., PCA, then having an image space doesn't seem to add much difficulty over having a structured representation.

[C2] Regarding the physics environment: what is the action space? F.2 talks about how an action can select a single block, but that can't be right, because it seems like the interventions have something to do with pushing _pairs_ of blocks (which presumably requires the ability to select a source block and a target, or at least a push direction). Also, in this environment, are positions 1D or 2D? Discrete or scalar? How does "pushing" work, physically?

[C3] The reward functions for both environments are defined in Appendix D.1 (methodology for reward prediction). These are really important and should be in the body of the paper!

[C4] If I understand the "unobserved" variant of the physics environment correctly, it seems like the challenge is that the agent gets some training data that (implicitly?) shows how certain colours map to weights. It is then confronted at test time with an unseen colour, for which it cannot possibly know the weight. Thus it must perform an intervention. This is an interesting challenge, but it's not the same as unobservability in the Pearlian (Bayes net/SCM) sense, where something being "unobserved" means that the agent does not have access to a particular variable in the SCM _at all_. The challenge here seems to be about generalising despite missing data rather than unobservability of some part of the SCM.

[C5] How do the Physics and Chemistry environments relate to physics and chemistry (the fields)? I was looking for physical and chemical intrepretations of the environments while reading the paper, but the environments were too abstract for me to understand in those terms and I may have misled myself somehow.

[C6] In the chemistry environment, why have distinct shapes and positions if they are fixed across all episodes? (line 764) Why not just have four coloured pixels?

[C7] Why does section D use an ad-hoc "one-step planning" approach instead of an existing RL algorithm? The current approach seems harder to implement and harder to explain than an off-the-shelf RL or MBRL algorithm.
relation to prior work: Discussion of related work was adequate.
additional feedback: I don't understand what this paper is doing well enough to give a confident evaluation. For now my rating is a weak reject with low confidence.

------

(review last checked/updated 2021-07-19)
review_title: Potentially interesting benchmark that suffers from a lack of clarity

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper designs a suite of benchmarking reinforcement learning environments to systematically probe the ability of methods in learning representations of high-level variables as well as causal structures among them.

The authors design two suites of environments, the physics environment and the chemistry environment.

The authors evaluate various representation learning algorithms from the literature and find that explicitly incorporating structure and modularity in models can help causal induction in model-based reinforcement learning.
strengths: This paper handles a significant problem of modulating environment causality structure and evaluating models's capacity to capture it.

The proposed RL development environment allows the user to choose and modulate various properties of the ground truth causal graph.

The authors evaluate two monolithic models(autoencoders and variational autoencoders) and two typical models with explicit structure: GNNs and modular models using their proposed platform.
weaknesses: The novelty of this paper is a little bit limited, since it only benchmarks several existing methods.

Missing references: Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems.
correctness: Yes, the evaluation methods and experiment designs are appropriate and performed correctly.
clarity: Yes, the paper is well written.
relation to prior work: Yes, relation to prior work is clearly discussed.
additional feedback: -
review_title: A new benchmark for studying causal induction for visual model based reinforcement learning

Meta-review: This paper introduced a new benchmark to systematically evaluate causal learning in visual model-based reinforcement learning. It addressed an important yet less studied problem. At the initial reviews, two of the reviewers (9jsz and LJ2q) leaned positive, and the third reviewer (Taj6) leaned negative. After the author response period, the third reviewer voted “weak reject with low confidence" while raising valid concerns about the writing clarity issues. The discussions did not sway this reviewer's overall assessment. Reviewer LJ2q has expressed concerns about the novelty of this benchmark over existing datasets, which the authors explained in their responses. The AC read the paper, the reviews, and the discussions carefully. The AC believed that this paper had addressed an important research direction. Nonetheless, the writing quality of this manuscript could significantly benefit from another round of revision to pass the bar of acceptance. Therefore, the AC would highly recommend the authors incorporate the reviewers' feedback and resubmit this work in round 2 of this track, and would expect this paper to be highly likely accepted if the next revision improves the writing quality.
Decision: Reject