Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: === Summary

This paper proposes a benchmark that aims to systematically evaluate models' ability in learning representations of high-level variables as well as causal structures among them. The authors introduce two benchmarking RL environments:
- One is in a physical domain where an agent is pushing blocks of different weights.
- Another one is to simulate a chemistry environment, where the state of an element can cause changes to another variable's state according to the underlying causal graph.

The authors evaluate several representation learning algorithms from the literature and find that explicitly incorporating structure and modularity in models can help causal induction in model-based reinforcement learning.


=== Strengths

This paper targets an important problem of assessing models' ability to automate the inference and identification of the causal variables from high-dimensional inputs like images.

The construction of the benchmark allows building causal graphs with varying complexity, such as the size of the graph, the sparsity of the graph, and the length of cause-effect chains.

The authors have evaluated several baseline models on the benchmark, including two typical monolithic models (autoencoders and variational autoencoders) and two models with explicit structure: graph neural networks (GNNs) and modular networks.

They have made several interesting observations, e.g., modular networks hold better scalability than other baselines, suggesting the benefits that explicit structure and modularity bring for causal induction in MBRL.


=== Weaknesses

My primary concern of this paper is that the dataset is a bit too contrived, which makes it hard to know whether the observations from this benchmark can generalize to more complicated real-world scenarios.

For example, in the Physics Environment proposed in this paper, only heavier objects can push lighter ones, not the other way around. I understand the authors' desire to make the underlying graph a directed acyclic graph (DAG), but it does not reflect what will happen in the real world. One could imagine sliding a lighter object to collide with a heavier one; the motions of both objects are likely to change, where the interaction between them is bi-directional.

Also, in the Chemistry Environment, a few objects are connected by a randomly generated DAG, where interventions can change the color of the target and the subsequent blocks. In chemistry, a molecule is a group of atoms held together by chemical bonds, which are also bi-directional relationships. Are there any specific examples in chemistry where the graph is a DAG? I would appreciate it if the authors can elaborate on the connection between the design of the environment and "chemistry."

In terms of the difficulty of the tasks, the results shown in Figure 6 suggest that the Greedy algorithm can achieve a near-perfect performance on the tasks, which consistently outperforms all other baselines. If a simple greedy algorithm can solve the tasks, does it mean that the benchmark may be a bit too simple, where a good understanding of the underlying causal structure may not be necessary? It would be better if the authors can discuss the necessity of causal induction in these tasks, and how is the ability to perform causal inference correlate with the metrics used in the benchmark.

Overall, I feel the environments proposed in the paper are a bit too artificial, which does not reflect what's likely to happen in the real world. While I like the goal of this paper, I think a set of more realistic environments could greatly improve the significance and potential impact of this paper.



=== Other comments


The font size of the image caption may be a bit too small.

Typo: Section 2.1, "Impotant" --> "Important"


=== Post rebuttal

The authors' rebuttal addressed some of my concerns, but my primary concern still remains that that benchmark may be a bit too contrived, where the observations made in this paper may not generalize to more complicated real-world situations. The authors also made some far-fetched arguments in the rebuttal by claiming some concurrent works [1, 2] as "the 'real-world' version of the environments used in the paper," which, to be honest, further lowers the rating of the paper on my side: why is this paper worthy of acceptance if there exist more realistic benchmarks?

I also agree with R1 and R3 that there are no new methods proposed in the paper, and the insights derived from benchmarking a set of existing methods may not be considered novel from the point of view of the ICLR audience. As a result, I keep my rating the same.

[1] Physically Embedded Planning Problems: New Challenges for Reinforcement Learning, https://arxiv.org/abs/2009.05524

[2] CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning, https://arxiv.org/abs/2010.04296
review_title: This paper introduces an interesting benchmark with an ambitious goal, but the tasks may be a bit too contrived.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper investigates the importance of incorporating structure and modularity in MBRL. Specifically, it compares monolithic models with models with explicit structure: GNNs and modular models. It also investigates the influences of varying complexity of graphs. 

Pros:
1. The authors state a general goal in the abstract: "the joint discovery of abstract representations and causal structure", which is promising and needs more investigation for sure.

Cons:
1. The writing is a bit confusing. From the title and the abstract, the readers may expect that this paper is about causal discovery, i.e., learning casual relationships from data, while the casual structure actually is given in the main text.  
2. In addition, this paper only compares existing models with or without structures, but I fail to see any novel or interesting ideas that this paper tries to deliver. 
3. Furthermore, from the experiments, the authors found that increasing the size of the graph impacts the performance significantly, e.g., increasing 3 nodes to 5 nodes. This result surprises me. I think if the structure is handled properly, the increase from 3 nodes to 5 nodes should not have such a big effect. The authors may investigate or propose other methods that make proper use of the structures. 

Based on the above reasons, I do not think this paper is ready to publish.


Post-rebuttal:
Thanks for the feedback. The goal of learning causal representation is ambiguous, and it is absolutely a good research topic. However, I fail to see an obvious contribution of the current version. Researchers in this field are usually clearly aware of the limitations for existing methods in causal learning. The problem is how to handle it, e.g., how to give a appropriate definition of the causal variables, how to theoretically show the identifiability and consistency, and then propose a practical solution. 


review_title: Right direction; writing is confusing; just comparing existing methods, but no new method

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper is a review of model-based approaches of integrating causal inference to reinforcement learning (RL) in different environments (application areas). The authors provide software to analyse how three types of models (“monolithic”, i.e. latent space models without a graph-like structure of the latent space, graph neural networks (GNN) and “modular”, i.e. the C-SWM model (Kipf et al., 2020)) perform in two artificial “environments” devised by the authors (physics and chemistry) based on a number of metrics, some of them also proposed by the authors. The main contributions are the platform for evaluating models in the environments and the insights from the experiments performed on the selected models (taken from existing literature).


*****Strengths:*****

The paper is very well-written, clear and easy to follow.

The paper provides a novel perspective on applications of causality modelling and the presented environments could be useful to practitioners in the respective fields of physics and chemistry.


*****Weaknesses:*****

As the paper has the form of a review and a collection of insights, there seems to be no novelty from the point of view of either machine learning, causal inference or reinforcement learning. I am therefore not sure whether ICLR is the right venue for this paper.

While the motivation for the causal problem tackled in the paper (inferring model-driven causal insights usable for RL agents from data available to agents/robots such as images) is well-defined, interesting and relevant, there seems to be little actual follow-up on it in the paper. The presented models do not seem to adhere to any causal formalism (Pearl’s graphs, Rubin’s potential outcomes) and their only “causal” interpretation is a naïve use of structures (graphs and DAGs) as descriptors of causal relationships without any formal justification. There appears to be no relation to causal structure learning. Neither GNN nor C-SWM were conceived as causal models, and if one wishes to use them as causal models, more justification for such use is needed (why should the discovered graphs correspond to some underlying causal relationships?). Moreover, there have been approaches to achieve what this paper seems to promise (discover causal relationships from images), e.g. (Discovering causal signals in images, Lopez-Paz et al., 2017).


*****Post Rebuttal*****
I would like to thank the authors for the detailed rebuttal.

The authors state: "The main goal of our paper is NOT to introduce novel models, but rather to introduce a NOVEL benchmark and insights/ingredients to study causal induction in model-based RL" and "It is true that the models we use do not learn an explicit structure for causal learning" which corresponds to my original reservations to the novelty of this paper. The authors introduce a benchmark / software for evaluating causal induction in RL models, where the user can specify a causal model and its influence on the environment can be examined. I remain unconvinced that the introduction of a RL evaluation benchmark (even one allowing for the presence of arbitrary causal networks) counts as novel at ICLR.

Further, the statement "we used some of the typically common models for this purpose, such as GNNs and modular networks. Though these models do not learn an explicit causal graph, they do learn structure that could allow them to discover causal relationships under certain assumptions" confirms my point that no causal formalism (e.g. connection to the data generating process) is accounted for. It merely means that directed relationships can be modelled.

I agree with the authors that (Lopez-Paz et al., 2017) only uses observational data and does not have any connection to RL, but it is an example of an approach to extracting causal relationships from images in a sound way when it comes to causal inference. This is a side comment and does not influence my assessment of the paper.

To sum up, my reservations towards the degree of novelty in this paper have not changed (I agree with the authors' summary of their contributions, but disagree as to whether proposing a new benchmark constitutes novelty at ICLR) and I recommend a rejection of the paper in its current form.

review_title: Software for reviewing performance of integrating causality to RL in a number of application-driven environments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper proposes a suite of RL benchmarks to facilitate inducing causal relationships from visual observations. They show that structural inductive biases are beneficial for causal relationship learning and model-based RL by testing a variety of representation learning algorithms on this benchmark.

##########################################################################

pros: 

+ The motivation is clear and interesting. Inducing causal relationships from pixel-level observations is an important topic to integrate causality into RL.

+ Overall, the paper is well written.

+ A comprehensive evaluation is conducted to highlight the usage of the proposed benchmark.

##########################################################################
 
cons: 

- Would you explain more about the physics environment about why the underlying graph is a causal graph, not just a statistical graphical model?

- Why is PHYRE[1] benchmark not mentioned in the related work? You seem to work in the same direction and they have proposed more complicated and interesting tasks.

[1] Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., & Girshick, R. (2019). Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems (pp. 5082-5093).


##########################################################################

Post rebuttal

I'm happy with the author's response and would like to keep my original score.
review_title: Another benchmark to facilitate causal induction for RL with pixel-level observations.

Meta-review: This paper proposes a suite of benchmark visual model-based RL tasks to evaluate causal discovery approaches under systematically varying causal graphs. Despite some disagreement on this point among reviewers, I would come down on the side of saying that a better-executed version of this paper would have been a good fit at ICLR. However, its current drawbacks make this a borderline reject. The most important of these drawbacks is: it is unclear to what extent results on these simple environments translate to more realistic complex ones.

Reviewers have also pointed to omitted relevant work that could be discussed in future versions, such as PHYRE. Another relevant benchmark in this vein: https://arxiv.org/abs/1907.09620


Decision: Reject