Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Summary: the paper proposes an algorithm for learning causl strutures from the environment and using it to act and plan. The papers presents experiments on OOD generalization on either spurious correlations or compositional generalization.
strengths_and_weaknesses: Pros:

- Causal models hold promise to better generalization, especially in RL. Causal models focuses on the right causal variables as well as causal mechanisms between variables.  Hence, this paper addresses an very important problem, espeically in RL.

- The experimental results are interesting, especially the OOD generalization experiments. 

Cons:

There are some limitations of the work that would be nice to be addressed.

- One of the main claims of the paper is the identifiability proof in that the algorithm can always identify the true graph. However, this proof is based on assumptions that generally may not hold true, espiecally in high-dimensional data. Another major reason why causal identifiability is so challenging in low-level data is that the causal variables are often latent, making the interventions unknown in the latent space. this is not discussed in the paper.

However, the proof that the authors presents in the paper assumes 2 things, one being that the mapping from the low-level observational space to latent causal variables are trivial and can be easily obtained, and second, that the independence test has perfect information, in that it can tell perfectly which variable cases which other variable upon intervention.

I think both of these assumptions generally do not hold true. Discovering latent states is non-trivial, and Im not sure if faithfulness generally holds either. 

Another point that the authors have not discussed is that the interventions are unknown. The intervention is only observed on low-level images, but it's unknown to the abstract (latent) causal variables. I do not see a discussion on this anywhere in the paper.

- The experiments are interesting. However, the spurious correlation experiments can be solved without learning the underlying causal mechanisms. The model would only need to discover the causal variables (but not necessarily mechanisms). The compositional generalization tasks that the authors proposed is more interesting, but again, the agent doesnt  necessarily have to know the undelrying causal graph in order to generalize. For example, stacking 2 objects and stacking 5 objects as proposed by the authors. A model that has not learned the underlying causal structure of the environment might also be able to generalize to this.

- It's also unclear if the underlying causal structure of the environment for the stacking and door opening experiments are "interesting", in the sense that changing the color or backgrounds of the object has no causal influence on anything else in the env. 

- In order to test to see if the model has indeed captured the underlying causal structure, I would propose that the authors use either 

1. "Alchemy: A structured task distribution for meta-reinforcement learning"
2. "Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning"

Both of these environments have more complicated and interesting causal graphs. There is also a set of interesting OOD experiments that can be performed on these experiments.
questions: - It's unclear if the underlying causal structure of the environment for the stacking and door opening experiments are "interesting", in the sense that changing the color or backgrounds of the object has no causal influence on anything else in the env. 

- Can't the spurious correlation experiment be solved by only focusing on learning what is a causal variables (compared to learning the causal mechanisms).


limitations: One of the major limitations of this paper is that the theoretical results the paper presents is based on somewhat unrealistic assumptions. However, this is not discussed properly in the paper.

 One of the main claims of the paper is the identifiability proof in that the algorithm can always identify the true graph. However, this proof is based on assumptions that generally may not hold true, espiecally in high-dimensional data. Another major reason why causal identifiability is so challenging in low-level data is that the causal variables are often latent, making the interventions unknown in the latent space. this is not discussed in the paper.

However, the proof that the authors presents in the paper assumes 2 things, one being that the mapping from the low-level observational space to latent causal variables are trivial and can be easily obtained, and second, that the independence test has perfect information, in that it can tell perfectly which variable cases which other variable upon intervention.

I think both of these assumptions generally do not hold true. Discovering latent states is non-trivial, and Im not sure if faithfulness generally holds either. 

Another point that the authors have not discussed is that the interventions are unknown. The intervention is only observed on low-level images, but it's unknown to the abstract (latent) causal variables. I do not see a discussion on this anywhere in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Paper investigates the problem of model-based RL or planning. It proposes to decompose the transition model into a causal graph and some parameterized functions. The causal discovery here is formulated as learning a latent variable model using VI, where the causal graph is the latent variable. Then the transition loss is used to find the best set of parameters. Experiments on several synthetic domains including one designated for causality show some promises over state-of-the-art model-based RL avenues.  
strengths_and_weaknesses: [Stengths]

+Paper is overall clear and well-written. The authors also provide the code and indeed it helps with understanding the idea here.

+I found the formulation intuitive and reasonable. The math on identifiability and performance guarantee could be a bit dense but I didn't find any major issue.   

+The advantages over MBRL baselines are impressive, especially in the challenging car crash domain. With the code already there, I'm sure the results are reproducible if the authors can release it at some point.

[Weaknesses]

I don't have major concerns at this point. But I will appreciate it if the following questions can be addressed in a rebuttal.

-The authors claim that their approach is for "goal-based RL". But it seems that goal is not necessary for all their formulations and implementations. Why not just call it model-based RL or planning in general? Some clarifications here are needed.

-The causal discovery part might be clear in the formulation but the implementation details are yet to be unveiled. I also check the code, but as far as I could find the causal graphs are **pre-defined**. The authors are encouraged to elaborate more on this and some pointers to the code will be helpful.


questions: See [Weaknesses]

limitations: The authors mentioned that their limitation could be the inability of scaling to causal graphs with more nodes. However, I also doubt whether this approach could be applied to real-world domains with non-symbolic input, ex. pixel-based or other raw sensory input. Extracting symbolic input from these sources could impose new challenges on observation uncertainty, which cold be pivotal to causal discovery and model learning.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to augment Goal-Conditioned RL (GCRL) with Causal Graph (CG), and treats the causal graph as a latent variable. 
The paper then proposes to use  interventional data to estimate the posterior over the underlying causal graph, and then proposes to use the causal graph to learn factorized action conditioned model and agent's policy. 


=====

After rebuttal: I've read the rebuttal, and I remain by my rating. 
strengths_and_weaknesses: **Clarify**

- The paper is easy to read and well written. 

**Originality**

- The individual elements are known in causality literature, and various other works (most of them are cited by the authors). The contribution lies in using these ideas (for ex. planning with a factorial action conditioned model) in the context of goal conditioned policies. 

**Significance**

- The reviewer like the experiments where the authors focussed on using the transition models for doing planning. The reviewer believes it's a step in the right direction. 

**Quality**

- The reviewer likes the experimental setup as it's tailored to test generalization (robustness to spurious correlations and compositional generalization).

questions: **Model learning**

- The authors proposes to represent the transition model, with an ensemble of recurrent models, such that they can take into account variable number of "causal parents" for each node of the causal graph. Such a design choice is commonly used in causality papers (already cited by authors [1, 2], where the authors learn structural and functional parameters. Structural parameters corresponds to a soft-adjacency matrix, and functional parameters correspond to a set of MLPs which take in input a set of causal parents. This design choice has also been used as an inductive bias for learning architectures which consists of ensemble of recurrent models, such that each model can decide which other models to pay attention to via the use of attention [3, 4]. 

[1] Differentiable causal discovery from interventional data, https://arxiv.org/abs/2007.01754  \
[2] Learning neural causal models from unknown interventions, https://arxiv.org/abs/1910.01075 \
[3] Neural Production Systems, https://arxiv.org/abs/2103.01937  \
[4] Recurrent Independent Mechanisms, https://arxiv.org/abs/1909.10893 

**Chosen Method for  learning causal graph**

It may be useful to employ gradient based methods of learning causal graph as compared to methods based on independence testing. Recently, few methods have been proposed which allows inferring posterior over the learned causal graph [1, 2, 3].

[1] DiBS: Differentiable Bayesian Structure Learning, https://arxiv.org/abs/2105.11839 \
[2] BCDNets, https://proceedings.neurips.cc/paper/2021/hash/39799c18791e8d7eb29704fc5bc04ac8-Abstract.html \
[3] Bayesian Structure Learning with Generative Flow Networks, https://arxiv.org/abs/2202.13903 


**RL benchmarks for causal reasoning**

It may be useful to evaluate the proposed method on set of problems which are specifically tailored towards measuring generalization in RL (in causal sense) [1]. It may also be useful to evaluate an "attention" (or GNN) based baseline which learns interactions between different causal variables to predict the future state for planning. 

[1] Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning, https://arxiv.org/abs/2107.00848 


limitations: **Limitations**

The paper discusses two limitations: (a) the method may not scale well to large number of nodes, (b) assuming factorized state and action space restricts the scope to scenarios where only observables are low level information i.e., pixels in the case of image data. It may also be useful to discuss that **independent testing** i.e., the method used to learn causal graph may not scale well as compared to gradient based approaches of learning causal graph.
review_title: null

Meta-review: metareview: Promising direction for incorporating casual reasoning and RL   although the scope of theory and experiments seems limited. 
confidence: Less certain
Decision: Accept