Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors describe a method for endowing an artificial agent with causal reasoning  when completing goal-directed tasks. Causal reasoning knowledge is encoded in a directed acyclic bipartite graph or slightly more complicated "master switch" variation.

Essentially the method is:
  - train a model F to predict causal graphs (for which we have ground truth data) from trajectories generated with a heuristic,
  - train an policy \pi_G attending over the causal graph to solving tasks.

At inference time F an \pi_G are frozen and are evaluated on similar tasks to the ones in training, where however the (unseen) causal graph is new. Thus the evaluation is in a meta-learning scenario, where the model has to learn how to learn to solve tasks.

The experimental setting is an agent controlling 5 or 7 switches in a simulated environment and observing 32x32x3 images of the environment.

The authors report significant improvements over an existing baseline (Dasgupta et al. (2019)). The key to the improvements is in the iterative prediction of the causal graph (for F) and in having attention over the causal graph (in \pi_G).

The paper is describes a very nice new method and makes interesting points about the role that causal reasoning can play in modeling agents in goal-directed tasks. The paper is strengthened by improvements on existing baselines based on their intuitions.

Questions:

* The authors claim that constructing an explicit causal structure, instead of a latent feature encoding, leads to better generalization in “long-horizon” tasks -- but they seem to only test on one task, fairly limited in complexity.

* On page 1, the authors claim that empirical evidence suggests the lack of correct causal modeling is an important factor for lack of generalization, generation of unrealistic captions, and difficulties in transfer learning. This seems to be a bit of an overreach. It’s possible that many empirical problems could be solved to a large degree by advances in machine learning without having to resort to explicit causal modeling.

* On page 2, is the reward function r the same for all MDPs?

* Could it be said from the start that pi_I is heuristic and described in section 4.1?

* There seem to be strong assumptions on the structure of C that are only stated late in the paper. C is initially presented as an arbitrary acyclic graph, but from Figure 2 it appears to instead be a bipartite graph with N source nodes and N target nodes, and potential connections from any source node to any target node, where N is the number of actions. This should be explained early on. The “master switch” variation does not seem to exactly fit the mathematical descriptions from the “Methods” section.

* \hat{C}_H does not seem to be defined except inside Figure 2. It should be defined in the text.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: [Summary]

This paper proposes an interactive agent that tries to infer the underlying causal structure by interacting with the environment; the authors called it "causal induction."  The inferred graph will later help the agent complete goal-directed tasks referred to as a "causal inference" stage. Notably, the agent directly learns from visual inputs.

Both the induction and inference phases heavily rely upon the attention mechanism, which ensures that the agent only focuses on the relevant components of the causal graph. During the induction phase, the agent incrementally updates the predicted causal graph through each interaction using an attention-based edge decoder. During the inference phase, the attention bottleneck also showed to improve the agent's generalization ability.
 
They have shown that the proposed model outperforms several baselines in a synthetic environment that uses switches to control lights. They have also demonstrated the model's generalization ability by operating on unseen causal graphs and new task goals.


[Major Comments]

My primary concern about this work is the scope of its applicability.

For the causal induction phase, the proposed method makes a strong assumption that it can access the ground truth causal relationship during training. The authors can directly read this information from the synthetic environments used in this paper, yet, in more complex real-world situations, we might not know the underlying causal structure for supervised training the induction model.

For learning the goal-conditioned policies, the authors also assume that they have access to the ground truth causal graph. They use this information to generate the expert demonstrations, which, I presume, are deterministic and unimodal (correct me if I'm wrong). In the real world, a human may be able to infer the underlying causal structure from the observations by interacting with the environment and provide the demonstration data. However, the demonstration may be noisy or form a multi-modal distribution. While I agree that learning from demonstration is an effective way of guiding the learning of the policy, I'm not sure if the method can generalize to more realistic scenarios.

For inferring the causal graph, the authors also assume that they know the "cause" set and the "effect" set, which is already a DAG by construction. Instead of inferring the direction of the edge, they are solving an easier problem of deciding whether a directed edge between a "cause" node and an "effect" node exists or not. The assumption on the graph structure also limits the method's applicability, as, in the real world, the direction of the edge is not always known in advance. Smoking may cause lung cancer, but it is possible that lung cancer may make people smoke more.

I feel this paper makes strong assumptions on both the induction and inference stages, as well as the structure of the causal graph, which greatly limits the applicability of the approach.

[Detailed Comments]

I also have a few questions regarding the details of this paper.

In Section 3.1, the authors said that "N is the number of actions in the environments," which is a bit confusing. Before this point, the authors did not discuss the relationship between the size of the graph and the size of the action set. Only until Section 3.2 did I realize that N is the number of both "cause" set and "effect" set. It would be better to discuss the size of the graph and the action set at an earlier position.

How is the expert planner implemented? Is the expert's policy deterministic and unimodal? I feel this is an important detail to include.

It is related to the previous question. What will happen if we learn the goal-conditional policy from scratch? How much does imitation learning help with policy learning? Again, the assumption that we know the ground truth causal graph may not be feasible in the real world.

In Section 3.3, the authors said that "the expert's action is added to the memory of the policy." However, the authors also noted that "the policy has no memory" in Section 3.2, which seems to contradict each other. Does the "memory" mean replay buffer in Section 3.3?

Does the number of switches fixed across all environments and always the same as the number of lights? Also, are all the lights mounted in the same location? I'm wondering if the model is invariant to the order of the cause nodes and effects nodes in the graph, and can it generalize to larger environments, more lights, and different room configurations.

In Figure 4, TCIN seems to have the best performance in the "Masterswitch" environment when there are 500 seen causal structures. What might be the reason?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This is a paper about a very interesting topic, involving both learning (in a supervised way) to induce a causal graph and taking advantage of it in a goal-conditioned policy.

This is clearly a timely topic and I loved the motivations of the paper. My main difficulty was with understanding the actual architecture and its motivation, but I believe this is fixable but is a serious impediment to being able to evaluate the paper, as it stands. I was a bit disappointed to see that training is mostly supervised (both providing the ground truth causal graph and an oracle policy as target) but on the other hand it is impressive to obtain these results with raw images as input and the comparative results are good.

First, I would like to better understand the insight behind the architecture of F and several things would need to be clarified to enable reproducibility and making sense of the equations. I would start by suggesting to add an example illustrating why simply seeing a (state,next-state,action) triplet is sufficient to obtain a bit of evidence in favour of a particular edge of the graph. Since this is supervised learning of the causal graph, I imagine that the semantics of the node is predetermined, which is a bit disappointing (but doing otherwise would be understandably much more challenging). Second, I don't understand the structure of the causal graph C. What are the input nodes? action values? action x state cross-product? What are the output nodes? Effect variables? Why would C be NxN and not have different input and output dimensions? All this really needs to be clarified. Based on the 1st eqn of page 4 (PLEASE NUMBER YOUR EQUATIONS!!!) it looks like C is number of actions by number of actions, which does not seem consistent with any reasonable interpretation. The authors should also clarify how R is computed (if it is a straight difference of the encoder output, put up an equation for example) and how delta e is computed.

Then in sec 3.2 the authors talk about a weighted sumn involving selected edges. I imagine this is soft-attenetion but it needs to be clarified with equations and explanations. What is the "content" associated with each edge e which gets averaged in the soft attention?

I found a possibly interesting parallel between the focus of attention on one edge of the graph  at a time (figure 3) and the ideas of the "Consciousness Prior" (Bengio 2017, on arXiv) bottleneck (where only a small tuple of variables, corresponding to an edge here, is considered at each time step in order to reason, plan, decide etc). The fact that in the experiments this attention mechanism helps seems to support the sparsity of dependencies hypothesis underlying the consciousness prior.

My rating is weak reject but I am ready to upgrade with appropriate explanations answering the above questions.

--- post-rebuttal  addition ---

The authors have satisfied most of my concerns and I have upgraded my rating to weak-accept.


review_title: Official Blind Review #2

Meta-review: The submission presents an approach to uncovering causal relations in an environment via interaction. The topic is interesting and the work is timely. However, the experimental setting is quite simplistic and the approach makes strong assumptions that limit its applicability. The reviewers are split. R2 raised their rating from 3 to 6 following the authors' responses and revision, but R1 maintained their rating of 3 and posted a response that justifies this position. In light of the limitations of the work, the AC recommends against accepting the submission.
Decision: Reject