Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper considers the problem of predicting a variable y given x where x may suffer from a policy change, e.g., x may follow a different distribution than the original data or suffer from a confounding variable. The flow of the paper proceeds in learning a causally correct model in the sense that the model is robust to any intervention changes. Specifically, the paper considers a setting called "partial model" meaning a generative model conditioned on functions of past observations. To make the partial model causally correct, the paper considers the partial model conditioned on the backdoor that blocks all paths from the confounding variables. 

1. The problem that this paper addresses seems to be new and interesting. The approach makes much sense: the problem is due to the confounding effect which can be addressed by introducing some other variables that implicitly blocks the confouders.  

2. The  paper assumes the existence of the backdoor variable which is crucial for causal correctness. Does the backdoor always exist? Pearl's book may have some discussions on this. It will be still useful to include some materials in case of unfamiliar readers. 


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: SUMMARY:
The authors apply ideas from causal learning to the problem of model learning in the context of sequential decision making problems. They show that models typically learned in this context can be problematic when used for planning. The authors then reformulate the model-learning problem using a causal learning framework and propose a solution to the above-mentioned problem using the concept of "backdoors." They perform experiments to demonstrate the advantages of doing so.

MAJOR COMMENTS:
Overall, I'm positive about the submission, though I think there's room for improvement.

The paper is well-written and does a very nice job of formulating the model-learning problem through the lens of causal learning, and it is convincing that the baseline model-learning procedure suffers from confounds. Moreover, the proposed modified procedure for model learning would seem to have the potential to fundamentally and positively impact model-learning in general. Finally, I found that limited experiments supported the points made in the paper.

That said, I think the paper as written could be substantially improved with a little extra effort. First, it does not -- in my opinion -- pass a basic reproducibility test, i.e., I am not confident that I could implement the proposed modified model-learning procedure even after reading the paper. The authors should seriously consider adding an algorithm box with pseudocode to both show the flow of training data to the model-learning steps and also how planning can be accomplished with the learned models.

Second, the paper would greatly benefit from any discussion (theoretical, experimental, or -- ideally -- both) of the tradeoffs that would arise when considering the proposed technique. What price is paid for the gain in planning accuracy? Is more data required for learning compared to non-causal approaches? If such a price is incurred, how was this made clear in the presented results?

Finally, an important aspect of the experiments seems to have gone un-discussed. Namely, what is the reason for the behavior of the proposed technique in Figure 4(a)? From much of the paper, I would have expected the red dots to live entirely on the horizontal dotted line $V^*_{env}$, but instead this only happens when the behavior policy has the same value. Why is this? Moreover, why do *better* behavior policies result in *worse* performance for the optimal model evaluation policies?

MINOR COMMENTS:
    * The authors should define "par_k", first referenced at the bottom of p3, before using it. Similarly for $\psi_k$
    * p5, second paragraph. I think there's a typo in "... sample $y_2$ from $q(\cdot | h_1)$ ..." (should probably be conditioned on $h_2$ not $h_1$).

POST-RESPONSE COMMENTS:
In my opinion, the authors adequately addressed both my own concerns, and also several valid concerns from the other reviewers. Therefore, I'm raising my score to "accept."
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper tackles the issue of identifying the causal reasoning behind why partial models in MBRL settings fail to make correct predictions under a new policy. The novel contribution was a framework for learning better partial models based on models learning an interventional conditional, rather than an observational conditional. The paper tried to provide both theoretical and experimental reasoning for this framework. 

I vote to (weak) reject the paper due to the major issues with section 2. Furthermore, the paper hard or at times almost impossible to understand as too many assumptions are made and too little is explained.

Recommendations

Because your graphs are not MDPs, you are not framing your example as an RL problem. This is causing a number of issues with notation and lack of clarity in the argument you're making.

1. It is unclear to me that the FuzzyBear example is correctly constructed as a RL example, reasons being that:  
- Figure 1 (a) & (b) do not correspond to an MDP as two different states, teddy vs grizzly, are both designated as s_1 and similarly, the two possible actions, hug or run, are both designated as a_1 and thus are not distinct.
- Note your terminal state for the episodes
- Have a reward for (s0, a0) as every s-a pair should have a reward
- It would be helpful to note that the environments in Figure 1 are stochastic

2. Clarify notation. There are a number of assumptions about what background knowledg the reader should have. Given the bridging of disciplines in the paper, it would be useful to provide more detail on notation in Section 3.

3. Add a section on reinforcement learning in Section 3. If it's the last subsection in section 3, you could describe the relationship between the various causal reasoning and RL principles. This would further clarify how you're bridging these subtopics.

4. For sentence,

"Fundamentally, the problem is due to causally incorrect reasoning: the model learns the observational condi- tional p(r|s0, a0, a1) instead of the interventional conditional given by p(r|s0, do(a0), do(a1)) = s1 p(s1|s0, a0)p(r|s1, a1)."

As you don't cover the meaning of the do() operator until a later paragraph, provide a quick description of it as it is not common knowledge to a general AI audience, e.g., where do() indicates that the action was taken.

5. Correct the following sentences,

"Mathematically, the model with learn the following conditional probability:"

"In Section 3, we review relevant concepts from causal reasoning based on which we propose solutions that address the problem."

6. I recommend putting the interventional conditional equation, p(r|s0, do(a0), do(a1)) = 􏰀s1 p(s1|s0, a0)p(r|s1, a1), on its own line as the reader is doing a comparison of it with the previous equation, p(r|s0, a0, a1), given on page 2.

7. Strengthen your abstract by aligning more with claims you make in your conclusion.

8. The experiments in Figure 5 are averaged over 5 seeds. This is not enough to be statistically significant - furthermore, there are no error bars in the Figure.

Question(s):
1. You've indicated two policies for Figure 1 (a):
- pi1: the agent knows it is encountering a teddy bear, so it will hug
- pi2: the agent knows it is encountering a grizzly bear, so it will run
Is this the "change in the behavior policy" that you're referring to? If so, make this clearer, this currently requires a lot of work by the reader to make sense of it.

2. What are the partially observable parts of the environments in Figure 1 (a) & (b)? Make this clear.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: *Summary*

This paper considers the effect of partial models in RL, authors claim that these models can be causally wrong and hence result in a wrong policy (sub optimal set of actions). Authors demonstrate this issue with a simple MDP model, and emphasize the importance of behavior policy and data generation process. Then authors suggest a simple solution using backdoors (Pear et al) to learn causally correct models.They also conduct experiments to support their claims.

*Decision*

I vote for rejection of this paper, based on the following argument:

To my understanding authors are basically solving the “off-policy policy evaluation” problem, without relating to this literature. For example, the MDP example is just an off-policy policy evaluation problem, and it is very well known that in this case you need to consider the behavior policy, for example with importance sampling.
Even authors definition of the problem at the end of the page 4, and beginning of page 5, is the problem of “off-policy policy evaluation” when y_t = r_t 
Authors have not cited any paper in this literature, and did not situate their work with respect to this literature. To my understanding, the proposed solution is basically importance sampling, that is very well known and studied in the field.

Additionally, I suggest that authors be more careful with their citations, for example, authors cited Silver et al 2017 [The Predictron: End-To-End Learning and Planning], as one recent paper using the method; however Silver et al 2017  is in MRP setting (Markov Reward process) where there is no action, so the described problem setting doesn't apply.  

Improvement

The current manuscript needs a major revision, mainly 1) situate the work with respect to off-policy policy evaluation literature, and then 2) Considering step 1, a clarification for what is the novelty/ contribution of the current paper is needed. 

review_title: Official Blind Review #1

Meta-review: The authors show that in a reinforcement learning setting, partial models can be causally incorrect, leading to improper evaluation of policies that are different from those used to collect the data for the model.  They then propose a backdoor correction to this problem that allows the model to generalize properly by separating the effects of the stochasticity of the environment and the policy.  The reviewers had substantial concerns about both issues of clarity and the clear, but largely undiscussed, connection to off-policy policy evaluation (OPPE).  

In response, the authors made a significant number of changes for the sake of clarity, as well as further explained the differences between their approach and the OPPE setting.  First, OPPE is not typically model-based.  Second, while an importance sampling solution would be technically possible, by re-training the model based on importance-weighted experiences, this would need to be done for every evaluation policy considered, whereas the authors' solution uses a fundamentally different approach of causal reasoning so that a causally correct model can be learned once and work for all policies.

After much discussion, the reviewers could not come to a consensus about the validity of these arguments.  Futhermore, there were lingering questions about writing clarity. Thus, in the future, it appears the paper could be significantly improved if the authors cite more of the off policy evaluation literature, in addition to their added textual clairifications of the relation of their work to that body of work.  Overall, my recommendation at this time is to reject this paper.
Decision: Reject