Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method called "counterfactual regularization" whereby the dynamics/transition model is encourage to not have degeneracies where the actions don't influence the state transitions.  Concretely, this is done by, for every state, computing the maximum deviation of Transition model under a different action than the action taken in the history, and encourage that deviation to be as large as possible.  If that maximum deviation is 0, then all actions lead to the same next state. Empirical results show reasonable improvements in the StarIntruders task.

My biggest complaint (and the only one barring me from supporting acceptance) is that I don't see the body of results as scientifically solid.  Example of additional results that I would find much more convincing are:

-- Experiments on more than one environment.  Currently, this paper should be judged solely for its empirical improvements, because there is little formal analysis or rigorous derivations.  But it's hard to judge that based on only one experiment.

-- Deeper investigation into the effects of counterfactual regularization, including its interaction with learning disentangled representations.  Right now, there is no investigation, just a numerical score of reward attained.  This does not lead to much scientific insight.

-- Exploration of the limitations of the approach.  Personally, I think this approach is reasonable for video games with a few discrete actions, but quickly runs into problems for more complex action spaces.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper considers regularization based on "counterfactual" trajectories.
Namely, it suggests two losses, action-control and disentanglement regularization.
It experimentally evaluates the benefits of such regularization in the StarIntruders environment.

The paper is well written and explained.

Issues:

1) Authors evaluated the two suggested regularizations in separate. 
I would like to also see numbers from a combination of these.

2) I think the related work is missing a large line of work on "auxiliary tasks".
It seems to me that this paper would exactly fit within that scope?

3) My main issue is the evaluation.
The evaluation is done on a in-house game and compares to very few methods.
For a paper that has very little theory and thus most of the value is in the empirical evaluation, I think that is a problem.
If authors opted for example for Space Invaders (they do say it is similar) or simply more games, one would have many more existing numbers to compare against.

Minor issues:

1) The first regularization - action control regularization is motivated by the idea that there is always an action that changes the state. While true for most environments, this does not hold in general.

Summary:

Overall, this paper has potential but I don not believe is good enough - I suggest a reject.
The main problem is that the idea is relatively simple, there is no theory and thus the crucial piece of the paper has to be the empirical evaluation.
And the evaluation only compares to a single method with no regularization, no auxiliary tasks and reports only experiments on a single game.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper presents regularization techniques for model based reinforcement learning which attempt to build counterfactual reasoning into the model. In particular, they present auxiliary loss terms which can be used in "what if" scenarios where the actual state is unknown. Given certain assumptions, they show that this added regularization can improve generalization to unseen problem settings. Specifically they propose two forms of regularization: (1) enforcing that for different actions the predicted next state should be different (action-control) and (2) enforcing that when certain parts of the low dimensional state are perturbed, over a model rollout the perturbation should only affect the perturbed parts of the state, essentially encouraging the latent space features to be independent (disentanglement). 

Overall the idea is well motivated - incorporating counterfactual reasoning into model based RL has potential to to improve generalization. Also, while the assumptions needed for the regularization to be correct are not always true, they do seem to hold in many cases. Lastly, the results do seem to indicate that generalization is slightly improved when using the proposed forms of regularization.

My criticisms are:

(1) As mentioned in the paper Action-Control assumes that at every single timestep the agent has potential to change the state. However there may be settings where the agent can always change state, but only a small component of the state. In these cases the states should be quite similar. For example a robot only moving a single object when the state consists of many objects. Also as mentioned in the paper Disentanglement will not work in stochastic environments. One concern I have is that since different environments can violate the assumptions to varying degrees, it seems like actually using the regularization and picking the correct hyperparameter to weight it will be very challenging. 

(2) The current results are only demonstrated in a single, custom environment. Additionally performance is shown on only 2 test tasks, and in all cases in Table 2 it is unclear how to interpret the reward. Does this performance constitute completing the task? What is the best possible cumulative reward in this case? The performance improvement seems small, but it is difficult to judge without knowing the details of the task.

I think the paper would be significantly improved by (1) adding experiments in more environments, especially standard model based RL environments where the performance of many existing methods is known and (2) adding comparisons to other forms of model regularization, for example using an ensemble of models. My current rating is Weak Accept.

Some other questions:
- In Table 2 does MPC amount to PlaNet?
- How sensitive are the current numbers to planning parameters (horizon, num samples)?
- Can you provide error bars for the numbers in the tables?

______________________________________________

After author responses and closer examination of the paper I have some additional concerns about experimental details.  Changing my score from 'Weak Accept' to 'Weak Reject'
review_title: Official Blind Review #2

Meta-review: I agree with the reviewers that this paper has serious limitations in the experimental evaluation.
Decision: Reject