Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper proposes to use ensembles of models to overcome a typical problem when training on a learned model: That the policy learns to take advantage of errors of the model.
The models use the same training data but are differentiated by a differente parameter initialization and by training on differently drawn minibatches.
To train the policy, at each step the next state is taken from an uniformly randomly drawn model.
For validation the policy is evaluated on all models and training is stopped early if it doesn't improve on enough of them. 

While the idea to use an ensemble of deep neural networks to estimate their uncertainty is not new, I haven't seen it yet in this context. They successfully show in their experiments that typical levels of performance can be achieved using much less samples from the real environment.

The reduction in required samples is over an order of magnitude for simple environments (Mujoco Swimmer). However, (as expected for model based algorithms) both the performance as well as the reduction in sample complexity gets worse with increasing complexity of the environment. It can still successfully tackle the Humanoid Mujoco task but my guess is that that is close to the upper limit of this algorithm?

Overall the paper is a solid and useful contribution to the field.

*Quality:*
The paper is clearly shows the advantage of the proposed method in the experimental section where it compares to several baselines (and not only one, thank you for that!). 

Things which in my opinion aren't absolutely required in the paper but I would find interesting and useful (e.g. in the appendix) are:
1. How does the runtime (e.g. number of total samples drawn from both the models and the real environment, including for validation purpuses) compare?
From the experiments I would guess that MB-TRPO is about two to three orders of magnitude slower, but having this information would be useful.
2. For more complex environments it seems that training is becoming less stable and performance degradates, especially for the Humanoid environment. A plot like in figure 4 (different number of models) for the humanoid environment could be interesting? Additionally maybe a short discussion where the major problem for further scaling lies? For example: Expressiveness of the models? Required number of models / computation feasibility? Etc... This is not necessarily required for the paper but would be interesting.

*Originality & Significance:*
As far as I can tell, none of the fundamental ideas are new. However, they are combined in an interesting, novel way that shows significant performance improvements.
The problem the authors tackle, namely learning a deep neural network model for model based RL, is important and relevant. As such, the paper contributes to the field and should be accepted.

*Smaller questions and notes:*
- Longer training times for MB-TRPO, in particular for Ant and Humanoid would have been intersting if computationionally feasibly.
- Could this in principle be used with Q-learning as well (instead of TRPO) if the action space is discrete? Or is there an obvious reason why not that I am missing?
review_title: Solid contribution in relevant field

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors combine an ensemble of DNNs as model for the dynamics with TRPO. The ensemble is used in two steps:
First to collect imaginary roll-outs for TRPO and secondly to estimate convergence of the algorithm. The experiments indicate superior performance over the baselines.

The paper is well-written and the experiments indicate good results. However, idea of using ensembles in the context of 
(model-based) RL  is not novel, and it comes at the cost of time complexity.  Therefore, the method should utilize 
the advantage an ensemble provides to its full extent. 
The main strength of an ensemble is to provide lower test error, but also some from of uncertainty estimate given by the spread of the predictions. The authors mainly utilize the first, but to a lesser extent the second advantage (the imaginary roll-outs will  utilize the spread to generate possible outcomes). Ideally the exploration should also be guided by the uncertainty (such as VIME).

Related, what where the arguments in favor of an ensemble compared to Bayesian neural networks (possibly even as simple as using MH-dropout)? BNNs provide a stronger theoretical justification that the predictive uncertainty is meaningful.

Can the authors comment on the time-complexity of the proposed methods compared to the baselines? In Fig. 2  the x-axis is the time step of the  real data. But I assume it took a different amount of time for each method to reach step t. The same argument can be made for Fig. 4. It seems here that in snake the larger ensembles reach convergence the quickest, but I expect this effect to be reversed when considering actual training time.

In total I think this paper can provide a useful addition to the literature.  However, the proposed approach does not have strong novelty and I am not fully convinced if the additional burden on time complexity outweighs the improved performance.

Minor:  In Sec. 2: "Both of these approaches assume a fixed dataset of samples which are collected
before the algorithm starts operating."  This is incorrect, while these methods consider the domain of fixed datasets, the algorithms themselves are not limited to this context.

review_title: well-written, good experiments, but limited novelty, doubts about time-complexity, and use of ensembles over BNNs

final_score: rating: 7: Good paper, accept
review_content: This paper presents a simple model-based RL approach, and shows that with a few small tweaks to more "typical" model-based procedures, the methods can substantially outperform model-free methods on continuous control tasks.  In particular, the authors show that by 1) using an ensemble of models instead of a single models, 2) using TRPO to optimize the policy based upon these models (rather that analytical gradients), and 3) using the model ensemble to validate when to stop policy optimization, then a simple model-based approach actually can outperform model-free methods.

Overall, I think this is a nice paper, and worth accepting.  There is very little actually new here, of course: the actual model-based method is entirely standard except with the additions above (which are also all fairly standard approaches in isolation).  But at a higher level, the fact that such simple model-based approaches work better than somewhat complex model free approaches actually is the point of the paper to me.  While the general theme of model-based RL outperforming model-free RL is not new (Atkeson and Santamaria (1997) comes to a similar conclusion) its good to see this same pattern demonstrated "officially" on modern RL benchmarks, especially since the _completely_ naive strategy of using a single model and more standard policy optimization doesn't perform as well.

Naturally, there is some question as to whether the work here is novel enough to warrant publication, but I think the overall message of the paper is strong enough to overcome fairly minimal contribution from an algorithmic perspective.  I did also have a few general concerns that I think could be discussed with a bit more detail in the paper:
1) The choice of this particular model ensemble to represent uncertainty seems rather ad-how.  Why is it sufficient to simply learn N models with different initial weights?  It seems that the likely cause for this is that the random initial weights may lead to very different behavior in the unobserved parts of the space (i.e., portions of the state space where we have no samples), and thus.  But it seems like there are much more principled ways of overcoming this same problem, e.g. by using an actual Bayesian neural net, directly modeling uncertainty in the forward model, or using generative model approaches.  There's some discussion of this point in the introduction, but I think a bit more explanation about why the model ensemble is expected to work well for this purpose.
2) Likewise, the fact the TRPO outperforms more standard gradient methods is somewhat surprising.  How is the model ensemble being treated during BPTT?  In the described TRPO method, the authors use a different model at each time step, sampling uniformly.  But it seems like a single model is used for each rollout in the proposed BPTT method?  If so, it's not surprising that this approach performs worse.  But it seems like one could backprop through the different per-timestep models just as easily, and it would remove one additional source of difference between the two settings.
review_title: A nice baseline in the epic model-based vs. model-free battle

Meta-review: The reviewers agree that the paper presents nice results on model based RL with an ensemble of models. The limited novelty of the methods is questioned by one reviewer and briefly by the others, but they all agree that this paper's results justify its acceptance.
Decision: Accept (Poster)