Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper proposes to use the latent representations learned by a model-free RL agent to learn a transition model for use in model-based RL (specifically MCTS). The paper introduces a strong model-free baseline (win rate ~80% in the MiniRTS environment) and shows that the latent space learned by this baseline does include relevant game information. They use the latent state representation to learn a model for planning, which performs slightly better than a random baseline (win rate ~25%).

Pros:
- Improvement of the model-free method from previous work by incorporating information about previously observed states, demonstrating the importance of memory.
- Interesting evaluation of which input features are important for the model-free algorithm, such as base HP ratio and the amount of resources available.

Cons:
- The model-based approach is disappointing compared to the model-free approach.

Quality and Clarity:

The paper in general is well-written and easy to follow and seems technically correct, though I found some of the figures and definitions confusing, specifically:

- The terms for different forward models are not defined (e.g. MatchPi, MatchA, etc.). I can infer what they mean based on Figure 1 but it would be helpful to readers to define them explicitly.
- In Figure 3b, it is not clear to me what the difference between the red and blue curves is.
- In Figure 4, it would be helpful to label which color corresponds to the agent and which to the rule-based AI.
- The caption in Figure 8 is malformatted.
- In Figure 7, the baseline of \hat{h_t}=h_{t-2} seems strange---I would find it more useful for Figure 7 to compare to the performance if the model were not used (i.e. if \hat{h_t}=h_t) to see how much performance suffers as a result of model error.

Originality:

I am unfamiliar with the MiniRTS environment, but given that it is only published in this year's NIPS (and that I couldn't find any other papers about it on Google Scholar) it seems that this is the first paper to compare model-free and model-based approaches in this domain. However, the model-free approach does not seem particularly novel in that it is just an extension of that from Tian et al. (2017) plus some additional features. The idea of learning a model based on the features from a model-free agent seems novel but lacks significance in that the results are not very compelling (see below).

Significance:

I feel the paper overstates the results in saying that the learned forward model is usable in MCTS. The implication in the abstract and introduction (at least as I interpreted it) is that the learned model would outperform a model-free method, but upon reading the rest of the paper I was disappointed to learn that in reality it drastically underperforms. The baseline used in the paper is a random baseline, which seems a bit unfair---a good baseline is usually an algorithm that is an obvious first choice, such as the model-free approach.
review_title: Interesting approach to learning a model, but underperforms model-free methods

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to use a pretrained model-free RL agent to extract the developed state representation and further re-use it for learning forward model of the environment and planning.
The idea of re-using a pretrained agent has both pros and cons. On one hand, it can be simpler than learning a model from scratch because that would also require a decent exploration policy to sample representative trajectories from the environment. On the other hand, the usefulness of the learned representation for planning is unclear. A model-free agent can (especially if trained with certain regularization) exclude a lot of information which is potentially useful for planning, but is it necessary for reactively taking actions.
A reasonable experiment/baseline thus would be to train a model-free agent with a small reconstruction loss on top of the learned representation. In addition to that, one could fine-tune the representation during forward model training. 
It would be interesting to see if this can improve the results.

I personally miss a more technical and detailed exposition of the ideas. For example, it is not described anywhere what loss is used for learning the model. MCTS is not described and a reader has to follow references and infer how exactly is it used in this particular application which makes the paper not self-contained. 
Again, due to lack of equations, I don’t completely understand the last paragraph of 3.2, I suggest re-writing it (as well as some other parts) in a more explicit way.
I also could find the details on how figure 1 was produced. As I understand, MCTS was not used in this experiment. If so, how would one play with just a forward model?

It is a bit disappointing that authors seem to consider only deterministic models which clearly have very limited applicability. Is mini-RTS a deterministic environment? 
Would it be possible to include a non-deterministic baseline in the experimental comparison?

Experimentally, the results are rather weak compared to pure model-free agents. Somewhat unsatisfying, longer-term prediction results into weaker game play. Doesn’t this support the argument about need in stochastic prediction? 

To me, the paper in it’s current form is not written well and does not contain strong enough empirical results, so that I can’t recommend acceptance. 

Minor comments:
* MatchA and PredictPi models are not introduced under such names
* Figure 1 that introduces them contains typos. 
* Formatting of figure 8 needs to be fixed. This figure does not seem to be referred to anywhere in the text and the broken caption makes it hard to understand what is happening there.

review_title: Interesting way of re-using pre-trained agents with a lot of room for improvement

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper studies learning forward models on latent representations of the environment, and use these for model-based planning (e.g. via MCTS) in partial-information real-time-strategy games. The testbed used is MiniRTS, a simulation environemnt for 1v1 RTS.

Forecasting the future suffers from buildup / propagation of prediction errors, hence the paper uses multi-step errors to stabilize learning.

The paper:
1. describes how to train strong agents that might have learned an informative latent representation of the observed state-space.
2. Evaluates how informative the latent states are via state reconstruction.
3. trains variatns of a forward model f on the hidden states of the various learned agents.
4. evaluates different f within MCTS for MiniRTS.

Pro:
- This is a neat idea and addresses the important question of how to learn accurate models of the environment from data, and how to integrate them with model-free methods.
- The experimental setting is very non-trivial and novel.

Con:
- The manuscript is unclear in many parts -- this should be greatly improved.
1. The different forward models are not explained well (what is MatchPi, MatchA, PredN?). Which forward model is trained from which model-free agent?
2. How is the forward model / value function used in MCTS? I assume it's similar to what AlphaGo does, but right now it's not clear at all how everything is put together.

- The paper devotes a lot of space (sect 4.1) on details of learning and behavior of the model-free agents X. Yet it is unclear how this informs us about the quality of the learned forward models f. It would be more informative to focus in the main text on the aspects that inform us about f, and put the training details in an appendix.

- As there are many details on how the model-free agents are trained and the system has many moving parts, it is not clear what is important and what is not wrt to the eventual winrate comparisons of the MCTS models. Right now, it is not clear to me why MatchA / PredN differ so much in Fig 8.

- The conclusion seems quite negative: the model-based methods fare *much* worse than the model-free agent. Is this because of the MCTS approach? Because f is not good? Because the latent h is not informative enough? This requires a much more thorough evaluation. 

Overall:
I think this is an interesting direction of research, but the current manuscript does provide a complete and clear analysis.

Detailed:
- What are the right prediction tasks that ensure the latent space captures enough of the forward model?
- What is the error of the raw h-predictions? Only the state-reconstruction error is shown now.
- Figure 6 / sect 4.2: which model-free agent is used? Also fig 6 misses captions.
- Figure 8: scrambled caption.
- Does scheduled sampling / Dagger (Ross et al.) improve the long-term stability in this case?

review_title: Interesting direction of research, but analysis is not complete and exposition is unclear.

Meta-review: There was certainly some interest in this paper which investigates learning latent models of the environment for model-based planning, particularly articulated by Reviewer3.  However, the bulk of reviewer remarks focused on negatives, such as:

--The model-based approach is disappointing compared to the model-free approach.
--The idea of learning a model based on the features from a model-free agent seems novel but lacks significance in that the results are not very compelling.
--I feel the paper overstates the results in saying that the learned forward model is usable in MCTS.
-- the paper in it’s current form is not written well and does not contain strong enough empirical results



Decision: Reject