Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - I would like the authors to relate formulations proposed in (1,2,3) to previous RL literature and models, and then adjust the claims. This approach could be crudely categorized as a UVFA (universal value function appx, Schaul et al) model trained with an on-policy method.

- In order to make the comparisons with DSR fair, DSR could be learnt without the autoencoder loss (although the original DSR paper does make use of this loss throughout the paper)

review_title: comparisons

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents an on-policy deep RL method with additional auxiliary intrinsic variables. 

- The method is a special case of an universal value function based approach and the authors do cite the correct references. Maybe the biggest claimed technical contribution of this paper is to distill many of the existing ideas to solve 3D navigation problems. I think the contributions should be more clearly stated in the abstract/intro

- I would have liked to see failure modes of this approach. Under what circumstances does the model have problems generalizing to changing goals? There are other conceptual problems -- since this is an on-policy method, there will be catastrophic forgetting if the agent dosen't repeatedly train on goals from the distant past. 

- Since the main contribution of this paper is to integrate several key ideas and show empirical advantage, I would have liked to see results on other domains like Atari (maybe using the ROM as intrinsic variables)

Overall, I think this paper does show clear empirical advantage of using the proposed underlying formulations and experimental insights from this paper might be valuable for future agents
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Deep RL (using deep neural networks for function approximators in RL algorithms) have had a number of successes solving RL in large state spaces. This empirically driven work builds on these approaches. It introduces a new algorithm which performs better in novel 3D environments from raw sensory data and allows better generalization across goals and environments. Notably, this algorithm was the winner of the Visual Doom AI competition.

The key idea of their algorithm is to use additional low-dimensional observations (such as ammo or health which is provided by the game engine) as a supervised target for prediction. Importantly, this prediction is conditioned on a goal vector (which is given, not learned) and the current action. Once trained the optimal action for the current state can be chosen as the action that maximises the predicted outcome according the goal. Unlike in successor feature representations, learning is supervised and there is no TD relationship between the predictions of the current state and the next state.

There have been a number of prior works both in predicting future states as part of RL and goal driven function approximators which the authors review in section 2. The key contributions of this work are the focus on Monte Carlo estimation (rather than TD), the use of low-dimensional ‘measurements’ for prediction, the parametrized goals and, perhaps most importantly, the empirical comparison to relevant prior work.

In addition to the comparison with Visual Doom AI, the authors show that their algorithm is able to learn generalizable policies which can respond, without further training, to limited changes in the goal.

The paper is well-communicated and the empirical results compelling and will be of significant interest.

Some minor potential improvements:
There is an approximation in the supervised training as it is making an on-policy assumption but it learns from a replay buffer (with the Monte Carlo regression the expectation of the remainder of the trajectory is assumed to follow the current policy, but is being sampled from episodes generated by prior versions of the policy). This should be discussed.
The algorithm uses additional metadata (the information about which parts of the sensory input are worth predicting) that the compared algorithms do not. I think this, and the limitations of this approach (e.g. it may not work well in a sensory environment if such measurements are not provided) should be mentioned more clearly.


review_title: Compelling empirically driven result

final_score: rating: null
review_content: The goal is always given to the agent, never learned. Is that correct? So it seems like this is providing additional information that other models didn't receive about the decomposition of the reward function (e.g. how much to relatively value ammo, health, frags).

Have you tried learning the goal?
review_title: Goals

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents an on-policy method to predict future intrinsic measurements. All the experiments are performed in the game of Doom (vizDoom to be exact), and instead of just predicting win/loss or the number of frags (score), the authors trained their model to predict (a sequence of) triplets of (health, ammunition, frags), weighted by (a sequence of) "goal" triplets that they provided as input. Changing the weights of the goal triplet is a way to perform/guide exploration. At test time, one can act by maximizing the long term goal only.

The results are impressive, as this model won the 2016 vizDoom competition. The experimental section of the paper seems sound:
 - There are comparisons of DFP with A3C, DQN, and an attempt to compare with DSR (a recent similar approach from Kulkarni et al., 2016). DFP outperforms other approaches (or equal them when they reach a ceiling / optimum, as for A3C in scenario D1).
 - There is an ablation study that supports the thesis that all the "added complexity" of the paper's model is useful.

Predicting intrinsic motivation (Singh et al. 2004), auxiliary variables, and forward modelling, are well-studied domains of reinforcement learning. The version that I read (December 4th revision) adequately references prior work, even if it is not completely exhaustive. 

A few comments (nitpicks) on the form:
 - Doom is described as a 3D environment, whereas it is actually a 2D environment (the height is not a discriminative/actionable dimension) presented in (fake) 3D.
 - The use of "P" in (2) (and subsequently) may be misleading as it stands for prediction but not probability (as is normally the case for P).
 - The double use of "j" (admittedly, with different fonts) in (6) may be misleading.
 - Results tables could repeat the units of the measurements (in particular as they are heterogenous in Table 1).

I think that this paper is a clear accept. One could argue that experiments could be conducted on different environments or that the novelty is limited, but I feel that "correct" (no-nonsense, experimentally sound on Doom, appendix providing details for reproducibility) and "milestone" (vizDoom winner) papers should get published.
review_title: Review

Meta-review: This paper details the approach that won the VizDoom competition - an on-policy reinforcement learning approach that predicts auxiliary variables, uses intrinsic motivation, and is a special case of a universal value function. The approach is a collection of different methods, but it yields impressive empirical results, and it is a clear, well-written paper.
Decision: Accept (Oral)