Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This paper is well written so I mainly have a high level discussion question.

If I understand correctly, the authors propose adding a reward prediction branch to a neural architecture that is known to predict future frames, and show that it can predict cumulative reward as well as future video frames.

My question is: what's the take-away here? The stated motivation is that agents in the real world may need to learn both system dynamics and rewards and the goal is to provide first steps in that direction. My main qualm is that I don't see how this paper makes a meaningful first step. Atari games are deterministic and the reward is tightly coupled in a deterministic fashion to the environment, with barely any hidden information. It has already been shown in prior work that the network can predict the dynamics of the game far ahead, so why is it any surprise that it can also learn the reward function? Why should this result generalize to a more realistic scenario?

The most interesting parts of this paper are the proposed next steps (not the immediate next steps, but the future ones about planning with uncertainty and doing planning in latent spaces). If the paper was about those, it would be far more interesting and impactful. Instead, the results seem very incremental. I understand that there's a lot of literature about playing Atari games, but one needs to be realistic about claims and impact with such an artificial setting.
review_title: What's the take-away here?

final_score: rating: 4: Ok but not good enough - rejection
review_content: The topic of the paper, model-based RL with a learned model, is important and timely. The paper is well written. I feel that the presented results are too incremental. Augmenting the frame prediction network with another head that predicts the reward is a very sensible thing to do. However neither the methodology not the results are novel / surprising, given that the original method of [Oh et al. 2015] already learns to successfully increment score counters in predicted frames in many games.

I’m very much looking forward to seeing the results of applying the learned joint model of frames and rewards to model-based RL as proposed by the authors. 
review_title: null

final_score: rating: null
review_content: No comparisons are done to other model-based or model-free reward prediction methods, nor to other frame-prediction algorithms. Right now I feel the quality of the results is hard to judge.
review_title: Comparison to state-of-the-art

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces an additional reward-predicting head to an existing NN architecture for video frame prediction. In Atari game playing scenarios, the authors show that this model can successfully predict both reward and next frames.

Pros:
- Paper is well written and easy to follow.
- Model is clear to understand.

Cons:
- The model is incrementally different than the baseline. The authors state that their purpose is to establish a pre-condition, which they achieve. But this makes the paper quite limited in scope.

This paper reads like the start of a really good long paper, or a good short paper. Following through on the future work proposed by the authors would make a great paper. As it stands, the paper is a bit thin on new contributions.
review_title: Final Review

final_score: rating: null
review_content: Does joint prediction of rewards and frames help? It would be instructive to compare the reward prediction error to the baseline of networks trained on reward prediction only.
review_title: Comparison to baseline

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper extends a recently proposed video frame prediction method with reward prediction in order to learn the unknown system dynamics and reward structure of an environment. The method is tested on several Atari games and is able to predict the reward quite well within a range of about 50 steps. The paper is very well written, focussed and is quite clear about its contribution to the literature. The experiments and methods are sound. However, the results are not really surprising given that the system state and the reward are linked deterministically in Atari games. In other words, we can always decode the reward from a network that successfully encodes future system states in its latent representation. The contribution of the paper is therefore minor. The paper would be much stronger if the authors could include experiments on the two future work directions they suggest in the conclusions: augmenting training with artificial samples and adding Monte-Carlo tree search. The suggestions might decrease the number of real-world training samples and increase performance, both of which would be very interesting and impactful.
review_title: Well written paper with a clear focus and interesting future work proposal but with an overall minor contribution.

Meta-review: The authors have combined two known areas of research - frame prediction and reward prediction - and combined them in a feedforward network trained on sequences from Atari games. The fact that this should train well is unsurprising for this domain, and the research yields no other interesting results. Pros - the paper is clearly written and the experiments are sound. Cons - there is very little novelty or contribution.
Decision: Reject