Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This is an interesting direction. There is still much to understand about the relative strengths and limitations of model based and model free techniques, and how best to combine them, and this paper discusses a new way to address this problem. The empirical results are promising and the ablation studies are good, but it also makes me wonder a bit about where the benefit is coming from.

Could you please put a bit of discussion in about the computational and memory cost. TDM is now parameterized with (state, action (goal) state, and the horizon tau). Essentially it is now computing the distance to each possible goal state after starting in state (s,a) and taking a fixed number of steps. 
It seems like this is less compact than learning a 1-step dynamics model directly.
The results are better than models in some places. It seems likely this is because the model-based approach referenced doesn’t do multi-step model fitting, but essentially TDM is, by being asked to predict and optimize for C steps away. If models were trained similarly (using multi-step loss) would models do as well as TDM?
How might this be extended to the stochastic setting?

review_title: interesting direction

final_score: rating: 7: Good paper, accept
review_content: The paper universal value function type ideas to learn models of how long the current policy will take to reach various states (or state features), and then incorporates these into model-predictive control. This looks like a reasonable way to approach the problem of model-based RL in a way that avoids the covariate shift produced by rolling learned transition models forward in time. Empirical results show their method outperforming Hindsight Experience Replay (which looks quite bad in their experiments), DDPG, and more traditional model-based learning. It also outperforms DDPG quite a bit in terms of sample efficiency on a real robotic arm. They also show the impact of planning horizon on performance, demonstrating a nice trade-off.

There are however a couple of relevant existing papers that the authors miss referencing / discussing:
- "Reinforcement Learning with Unsupervised Auxiliary Tasks" (Jarderberg et al, ICLR 2017) - uses predictions about auxiliary tasks, such as effecting maximum pixel change, to obtain much better sample efficiency.
- "The Predictron: End-To-End Learning and Planning" (Silver et al, ICML 2017), which also provides a way of interpolating between model-based and model-free RL.

I don't believe that these pieces of work subsume the current paper, however the authors do need to discuss the relationship their method has with them and what it brings.

** UPDATE Jan 9: Updated my rating in light of authors' response and updated version. I recommend that the authors find a way to keep the info in Section 4.3 (Dynamic Goal and Horizon Resampling) in the paper though, unless I missed where it was moved to. **

review_title: Nice ideas and execution; needs some more discussion of existing work

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper proposes a "temporal difference model learning", a method that aims to combine the benefits of model-based and model-free RL.  The proposed method essentially learns a time-varying goal-conditional value function for a specific reward formulation, which acts as a surrogate for a model in an MPC-like setting.  The authors show that the method outperforms some alternatives on three continuous control domains and real robot system.

I believe this paper to be borderline, but ultimately below the threshold for acceptance.  On the positive side, there are certainly some interesting ideas here: the notion of goal-conditioned value functions as proxies for a model, and as a means of merging model-free and model-based approaches is very really interesting, and hints at a deeper structure to goal-conditioned value functions in general.  Ultimately, though, I feel that there are two main issues that make this research feel as though it is still ultimately in the earlier stages: 1) the very large focus on the perspective that this approach is unifying model-based and model-free RL, when it fact this connection seems a bit tenuous; and 2) the rather lackluster experimental results, which show only marginal improvement over purely model-based methods (at the cost of much additional complexity), and which make me wonder if there's an issue with their implementation of prior work (namely the Highsight Experience Replay algorithm).

To address the first point, although the paper stresses it to a very high degree, I can't help but feel that the connection that the claimed advance of "unifying model-based and model-free RL" is overstated.  As far as I can tell, the connection is as follows: the learned quantity here is a time-varying goal-conditioned value function, and under some specific definition of reward, we can interpret the constraint that this value function equal zero as a proxy for the dynamics constraint in MPC.  But the exact correspondence between this and the MPC formulation only occurs for a horizon of size zero: longer horizons require a multi-step MPC for the definition of the model-free and model-based correspondence.  The fact that the action selection of a model-based method and this approach have some function which looks similar (but only under certain conditions), just seems like a fairly odd connection to highlight so heavily.

Rather, it seems to me that what's happening here is really quite simple: the authors are extending goal-conditioned value functions to the case of non-stationary finite horizon value functions (the claimed "key insight" in eq (5) is a completely standard finite-horizon MDP formulation).  This seems to describe perfectly well what is happening here, and it does also seem intuitive that this provides an advantage over stationary goal-conditioned value functions: just as goal conditioned value functions offer the advantage of considering "every state as a goal", this method can consider "every state as a goal for every time horizon".  This seems interesting enough on its own, and I admit I don't see the need for the method to be yet another claimed unification of model-free and model-based RL.

I would also suggest that the authors look into the literature on how TD methods implicitly learn models (see e.g. Boyan 1997 "Least-squares temporal difference learning", and Parr et al., 2007 "An analysis of linear models...").  In these works it has been shown that least squares TD methods (at least in the linear feature setting), implicitly learn a dynamics model in feature space, but only the "projection" of the reward function is actually needed to learn the TD weights.  In building the proposed value functions, it seems like the authors are effectively solving for multiple rewards simultaneously, which would effectively preserve the learned dynamics model.  I feel like this may be an interesting line of analysis for the paper if the authors _do_ want to stick with the notion of the method as unifying model-free and model-based RL.

All these points may ultimately just be a matter of interpretation, though, if not for the second issue with the paper, which is that the results seem quite lackluster, and the claimed performance of HER seems rather suspicious.  But instead, the authors evaluate the algorithm on just three continuous control tasks (and a real robot, which is more impressive, but the task here is still so extremely simple for a real robot system that it really just qualifies as a real-world demonstration rather than an actual application).  And in these three settings, a model-based approach seems to work just as well on two of the tasks, and may soon perform just as well after a few more episodes on the last task (it doesn't appear to have converged yet).  And despite the HER paper showing improvement over traditional policy approaches, in these experiments plain DDPG consistently performs as well or better than HER.  
review_title: Some interesting ideas, but not clear just how strong is the model-based/model-free connection here

Meta-review: There is a concern from one of the reviewers that the paper needs deeper analysis. On the other hand, applying finite horizon techniques to deep RL is relatively unexplored, and the paper does provide some interesting results in that direction.

Decision: Accept (Poster)