Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Value-Driven Hindsight Modelling proposes a method to improve value function learning. The paper introduces the hindsight value function which estimates the expected return at a state conditioned on the future trajectory of the agent. How use this hindsight value function is not obvious, since an agent does not have access to the future states needed in order to take actions (for Q-Learning) and the hindsight value function is a biased gradient estimator for training policy gradient methods. 

The authors train the standard value function (which does not have access to future information) to predict the features which the highsight value function learns to summarize the value relevant parts of the future trajectory. These predicted features can then be used in place of the actual hindsight value function, circumventing the issues discussed above. The authors argue that this auxiliary objective provides a richer training signal to the normal value function, helping it to better learn what information in a given state is relevant to predicting future rewards.

The paper is well structured and written, flowing from high level motivation and review into the core of the method, followed by analysis of the approach, and then proceeds through three experiments. The first two are toy / crafted experiments which build intuition and probe the behavior of the method and finally a large scale test on the Atari 57 benchmark demonstrating improvements when augmenting a state-of-the-art method with HiMo.

This reviewer recommends acceptance (I would give a 7 given more granularity) based on the contribution of a new auxiliary objective for value functions and the strength of the experimental suite. The Portal Choice environment is well crafted and instrumented with the graphs of figure 5b and 5c to show the behavior of the approach and the clean demonstration of an improvement over a previously SOTA method for Atari 57 is encouraging (the same architecture and the ablation simply sets the auxiliary objective’s weight to 0). However, the reviewer has some caution and concerns as follows:

1) The lack of a large scale experiment demonstrating improvement with an actor-critic method. While the Portal Choice experiments are informative and use Impala, it is a bit toy, and it would increase the reviewer’s confidence in the generality and robustness of the approach if improvements were also demonstrated for an actor-critic method on a large environment suite. Atair 57 could work but ideally a different setting such as DMLab 30 or continuous control from pixels. Demonstrating improvements in one of these additional settings would raise the reviewer to a strong acceptance.

2) The potential sensitivity of the approach to the two important hyperparameters that the authors mention, the dimensionality of the hindsight feature space (to reduce approximation error) and the # of future states it conditions on (to avoid just observing the full return directly). The very low dimensionality of the hindsight feature space (d=3 for Atari) seems a bit at odds with the explanation that the hindsight features provide a strong training signal for learning to better extract value relevant information from the state. Experiments that studied sensitivity to these would provide better perspective on the robustness of HiMo.

Questions and suggestions for improving the paper:

For Figure 6 the dynamic range gets squashed by a few games with relatively large performance improvements or regressions. Changing to a log-scale on the y-axis could be more informative? For instance, I find it pretty difficult to eyeball the ~1 human normalized score median improvement according to Table 1 from the chart.

Figure 3 could also be improved. It requires significant context from definitions in the paper in order to understand. It could be reworked into a stand alone expository overview of HiMo that helps readers quickly grok the idea of the paper such that abstract + figure is enough.

Could the authors consider showing / adding full learning curves (median human normalized score?) for HiMO vs the baseline on Atari 57? This would help readers get a qualitative feel for the learning dynamics of the algorithm instead of only having a final scalar measure at the end of training.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:

The paper proposes a way to learn better representation for RL by employing a hindsight model-based approach. The reasoning is that during training, we can observe the future trajectory and use features from it to better predict past values/returns. However to make this practical, the proposed approach fits an approximator to predict these features of the future trajectory from the current state and then subsequently, use them to predict the value. The authors claim that this extra information can be used to learn a better representation in some problems and lead to faster learning of good policies (or optimal value functions)

Decision:
Weak Accept
My decision is influenced by the following two key reasons

(1) I like the idea of hindsight modeling a lot. It is true that a trajectory gives much more information than just a weak scalar signal indicating return from each state in the trajectory. Identifying a way to make use of all the extra information in the trajectory to aid in value prediction is useful. The proposed approach is a step towards that, and I think the community should be made aware of that for sake of future research in this direction.

(2) Having said that, I am not super satisfied with the way the authors have presented their approach. The explanation is jumbled and confusing, at times. The paper needs careful rewriting to communicate ideas better and notation needs to be standardized earlier. Some of the sections are either redundant or lack insights. Even if they do have insights, they are not highlighted leaving the reader to search for them. The experimental setup is not clear and the authors could have spent more space in the paper dedicated to how the hindsight modeling approach can be implemented within an existing RL method.

Comments:

(1) The line in abstract "but this approach is usually not sensitive to reward function" doesn't make sense. Isn't reward function part of the model? So you are learning the reward function, so how is it not sensitive to it? I think I understand what the authors are saying but it took me until the end of Sec 3.2 to get that. 

(2) How does this work relate to Value-aware model learning works from Farahmand (AISTATS 2017, NeurIPS 2018). The premise seems to be similar: learn a model taking into account the underlying decision-making problem to be solved and the structure of the value function. The paper needs a discussion of these set of works

(3) In Section 3.3, \phi_{\theta_2} has conflicting function parameters in eq (2) and (3). 

(4) Section 3.4 is very confusing. I understood the setup of the problem and it seemed like it was very illustrative of an example where proposed approach will excel. However, Fig 1 and its caption are unclear and I found it hard to understand what the figure is conveying. The paragraph underneath the figure had no explanation for the Fig 1, and instead directly jumped to the results in Fig 2. The paper could use a better explanation of Fig 1. and explain why the proposed approach can learn the structure of s' and better predict value at s

(5) Section 3.5 partially answers the question "when is it advantageous to model in hindsight?" In cases, where L_model is low, of course its advantageous to model in hindsight! But the real question that needs to be answered is buried in the last paragraph. What if learning a good \phi is as hard as predicting the return? In this case, do we still gain any advantage? I am not sure how having a limited view of future observations and low dimensional \phi helps. If the feature that decides future return lies beyond the limited view of future observations, does it still not give any advantage? Questions like these might be useful in aiding the reader to understand why hindsight modeling is better

(6) Section 4 needs more text to explain what components of the architecture are learnt using what losses, and provide intuitions for why that is the case. It seems like that is very crucial to ensure that \phi doesn't learn something trivial and non-useful. I am surprised section 4 is so small, and Fig 3 is not useful. Maybe, you can combine section 3.4, 3.5 and condense them, and using the obtained space in expanding sec 4.

(7) The experiments section immediately dives into the problem setup and results. It will be useful to have a subsection explaining how the proposed hindsight model is implemented within an RL algorithm. Currently, it is hard for the reader to connect what he/she has read until Section 4 with what's presented in Section 5.

(8) The results are convincing. However, my biggest concern is the experiments were not designed carefully to analyze how much the hindsight modeling contributed in the increase of performance? Are the number of parameters in the value function approximator the same between the hindsight RL algorithm and the baseline? Can we have a simplistic example that is amenable to isolate the influence of hindsight modeling from other factors? Fig. 2 does a reasonable job at it but I think the hindsight modeling approach can achieve improvement in more diverse problems. In a way, the proposed feature is doing state space augmentation so that value can be easily predicted from the features of the augmented state. So, identifying the characteristics of the problems where this can be done is very useful to the RL practictioner. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a new model-based reinforcement learning method, termed hindsight modelling. The method works by training a value function which, in addition to depending on information available at the present time is conditioned on some learned embedding of a partial future trajectory. A model is then trained to predict the learned embedding based on information available at the current time-step. This predicted value is fed in place of the actual embedding to the same value model, to generate a value prediction for the current time-step. So instead of just learning a value function based on future returns, the method uses a two-step process of learning an embedding of value relevant information from the future and then learns to predict that embedding.

The paper gives some motivating examples of why and when such an approach could yield an advantage over standard value learning methods like Monte-Carlo or temporal difference learning. The basic idea is that when the returns obey some causal structure like X->Y->Z it may be easier to learn P(Y|X) and P(Z|Y) than to learn P(Z|X) directly. In particular, the authors point out that in the discrete case when Y takes relatively few values the size of the respective probability tables can be smaller in the former case than the latter. This motivates the approach of discovering a set of future variables to predict which are themselves predictive of return, rather than predicting the expected future return directly.

On a high level, I like the idea of specifically learning to model relevant aspects of the environment. However, I lean toward rejecting this work in its current form because I feel the motivation for when this particular method would be useful is unclear.

In particular, I don't really understand how this method could, in general, be expected to improve on regular bootstrapping. Why learn a prediction of the return at time t based on future information when we could just use the value function at a later time to improve the prediction at time t? It seems to me that the future value function itself concisely summarizes the information in the future state that is relevant for predicting the past return, while better exploiting the structure of the problem. Of course in cases like partial observability, it could be that the future value function lacks information from the past that is important for accurately predicting the return (for example in the portal example of this paper). However, if partial observability is really the case of interest, the method presented in this paper seems like a rather roundabout solution method. For example, instead of conditioning v+ on a future hidden state h_{t+k} (as the authors do in the experiments) perhaps one could simply condition the value function on a past hidden state h_{t-k} and obtain similar benefit from bootstrapping?

Aside from partial observability, for which I feel there are better approaches, the only situation I can understand the method having an advantage is when later states contain information which helps to predict earlier rewards. This is essentially the situation presented in the illustrative example. However, currently I feel such situations are rather contrived and unintuitive so I would need more supporting evidence to accept these situations as a good motivation.

On a deeper level, I don't see how the probability table motivation given in the introduction applies when what is being learned is an expectation (i.e. a value function) and not a distribution.

The approach also suffers from well-known issues with using the output of an expectation model of a variable as the input to a nonlinear function approximator in place of the variable itself. Namely, there is no guarantee that the expectation value of a variable is a possible value for the variable so giving it as input to a predictor trained on the variable itself could easily yield nonsense output in the stochastic case. As far as I can tell the method does nothing to mitigate this (please correct me if I'm wrong), so there is no reason to assume the method is generally applicable in settings with nontrivial stochasticity.

Despite these concerns, I feel the experiments are for the most part quite well thought out and executed. The paper is also quite well written, motivation issues aside, so I would not be upset if it was accepted with the hope that it leads to future work addressing the above-mentioned concerns.

If possible I think this paper would benefit significantly from a detailed explanation of how and when the proposed approach should be expected to improve on bootstrapping, including bootstrapping off a value function which uses an analogous architecture to v+.

Questions for authors:
Given the hyper-parameters of R2D2 deviate somewhat from those used in the original paper, and nothing is said about how they were chosen, how confident can we be that the observed advantage of hindsight modelling is not simply due to hyper-parameters being selected which are more favourable for the proposed method?

Given that you are not learning distributions but expectations in the form of value functions, how pertinent is the motivation of learning P(Y|X) and P(Z|Y) instead of P(Z|X) directly described in the introduction?

How much of the benefit observed in the portal example an ATARI could also be gained from simply providing the value function approximation with h_{t-k} as input to help span larger time-gaps?

Update:

While I still feel the exposition could be improved to make the underlying idea clearer, I feel the authors did a good job of addressing my major concerns in their reply, hence I have raised my score to a weak accept.

I have to admit I missed the point that v^+ and v^m were using entirely different parameter sets. In light of this, I agree that the expectation model issue I mentioned is not a major concern.

I also appreciate the clarification of the hyperparameters, if they were really tuned to improve the baseline then this detail should be added to the paper and would negate my concern there. 

Finally, I thank the authors for providing the value-function oriented example. I found this example to be more illustrative than the one in the introduction of the paper, and I now feel that I have a better grasp of the motivation. I still have doubts about the general benefit of the approach over bootstrapping but since it is not entirely clear to me one way or the other I feel the idea at least warrants further exploration, and it would be reasonable to accept the paper to make the community aware of it.

review_title: Official Blind Review #2

Meta-review: This paper studies the problem of estimating the value function in an RL setting by learning a representation of the value function. While this topic is one of general interest to the ICLR community, the paper would benefit from a more careful revision and reorganization following the suggestions of the reviewers.
Decision: Reject