Value-Driven Hindsight Modelling
Authors: Arthur Guez
Fabio Viola
Theophane Weber
Lars Buesing
Steven Kapturowski
Doina Precup
David Silver
Nicolas Heess
Venue:  (2020)

Abstract: Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn predictors for value from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future but have to compose with a potentially weak scalar signal (an estimate of the return). In this paper we develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end we determine which features of the future trajectory provide useful information to predict the associated return. This provides us with tractable prediction targets that are directly relevant for a task, and can thus accelerate learning of the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.

Under review as a conference paper at ICLR 2020
VALUE-DRIVEN HINDSIGHT MODELLING
Anonymous authors
Paper under double-blind review
ABSTRACT
Value estimation is critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn predictors for value from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of ob- servations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future but have to compose with a potentially weak scalar signal (an estimate the return). In this paper we develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end we determine which fea- tures of the future trajectory provide useful information to predict the associated return. This provides us with tractable prediction targets that are directly relevant for task, and can thus accelerate learning of the value function. The idea can be understood as reasoning, in hindsight, about which aspects the future observa- tions could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale challenging domains, including on 57 Atari 2600 games.
INTRODUCTION
Consider a baseball player trying to perfect their pitch. The player performs arm motion and releases the ball towards the batter, but suppose that instead of observing where the ball lands and the reaction of the batter, the player only gets told the result of the play in terms of points or, worse, only gets told the final result of the game. Improving their pitch from this experience appears hard and inefficient, yet this is essentially the paradigm model-free reinforcement learning. The scalar feedback that estimates the return from a state (and action), encoding how well things went, drives the learning while the accompanying observations that may explain that result (e.g. flight path of the ball or the way the batter anticipated and struck the incoming baseball) are ignored. To intuitively understand how such information could help value prediction, consider a simple discrete Markov chain X - Y + Z, where Z is the scalar return and X is the observation from which we are trying to predict Z. If the space of possible values of Y is smaller than X, then it may be more efficient to estimate both P(Y and P(Z rather than directly estimating P(Z } In other words observing and then predicting can be advantageous to directly estimating the signal of interest Z. Model-based RL approaches would duly exploit the observed Y (by modeling the transition Y X),, but would, in general scenarios, contain information that is irrelevant to Z and hard to predict. Building full high-dimensional predictive model to indiscriminately estimate all possible future observations, including potentially chaotic details of the ball trajectory and the spectators' response, is challenge that may not pay off if the task-relevant predictions (e.g., was the throw accepted, was the batter surprised) are error-ridden. Model-free RL methods directly consider the relation X to Z, and focus solely upon predicting and optimising this goal, rather than attempting to learn the full dynamics. These methods have recently dominated the literature, and have attained the best performance in wide array of complex problems with digh-diimesiooal observations (Mnih et al. 2015 Schulman et al. 2017 Haarnoja al. 2018 Guez et al. 2019).
employ when optimizing policies in
In the discrete case, this follows from counting argument from the size of the prohahility tables involved.

---

Under review as a conference paper at ICLR 2020
In this paper, we propose to augment model-free methods with a lightweight model of future quan- tities of interest. The motivation is to model only those parts of the future observations (Y) that needed to obtain better value predictions. The major research challenge is to learn, from obser- vational data, which aspects of the future are important to model (i.e. what Y should be). To this end, we propose learn special value function in hindsight that receives future observations as an additional input. This learning process reveals features of the future observations that would be most useful for value prediction (e.g. flight path of the ball or the reaction of the batter), if provided by an oracle. These important features are then predicted, in advance, using only information available at test time (at the time of releasing the baseball, we knew the identity of the batter, the type of throw and spin given to the ball). Learning these value-relevant features can help representation learning for an agent and provide an additional useful input to its value and policy. Experimentally, hindsight value functions surpassed the performance of model-free RL methods in challenging association task (Portal Choice). When hindsight value functions were added to the prior state-of-the- RL method for Atari games, they significantly increased median performance from 833% to 965%.
2 BACKGROUND AND NOTATION
We consider a reinforcement learning setting whereby an agent learns from interaction in a sequen- tial decision-making environment (Sutton & Barto 201T). An agent's policy æ, mapping states to an action distribution, executed to obtain a sequence rewards and observations as follows. At each step after observing state st, the policy outputs an action a,, sampled from π (A[s;). and obtains a scalar reward rt and the next-state $t+1 from the environment. The sum of discounted rewards from state is the return denoted by G = =t=u" 7 RG, with y denoting the discount factor. Its expectation, as a function of the starting state, is called the value function, (s) = Ex [G]S, = s]. An important related quantity is the action-vall pectation with a particular action executed first: (s a) = Ex [G]So = 8, Ao = a]. The learning problem consists in adapting the policy π in order achieve higher value v". This usually entails learning an estimate of e for the current policy r,, this is the problem we focus on in this paper.
or Q-value, which corresponds to the same
Note that in practice we are interested in partially-obsereed environments where the state of the world is not directly accessible. For this case, we can think of replacing the observed state in the case of the observobeervedd case by a learned function that depends on past observations.
3 VALUE LEARNING
3.1 DIRECT LEARNING
A common approach to estimate v (or ②) is to represent it as a parametric function (or qo) and directly update its parameters based on sample returns of the policy of interest. Value-based RL algorithms vary in how they construct value target Y from single trajectory. They may regress to towards the Monte-Carlo return (Y, Gt), or exploit sequentiality in the reward process by relying on form of temporal-differene
learning to reduce variance (e.g. the TD(0) target Y, R2 + ( (S++1)). For a given target definition Y. the value loss C. to derive an update for is: )) In constructing target Y, based on trajectory of observations and rewards from time t, the observations are either unused (for a Monte Carlo return) or only indirectly exploited (when bootstrapping obtain a value estimate). In all cases, the trajectory is distilled into a scalar signal that estimates the return of a policy, and other relevant aspects of future observations are discarded. In particular in partially observed domains or domains with high- dimensional observation spaces it can be difficult to discover correlations with this noisy signal.
(0) (v)
3.2 MODEL-BASED APPROACH
An indirect way of estimating the value is to first learn a model of the dynamics. For example a 1-step observation model me learns to predict the conditional distribution St+1, ralst, aq. Then value estimate v(s) for state s can be obtained by autoregressively rolling out the model (until the end of the episode or fixed depth with parametric value bootstrap).
2

---

Under review as a conference paper at ICLR 2020
The model is trained potentially much richer data than the return signal since it exploits all information in the trajectory. Indeed, the observed transitions between states can reveal the structure behind sparse reward signal. A drawback of classic model-based approaches is that they predict high-dimensional signal, a task which may be costly and harder than directly predicting the scalar value As result, the approximation of the dynamics my may contain errors where it matters most for predicting the value (Talvitic] 2014). Although the observations carry all the data from the environment, most of it not essential to the task (Gelada al. 2019). The concern that modeling all observations is expensive also applies when the model is not used for actual rollouts but merely for representation learning. So while classic model- methods fully use this high- dimensional signal at some cost, model-free methods take the other extreme to focus only on the most relevant low-dimensional signal (the scalar return). Below we propose a method that strikes a balance between these paradigms.
3.3 HINDSIGHT VALUE AND MODEL
We introduce a new value function estimate that can only be computed at training time, the hindsight value function vt. This value still estimates the expected return from state 8, but it is further conditioned on additional observations Tr 81+1, 8 S occurring after time i
u+ (st, r = B = st.... S = s
(1)
Furthermore, we require et follow this particular parametric structure:
(St, To 0) = de, (s:), Φ0. (r.t))
(2)
where - (0,, 02), (00)), which forces information about the future trajectory through some vector-valued function E Rd. Intuitively, v+ is estimating the expected return from a past time point using privileged access to future observations. Note that if l is large enough, then et simply estimates the empirical return from time t given access to the state trajectory. However, if k is small and Φ is low- dimensional, then & becomes a bottleneck representation of the future trajectory T By learning in hindsight, we identify features that are maximally useful to predict the return the trajectory from time f. The hindsight value function is not useful quantity by itself, since - because of its use of privileged future observations we cannot readily use test time. Furthermore, it cannot be used as a baseline either, as when computing the policy gradient it will yield a biased gradient estimator. Instead, the idea is to learn a model o of Φ that can be used at test time. We conjecture that if privileged features & are useful for estimating the value, then the model of those features will also be useful for estimating the value function. We propose to learn the approximate expectation model 6 (s) conditioned on the current state and parametrized by n2, minimizing the following squared loss:
Lmodel (72) Eg,r+ (1l (8, r+) Pop (5)
where the expectation is taken over the distribution of states and partial trajectories -+ resulting from that state.
The approximate model & can then be leveraged to obtain better model-based value estimate wn (s: v) (f)), & (s)). Although d(s) cannot contain more information than included already in the state 8, it can still benefit from having being trained using a richer signal before the value converges. Figure summarizes the relation between the different quantities.
3.4 ILLUSTRATIVE EXAMPLE
We consider the following example to illustrate how the approaches to estimating the value function can differ. There are no actions in this example] and each episode consists a single transition from initial state to terminal state s', with reward r(s, 5') on the way.
Each instance of this example is parametrized by a square matrix W and a vector b sampled from a unit normal distribution, which determine the uncontrolled MDP. Initial states are of dimension D and sampled from a multivariate unit normal distribution (s; ~ N (O, 1) for all state dimension i).
In general 7/ can be defined to include full future transitions, actions and rewards. Thii he understood Markov Reward Process or policy evaluation setting
3

---

Under review as a conference paper at ICLR 2020
Given (#2), where $1 and 82 of dimension D1 and D2 (D D1 D2). the next state is determined according to the transition function: s = MLP(s) +c and 2/ a (W&s + b) where a is the Heaviside function. 511 acts as a distractor here, with additive noise ~ [(,, 1). The reward obtained is r(8,8') Σ, $ (6) o(s) = s').
Σ, 82 (() /VD. The true value in the start state is also
The key aspect of this domain is that s' reveals structure that helps predict the value function in the start state 8. This is made visually obvious in the trajectories sampled in this domain shown in Figure [
Direct learning
Start states
Next states (reuealle-structuree structure)
Rewards
Hindsight learning
Figure 1: Visualization of episodes in the illustrative example of Section 3.4 the start state on the left and must predict the corresponding color-coded reward on the right. Hindsight value prediction can leverage the observed structure in the intermediate state obtain a better value prediction. In more detail, this plot shows the second half #2 initial state on left. In the middle, superimposed the observed reward-relevant quantity Σ, 82 that been color-coded on the 82 vectors. On the right is the color-coded reward for each trajectory. The dimension states
Model-free value prediction see
in this example.
Let us consider how the different approaches to learning values presented above fare this problem. For direct learning, the value from v(s') is 0 since s' is terminal, so any n-step return is identical to the Monte-Carlo return, that is, the information present in observation s' is not leveraged. Results from learning from 8 given the return presented in Figure 2 (blue curve). approach first predicts from s, then attempts to predict the value given and the estimated next state. When increasing the input dimension, given a fixed capacity, the model does not focus its attention on the reward-relevant structure in s' and makes error where it matters most. As result, it can struggle to learn faster than model-free estimate (cf. red curve in Figure 2). When learning in hindsight, a+ can directly exploit the revealed structure in the observation of 7*, and as result the hindsight value learns faster than the regular causal model-free estimate (cf. dotted yellow curve in Figure 2). This drives the learning of & and its model o. which directly gets trained to predict these useful features for the value. As result, nthh also benefits and learns faster than the regular estimate on this problem (cf. green curve in Figure 2b.
model-based
3.5 WHEN IS IT ADVANTAGEOUS TO MODEL IN HINDSIGHT?
To understand the circumstances in which hindsight modelling provides a better value estimate, we first consider an analysis that relies on the following assumptions. Suppose that vy is sharing the
same function sa as w (i.e., 01 = 77), and let & be linear. If we write the, (f. Φ) (w) T ④ + b,
where 0, (Wi, w2), then we have for fixed values of the parameters:
E E(( (s; - wt os, 0 = [[ ( Φ(s; 92)))11)
(4)
I EE Iwo (1( +
(5)
|aaz || (mnndea
4

---

Under review as a conference paper at ICLR 2020
0. 40
HiMo value with
0. 35
Hindsight value of
Model-free value
0.30
Value error
0.25
Model-based value
0.20
0.15
0.10
0.05
0.
2000
4000
6000
8000
Steps
Figure 2: Learning the value of the initial state in the example of Section 3.4 The dimension of the data is
D 32 for this experiment, with the dimension of the useful data in the next state D2 4. The results averaged over 4 different instances, each repeated twice. Note that ut (dotted line) using privileged information (the next state).
using the Schwarz inequality. Let define the value error for a particular value function ut [(v) = [[(v(s) G) and [(e+) = [[( t (s, r+) G) Then we have:
(( B[(u"* (s) - wt (s, +) t
2 2(1 || 2 mmndee (12) + L(x+)).
using the fact that E[ + Y < ] + E| for random variables and Y. If we assume L(o+)= CL(v) with C < 0.5 (i.e., estimating the value in hindsight with more information is easier learning problem), then the following holds:
(1-2C)((c)) L(0 < L(v).
Lemoded (1p2)
2 1
In other words, this relates how small the modeling error needs to be to guarantee that the value error for v is smaller than the value error for the direct estimate The modeling error can be large different reasons. the environment or the policy is stochastic, then there some irreducible modeling eтror for the deterministic model. Even in these cases, small C' can make hindsight modeling advantageous. The modeling error could also be high because predicting & is hard. For example, it could be that & essentially encodes the empirical return, which means predicting & is at least as hard as predicting the value function (Lemotel) (72) ) 2 L(v)). Or it could be that Φ is high- dimensional, this could cause both hard prediction problem but also would cause the acceptable threshold for Cmodel to decrease (since |/02|² will grow). We address some of these concerns with specific architectural choices like v+ having limited view on future observations and having low dimensional & (see next section). Note that the analysis above ignores any advantage that could be obtained from representation learning when training o (if the state encoding function f shares parameters with ②).
ARCHITECTURE
The architecture for Hindsight Modelling (HiMo) we found to work at scale and tested in the ex- perimental section of the paper is described here. To deal with partial observability, we employ recurrent neural network, the state-RNN, which replaces the state with learned internal state h4, a function of the current observation o; and past observations through h4-1: h4 = f(o,, h:-1; 773), where we have extended the parameter description of y function TM and the hindsight value function t share the same internal state representation h, but the learning of t assumes is fixed (we do not backpropagate through the state- state-RNN in hindsight). In addition, force & only be learned through Lmudel, that T uses it as additional input.
(72, 7/, 73). The model-based value
5

---

Under review as a conference paper at ICLR 2020
To summarize:
ut (he, hetxi 0) = wh, (he: Φ8, (h+++)))
(10)
o"'(he;; th (he, 2 (he)).
(11)
with the bar notation denoting quantities treated as non-differentiable (i.e. where the gradient is stopped). The different losses in the HiMo architecture combined in the following way:
[(0, n) L.(n) + aCC,+ (0) + 3moodee
(12)
diagram of the architecture presented in Figure 3 and further implementation details can be found in the appendix.
Y
4
Cusa
ht
h44k
Or
O++1
Figure 3: Network architecture for HiMo. Double blue arrows denote losses different outputs of the network. Red denote quantities which are only computed in hindsight at train time (using parameters 0). The @ symbol on arrow means imput is assumed be non-differentiable (also sometimes called stop gradient).
This architecture can be straightforwardly generalized to cases where we also output policy T. for actor-critic setup, providing h and o inputs policy network 4 For Q-value based algorithm like Q-learning, we predict vector of values q"h and qt instead of T and vt. Computing v and training & can be done in an online fashion by simply delaying the updates by k steps (just like the computation of an n-step return).
EXPERIMENTS
The illustrative example Section 3.4 demonstrated the positive effect of hindsight modeling in simple policy evaluation setting. In this section, we now explore these benefits in the context of policy optimization challenging domains, a custom navigation task called Portal Choice, and Atari 2600. To demonstrate the generality and scalability of our approach we test hindsight value functions the context two high high-perrforance RL algorithms, IMPALA (Espeholt et al. 2018) and R2D2 (Kapturosski et al. 2019).
5.1 PORTAL CHOICE TASK
The Portal Choice (Fig. 4 is a two-phase navigation task where, in phase one an agent is presented with a contextual choice between two portals, whose positions vary between episodes. The position of the portal determines its destination in phase two, one of two different goal rooms (green and red rooms). Critically, the reward when terminating the episode in the goal room depends on both the color of the goal room in phase two and visually indicated combinatorial context shown in the first context matches the goal room color, then a reward of is given, otherwise the reward is 0 when terminating the episode (see appendix for the detailed description).
phase.
An easy suboptimal solution is to select the portal at random and finish the episode in the resulting goal room by reaching the goal pixel, which will result in positive reward of on average. A more
'In this case, the total loss also contains loss update n, and negative entropy loss.

---

Under review as a conference paper at ICLR 2020
Agont
Taleorts to
Context
Figure 4: Portal Choice task. Left: observation in the starting of the Portal Choice task. Two portals (cyan squares) are availahle to the agent (orange), each them leading to different room deterministically based on their position. Right: The two possible goal rooms are identified by green and red pixel. The reward upon reaching the goal (blue square) is a function of the room and the initial context.
difficult strategy is to be selective about which portal to take depending on the context, in order to get the reward of at each every episode. A model-free agent has to learn the joint mapping from contexts and portal positions to rewards. Even if the task is not visually complex, the context is combinatorial in nature (the agent needs count randomly placed pixels) and the joint configuration space of context and portal is fairly large (around 250M). Since the mapping from portal position to rooms does not depend on context, learning the portal-roon mapping independently is more efficient in this scenario.
For this domain, we implemented the HiMo architecture within a distributed actor-critic agent, named IMPALA proposed by |Espeholt et al. (2018). In this case, the target Y, to train y (used as a critic in this context) and v* is the V-trace target (Espeholt et al. 2018) to account for off- policy corrections between the behavior policy and the learner policy. The actor shares the same network as the critic and receives h and Φ inputs.
(baseline)
Hirdsight Nodeling (HMM IMPALA)
0.6
(rasellne)
L4
0.30
Reguiar value
cotic
0.10
0.2
0.6
0.60
8.0
0.6
1.2
steps
steps
57695
(b)
(c)
Figure 5: Results in the Portal Choice task. (a) shows the median performance as function of environment steps out of 4 seeds. (b) shows the value error averaged across states on the same x-axis scale for different value function estimate. (c) is analysis that shows the cross-entropy loss of a classifier that takes input & (solid line) or è (dotted line) and predicts the identity of the goal room (red green) as binary classification task. The HiMo curves (blue) show that information about the room identity hecomes present first in and then gets captured in its model é. For the baseline (where we set = B 0), à is not trained based on & and only achieves to classify the room identity at chance level.
We found that HiMo+IMPALA learned reliably faster to reach the optimal behavior, compared to the vanilla IMPALA baseline that shared the same network capacity (see Figure Sap. The hindsight value a rapidly learns to predict whether the portal-context association is rewarding based on seeing the goal room color in the future. Then & learns to predict the new information from the future that helps that prediction: the identity of the room (see analysis Fig 5c).. The prediction of & becomes effectively model of the mapping from portal room identity (since the context does correlate with the room identity). Having access to such mapping through ò helped the value prediction (Fig (b), which led to better action selection. Note that if the two rooms were visually indistinguishable, for example with no red/green rooms separation, HiMo would not able to offer any advantage over its model-free counterpart.
5.2 ATARI
We tested our approach in Atari 2600 videogames using the Arcade Learning Environment (Belle- mare et al. 2013). We added HiMo top of Recurrent Replay Distributed DQN (R2D2), DQN-
7

---

Under review as a conference paper at ICLR 2020
based distributed architecture introduced by Kapturowski al. (2019) which achieved state-of-the- art scores in Atari games.
In this value-based setting, HiMo trains q" (s+7)) and q² (.::0) based n-step return targets:
Y
Σ m==
7" Rumm + + (f" Ar;)))
(13)
where g is an invertible function, " are the periodically updated target network parameters (as in DQN by Mnih et al. (2015)), and A arg max, (St+n: a: 7) (the Double DQN update proposed by Van Hasselt et al. (2016)). The details of the architecture and hyperparameters are described in the appendix.
25
phoente
chogper command vidha ginbal
SOACE Invaders
bank hele
lungdd master
Rrettitee
20
Holo RZ02
K202
ployesor
our
Figure 6: Difference in human normalized score per game in Atari, HiMo versus the improved R2D2 after 200k learning steps, alongside learning curves for selection of HiMo worst and top performing games. Note that the high variance of the curves in Atari between seeds usually he explained by the variahle timestep which different seeds jump from one performance plateau the next.
We ran our approach on 57 Atari games for 200k gra- dient steps (around day of training), with 3 seeds for each game. The evaluation averages the score between 200 episodes across seeds, each lasting maximum of 30 minutes each and starting a random number (up to 30) of no-op actions. In order compare scores be- tween different games and aggregated results, we com- puted normalized scores for each game based ran- dom and human performance so that 0% corresponds random performance and 100% corresponds hu- man. We observed an increase of 132.5% in the me- dian human normalized score compared the R2D2 baseline with the same network capacity, ag- gregate results are reported in Table Figure 6 details the difference in normalized score between HiMo and our R2D2 baseline for all games individually. We note that the original R2D2 results reported by Kapturowski et al. (2019). which used similar hardware configuration but different network architecture, were around 750% median human normalized score after a day of training.
Table I: Median and mean human normalized scores across 57 Atari2600 games for HiMo versus the R2D2 baseline after day of train- ing.
R2D2
R2D2 + HiMo
Median Mean
832.5%
965 %
2818.5%
2980 %
In experimental evaluation we observed that HiMo typically either offers improved data ef- ficiency or has no overwhelming adverse effects in training performance. In Figure 6 we show training curves for selection of representative Atari environments where at evaluation time HiMo both under-performed (left) and out-performed R2D2 (right); these seem to indicate that in the worst scenario HiMo's training performance reduces to R2D2's
Bowling is one of the Atari games where rewards are delayed with relevant information being com- municated through intermediate observations (the ball hitting the pins), just like the baseball example
We will include performance analysis over longer training regimes in future version of the paper.
8

---

Under review as a conference paper at ICLR 2020
have used in the introduction. We found HiMo to perform better than the R2D2 baseline in this particular game. We also ran HiMo with the actor-critic setup (IMPALA) described the previous section, finding similar performance gain with respect to the model-free baseline. Learning curves for these experiments are presented in Figure 7
100
199
140
120
150
100
Hilo + IMPALA
INPALA
S2002
133000
250000
300300
50066
383300
350000
300000
Parmiater
Parameter
(a)
(b)
(c)
Figure 7: (a) The bowling game in Atari, where delayed reward be predicted the intermediate event of the ball hitting the pins. (b-c) learning curves for HiMo in the howling game using different RL methods: a value-based method (R2D2) in (b) and policy-gradient method (IMPALAA) (c).
RELATED WORK
Recent work have used auxiliary predictions successfully in RL as mean to obtain richer signal for representation learning (Jaderberg et al. [016]] /Sttton et al.) [00I]) However these additional prediction tasks are hard-coded and so they cannot adapt to the task demand when needed. We see them as a complementary approach to more efficient learning RL.
Buesing et al. (2018) have considered using observations in an episode trajectory in hindsight to infer variables in a structural causal model of the dynamics, allowing to reason more efficiently in model-based way about counterfactual actions. However this approach requires learning accurate generative model of the environment.
In supervised learning, the learning using privileged information (LUPI) framework introduced by (Vapnik& & Izmailov] 2015) considers ways of leveraging privileged information at train time. Al- though the techniques developed in that work do not apply directly in the RL setting, some of our approach can be understood in that setting as considering the future trajectory as the privileged information for value prediction problem.
Privileged information coming from full state observation has been leveraged in RL to learn better critic asymmetric actor-critic architectures (Pinto al. 2017 Zhu et al., (018). However this does not use future information and only applies to settings where special side-information (full state) is available train time.
CONCLUSION
High-dimensn High-dimensional observations in the intermediate future often contain task-relevant features that can facilitate the prediction of an RL agent's final return. We introduced a reinforcement learning algorithm, HiMo, that leverages this insight by the following two-stage approach. First, by reason- ing in hindsight, the algorithm learns to extract relevant features future observations that would be been most helpful for estimating the final value. Then, forward model is learned to predict these features, which in turn used as input to an improved value function, yielding better policy evaluation and training test time. We demonstrated that this approach can help tame complexity environments with rich dynamics at scale, yielding increased data efficiency and improving the performance of oate-offttheart model-free architectures.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactually-guue policy search. arXiv preprint 10I 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint 1802.0:800.00100 2018.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. arXiv preprint arXiv:1906.022 02736, 2019.
Arthur Guez, Mchdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, et al. An investigation of model-free planning. In International Conference Machine Learning, pp. 2464-244222 2473, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 161 .05397, 2016.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent ex- perience replay in distributed reinforcement learning. International Conference on Learning Representations, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2011.
Richard S Sutton, Joseph Modayil, Michael Delp. Thomas Degris, Patrick Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsu- pervised sensorimotor interaction. The 10th International Conference on Autonomous Agents and Multiagent Systems- Dostems-Volume 2, pp. 761-768, 2011.
Erik Talvitie. Model regularization for stable sample rollouts. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pp. 780-789. AUAI Press, 2014.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: similarity control and knowledge transfer. Journal of machine learning research, 10 1 2015.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvu- nakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/ fabs/1802... 008.802..995644 2018.
10

---

3. The Under review as a conference paper at ICLR 2020
A APPENDIX
A. GENERAL ARCHITECTURE DETAILS
To compute at and train é in an online fashion, we process fixed-length unrolls of the state-RNN and compute the hindsight value and corresponding updates at time t if i t++ k is also within that same unroll. Also, we update ot at slower rate (i.e., a < 8) to give enough time for the model Φ adapt to the changing hindsight features Φb In our experiments we found that even a low-dimensional &b (in the order of d = 3) and relatively short hindsight horizon & (in the order of 5) are sufficient to yield significant performance boosts, whilst keeping the extra model computational costs modest.
A.2 PORTAL CHOICE
Environment
The observation is a 7 23 RGB frame (see Figure 4). There are possible spawn- ing points for the agent in the center and 42 possible portal positions (half of which lead to the green room, the other half leading to the red room). At the start of an episode, two portals, each leading to different room, are chosen are random. They are both displayed as cyan pixels. In- cluded in the observation in the first phase is the context, a random permutation in a 5 x 5 grid of N pixels, where is uniformly sampled at the start of each episode: N ~ U11, 10). A fixed map f (I,..., 10} - {0, I} determines which contexts are rewarding with the green room, the rest being rewarding with the red room. The reward when reaching the goal is determined according to:
R 2(J(N)G + (1 - f(N)) G)),
(14)
where G E {0, is whether the reached room is green.
Network architecture The policy and value network takes in the observation and passes to a ConvNet encoder (with filter channels [32, 32, 32], kernel shapes [4, 3, 3] applied with strides [2, 1, ID) before being passed to a ConvLSTM network with 32 channels and 3x3 filters. The output of the ConvLSTM is the internal state h. The & network is a ConvNet with [32, 32, 32, 1] filter channels with kernels of size except for final Ixl filter, whose output is flatten and passed to an MLP with 256 hidden units with ReLu activation, before a linear layer with dimension & network is similarly configured network with one less convolution layer and 128 hidden units in the MLP. The ,i network is an MLP with 256 hidden units followed by a linear layer that takes and & input and outputs the policy Th and the value n w+ is obtained similarly with a similar MLP that has a single scalar output. We used a future observation window of = 5 steps in this domain and loss weights was done with the Adam optimizer (learning rate of 5e - 4), with batch size 32. The model-free
0.25, 8 0.5. Unroll length was 20, and
0.99. Optimization
baseline is obtained by using the same code and network architecture, and setting the modeling loss and hindsight value loss to 0 (a = 3 = -).
A.3 ATARI
Hyper-pan paremmearaeters and infrastructure are the same as reported in Kapturowski et al. (2019), with deviations as listed in table 2 For our value target, we also average different n-step returns with exponential averaging as in Q(2) (with the return being truncated at the end of unrolls). The Q network is composed of a convolution network (cf. Vision ConvNet in table) which is followed by an LSTM with 512 hidden units. What we refer to in the main text as the internal state h is the output of the LSTM. The & and o networks are MLPs with a single hidden layer of 256 units and ReLu activation function, followed by linear which outputs vector of dimension d. The the functioon concatenates h and & as inputs to an MLP with 256 hidden units with ReLu activation function, followed by linear which outputs o (a vector of dimension 18, the size of the Atari action set). 7Thh obtained by passing and & to a dueling network as described by Kapturowski et al. (2019).
Other HiMo parameters are described in table g
The R2D2 baseline with the same capacity is
obtained by running the same architecture with a 3
11

---

Under review as a conference paper at ICLR 2020
Table 2: values used for our R2D2 implementation.
Number actors
320
Sequence length
80 (+ prefix of = 20 in burn-in experiments)
Learning rate
2e-1
Adam optimizer $1
0.9
Adam optimizer $2
0.999
0.7
Target update interval
400
Value function rescaling
g(x) sign(x)
2
+ 1 - + cx, 6 10-3
Frame pre-processing
None (full res. including no frame stacking)
Vision ConvNet filters sizes
[, 5, 3] [4. 2, 2, 1]
Vision ConvNet filters strides
Vision ConvNet filters channels
[32, 64, 128, 128]
Table 3; Hindsight modeling parameters for Atari
0.01
1.0
k
5
12


Reviews:
[{"review_id": "Skg7ydsXcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Value-Driven Hindsight Modelling proposes a method to improve value function learning. The paper introduces the hindsight value function which estimates the expected return at a state conditioned on the future trajectory of the agent. How use this hindsight value function is not obvious, since an agent does not have access to the future states needed in order to take actions (for Q-Learning) and the hindsight value function is a biased gradient estimator for training policy gradient methods. \n\nThe authors train the standard value function (which does not have access to future information) to predict the features which the highsight value function learns to summarize the value relevant parts of the future trajectory. These predicted features can then be used in place of the actual hindsight value function, circumventing the issues discussed above. The authors argue that this auxiliary objective provides a richer training signal to the normal value function, helping it to better learn what information in a given state is relevant to predicting future rewards.\n\nThe paper is well structured and written, flowing from high level motivation and review into the core of the method, followed by analysis of the approach, and then proceeds through three experiments. The first two are toy / crafted experiments which build intuition and probe the behavior of the method and finally a large scale test on the Atari 57 benchmark demonstrating improvements when augmenting a state-of-the-art method with HiMo.\n\nThis reviewer recommends acceptance (I would give a 7 given more granularity) based on the contribution of a new auxiliary objective for value functions and the strength of the experimental suite. The Portal Choice environment is well crafted and instrumented with the graphs of figure 5b and 5c to show the behavior of the approach and the clean demonstration of an improvement over a previously SOTA method for Atari 57 is encouraging (the same architecture and the ablation simply sets the auxiliary objective’s weight to 0). However, the reviewer has some caution and concerns as follows:\n\n1) The lack of a large scale experiment demonstrating improvement with an actor-critic method. While the Portal Choice experiments are informative and use Impala, it is a bit toy, and it would increase the reviewer’s confidence in the generality and robustness of the approach if improvements were also demonstrated for an actor-critic method on a large environment suite. Atair 57 could work but ideally a different setting such as DMLab 30 or continuous control from pixels. Demonstrating improvements in one of these additional settings would raise the reviewer to a strong acceptance.\n\n2) The potential sensitivity of the approach to the two important hyperparameters that the authors mention, the dimensionality of the hindsight feature space (to reduce approximation error) and the # of future states it conditions on (to avoid just observing the full return directly). The very low dimensionality of the hindsight feature space (d=3 for Atari) seems a bit at odds with the explanation that the hindsight features provide a strong training signal for learning to better extract value relevant information from the state. Experiments that studied sensitivity to these would provide better perspective on the robustness of HiMo.\n\nQuestions and suggestions for improving the paper:\n\nFor Figure 6 the dynamic range gets squashed by a few games with relatively large performance improvements or regressions. Changing to a log-scale on the y-axis could be more informative? For instance, I find it pretty difficult to eyeball the ~1 human normalized score median improvement according to Table 1 from the chart.\n\nFigure 3 could also be improved. It requires significant context from definitions in the paper in order to understand. It could be reworked into a stand alone expository overview of HiMo that helps readers quickly grok the idea of the paper such that abstract + figure is enough.\n\nCould the authors consider showing / adding full learning curves (median human normalized score?) for HiMO vs the baseline on Atari 57? This would help readers get a qualitative feel for the learning dynamics of the algorithm instead of only having a final scalar measure at the end of training.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylXiem0KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a way to learn better representation for RL by employing a hindsight model-based approach. The reasoning is that during training, we can observe the future trajectory and use features from it to better predict past values/returns. However to make this practical, the proposed approach fits an approximator to predict these features of the future trajectory from the current state and then subsequently, use them to predict the value. The authors claim that this extra information can be used to learn a better representation in some problems and lead to faster learning of good policies (or optimal value functions)\n\nDecision:\nWeak Accept\nMy decision is influenced by the following two key reasons\n\n(1) I like the idea of hindsight modeling a lot. It is true that a trajectory gives much more information than just a weak scalar signal indicating return from each state in the trajectory. Identifying a way to make use of all the extra information in the trajectory to aid in value prediction is useful. The proposed approach is a step towards that, and I think the community should be made aware of that for sake of future research in this direction.\n\n(2) Having said that, I am not super satisfied with the way the authors have presented their approach. The explanation is jumbled and confusing, at times. The paper needs careful rewriting to communicate ideas better and notation needs to be standardized earlier. Some of the sections are either redundant or lack insights. Even if they do have insights, they are not highlighted leaving the reader to search for them. The experimental setup is not clear and the authors could have spent more space in the paper dedicated to how the hindsight modeling approach can be implemented within an existing RL method.\n\nComments:\n\n(1) The line in abstract \"but this approach is usually not sensitive to reward function\" doesn't make sense. Isn't reward function part of the model? So you are learning the reward function, so how is it not sensitive to it? I think I understand what the authors are saying but it took me until the end of Sec 3.2 to get that. \n\n(2) How does this work relate to Value-aware model learning works from Farahmand (AISTATS 2017, NeurIPS 2018). The premise seems to be similar: learn a model taking into account the underlying decision-making problem to be solved and the structure of the value function. The paper needs a discussion of these set of works\n\n(3) In Section 3.3, \\phi_{\\theta_2} has conflicting function parameters in eq (2) and (3). \n\n(4) Section 3.4 is very confusing. I understood the setup of the problem and it seemed like it was very illustrative of an example where proposed approach will excel. However, Fig 1 and its caption are unclear and I found it hard to understand what the figure is conveying. The paragraph underneath the figure had no explanation for the Fig 1, and instead directly jumped to the results in Fig 2. The paper could use a better explanation of Fig 1. and explain why the proposed approach can learn the structure of s' and better predict value at s\n\n(5) Section 3.5 partially answers the question \"when is it advantageous to model in hindsight?\" In cases, where L_model is low, of course its advantageous to model in hindsight! But the real question that needs to be answered is buried in the last paragraph. What if learning a good \\phi is as hard as predicting the return? In this case, do we still gain any advantage? I am not sure how having a limited view of future observations and low dimensional \\phi helps. If the feature that decides future return lies beyond the limited view of future observations, does it still not give any advantage? Questions like these might be useful in aiding the reader to understand why hindsight modeling is better\n\n(6) Section 4 needs more text to explain what components of the architecture are learnt using what losses, and provide intuitions for why that is the case. It seems like that is very crucial to ensure that \\phi doesn't learn something trivial and non-useful. I am surprised section 4 is so small, and Fig 3 is not useful. Maybe, you can combine section 3.4, 3.5 and condense them, and using the obtained space in expanding sec 4.\n\n(7) The experiments section immediately dives into the problem setup and results. It will be useful to have a subsection explaining how the proposed hindsight model is implemented within an RL algorithm. Currently, it is hard for the reader to connect what he/she has read until Section 4 with what's presented in Section 5.\n\n(8) The results are convincing. However, my biggest concern is the experiments were not designed carefully to analyze how much the hindsight modeling contributed in the increase of performance? Are the number of parameters in the value function approximator the same between the hindsight RL algorithm and the baseline? Can we have a simplistic example that is amenable to isolate the influence of hindsight modeling from other factors? Fig. 2 does a reasonable job at it but I think the hindsight modeling approach can achieve improvement in more diverse problems. In a way, the proposed feature is doing state space augmentation so that value can be easily predicted from the features of the augmented state. So, identifying the characteristics of the problems where this can be done is very useful to the RL practictioner. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byev_nUoFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new model-based reinforcement learning method, termed hindsight modelling. The method works by training a value function which, in addition to depending on information available at the present time is conditioned on some learned embedding of a partial future trajectory. A model is then trained to predict the learned embedding based on information available at the current time-step. This predicted value is fed in place of the actual embedding to the same value model, to generate a value prediction for the current time-step. So instead of just learning a value function based on future returns, the method uses a two-step process of learning an embedding of value relevant information from the future and then learns to predict that embedding.\n\nThe paper gives some motivating examples of why and when such an approach could yield an advantage over standard value learning methods like Monte-Carlo or temporal difference learning. The basic idea is that when the returns obey some causal structure like X->Y->Z it may be easier to learn P(Y|X) and P(Z|Y) than to learn P(Z|X) directly. In particular, the authors point out that in the discrete case when Y takes relatively few values the size of the respective probability tables can be smaller in the former case than the latter. This motivates the approach of discovering a set of future variables to predict which are themselves predictive of return, rather than predicting the expected future return directly.\n\nOn a high level, I like the idea of specifically learning to model relevant aspects of the environment. However, I lean toward rejecting this work in its current form because I feel the motivation for when this particular method would be useful is unclear.\n\nIn particular, I don't really understand how this method could, in general, be expected to improve on regular bootstrapping. Why learn a prediction of the return at time t based on future information when we could just use the value function at a later time to improve the prediction at time t? It seems to me that the future value function itself concisely summarizes the information in the future state that is relevant for predicting the past return, while better exploiting the structure of the problem. Of course in cases like partial observability, it could be that the future value function lacks information from the past that is important for accurately predicting the return (for example in the portal example of this paper). However, if partial observability is really the case of interest, the method presented in this paper seems like a rather roundabout solution method. For example, instead of conditioning v+ on a future hidden state h_{t+k} (as the authors do in the experiments) perhaps one could simply condition the value function on a past hidden state h_{t-k} and obtain similar benefit from bootstrapping?\n\nAside from partial observability, for which I feel there are better approaches, the only situation I can understand the method having an advantage is when later states contain information which helps to predict earlier rewards. This is essentially the situation presented in the illustrative example. However, currently I feel such situations are rather contrived and unintuitive so I would need more supporting evidence to accept these situations as a good motivation.\n\nOn a deeper level, I don't see how the probability table motivation given in the introduction applies when what is being learned is an expectation (i.e. a value function) and not a distribution.\n\nThe approach also suffers from well-known issues with using the output of an expectation model of a variable as the input to a nonlinear function approximator in place of the variable itself. Namely, there is no guarantee that the expectation value of a variable is a possible value for the variable so giving it as input to a predictor trained on the variable itself could easily yield nonsense output in the stochastic case. As far as I can tell the method does nothing to mitigate this (please correct me if I'm wrong), so there is no reason to assume the method is generally applicable in settings with nontrivial stochasticity.\n\nDespite these concerns, I feel the experiments are for the most part quite well thought out and executed. The paper is also quite well written, motivation issues aside, so I would not be upset if it was accepted with the hope that it leads to future work addressing the above-mentioned concerns.\n\nIf possible I think this paper would benefit significantly from a detailed explanation of how and when the proposed approach should be expected to improve on bootstrapping, including bootstrapping off a value function which uses an analogous architecture to v+.\n\nQuestions for authors:\nGiven the hyper-parameters of R2D2 deviate somewhat from those used in the original paper, and nothing is said about how they were chosen, how confident can we be that the observed advantage of hindsight modelling is not simply due to hyper-parameters being selected which are more favourable for the proposed method?\n\nGiven that you are not learning distributions but expectations in the form of value functions, how pertinent is the motivation of learning P(Y|X) and P(Z|Y) instead of P(Z|X) directly described in the introduction?\n\nHow much of the benefit observed in the portal example an ATARI could also be gained from simply providing the value function approximation with h_{t-k} as input to help span larger time-gaps?\n\nUpdate:\n\nWhile I still feel the exposition could be improved to make the underlying idea clearer, I feel the authors did a good job of addressing my major concerns in their reply, hence I have raised my score to a weak accept.\n\nI have to admit I missed the point that v^+ and v^m were using entirely different parameter sets. In light of this, I agree that the expectation model issue I mentioned is not a major concern.\n\nI also appreciate the clarification of the hyperparameters, if they were really tuned to improve the baseline then this detail should be added to the paper and would negate my concern there. \n\nFinally, I thank the authors for providing the value-function oriented example. I found this example to be more illustrative than the one in the introduction of the paper, and I now feel that I have a better grasp of the motivation. I still have doubts about the general benefit of the approach over bootstrapping but since it is not entirely clear to me one way or the other I feel the idea at least warrants further exploration, and it would be reasonable to accept the paper to make the community aware of it.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of estimating the value function in an RL setting by learning a representation of the value function. While this topic is one of general interest to the ICLR community, the paper would benefit from a more careful revision and reorganization following the suggestions of the reviewers.
Decision: Reject