Title: The Predictron: End-To-End Learning and Planning
Authors: David Silver
Hado van Hasselt
Matteo Hessel
Tom Schaul
Arthur Guez
Tim Harley
Gabriel Dulac-Arnold
David Reichert
Neil Rabinowitz
Andre Barreto
Thomas Degris
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Supervised Learning
Semi-Supervised Learning

Abstract: One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple "imagined" planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. 
The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function, thereby focusing the model upon the aspects of the environment most relevant to planning. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.


Review discussion:
- I interpret setting the different discount factors as forcing the internal representation to represent different time-scales. Is this right?
- If not, how is the model deciding its own timescales? (which is more or less suggested in the intro)
- (and if so, do you think there a simple extension to this where the model learns its own timescales?)
- both tasks you show are supervised, and have arguably strong, high-level signals as supervision. What do you think will happen if the only supervision signal is a sparse reward? (as in a typical control task)
In the pool domain, we indeed make predictions that span different time scales.
However, this does not force the predictron to represent different time scales internally.  It is better to think of the internal steps as reasoning steps, rather than time steps.  Some long-term predictions may be made in a single reasoning step.
Conversely, some near-time events may require multiple reasoning steps to predict accurately.
When certain predictions are made with few internal reasoning steps, this can free up representation capacity on later steps for more challenging predictions.
So although the predictron’s internal states may indeed capture information that is relevant for predictions that span different external time scales, perhaps not all internal steps have to retain information about all external time scales.
Our analysis on predictions in pool does show a correlation between external time scales and internal reasoning steps (see Fig. 6). Interestingly, sometimes the correlation is reversed (right-most column in Fig. 6).  These correlations emerge automatically, probably because these help make more accurate predictions.
To answer the last question, the experiments can be interpreted as using a Monte Carlo algorithm to update the predictions.  Some of the signals are already quite sparse (e.g., only a single reward at the end, in the maze task).
I hope this clarifies things, thanks for your questions!
It is still a bit unclear to me how this architecture works, so here are a few questions:
- when you say (in the appendix) "we allow 16 model steps", this means you set K=16?
- Do you perform these K "models steps", at each "environment step"?
- Is \mathbf{v} trained to match the external "reward"? (or in your case maze connectedness or pool events) If so, how?
- My interpretation of your work is that your model is akin to a recurrent network (similar to Graves 2016) with input only at the first time step, that is strongly regularized to match the dynamics of an MRP, as well as \lambda-return. Is this correct?
- why don't you compare to Graves 2016? Or mention them in the related work? It seems like the prior-free method that is closest to yours.
I thought these were unclear in the paper. If you also think it might be the case it would be nice to explicit those in the paper. If it's just me, I apologize. I really like the main idea so far.
Thanks for your insightful comments and questions!  It is very helpful to learn which parts of the paper were clear, and which could perhaps be explained more clearly.
I'll reply to your questions one by one below.
Q: when you say (in the appendix) "we allow 16 model steps", this means you set K=16?
A: Yes.
Q: Do you perform these K "models steps", at each "environment step"?
A: Yes, we take K reasoning steps for each new input.
Q: Is \mathbf{v} trained to match the external "reward"? (or in your case maze connectedness or pool events) If so, how?
A: The internal value function v is trained end to end together with the internal rewards and discounts so that the predictions made by the predictron are close to the external targets. These internal predictions are what we call preturns, as defined in Equation (2), which can include rewards and discounts in addition to values.
All parameters of the model and value functions are jointly trained to minimize the difference between each preturn g^k and the target g (using a variant of stochastic gradient descent - in particular we used Adam); see Equations (3) and (4).
In particular, the total loss is a sum of squared losses (g - g^k)^2.
The weight w^k on each loss (g - g^k)^2 is defined by the λ parameters (results in Figure 3, left plots):
w^k  =  λ^0 * … * λ^{k-1} * (1 - λ^k) .
Or, alternatively, we can weight each loss equally so that w^k = 1/K (Figure 3, right plots). This is technically explained in the current version paper but we intend to explain it more clearly in our next update.
So, the values on the first abstract state, v^0 = v(s^0), are indeed trained to match the external targets, because the 0-step preturn is equal to this value: g^0 = v^0.  Values at later states are trained only in the context of internal rewards and discounts.
For instance, the 2-step preturn is defined as:
g^2  =  r^1  +  𝜸^1 r^2  +  𝜸^1 𝜸^2 v^2 .
All components of this preturn are updated so that g^2 is close to the external target, but v^2 does not itself have to match the target.
In Section 5.1, we examine what happens if there are no rewards (r^k = 0) and no discounts (gamma^k = 1).
Then each preturn is equal to the corresponding value: g^k = v^k, and then all the values are indeed updated to be close to the corresponding targets.  We call this the λ-predictron; see the purple lines in Figure 3 for its relative performance.
The external targets can themselves be (discounted) accumulations of external ‘reward’ signals.
For instance, in pool one such target is the discounted cumulative number of steps the white ball is in the top-left quadrant in a certain trajectory.  In the random mazes, each target is equal to one when the maze is connected from the corresponding starting location, and equal to zero otherwise.
Q: My interpretation of your work is that your model is akin to a recurrent network (similar to Graves 2016) with input only at the first time step, that is strongly regularized to match the dynamics of an MRP, as well as \lambda-return. Is this correct?
A: Yes, the repeated application of a model can indeed be interpreted as the unrolling of a recurrent neural network, similar to the ACT algorithm proposed by Alex Graves. The addition of learnable rewards and discounts additionally increase flexibility compared to just having values; Figure 3 shows that this improved performance.
In both cases there is the restriction of sharing the same model (or recurrent core) on each step.
However, the results in Figure 4 (compare the left plots to the right plots) indicate that the MRP core was more important than whether or not the same core was shared across all steps.
Q: why don't you compare to Graves 2016? Or mention them in the related work? It seems like the prior-free method that is closest to yours.
A: Alex Graves’ ACT algorithm is similar to what we call the λ-predictron, because it does feature multiple outputs (similar to what we call values) but does not include internal rewards or discounts. The results in Figure 3 show that these additions help improve performance. It is a good idea to discuss the similarities and differences to ACT in more detail in the paper, thanks for the suggestion.
Thank you for your answers, they helped me understand your work better.
When I read: "The main idea of the predictron is that each preturn g^k should accurately estimate the true value of state s", I understood "the true value" as being some latent intrinsic "value"/representation that the predictron would learn to model, instead of just some external supervised targets. That's on me, but maybe it would be worth adding a few words so that readers unfamiliar with the subject do not make the same mistake as I.
(Similarly in the abstract, I interpreted "predictions about the future" as solely predictions in the latent space/state space of the MRP rather than the target/reward space.)
Everything makes sense now, thanks again!
This work proposes a computational structure of function approximator with a strong prior: it is optimized to act as an abstract MRP, capable of learning its own internal state, model, and notion of time-step. Thanks to the incorporation of a \lambda-return style return estimation, it can effectively adapt its own "thinking-depth" on the current input, thus performing some sort of soft iterative inference.
Such a prior, maintained by strong regularization, helps perform better than similar baselines or some prediction tasks that require some form of sequential reasoning.
The proposed idea is novel, and a very interesting take on forcing internal models upon function approximators which begs for future work. The experimental methodology is complete, showcases the potential of the approach, and nicely analyses the iterative/adaptative thinking depth learned by the model.
As pointed out by my previous comments, the paper reads well but utilizes language that may confuse a reader unfamiliar with the subject. I think some rewording could be done without having much impact on the depth of the paper.
In particular, introducing the method as a regularized model pushed to act like an MRP, rather than an actual MRP performing some abstract reasoning, may help confused readers such as myself.
I think there may be a nice paper to made from this, but as it is, it should not be accepted.   The authors describe a new architecture for regression, inspired by techniques for estimating the value function of an Markov reward process.   The connection is interesting, and there is certainly merit in the idea.
However, the writing is confusing,  and as far as I can tell, the experiments and discussion are inadequate.  It is quite possible that I am misunderstanding some things, so I am not putting high confidence.
Because of all the discussion of MRP's and the background that inspired the model, it is difficult to see that the authors are in a pure, i.i.d. regression setting, where they sample inputs i.i.d. (with deterministic outputs given the input) from a distribution, and try to match  a parameterized function to the input output pairs.
Because they are in this setting, there is a lot lacking from the experiments.
For example, they report l2 loss on the maze problem;
but not "percent correct";
indeed, it looks like the deep net with skips goes to about .001 average l2 loss on the 0-1 output maze problem.   This is an issue because because it suggests that by simply thresholding the outputs, you could get nearly perfect results, which would point to a model specification error of the baseline.  Are there sigmoids at the end of the baseline plain deep network?  Note that the proposed models do have sigmoids in the outputs in the multiplicative weightings.
How do the number of parameters of the proposed network compare to the baselines?   Is the better performance (and again, better is really marginal if I am understanding the way loss is measured) simply an issue of modeling power (perhaps because of the multiplicative connections of the proposed model vs. the baseline)?  Because the input is taken i.i.d and the test distribution exactly matches the train, this is an important part of the discussion.
Moreover, there do not seem to be experiments where the size of the training set is fixed- the axis in the graphs is number of samples seen, which is tied to the number of optimization steps.
Thus there is no testing of over-fitting.
Why not try the model on more standard regression problems (as at heart, the paper seems to be about a new convnet architecture for regression)?   Show imagenet or cifar accuracies, for example.  If  the proposed model does worse there, try to explain/understand what it is about the reported tasks that favor the proposed model?
edited with increased confidence in post review discussions
We thank the reviewer for the constructive criticism, and hope that we can address all their concerns.
Quote: The authors describe a new architecture for regression [...] Why not try the model on more standard regression problems (as at heart, the paper seems to be about a new convnet architecture for regression)?
Response: The predictron is not aiming to be a generic architecture for regression.
Instead it is intended for domains that can benefit from a flexible number of reasoning steps.  The domains we test on reflect this, where appropriate solutions might include sequential reasoning or planning steps.
Quote: the writing is confusing, and as far as I can tell, the experiments and discussion are inadequate.  It is quite possible that I am misunderstanding some things, so I am not putting high confidence.
Response: We revised the paper for clarity. If the reviewer is prepared to take another look, we hope that the presentation is now easier to digest.
Quote: it is difficult to see that the authors are in a pure, i.i.d. regression setting, where they sample inputs i.i.d. (with deterministic outputs given the input) from a distribution, and try to match  a parameterized function to the input output pairs.
Response: We now state this more clearly in the paper (I.e., “In both domains, inputs are presented as mini-batches of i.i.d. samples with their regression targets.”).
Of course, this holds only for the experiments - the predictron itself is not specific to the i.i.d. case.
Quote: Because they are in this setting, there is a lot lacking from the experiments.
For example, they report l2 loss on the maze problem;
but not "percent correct";
indeed, it looks like the deep net with skips goes to about .001 average l2 loss on the 0-1 output maze problem.   This is an issue because because it suggests that by simply thresholding the outputs, you could get nearly perfect results, which would point to a model specification error of the baseline. Are there sigmoids at the end of the baseline plain deep network?  Note that the proposed models do have sigmoids in the outputs in the multiplicative weightings.
Response: Only discount and lambda components are passed through sigmoids, not the value and reward components.  The baseline deep net is not hindered by its functional form, but by the fact that it is unable to get the correct predictions on a small fraction of mazes.
The baseline only differs from the predictron in terms of the differences that are explicitly stated in the paper - all experiments are conducted with the same codebase, by only changing the relevant parameters (e.g., boolean flags), so the reader can be assured that there are no other hidden differences in the comparisons.
I am not sure I understand why reporting ‘percent correct’ would be more informative - we show the MSE because that is what all the models are trying to minimize.
However, if the reviewer prefers to have the classification error as well, we can add those results to the appendix.
Quote: How do the number of parameters of the proposed network compare to the baselines?   Is the better performance (and again, better is really marginal if I am understanding the way loss is measured) simply an issue of modeling power (perhaps because of the multiplicative connections of the proposed model vs. the baseline)?
Response: Yes, the predictron variants have more parameters compared to the baseline. We will add an experiment to the supplementary material comparing two variants with near-identical numbers of parameters - this experiment will be included in a later revision.
In the meanwhile, Figure 8 compares different depths and shows that even shallow predictron variants outperform much deeper (unshared) baselines networks.
Additionally, in Figure 4 shows the predictron with a shared code performs better than a deep net with an unshared core. Perhaps it is good to note that the amount of compute time is very similar for all variants, because most of the compute is in the convolutions and all models have the same number of convolutional layers.
Quote: Moreover, there do not seem to be experiments where the size of the training set is fixed- the axis in the graphs is number of samples seen, which is tied to the number of optimization steps.
Thus there is no testing of over-fitting.
Response: In our experiments we were using a simulator and could generate as many samples as we required;
in practice each input was presented only once. The reported errors were measured on unseen samples (before the network is updated on them).
Therefore our results are valid for the large data regime.  The question of (reducing) overfitting in small data regimes is an interesting one to investigate, and perhaps the shared cores or additional (consistency) constraints help in that case, but this was not the focus of this paper.
Quote: Show imagenet or cifar accuracies, for example.  If  the proposed model does worse there, try to explain/understand what it is about the reported tasks that favor the proposed model?
Response: It is indeed possible that a predictron-like architecture could be applicable to classification problems as well, although perhaps domains such as ImageNet or CIFAR are more about vision than about reasoning.
Dear reviewer,
I see you have increased the confidence of the review after discussions, but have kept the same rating.
I just wanted to check whether you have seen the updated revision of the paper which we uploaded on January 19th.  It addresses at least some of the issues you raised in your review.
For instance, this newer version features an experiment in which we compared the predictron architecture to more conventional deep neural network architectures with different numbers of parameters.  The results (Figure 9 in the paper) showed that the performance of the baselines would improve with larger networks on pool (the signal is less clear on the mazes), but that all baselines performed less well than similar predictron architectures, even if the predictron had fewer parameters.
You never responded to our comments on your initial review, and so we would appreciate if you could let us know if there is anything else that we can clarify before you come to a final decision.
Thanks!
I have seen the updated version, but it does not alleviate my concerns;
while the paper *is* clearer, I am unfortunately also more confident that the paper is missing evidence.
The fundamental problem is that the central hypothesis of the paper, that this architecture can do better at certain kinds of problems *because* it is running a MRP in the hiddens, is simply unsubstantiated.  You do present evidence that adding sigmoid gating to your networks improves performance on those regression tasks, but:
On one hand:
1: you are working in the "large data" domain, where any increase in modeling power shows up as improvement
2: in general, multiplicative interactions and/or gating can increase the modeling power per parameter of a network
3: in your particular case, you are modeling a binary valued function in l2 with a relunet.  Look at equation (3) in the paper, and note that the w terms have sigmoids.  The model doesn't care which is the lambdas and discounts, and which is the value and reward.  It will use its parameters to approximate the function as best as possible.  If that means making the reward terms constant and putting all the representation into the lambdas, so be it.  In general, a deep network is going to find all sorts of clever ways of using use sigmoids if the sigmoids are in the network, and they are useful for the approximation.  And undoubtedly, sigmoids are useful for a (piecewise linear!) ReLU net to approximate a 0-1 function.
On the other hand, you do not even define the MRP's underlying the example problems you show, much less give any evidence that the model actually runs these MRP's.  More generally, you do not give any indication of what kinds of problems have an implicit "planning" or "prediction" component and which do not, or how to tell the difference.  In my view, the problems you use are no less pure pattern recognition problems than CIFAR 10 or Imagenet.  It is neither clear to me that your problems  may "benefit from a flexible number of reasoning steps" or "appropriate solutions might include sequential reasoning ", as in the extremely high data regime where they work (with millions of examples)  even nearest neighbors might do well.  Turning this around, the orthodoxy in vision for quite a while was that in order to solve imagenet, we might need to be able to do sequential reasoning over object parts and subparts.
Indeed, that continues to be a standard interpretation of the action of a convnet on image processing tasks.
I do think that you have an interesting hypothesis, and that it is worth studying.  I don't believe the experiments you have presented support or contradict your hypothesis: you have to show that it is really the prediction/planning making things better.
No questions
The paper proposes an approach to learning models that are good for planning problems, using deep netowork architectures. The key idea is to ensure that models are self-consistent and accurately predict the future. The problem of learning good planning models (as opposed to simply good predictive models is really crucial and attempts so far have failed. This paper is conceptually interesting and provides a valuable perspective on how to achieve this goal. Its incorporation of key RL concepts (like discounting and eligibility traces) and the flexibility to learn these is very appealing.
Hence, I think it should be accepted. This being said, I think the paper does not quite live up to its claims. Here are some aspects that need to be addressed (in order of importance):
1. Relationship to past work: the proposed representation seems essentially a non-linear implementation of the Horde architecture. It is also very similar in spirit to predictive state representations.
Yet these connections are almost not discussed at all. The related work paragraph is very brief and needs expansion to situate the work in the context of other predictive modelling attempts that both were designed to be used for planning and (in the case of PSRs) were in fact successsfully used in planning tasks. Some newer work on learning action-conditional models in Atari games are also not discussed. Situating the paper better in the context of existing model learning would also help understand easier both the motivations and the novel contributions of the work (otherwise, the reader is left to try and elucidate this for themselves, and may come to the wrong conclusion).
2. The paper needs to provide some insight about the necessity of the recurrent core of the architecture. The ideas are presented nicely in general fashion, yet the proposed impolementation is quite specific and "bulky" (very high number of parameters). Is this really necessary in all tasks? Can one implement the basic ideas outside of the particular architecture proposed? Can we use feedforward approximations or is the recurrent part somehow necessary? At the very least the paper should expand the discussion on this topic, if not provide some empirical evidence.
3. The experiments are very restricted in their setup: iid data drawn from fixed distributions, correct targets.
So, the proposed approach seems like an overkill for these particular tasks. There is an indirect attempt to provide evidence the learned models would be useful for planning, but no direct measurement to support this'd claim (no use of the models in planning). Compared to the original Horde paper, fewer predictions are learned, and these are more similar to each other.
While I sympathize with the desire to go in steps, I think the paper stops short of where it should. At the very least, doing prediction in the context of an actual RL prediction task, with non-iid inputs, should be included in the paper. This should only require minor modifications to the experiments (same task, just different data). Ideally, in the case of the mazes, the learned models should be used in some form of simplified planning to learn paths. This would align the experiments much better with the claims in the presentation of the architecture.
Thank you for the constructive comments and questions!
I'll answer the questions as numbered above in the review.
We intend to flesh out the related work section a little more, and I agree it would be good to discuss the Horde and other related work.
You mention “newer work on learning action-conditional models in Atari games”.  Are there specific papers you are thinking about?
In addition, could you perhaps elucidate what you would want us to discuss about these papers?  I ask, because we are neither considering action-conditional models nor applying our abstract models to Atari. We do cite two recent papers on action-conditional models applied to Atari (Oh et al., 2015; Chiappa et al., 2016) in the introduction, but these are more intended as recent examples of work in model-based RL more generally.
Similarly, is there any specific work on model learning that you feel should in any case be discussed?  The literature on model-based RL is, of course, too large to cover comprehensively.
In addition, we focus on learning models only to generate accurate predictions and there does not seem to be much prior work on learning such fully abstract models that are not intended to approximate the state dynamics. Are there specific connections that you feel we should be, but are not yet, discussing?
We too had these questions, and indeed there are experiments about this in the paper.  Please see Figure 4.  Here “shared core” refers to using a recurrent core, and “unshared core” to using a feedforward architecture (that is otherwise the same).  As you can see, the performance is very similar and so indeed it is not essential to use a recurrent core.
We did more experiments with different variants, including runs with far fewer parameters.  The qualitative conclusions remained the same, in the sense that the relative performances of the variants we consider remain very similar.
Of course, if we reduce the number of parameters too far, the performance of all variants (including more conventional baselines) deteriorates.
Incidentally, please note that the number of parameters in the model is maybe not as high as you may think, because the cores consist of convolutions.
To be precise, the full (r, gamma, lambda)-predictron used for the maze task has a total of 1,678,400 parameters, which is similar to the number of parameters in DQN.  The predictron for pool has about twice as many parameters (3,285,696, to be exact), because the pool domain has larger inputs and we make more predictions.
The experiments are meant to show, in a clean way, that reasoning with an internal MRP structure can be beneficial in tasks in which we would expect that good solutions consist of sequential reasoning steps. The important point here is that the nature of the task (that the solution has a sequential structure) is more important than the specific learning regime used (i.i.d. inputs or not). It will be interesting to extend this work in various ways, but we feared that trying to include too much at the same time would make the paper less clear.
In addition, the i.i.d. setting is somewhat of a best case for the conventional baselines we compare against, which is nice when we want to fairly judge the merits of the proposed ideas.
In terms of learning paths, I’m not 100% sure what you mean. The predictron can learn accurate predictions.
However, the internal model steps are - deliberately, and by design - not tied to external (time) steps.  Perhaps you mean that we should move towards control, in which the output of the system is not (only) a set of predictions, but also a sequence of actions?  We do intend to move towards control, but deliberately left this for future work.  Please clarify if I misunderstood what you were intending.
I hope this helps clarify things.  Please don’t hesitate to ask more questions or to comment more, the feedback is greatly appreciated.
We have uploaded a new revision of the paper.  The paper was edited to improve clarity and to address comments made by the reviewers and others, and the section on related work was expanded.
Additionally, the appendix includes results on a new experiment that shows the predictron compares favourably to more conventional deep networks architectures, even when the latter have more parameters.

Meta-review:
role: PC
data: comment: There is potential here for a great paper, unfortunately in its current form there is too deep of a disconnect between the framing and promise of the presentation, and the empirical validation actually delivered by the experiments.
 The choice of the experimental setting (iid input from fixed distributions in a pattern recognition setting where targets are between 0 and 1) is too narrow to allow for convincing validation of the central hypothesis of the paper, namely, that the architecture (a recurrent convnet with sigmoid gating) can be useful for problems involving planning. Instead, it simply shows that the architecture is better at outputting the targets than other deep architectures without sigmoid gating.
 I strongly encourage the authors to add more ambitious experiments to keep the empirical arm more in step with the stated promises of the set-up.

Decision: Reject