Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: No questions
review_title: No questions

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This work proposes a computational structure of function approximator with a strong prior: it is optimized to act as an abstract MRP, capable of learning its own internal state, model, and notion of time-step. Thanks to the incorporation of a \lambda-return style return estimation, it can effectively adapt its own "thinking-depth" on the current input, thus performing some sort of soft iterative inference.

Such a prior, maintained by strong regularization, helps perform better than similar baselines or some prediction tasks that require some form of sequential reasoning.

The proposed idea is novel, and a very interesting take on forcing internal models upon function approximators which begs for future work. The experimental methodology is complete, showcases the potential of the approach, and nicely analyses the iterative/adaptative thinking depth learned by the model.

As pointed out by my previous comments, the paper reads well but utilizes language that may confuse a reader unfamiliar with the subject. I think some rewording could be done without having much impact on the depth of the paper. In particular, introducing the method as a regularized model pushed to act like an MRP, rather than an actual MRP performing some abstract reasoning, may help confused readers such as myself.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an approach to learning models that are good for planning problems, using deep netowork architectures. The key idea is to ensure that models are self-consistent and accurately predict the future. The problem of learning good planning models (as opposed to simply good predictive models is really crucial and attempts so far have failed. This paper is conceptually interesting and provides a valuable perspective on how to achieve this goal. Its incorporation of key RL concepts (like discounting and eligibility traces) and the flexibility to learn these is very appealing. Hence, I think it should be accepted. This being said, I think the paper does not quite live up to its claims. Here are some aspects that need to be addressed (in order of importance):
1. Relationship to past work: the proposed representation seems essentially a non-linear implementation of the Horde architecture. It is also very similar in spirit to predictive state representations. Yet these connections are almost not discussed at all. The related work paragraph is very brief and needs expansion to situate the work in the context of other predictive modelling attempts that both were designed to be used for planning and (in the case of PSRs) were in fact successsfully used in planning tasks. Some newer work on learning action-conditional models in Atari games are also not discussed. Situating the paper better in the context of existing model learning would also help understand easier both the motivations and the novel contributions of the work (otherwise, the reader is left to try and elucidate this for themselves, and may come to the wrong conclusion).
2. The paper needs to provide some insight about the necessity of the recurrent core of the architecture. The ideas are presented nicely in general fashion, yet the proposed impolementation is quite specific and "bulky" (very high number of parameters). Is this really necessary in all tasks? Can one implement the basic ideas outside of the particular architecture proposed? Can we use feedforward approximations or is the recurrent part somehow necessary? At the very least the paper should expand the discussion on this topic, if not provide some empirical evidence.
3. The experiments are very restricted in their setup: iid data drawn from fixed distributions, correct targets. So, the proposed approach seems like an overkill for these particular tasks. There is an indirect attempt to provide evidence the learned models would be useful for planning, but no direct measurement to support this'd claim (no use of the models in planning). Compared to the original Horde paper, fewer predictions are learned, and these are more similar to each other. While I sympathize with the desire to go in steps, I think the paper stops short of where it should. At the very least, doing prediction in the context of an actual RL prediction task, with non-iid inputs, should be included in the paper. This should only require minor modifications to the experiments (same task, just different data). Ideally, in the case of the mazes, the learned models should be used in some form of simplified planning to learn paths. This would align the experiments much better with the claims in the presentation of the architecture.
review_title: A very worthwhile idea, but the empirical results could have been better aligned with the main message

final_score: rating: null
review_content: - I interpret setting the different discount factors as forcing the internal representation to represent different time-scales. Is this right?
- If not, how is the model deciding its own timescales? (which is more or less suggested in the intro)
- (and if so, do you think there a simple extension to this where the model learns its own timescales?)
- both tasks you show are supervised, and have arguably strong, high-level signals as supervision. What do you think will happen if the only supervision signal is a sparse reward? (as in a typical control task)
review_title: Questions on gamma and supervision

final_score: rating: 4: Ok but not good enough - rejection
review_content: I think there may be a nice paper to made from this, but as it is, it should not be accepted.   The authors describe a new architecture for regression, inspired by techniques for estimating the value function of an Markov reward process.   The connection is interesting, and there is certainly merit in the idea.   However, the writing is confusing,  and as far as I can tell, the experiments and discussion are inadequate.  It is quite possible that I am misunderstanding some things, so I am not putting high confidence.

Because of all the discussion of MRP's and the background that inspired the model, it is difficult to see that the authors are in a pure, i.i.d. regression setting, where they sample inputs i.i.d. (with deterministic outputs given the input) from a distribution, and try to match  a parameterized function to the input output pairs.    

Because they are in this setting, there is a lot lacking from the experiments.  For example, they report l2 loss on the maze problem; but not "percent correct"; indeed, it looks like the deep net with skips goes to about .001 average l2 loss on the 0-1 output maze problem.   This is an issue because because it suggests that by simply thresholding the outputs, you could get nearly perfect results, which would point to a model specification error of the baseline.  Are there sigmoids at the end of the baseline plain deep network?  Note that the proposed models do have sigmoids in the outputs in the multiplicative weightings.     

How do the number of parameters of the proposed network compare to the baselines?   Is the better performance (and again, better is really marginal if I am understanding the way loss is measured) simply an issue of modeling power (perhaps because of the multiplicative connections of the proposed model vs. the baseline)?  Because the input is taken i.i.d and the test distribution exactly matches the train, this is an important part of the discussion.  Moreover, there do not seem to be experiments where the size of the training set is fixed- the axis in the graphs is number of samples seen, which is tied to the number of optimization steps.  Thus there is no testing of over-fitting.

Why not try the model on more standard regression problems (as at heart, the paper seems to be about a new convnet architecture for regression)?   Show imagenet or cifar accuracies, for example.  If  the proposed model does worse there, try to explain/understand what it is about the reported tasks that favor the proposed model?




**********************************************************************************
edited with increased confidence in post review discussions
**********************************************************************************
review_title: null

Meta-review: There is potential here for a great paper, unfortunately in its current form there is too deep of a disconnect between the framing and promise of the presentation, and the empirical validation actually delivered by the experiments.
 The choice of the experimental setting (iid input from fixed distributions in a pattern recognition setting where targets are between 0 and 1) is too narrow to allow for convincing validation of the central hypothesis of the paper, namely, that the architecture (a recurrent convnet with sigmoid gating) can be useful for problems involving planning. Instead, it simply shows that the architecture is better at outputting the targets than other deep architectures without sigmoid gating.
 I strongly encourage the authors to add more ambitious experiments to keep the empirical arm more in step with the stated promises of the set-up.
Decision: Reject