Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: I really enjoyed reading this paper and stopped a few time to write down new ideas it brought up. Well written and very clear, but somewhat lacking in the experimental or theoretical results.

The formulation of AdaGain is very reminiscent of the SGA algorithm in Kushner & Yin (2003), and more generally gradient descent optimization of the learning rate is not new. The authors argue for the focus on stability over convergence, which is an interesting focus, but still I found the lack of connection with related work in this section a strange.

How would a simple RNN work for the experimental problems? The first experiment demonstrates that the regularization is using fewer features than without, which one could argue does not need to be compared with other methods to be useful. Especially when combined with Figure 5, I am convinced the regularization is doing a good job of pruning the least important GVFs. However, the results in Figure 3 have no context for us to judge the results within. Is this effective or terrible? Fast or slow? It is really hard to judge from these results. We can say that more GVFs are better, and that the compositional GVFs add to the ability to lower RMSE. But I do not think this is enough to really judge the method beyond a preliminary "looks promising".

The compositional GVFs also left me wondering: What keeps a GVF from being pruned that is depended upon by a compositional GVF? This was not obvious to me.

Also, I think comparing GVFs and AdaGain-R with an RNN approach highlights the more general question. Is it generally true that GVFs setup like this can learn to represent any value function that an RNN could have? There's an obvious benefit to this approach which is that you do not need BPTT, fantastic, but why not highlight this? The network being used is essentially a recurrent neural net, the authors restrict it and train it, not with backprop, but with TD, which is very interesting. But, I think there is not quite enough here.

Pros:
Well written, very interesting approach and ideas
Conceptually simple, should be easy to reproduce results

Cons:
AdaGain never gets analyzed or evaluated except for the evaluations of AdaGain-R.
No experimental context, we need a non-trivial baseline to compare with


review_title: Thought provoking but limited

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a suite of algorithmic ideas to learn a network of generalized value functions.  The majority of the technical contribution is dedicated to a new algorithm, AdaGain, that adaptively selects a step size for an RL algorithm; the authors claim this algorithm has sparsifying properties.  AdaGain is used in the context of a larger algorithm designed to search for GVFs by constructing a simple grammar over GVF components.  By creating large numbers of random GVFs and then pruning away useless ones, the authors discover a state representation that is useful for prediction.

While I am deeply sympathetic to the utility and difficulty of the discovery problem in this sort of state-space modeling, this paper ultimately felt a bit weak.  

On the positive side, I felt that it was well-written.  The work is well localized in the literature, and answers most questions one would naturally have.  The AdaGain algorithm is, to the best of my knowledge, novel, and the focus on "stability, not convergence" seems like an interesting idea (although ultimately, not well fleshed-out).

However, I felt that the central ideas were only thinly vetted.  For example:

* It seems that AdaGain is designed to tune a single parameter (\alpha) adaptively.  This raises several questions:
  - State-of-the-art stochastic optimizers (eg, Adam) typically introduce one step size per parameter; these are all tuned.  Why wasn't that discussed?  Would it be possible to apply something like Adam to this problem?
  - How does AdaGain compare to other adaptive gain algorithms?
  - There are ways to sparsify a representation - simple SGD + L1 regularization is a natural option.  How do we know how well AdaGain compares to this more common approach?

* The experiments seemed thin.  While I appreciated the fact that it seems that AdaGain was pruning away something, I was left wondering:
  - How generalizable are these results?  To be honest, the CompassWorld seems utterly uninteresting, and somewhat simplistic. 
  - I am convinced that AdaGain is learning.  But it would be interesting to know *what* it is learning.  Do the learned GVFs capture any sort of intuitive structure in the domains?
review_title: Ultimately falls short of convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: I have to say that I do not have all the background of this paper, and the paper is not written very clearly. I think the major contribution of the paper is represented in a very vague way.
review_title: not quite understand the paper

Meta-review: There was substantial disagreement between reviewers on how this paper contributes to the literature; it seems (having read the paper) that the problem tackled here is clearly quite interesting, but it is hard to tease out in the current version exactly what the contribution does to extend beyond current art.
Decision: Reject