Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a simple problem to demonstrate the short-horizon bias of the learning rate meta-optimization.

- The idealized case of quadratic function the analytical solution offers a good way to understand how T-step look ahead can benefit the meta-algorithm.
- The second part of the paper seems to be a bit disconnected to the quadratic function analysis. It would be helpful to understand if there is gap between gradient based meta-optimization and the best effort(given by the analytical solution)
- Unfortunately, no guideline or solution is offered in the paper.

In summary, the idealized model gives a good demonstration of the problem itself. I think it might be of interest to some audiences in ICLR.
review_title: simplified model demonstrating a problem of meta-learning learning rate

final_score: rating: 7: Good paper, accept
review_content: The paper discusses the problems of meta optimization with small look-ahead: do small runs bias the results of tuning? The result is yes and the authors show how differently the tuning can be compared to tuning the full run. The Greedy schedules are far inferior to hand-tuned schedules as they focus on optimizing the large eigenvalues while the small eigenvalues can not be "seen" with a small lookahead. The authors show that this effect is caused by the noise in the obective function.

pro:
- Thorough discussion of the issue with theoretical understanding on small benchmark functions as well as theoretical work
- Easy to read and follow

cons:
-Small issues in presentation: 
* Figure 2 "optimal learning rate" -> "optimal greedy learning rate", also reference to Theorem 2 for increased clarity.
* The optimized learning rate in 2.3 is not described. This reduces reproducibility.
* Figure 4 misses the red trajectories, also it would be easier to have colors on the same (log?)-scale. 
  The text unfortunately does not explain why the loss function looks so vastly different
  with different look-ahead. I would assume from the description that the colors are based
  on the final loss values obtaine dby choosing a fixed pair of decay exponent and effective LR. 

Typos and notation:
page 7 last paragraph: "We train the all" -> We train all
notation page 5: i find \nabla_{\theta_i} confusing when \theta_i is a scalar, i would propose \frac{\partial}{\partial \theta_i}
page 2: "But this would come at the expense of long-term optimization process": at this point of the paper it is not clear how or why this should happen. Maybe add a sentence regarding the large/Small eigenvalues?
review_title: Interesting work on meta-optimization

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper studies the issue of truncated backpropagation for meta-optimization. Backpropagation through an optimization process requires unrolling the optimization, which due to computational and memory constraints, is typically restricted or truncated to a smaller number of unrolled steps than we would like.

This paper highlights this problem as a fundamental issue limiting meta-optimization approaches. The authors perform a number of experiments on a toy problem (stochastic quadratics) which is amenable to some theoretical analysis as well as a small fully connected network trained on MNIST.  

(side note: I was assigned this paper quite late in the review process, and have not carefully gone through the derivations--specifically Theorems 1 and 2).

The paper is generally clear and well written.

Major comments
-------------------------
I was a bit confused why 1000 SGD+mom steps pre-training steps were needed. As far as I can tell, pre-training is not typically done in the other meta-optimization literature? The authors suggest this is needed because "the dynamics of training are different at the very start compared to later stages", which is a bit vague. Perhaps the authors can expand upon  this point?

The conclusion suggests that the difference in greedy vs. fully optimized schedule is due to the curvature (poor scaling) of the objective--but Fig 2. and earlier discussion talked about the noise in the objective as introducing the bias (e.g. from earlier in the paper, "The noise in the problem adds uncertainty to the objective, resulting in failures of greedy schedule"). Which is the real issue, noise or curvature? Would running the problem on quadratics with different condition numbers be insightful?

Minor comments
-------------------------
The stochastic gradient equation in Sec 2.2.2 is missing a subscript: "h_i" instead of "h"

It would be nice to include the loss curve for a fixed learning rate and momentum for the noisy quadratic in Figure 2, just to get a sense of how that compares with the greedy and optimized curves.

It looks like there was an upper bound constraint placed on the optimized learning rate in Figure 2--is that correct? I couldn't find a mention of the constraint in the paper. (the optimized learning rate remains at 0.2 for the first ~60 steps)?

Figure 2 (and elsewhere): I would change 'optimal' to 'optimized' to distinguish it from an optimal curve that might result from an analytic derivation. 'Optimized' makes it more clear that the curve was obtained using an optimization process.

Figure 2: can you change the line style or thickness so that we can see both the red and blue curves for the deterministic case? I assume the red curve is hiding beneath the blue one--but it would be good to see this explicitly.

Figure 4 is fantastic--it succinctly and clearly demonstrates the problem of truncated unrolls. I would add a note in the caption to make it clear that the SMD trajectories are the red curves, e.g.: "SMD trajectories (red) during meta-optimization of initial effective ...". I would also change the caption to use "meta-training losses" instead of "training losses" (I believe those numbers are for the meta-loss, correct?). Finally, I would add a colorbar to indicate numerical values for the different grayscale values.

Some recent references that warrant a mention in the text:
- both of these learn optimizers using longer numbers of unrolled steps:
Learning gradient descent: better generalization and longer horizons, Lv et al, ICML 2017
Learned optimizers that scale and generalize, Wichrowska et al, ICML 2017
- another application of unrolled optimization:
Unrolled generative adversarial networks, Metz et al, ICLR 2017

In the text discussing Figure 4 (middle of pg. 8) , "which is obtained by using..." should be "which are obtained by using..."

In the conclusion, "optimal for deterministic objective" should be "deterministic objectives"
review_title: A clear and well written demonstration of a fundamental issue with meta-optimization.

Meta-review: An interesting analysis of the issue of short-horizon bias in meta-optimization that highlights a real problem in a number of existing setups. I concur with Reviewer 3 that it would be nice to provide a constructive solution to this issue: if something like K-FAC does indeed work well, it would be a great addition to a final version of this paper. Nonetheless, I think the paper would be a interesting addition to ICLR and recommend acceptance.
Decision: Accept (Poster)