Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The authors propose an extension to the lookahead optimizer called generalized lookahead. The key idea is to utilize all parameters computed during the fast training process (instead of selecting the last one) to update the slow weight. The generalized lookahead achieves this by constructing an affine hull and performing quadratic approximation.
strengths_and_weaknesses: *Strengths*
- The findings are clearly communicated, and the paper includes an appropriate discussion of other relevant literature. 
- The paper is reproducible. 
- The ideas presented in this paper are interesting. 

*Weaknesses*
- Some justifications in the paper are not well justified, which I mention below. However, for TinyPapers, I don't think this is a significant weakness.
- The empirical analysis is extremely limited, as the authors use a different training recipe for image classification. Hence, the numbers reported in the table are quite low than the ones presented in the literature. I don't understand why using a learning rate decay on ResNet-18 results in a higher tuning cost; the authors can use the standard scheme, which is to decay at epochs 60, 120, and 180. The current results can be biased towards the proposed methods as I imagine the generalized lookahead to have less variance update, making it less susceptible to learning rate decay. I strongly encourage the authors to evaluate the algorithm on a competitive baseline.  
suggested_changes: *Suggestions*
- As mentioned above, I recommend the authors use the competitive baseline.
- The justification for why the Generalised Lookahead optimizer is weak; I recommend looking into the Noisy Quadratic Model (NQM) derivation and investigating whether the proposed method can reduce the variance. 
- Authors can try gradient descent on $\alpha$s to find the best linear combination of the slow weights. Authors can use the fact that it is convex and deterministic (given a fixed batch).

*Minor*
- The sequence formula after "Lookahead works by resetting some of the progress every k steps without considering the intermediate path" should have an index $i=m$ and $m + k - 1$.
- It would help clarify if the authors describe what an affine-hull search region is (although it can be straightforward for people familiar with lookahead).
review_title: null

Meta-review: Novelty is expressly not a reviewing criteria, so is not considered in this decision. However, given the areas identified for improvement by the reviewers, the authors are invited to make improvements before archiving.
Decision: Invite to archive