Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new method for gradient-based meta-learning that allows for multiple steps of inner-loop optimization while avoiding expensive backpropagation through these updates. The proposal is based on a teacher-student scheme, where the student does many steps of inner loop gradient descent, and the teacher does only a single update step where the single step is computed using the regions visited by the student. The authors propose variants of this setup for both Maml and Reptile. The meta-learning model is trained by back-propagating through the teacher's updates. Experiments are conducted on few-shot learning, long-tailed classification, and meta-attack benchmarks. 

Pros
* The paper considers a variety of different benchmarks (other than simply few-shot learning), including memorization in few-shot learning, long-tailed classification and meta-attack.

Cons
* It is not straightforward to me what the benefit of the proposed method is compared to first-order maml (fomaml) or reptile. Both fomaml and reptile do not require expensive back-propogation through the inner-loop updates. It is mentioned in the paper that for their model, "one can update the meta-model by backpropagating gradients through the teacher’s “leap”, not the student’s update chain (ignoring that the chain starts from the meta-model)." This to me means that the second-order term is being ignored and so the update is indeed first-order.  Then, the only difference (as far as I can see), is that the proposed method uses the convex hull of the last b checkpoints and a linear combination that includes the initial parameters (which is a form of a lookahead optimizer) instead of simply the last checkpoint after k steps of gradient descent in the inner-loop. This could indeed offer some benefit but doesn't seem to offer the claimed benefit of "enabling...exploring long horizons in the inner loop" compared to fomaml or reptile as training with long inner loops seem similarly computationally expensive across all these first-order methods. If what I said above is not the case, I think the outer-gradient for the proposed method should be specifically stated and compared against the fomaml and second-order maml cases to highlight how this gradient is different and what benefit this offers (similar to the analysis that was done in the Reptile paper). In general, I believe a better argument needs to be made about the benefits of the proposed method compared to first-order methods such as fomaml and reptile, given that they all seem computationally similar in terms of cost.
* My interpretation is that their "lazy maml" is essentially inserting a lookahead optimizer in the inner loop while having a first-order outer update; while "lazy reptile" is inserting a lookahead optimized in the inner loop and the outer-loop is also a lookahead optimizer by definition of reptile. However, in addition to the lookahead optimizer, they add calculating the convex hull of the last b checkpoints and hyperparmeters for this calculation and the lookahead optimizer are computed during meta-training by doing grid search on the perfomance on the corresponding validation set in each episode. It is unclear what the value of this hyperparameter search is and since it complicates the implementation of their method, it would be useful to have experiments to quantify the necessity of this hyperparameter search.
* The experiments for TiredImageNet seem to only compare against second-order maml where the inner-loop consists of very few steps? I think a natural comparison is also fomaml where the number of inner-loop steps is the same as what was used for their model. Additionally, I believe results from other state-of-the-art models should also be shown to put the proposed model's results in context (a table is probably better for comparing across the metrics from different models). I had a similar observation about the few-shot memorization case - that fomaml with same number of inner-loop steps is a natural baseline. I would also be interested in knowing the metrics of iMAML for these benchmarks as it seems like a natural method for comparison.

To conclude, though some of the experimental results seem interesting, I believe the discussion of the method could be greatly improved as some of the claims seem a bit exaggerated or confusing. Additionally, I believe the setup of the experiments could be improved as stated above.
review_title: Interesting work but some claims seem exaggerated/confusing

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the problem setting of gradient based meta learning (GBML) and presents a new algorithmic approach that is computationally tractable. The proposed approach follows a two time-scale algorithm, where the inner level takes multiple gradient steps of the task loss function, while the outer level takes smaller "leaps" following a weighted average of the inner level iterates. Experimental results are provided for few-shot classification and query efficient "meta attack" on deep networks which suggest that the proposed algorithms perform favorably when compared to baselines.

**Assessment**

The proposed algorithm and motivations are interesting. The experimental results are convincing. However, the writing lacks clarity (some of these are below). Overall, my recommendation is borderline. I am happy to reconsider based on author response.

**Questions and Comments**

(1) Having very long inner loops can lead to divergence in the bi-level optimization setup, since the inner level parameters are no longer influenced by the meta-parameters. For example, if sufficient number of steps are used, the inner level will converge to a stable point regardless of initialization. This is related to the vanishing gradient problem and is explained in greater detail in the iMAML paper. This suggests that we either need early stopping (like in MAML) or some other explicit regularization (like in iMAML). How does your problem formulation consider this problem and how do you propose to use long inner loop training without any explicit or implicit regularization?

(2) The paper writes $\phi_i(\theta) = \hat{\gamma}_i \theta + (1-\hat{\gamma}_i) \hat{\phi}_i$ where $\hat{\phi}_i$ is a convex combinations of iterations in SGD for task $i$. Is $\hat{\phi}_i$ considered to be a function of $\theta$, or do you pass stop-grad through the computation? This seems like a crucial detail that is missing, since if stop grad is not used, then one would still be differentiating through the optimization path. If stop grad is indeed used, why does the proposed algorithm solve the bi-level optimization?

(3) "The teacher’s role is similar to the skip connection in ResNet (He et al., 2016). They bridge two otherwise distant points such that gradients can effectively propagate between them." -- this seems like an unsubstantiated claim. In ResNets, gradients are passed through the skip connections. Do you also do this (see pt. 2 above)? Secondly,  ResNets perform skip connections from input to output, whereas GBML deals with optimization iterates. In my opinion, the correct analogy to ResNets would look something like Nesterov's momentum (where previous iterates are used to find the next iterate).

(4) Lazy MAML requires hyper-parameter optimization over the different $\alpha_i$ and $\gamma$ for each **meta-training iteration**. This presumably would add significant computational overhead compared to MAML. This contradicts with the compute results in Appendix A.3. The cost of back-propagation through a graph is typically no more than 4-5 times as forward prop (regardless of the graph). If 4-5 hyperparameter choices are tried out, then the number of forward prop in case of lazy MAML would exceed this compute factor for backprop. Can the authors clarify as to why lazy MAML is more compute efficient? 

(5) Finally, I believe the authors may have a misunderstanding about MAML and related algorithms. Having a longer inner loop does not imply that higher order gradients are needed. As long as SGD is the inner loop optimizer, regardless of the size of the inner loop, only Hessian-vector products would be needed. See the iMAML paper for further discussion about these aspects. However, the memory footprint will increase linearly with the length of the inner loop. The authors of the submission point out the memory drawback correctly, but IMHO are a bit imprecise about the higher-order derivative claim. I recommend the authors to rephrase for correctness and better clarity.
review_title: Review of "Lazy" GBML

final_score: rating: 7: Good paper, accept
review_content: Computation of higher-order gradients is a major bottleneck in gradient-based meta-learning algorithms like MAML. This work proposes a simple approximation algorithm to alleviate the issue. They use a student network to explore the search space of task-specific models. Unlike MAML, the computation graph of the task-specific adaptation is not tracked and this allows the number of task-specific updates to be larger. Once the task-specific adaptation is done, they simply take a convex combination of the task-adapted parameters and the original parameters and evaluate the resulting model on the validation data of the task. The validation error is then backprogated for training. The proposed approach seems to be applicable to different meta-learning scenarios including few-shot learning, unbalanced classification and meta-attack. On the unbalanced classification and the meta-attack benchmarks, they showed marginal improvements over existing methods.

Pros:
- The paper is well written and clear
- They evaluated the algorithm on multiple different meta-learning problems
Strong results on the unbalanced classification and the meta-attack benchmarks 

Cons:
- Although the approach is motivated by the look ahead optimization, it feels a bit unintuitive to take a simple convex combination of the adapted and the original parameters for the sake of skip connection
- I am not fully convinced that the method alleviates the short-horizon biases of the gradient-based meta-learning algorithm. The meta-update still seems to favor a solution that is closest to the original model due to the skip connection.
- Evaluation on multi-shot learning benchmarks can make the work strong

review_title: Simple approach to alleviate higher-order gradient computation

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary

The paper proposes a method to address the short horizon limitation of gradient based meta-learning algorithms. To avoid the multiple problems of back-propagating the gradient through inner loops with many steps, the authors propose a teacher-student solution that decouples the exploration (done by the student) from the actual computation of the gradient (done by the teacher). In particular, the inner loop is run using the student on the training set. The teacher then optimises the validation loss using grid search in a reduced search space defined by the convex hull of last points explored by the student and the common initialization. Finally, the metagradient is taken with respect to the initialization avoiding the backpropagation through the inner loop. In the experimental section, the authors apply the methods in three different use cases (few shot learning, long-tail classification and meta-attack)

## Comments

The paper is well motivated and the writing is clear.

From the technical perspective, the model proposal seems reasonable: focus on the gradient information toward the end of the trajectories defined by the task specific inner loops. One of the confusing aspect of the paper is that despite the general description of the method based on the convex hull of the last "b" updates of the inner loop, in the end only the last update is used (according to the remark at the end of section 4.1). If there is not advantage in performance by taking b>1 and assuming there is a significant cost in increasing b (the computational complexity of grid search grows exponentially with the dimensionality b), why to include this extension in the first place? Unless I am missing something it may make sense to start with the description of the simpler algorithm that uses only the last update. 

I would also encourage the authors to elaborate on the similarities/differences between Reptile and their method in the same way they do it with MAML. In particular, when b=1 their method seems to be pretty close to Reptile (reptile uses \theta \leftarrow \theta + \epsilon(\widehat{\phi} -\theta) = (1 - \epsilon) \theta + \epsilon \widehat{\phi}), and so a deeper analysis of why the results are better would add value to the paper.

Regarding the experimentation, i would recommend the authors to move some of the comparisons with state-of-the-art methods from the appendix back to the manuscript. In particular, given that it is mentioned in Figure 1, I would include iMAML results in the manuscript rather than the appendix. Also, in the experimental setting, it is unclear why iMAML and FOMAML are incomplete in tables 5 and 6.

Finally, one of the relevant baselines that the authors missed is [1], which also addresses the short horizon bias by minimising the total length of the inner trajectories avoiding higher-order derivatives.

## Minors

* Figure 2: TAML is not defined
* Figure 1: The figure is quite confusing and it is not clear what the authors try to convey (e.g. should not be the task specific gradient be pointing in the same direction across the three plots?). Why is iMAML included but not Reptile, when it is actually Reptile and not iMAML the one baseline that appears in the main manuscript.
* At the end of page 4, the authors mention the importance of shuffling the N classes and then that their method performs better than MAML even without shuffling. Does that mean that all the experiments (for all methods) are done without shuffling? How do the results vary when shuffling is added?
* Equation 4: k-b+1...k should be k-b+j for j in {1,k}

## References
[1] Transferring Knowledge across Learning Processes
review_title: Official Review for A Lazy Approach to Long-Horizon Gradient-Based Meta-Learning

Meta-review: This paper presents a variant of MAML or Reptile, where the meta-update along the long trajectory of the inner-loop optimization is bypassed to reduce the computational overhead appeared in MAML. The main idea is to use the look-ahed optimizer with careful tuning of relevant hyperparameters, which is done by a teacher-student scheme. Lazy MAML/Reptile are presented and experiments  demonstrated their validity. While the paper contains interesting ideas, most of reviewers have a few concerns which are not even resolved even after the author responses. First of all, ResNet analogy with respect to teacher update was claimed but it was never clearly shown in the paper. The method needs careful tuning of hyperparameters in the inner loop, but  the study about the computation requirements is not convincing yet.  Long inner loops are computationally feasible for both fomaml and reptile so its not clear in which way the proposed method is improving lengthy exploration in the inner loop other than the performance being better in the experiments. Improving the paper, taking these comments into account, will lead to a good work in near future. 
Decision: Reject