Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
This paper builds on recent work in gradient-based meta-learning and aims to meta-learn a gradient correction module in a similar manner to Meta-SGD [1] and Meta-Curvature [2]. The method proposed in this paper, GECCO, differs in two regards. First, in terms of how it does preconditioning: it differs from Meta-SGD and Meta-Curvature in that preconditioning is parametrized by a neural network, as in WarpGrad [3]. In contrast to WarpGrad, the GECCO preconditioner is only active in the backward pass. Second, as far I know, GECCO is unique in that it builds in a two-stage gradient update process during adaptation akin to Nesterov momentum: it first computes the gradient with a default preconditioner, measures the error of the obtained parameter update, adjust the preconditioner via a context variable (by backpropagating through the parameter update), then makes a final parameter update with the adjusted preconditioner. The authors show that this approach achieves strong convergence on a variety of tasks and is robust to gradient corruption in the form of additive noise. The proposed method is interesting and the results are compelling. However, the manuscript does not motivate the components of the proposed method nor make sufficient connections to the literature. As such, as it currently stands I recommend rejection. If my concerns are addressed I will increase my score. My main concern is that the paper presents rather than explores the method, while being written as if it is the only meta-learner that produces gradient correction. It is well known that gradient preconditioning is a beneficial mechanism and is not a novelty of GECCO. In particular, the GECCO preconditioning formula (Eq. 2), which is of the same form as prior works on meta-learned preconditioning [1, 2], is presented without an explicit comparison to these. As far as I understand, GECCO uses an element-wise operation to adjust gradients, which renders it equivalent to Meta-SGD, except that GECCO uses a Nesterov-like correction mechanism. How does Meta-SGD with Nesterov/Adam/RMSProp compare to GECCO? What is the computational trade-off? Further, imposing low-rank preconditioning is a non-trivial departure from the progression of prior works that have sought the *increase* the expressiveness of preconditioning. Similarly, it seems to go against the conventional wisdom of over-parametrization [5]. As such, the low-rank structure needs a motivation and a demonstration as to its usefulness. While the authors do provide an ablation with regards to rank, I am left wondering if not a simpler method that directly produces a correction vector would be as effective if tuned appropriately. With regards to this point, the architecture used for generating preconditioning is similarly shrouded in mystery. Eq.3 defines preconditioning as an outer product between two low-rank matrices that are produced by two MLPs. Each matrix is then modulated by a softmax operation after which it is flattened into a vector. This is a complicated operation with many moving parts and it is not clear to me that what these parts bring to the table. What are the benefits to doing this, as poosed to directly output a gradient correction vector? What is the role and purpose of the softmax operator? Why do we need to produce two correction terms (phi_1 and phi_2)? I would expect the ablation study to separate the effect of each of these components, not just vary the rank of the preconditioner. Finally, making comparisons to prior methods with SGD in the inner loop seems unfair as GECCO uses a double-step type of update that involves a one-step MAML-like operation. Hence in wall-clock time GECCO might not be converging faster. GECCO would also have a higher computational cost in during adaptation - how does the computational cost of GECCO compare to related methods? Recommended Improvements - Abstract: “while previous approaches address this problem by altering the learning rates [...], GECCO is an off-the shelf unit that performs element-wise gradient-correction”. I don’t believe this distinction can be made, adapting element-wise learning rates is equivalent to element-wise gradient correction. - Abstract “without the need to observe the gradient”. If we don’t observe the gradient, then what do we apply the gradient correction term to? - Impurity of gradient information? What does this mean? If we simply mean noise, use conventional terminology. If we mean something else, we need to define what impurity means here. - Intro, paragraph 3, altering the step size is not just about correcting for noise. It also corrects for ill-conditioned curvature. - Intro, paragraph 4, I don’t see its relevance to this work. - Intro, paragraph 5, “the more data samples are provided, the more gradient steps are required to converge to a good solution”. I believe this is incorrectly phrased, more data means we can train for longer to obtain a *better* solution, not that we *must* train for longer to get an equally good solution as if we had less data. - contribution ii. It is unclear what we mean by “1-step scenario”, nor what “numerous experiments” mean or “higher-shot” scenarios. - Related work, paragraph 2. appleis -> applies - Related work, paragraph 3. “This work is also aligned with (M)T-Nets, where only several layers are updated and the rest is fixed. These approaches have lesser computational cost than updating the whole network”. This needs qualification: GECCO and MT-Nets have *higher* computational cost during meta-training, since they both learn the initialisation and gradient preconditioning. Moreover, because GECCO computes a one-step MAML loss at each step of adaptation, even during meta-testing, for large enough networks these second-order derivatives will dominate and GECCO will be more expensive than comparable methods. - Method, paragraph 3, ‘functionality of the model of interest’. What does functionality mean? Don’t you just mean that h is the model? - Remark 1. I am not convinced by this remark. I can easily define a meta problem where the number of steps required to adapt the task learner is beyond the budget of either MAML or GECCO. - Equation 2: what is Psi? I can’t find a definition in the manuscript. - Remark 2. Many prior works can be viewed as a gradient correction mechanism, not only GECCO. In general, a clearer distinction between GECCO and prior works is needed. - Eq. 3. This needs a motivation, see above. - Eq. 4. It is not clear at this point how the inner loop works. I would suggest you detail the process more clearly. For instance, if I understand correctly, the context variable is reset at each adaptation step? This means each adaptation step involves two forward-backward passes and a MAML update to the context vector. This seems quite expensive to me, what is the computational complexity of this method at test? - Remark 3. This is a sweeping remark without any support or further analysis. Every preconditioner represents a gradient field. - Fig. 6 is not in the main paper. - Few-shot classification ‘the size of 
u and alpha are 300 and 0.1’ - alpha is a scalar - Number of steps. This is a problematic comparison, since the computational cost of GECCO is much higher during adaptation. It needs to be set in a context of a computational budget. - Image completion. Why is GECCO only applied to a single layer? - Section 4.4: more details are needed. Are gradients corrupted during meta-training or only during meta-testing? The baselines appear unsuitable, Meta-SGD, Meta-Curvature and MT-Nets would be the relevant comparisons. - The paper frequently makes sweeping statements that give the impression GECCO is the only meta-learner that learns a gradient correction term or that GECCO is the only meta-learner that can be seen as incorporating the gradient field of a manifold. It would be more useful to have a nuanced discussion as to how GECCO compares to other methods that also have these properties. References [1] Li et al. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. 2017. [2] Park & Oliva. Meta-Curvature. 2019. [3] Flennerhag et al. Meta-Learning with Warped Gradient Descent. 2019. [4] Lee et al. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace. 2018. [5] Arora et al. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. ICML 2018.