EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
Year: 2021

Abstract:
Gradient-based meta-learning and hyperparameter optimization have seen significant progress recently, enabling practical end-to-end training of neural networks together with many hyperparameters. Nevertheless, existing approaches are relatively expensive as they need to compute second-order derivatives and store a longer computational graph. This cost prevents scaling them to larger network architectures. We present EvoGrad, a new approach to meta-learning that draws upon evolutionary techniques to more efficiently compute hypergradients. EvoGrad estimates hypergradient with respect to hyperparameters without calculating second-order gradients, or storing a longer computational graph, leading to significant improvements in efficiency. We evaluate EvoGrad on three substantial recent meta-learning applications, namely cross-domain few-shot learning with featurewise transformations, noisy label learning with Meta-Weight-Net and low-resource cross-lingual learning with meta representation transformation. The results show that EvoGrad significantly improves efficiency and enables scaling meta-learning to bigger architectures such as from ResNet10 to ResNet34.

heading: 1 Introduction
text: Gradient-based meta-learning and hyperparameter optimization have been of long-standing interest in neural networks and machine learning [11, 21, 4]. Hyperparameters (aka meta-parameters) can take diverse forms, especially under the guise of meta-learning, where there has recently been an explosion of successful applications addressing diverse learning challenges [9]. For example to name just a few: training optimizer initial condition in support of few-shot learning [7, 1, 15]; training instance-wise weights for cleaning noisy datasets [31, 26]; training loss functions in support of generalisation [14] and learning speed; and training stochastic regularizers in support of cross-domain robustness [33].
Most of these applications share the property that meta-parameters impact validation loss only indirectly through their effect on model parameters, and so computing validation loss gradients with respect to meta-parameters usually leads to the need to compute second-order derivatives, and store longer computational graphs for backpropagation. This eventually becomes a bottleneck to execution time, and – more severely – to scaling the size of the underlying models, given the practical limitation of GPU memory. There has been steady progress in the development of diverse practical algorithms for computing validation loss with respect to meta-parameters [20, 19, 21]. Nevertheless they mostly share some form of the aforementioned limitations. In particular, the majority of recent successful practical applications [31, 33, 14, 2, 5, 17, 30] essentially use some variant of the T1 − T2 algorithm [20] to estimate the gradient ∂ℓV∂λ of validation loss w.r.t. hyperparameters. This approach computes the gradient online at each step of updating the base model θ, and estimates it as ∂ℓV∂λ ≈ ∂ℓV ∂θ ∂2ℓT ∂θ∂λ , for training loss ℓT . As with many alternative estimators, this requires second-order derivatives, and extending the computational graph. Besides the additional computation cost, this limits the size of the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
base model that can be used in a given GPU, since the memory cost of meta-learning is now multiple times the size of vanilla backpropagation. This in turn prevents the application of meta-learning to problems where large state-of-the-art model architectures are required.
To address this issue, we draw inspiration from evolutionary optimization methods [27] to develop EvoGrad, a meta-gradient algorithm that requires no higher-order derivatives and as such is significantly faster and lighter than the standard approaches. In particular, we take the novel view of estimating meta-gradients via a putative inner-loop evolutionary update to the base model. As this requires no gradients itself, the meta-gradient can then be computed using first-order gradients alone, and without extending the computational graph – leading to efficient hyperparameter updates. Meanwhile for efficient and accurate base model learning, the real inner-loop update can separately be carried out by conventional gradient descent.
Our EvoGrad is a general meta-optimizer applicable to many meta-learning applications, among which we choose three to demonstrate its impact: the LFT model [33] observes that a properly tuned stochastic regularizer can significantly improve cross-domain few-shot learning performance. We show that by training those regularizer parameters with EvoGrad, rather than the standard secondorder approach, we can obtain the same improvement in accuracy with significant reduction in time and memory cost. This allows us to scale LFT from the original ResNet10 to ResNet34 within a 12GB GPU. Second, the Meta-Weight-Net (MWN) [31] model deals with label noise by meta-learning an auxiliary network that re-weights instance-wise losses to down-weight noisy instances and improve validation loss. We also show that EvoGrad can replicate MWN results with significant cost savings. Third, we demonstrate the benefits of EvoGrad on an application from NLP, in addition to the ones from computer vision: low-resource cross-lingual learning using MetaXL approach [35].
To summarize, our main contributions are: (1) We introduce EvoGrad, a novel method for gradientbased meta-learning and hyperparameter optimization that is simple to implement and efficient in time and memory requirements. (2) We evaluate EvoGrad on a variety of illustrative and substantial meta-learning problems, where we demonstrate significant compute and memory benefits compared to standard second-order approaches. (3) In particular, we illustrate that EvoGrad allows us to scale meta-learning to bigger models than was previously possible on a given GPU size, thus bringing meta-learning closer to the state-of-the-art frontier of real applications. We provide source code for EvoGrad at https://github.com/ondrejbohdal/evograd.

heading: 2 Related work
text: Gradient-based meta-learning solves a bilevel optimization problem where validation loss is optimized with respect to the meta-knowledge by backpropagating through the update of the model on training data and with meta-knowledge. The meta-knowledge updates form an outer loop, around an inner loop of base model updates. The inner loop can run for one [20], few [29, 21], or many [19] steps within each outer-loop iteration. Meta-knowledge can take many forms, for example, it can be an initialization of the model weights [7], feature-wise transformation layers [33], regularization to improve domain generalization [2] or even a synthetic training set [34, 5]. Most substantial practical applications use a one or few-step inner loop for efficiency.
More recently, several methods [19, 25] have utilized Implicit Function Theorem (IFT) to develop new gradient-based meta-learners. These methods use multiple inner-loop steps without the need to backpropagate through whole inner loop, which significantly improves memory efficiency over methods that need keep track of the whole inner loop training process. However, IFT methods assume the model has converged in the inner loop. This makes them unsuited for the majority of practical applications above where training the inner loop to convergence for each hypergradient step is infeasible. Furthermore, the hypergradient is still more costly compared to one-step T1 − T2 method. The costs come from the associated overhead with approximating an inverse Hessian of the training data with respect to the model parameters. Note that the Hessian itself does not need to be stored due to the mechanics of reverse-mode differentiation [8, 3]. However, this does not eliminate the remaining calculations which still require higher-order gradients that result in backpropagation via longer graphs due to additional gradient nodes. For these reasons, we focus comparison on the more widely used T1 − T2 strategy which is oriented at single-step inner loops similar to EvoGrad. Theoretically it is also possible to use hypernetworks [18] to find good hyperparameters in a first-order way. Hypernetworks take hyperparameters as inputs and generate model parameters. However, the
approach is not commonly used, likely due to the difficulty of generating well-performing model parameters. We provide experimental results to support this hypothesis in the supplementary material.
Meta-learning can be categorized into several groups, depending on the type of meta-knowledge and also if the model is trained from scratch as part of the inner loop [9]. Offline meta-learning approaches train a model from scratch per each update of the meta-knowledge, while online metalearning approaches train the model and meta-knowledge jointly. As a result, offline meta-learning is extremely expensive [6, 37] when scaled beyond few-shot learning problems where only a few iterations are sufficient for training [7, 1, 15]. Therefore most larger-scale problems [16, 10, 33, 35] use online learning in practice, and this is where we focus our contribution.
The meta-knowledge to learn can take different forms. A particular dichotomy is between the special case where the meta-knowledge corresponds to the base model itself, in the form of an initialization; and the more general cases where it does not. The former initialization meta-learning has been popularized by MAML [7], and is widely used in few-shot learning. This can be solved relatively efficiently, for example using a first-order approximation of MAML [7], Reptile [23] or minibatch proximal update [36]. On the other hand, there are vastly more cases [9] where the meta-knowledge is different from the model itself, such as LFT’s stochastic regularizer to improve cross-domain generalization [33], MWN’s instance-wise loss weighting network for label noise robustness [31], a label generation network to improve self-supervised generalization [17], a Feature-Critic loss to improve domain generalization [14] and many others. In this more general case, most applications rely on a T1 − T2-like algorithm, as the efficient approximations specific to MAML do not apply. The ability to significantly improve the efficiency of gradient-based meta-learning would have a large impact as methods like these would directly benefit from it in runtime and energy consumption. More crucially, they could scale to bigger and more state-of-the-art neural network architectures.

heading: 3 Methods

heading: 3.1 Background: meta-learning as bilevel optimization
text: We aim to solve a bilevel optimization problem where our goal is to find hyperparameters λ that minimize the validation loss ℓV of the model parametrized by θ and trained with loss ℓT and λ:
λ∗ = argmin λ ℓ∗V (λ), where ℓ ∗ V (λ) = ℓV (λ,θ ∗(λ)) and θ∗(λ) = argmin θ ℓT (λ,θ). (1)
In order to meta-learn the value of λ using gradient-based methods, we need to calculate the hypergradient ∂ℓV∂λ . We can expand its calculation as follows:
∂ℓ∗V (λ)
∂λ =
∂ℓV (λ,θ ∗(λ))
∂λ +
∂ℓV (λ,θ ∗(λ))
∂θ∗(λ)
∂θ∗(λ)
∂λ . (2)
In meta-learning and hyperparameter optimization more broadly, the direct term ∂ℓV (λ,θ ∗(λ))
∂λ is typically zero because the hyperparameter does not directly influence the value of the validation loss – it influences it via the impact on the model weights θ. However, the model weights θ are themselves trained using gradient optimization, which gives rise to higher-order derivatives. We propose a variation on this step where the update of the model weights is inspired by evolutionary methods, allowing us to eliminate the need for higher-order derivatives. We consider the setting where the hypergradient of hyperparameter λ is estimated online [20] together with updating the base model θ, as this is the most widely used setting in substantial practical applications [31, 33, 14, 2, 5, 30, 16, 35].

heading: 3.2 The EvoGrad update
text: Given the current model parameters θ ∈ RM , hyperparameters λ ∈ RN , training loss ℓT and validation loss ℓV , we aim to estimate ∂ℓV∂λ for efficient gradient-based hyperparameter learning. The key idea is – solely for the purpose of hypergradient estimation – to consider a simple evolutionary rather than gradient-based inner-loop step on θ.
Evolutionary inner step First, we sample random perturbations ϵ ∈ RM ∼ N (0, σI), and apply them to θ. Sampling K perturbations, we can create a population of K variants {θk}Kk=1 of the current model as θk = θ + ϵk. We can now compute the training losses {ℓk}Kk=1 for each of the K models, ℓk = f(DT |θk,λ) using the current minibatch DT drawn from the training set. Given these loss values, we can calculate the weights (sometimes called fitness) of the population of candidate models as
w1, w2, . . . , wK = softmax([−ℓ1,−ℓ2, . . . ,−ℓK ]/τ), (3) where τ is a temperature parameter that rescales the losses to control the scale of weight variability.
Given the weights {wk}Kk=1, we complete the current step of evolutionary learning by updating the model parameters via the affine combination
θ∗ = w1θ1 + w2θ2 + · · ·+ wKθK . (4)
Computing the hypergradient We now evaluate the updated model θ∗ for a minibatch from the validation set DV and take gradient of the validation loss ℓV = f(DV |θ∗) w.r.t. the hyperparameter:
∂ℓV ∂λ
= ∂f(DV |θ∗)
∂λ (5)
One can easily verify that the computation in Eq. 5 does not involve second-order gradients as no first-order gradients were used in the inner loop. This is in contrast to the typical approach [20, 21] of applying gradient-based updates in the inner loop and differentiating through it (in either forward-mode or reverse-mode), or even applying the implicit function theorem (IFT) [19], all of which trigger higher-order gradients and an extended computation graph.
Algorithm flow In practice we follow the flow of T1−T2 [20] used by many substantive applications [33, 31, 2, 16, 35]. We take alternating steps on θ using the exact gradient ∂ℓT∂θ , and on λ using the hypergradient ∂ℓV∂λ , which in EvoGrad is estimated as in Eq. 5.

heading: 3.3 EvoGrad hypergradient as a random projection
text: To understand EvoGrad, observe that the hyper-gradient in Eq. 5 expands as
∂ℓV ∂λ = ∂ℓV ∂θ∗
∂θ∗ ∂λ = ∂ℓV ∂θ∗ E ∂w ∂λ = ∂ℓV ∂θ∗ E ∂w ∂ℓ ∂ℓ ∂λ (6)
where E = [ϵ1, ϵ2, . . . , ϵK ] is the M × K matrix formed by stacking ϵk’s as columns, w is the K-dimensional (w = [w1, w2, . . . , wK ]) vector of candidate model weights, and ℓ = [ℓ1, ℓ2, . . . , ℓK ] is the K-dimensional vector of candidate model losses.
Recall that E is a random matrix, so the operation ∂ℓV∂θ∗ E can be understood as randomly projecting the M -dimensional validation loss’ gradient to a new low-dimensional space of dimension K ≪ M . Alternatively, we can interpret the update as factorising the model-parameter-to-hyperparameter derivative ∂θ ∗
∂λ (sized M ×N ) into two much smaller matrices E and ∂w ∂λ of size M ×K and K ×N .
In terms of implementation, ∂ℓV∂θ∗ is obtained by backpropagation and E is sampled on the fly. The term ∂w∂λ = ∂w ∂ℓ ∂ℓ ∂λ is computed by the softmax-to-logit derivative (K ×K) and the derivative of the K candidate models training losses w.r.t. hyperparameters. It is noteworthy that the K elements of ∂ℓ ∂λ are completely independent, and can be computed in parallel where multiple GPUs are available.

heading: 3.4 Comparison to other methods
text: We compare EvoGrad to the most related and widely-used alternative T1−T2 [20] in Table 1. T1−T2 requires higher-order gradients and associated longer computational graphs – due to the need to backpropagate through gradient nodes. This leads to increased memory and time cost compared to vanilla backpropagation. In contrast, EvoGrad requires no higher-order gradients, no large matrices, and no substantial expansion of the computational graph.
We analyse the asymptotic big-O time and memory requirements of EvoGrad vs T1 − T2 in Table 2. The dominant cost in terms of both memory and time is the cost of backpropagation. Backpropagation is significantly more expensive than forward propagation because forward propagation does not need to store all intermediate variables in memory [25, 8]. Note that even if EvoGrad keeps multiple copies of the model weights in memory, this cost is small compared to the cost of backpropagation, and the latter is done with only one set of weights θ∗. We remark that our main empirical results are obtained with only K = 2 models, so we can safely ignore this in our asymptotic analysis.
In addition, we elaborate on how higher-order gradients contribute to increased memory and time costs. Results from computing the first-order gradients are added into the computation graph as new nodes in the graph so that we can calculate the higher-order gradients. When calculating the higher-order gradients, we backpropagate through this longer computational graph, which directly increases the memory and time costs. The current techniques [20] rely on longer computational graphs, while EvoGrad significantly shortens the graph and reduces memory cost by avoiding this step. This consideration is not visible in the big-O analysis, but contributes to improved efficiency.

heading: 4 Experiments
text: We first consider two simple problems: 1) a 1-dimensional problem where we try to find the minimum of a function, and 2) meta-learning a feature-transformer to find the rotation that correctly aligns images whose training and validation sets differ in rotation. This serves as a proof-of-concept problem to show our method is capable of meta-learning suitable hyperparameters. We then consider three real problems where meta-learning has been used to solve different learning challenges. We show that EvoGrad makes a significant impact in terms of reducing the memory and time costs (while keeping the accuracy improvements brought by meta-learning): 3) Cross-domain few-shot classification via learned feature-wise transformation [33], 4) Meta-Weight-Net: learning an explicit mapping for sample weighting [31], 5) MetaXL: meta representation transformation for low-resource cross-lingual learning [35]. We provide a brief overview of each problem, together with evaluation and analysis. Further details and experimental settings are described in the supplementary material.

heading: 4.1 Illustration using a 1-dimensional problem
text: In this problem we minimize a validation loss function fV (x) = (x − 0.5)2 where parameter x is optimized using SGD with training loss function fT (x) = (x − 1)2 + λ∥x∥22 that includes a meta-parameter λ. A closed-form solution for the hypergradient is available and is equal to g(λ) = (λ− 1)/(λ+ 1)3, which allows us to compare EvoGrad against the ground-truth gradient. Our first analysis studies the estimated EvoGrad hypergradient for a grid of λ values between 0 and 2. For each value of λ we show the mean and standard deviation of the estimated ∂fV /∂λ over 100 repetitions (with random choice of x). We use temperature τ = 0.5, ϵ ∈ R ∼ N (0, 1) and consider between 2 and 100 models copies in the population. The results in Figure 2 show that EvoGrad estimates have a similar trend to the ground-truth gradient, even if the EvoGrad estimates are noisy. The level of noise decreases with more models in the population, but the correct trend is visible even if we use only 2 models.
Our second analysis studies the trajectories that parameters x, λ follow if they are both optimized online using SGD with learning rate of 0.1 for 5 steps, starting from five different positions (circles). The hypergradients are either estimated using EvoGrad or directly using the ground-truth formula. Figure 3 shows that the trajectories of both variations are similar, and they become more similar as we use more models in the population. In all cases the parameters converge towards the lightly-coloured region where the validation loss is the lowest at x = 0.5.

heading: 4.2 Rotation transformation
text: In this task we work with MNIST images [12], and assume that the validation and test sets are rotated by 30◦ compared to the conventionally oriented training images. Clearly, directly training a model and applying it will lead to low performance. We therefore assume meta-knowledge in the form of a hidden rotation. The rotation transformation is applied to the training images before learning, and should itself be meta-learned by the validation loss obtained by the CNN trained on the rotated training set. Thus solving the meta-learning problem should result in a 30◦ rotation, and a base CNN that generalises to the rotated validation set.
The problem is framed as online meta-learning where each update of the base model is followed by a meta-parameter update using EvoGrad. We use EvoGrad with 2 model copies, temperature τ = 0.05 and σ = 0.001 for ϵ ∼ σsign(N (0, I)). Our LeNet [13] base model is trained for 5 epochs. We repeat the experiments 5 times and show a comparison of the results in Table 3. A baseline model achieves 98.40± 0.07% accuracy if the test images are not rotated, but its accuracy drops to 81.79± 0.64% if the same images are rotated by 30◦. A model trained with EvoGrad and the rotation transformer is able to accurately classify rotated images, with a similar accuracy as the baseline model can classify unrotated images. This confirms we can successfully optimize hyperparameters with EvoGrad. The meta-learned rotation is also close to the true value.

heading: 4.3 Cross-domain few-shot classification via learned feature-wise transformation
text: As the next task we consider cross-domain few-shot classification (CD-FSL). CD-FSL is considered an important and highly challenging problem at the forefront of computer vision. The state-of-the-art approach learned feature-wise transformation (LFT) [33] aims to meta-learn stochastic featurewise transformation layers that regularize metric-based few-shot learners to improve their few-shot learning generalisation in cross-domain conditions. The method includes two key steps: 1) updating the model with the meta-parameters on a pseudo-seen task and 2) updating the meta-parameters by evaluating the model on a pseudo-unseen task by backpropagating through the first step. As feature-wise transformation is not directly used for the pseudo-unseen task, this leads to higher-order gradients. Note that the problem itself is memory-intensive because we work with larger images of size 224× 224 within episodic learning tasks. As a result, a significantly more efficient meta-learning approach could allow us to scale from the ResNet10 model used in the paper to a larger model.
We experiment with the LFT-RelationNet [32] metric-based few-shot learner and consider the exact same experiment settings as [33] using the official PyTorch implementation associated with the paper. LFT introduces 3712 hyper-parameters to train for ResNet10, and 9344 for ResNet34. All our experiments are conducted on Titan X GPUs with 12GB of memory using K = 2 for EvoGrad.
Table 4 shows the baseline performance of vanilla unregularised ResNet (-), manually tuned FT layers (FT), FT layers meta-learned by second-order gradient (LFT) and by EvoGrad. The results show that EvoGrad matches the accuracy of the original LFT approach, leading to clear accuracy improvements over training with no feature-wise transformation or training with fixed feature-wise parameters selected manually. At the same time EvoGrad is significantly more efficient in terms of the memory and time costs as shown in Figure 4. The memory improvements from EvoGrad allow us to scale the base feature extractor to ResNet34 within the standard 12GB GPU. The original LFT with its T1−T2 style second-order algorithm cannot be extended in the available memory if we keep the same settings of the few-shot learning tasks. Thus, we are able to improve state-of-the-art accuracy on both 5-way 1 and 5-shot tasks. For ResNet34, we include baselines without any feature-wise transformation and with manually chosen feature-wise transformation to confirm the benefit of meta-learning.

heading: 4.4 Label noise with Meta-Weight-Net
text: We consider a further highly practical real problem where online meta-learning has led to significant improvements – learning from noisy labelled data. The Meta-Weight-Net framework trains an auxiliary neural network that performs instance-wise loss re-weighting on the training set [31]. The base model is updated using the sum of weighted instance-wise losses for noisy data, while the Meta-Weight-Net itself is updated by evaluating the updated model on clean validation data and by backpropagating through the model update. We use the official implementation of the approach [31] and follow the same experimental settings, using K = 2 for EvoGrad.
Our results in Table 5 confirm we replicate the benefits of training with Meta-Weight-Net, clearly surpassing the accuracy of the baseline when there is label noise. We also note that EvoGrad can improve the accuracy over the T1 − T2-based approach because the two approaches are distinct and provide different estimates of the true hypergradient. Figure 5 shows that our method leads to significant improvements in memory and time costs (over half of the memory is saved and the runtime is improved by about a third).

heading: 4.5 Low-resource cross-lingual learning with MetaXL
text: The previous two real applications of meta-learning considered computer vision problems. To highlight EvoGrad is a general method that can make an impact in any domain, we also demonstrate its benefits on a meta-learning application from NLP. More specifically, we use EvoGrad for MetaXL [35], which meta-learns meta representation transformation to better transfer from source languages to low-resource target languages.
We have selected the named entity recognition (NER) task with English source language (WikiAnn dataset [24]), which is one of the key experiments in the MetaXL paper [35]. Table 6 shows EvoGrad matches and in fact surpasses the average test F1 score of MetaXL with the original T1 − T2 metalearning method. Figure 6 shows EvoGrad significantly improves both memory and time consumption compared to MetaXL T1 − T2. Overall these results confirm EvoGrad is suitable for meta-learning in various domains, including both computer vision and NLP.

heading: 4.6 Scalability analysis
text: We use the Meta-Weight-Net benchmark to study how the number of model parameters affects the memory usage and training time of EvoGrad, comparing it to the standard second-order T1 − T2 approach. We vary model size by changing the number of filters in the original ResNet32 model, multiplying the filter number ×1, . . . ,×5. The smallest model had around 0.5M parameters and the largest one around 11M parameters.
The results in Figure 7 show our EvoGrad leads to significantly lower training time and memory usage, and that the margin over the standard second-order optimizer grows as the model becomes larger. Further, we have analysed the impact of modifying the number of hyperparameters – from 300 up to 30,000. The impact on memory and time was negligible, and both remained roughly constant, which is caused by the main model being significantly larger. It is also because of the fact that reverse-mode differentiation costs scale with the number of model parameters rather than hyperparameters [22] – recall that backpropagation is the main driver of memory and time costs [25]. Moreover, we have done experiments that varied the number of model copies in EvoGrad. The results showed the training time per epoch increased slightly, while the memory costs remained similar.

heading: 5 Discussion and limitations
text: Similar to many other gradient-based meta-learning methods, our method is greedy as it considers only the current state of the model when updating the hyperparameters – rather than the whole training process. However, this greediness allows the method to be used in larger-scale settings where we train the hyperparameters and the model jointly. Further, our method approximates the hypergradient stochastically. While results were good for the suite of problems considered here using only K = 2, the gradient estimates may be too noisy in other applications. This could lead to poor outcomes which could be a problem in socially important applications. Alternatively, it may necessitate using a larger model population (Figure 2). While as we observed in Section 3.3 the candidate models can be trivially parallelized to scale population size, this still imposes a larger energy cost [28]. Another limitation is that similarly to IFT-based estimators [19], EvoGrad is not suitable for optimizing learner hyperparameters such as learning rate. Currently we have used the simplest possible evolutionary update in the inner loop, and upgrading EvoGrad to a state-of-the-art evolutionary strategy may lead to better gradient estimates and improve results further.

heading: 6 Conclusions
text: We have proposed a new efficient method for meta-learning that allows us to scale gradient-based meta-learning to bigger models and problems. We have evaluated the method on a variety of problems, most notably meta-learning feature-wise transformation layers, training with noisy labels using MetaWeight-Net, and meta-learning meta representation transformation for low-resource cross-lingual learning. In all cases we have shown significant time and memory efficiency improvements, while achieving similar or better performance compared to the existing meta-learning methods.
Acknowledgments and Disclosure of Funding
This work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh.

Review summaries:
The authors introduce EvoGrad, a method for approximating second order gradients for use in Meta-Learning applications. The method approximates the inner loop of optimization with "evolutionary" update. The authors show their method is more computationally efficient than the T1-T2 method, yields good approximations to the second order gradients and is able to converge in the toy setting, and achieves better results than MetaWeightNet, a second order method, on a cross-domain few-shot classification task. 

The paper presents an efficient gradient-based meta-learning algorithm based on evolutionary techniques. The proposed algorithm doesn’t require second order gradient computation, and thus, is more efficient. The proposed approach was evaluated on two meta-learning tasks for few-shot feature-wise transformation and noisy label learning, on which time and memory efficiencies were demonstrated.