EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization

1 Introduction . Gradient-based meta-learning and hyperparameter optimization have been of long-standing interest in neural networks and machine learning [ 11 , 21 , 4 ] . Hyperparameters ( aka meta-parameters ) can take diverse forms , especially under the guise of meta-learning , where there has recently been an explosion of successful applications addressing diverse learning challenges [ 9 ] . For example to name just a few : training optimizer initial condition in support of few-shot learning [ 7 , 1 , 15 ] ; training instance-wise weights for cleaning noisy datasets [ 31 , 26 ] ; training loss functions in support of generalisation [ 14 ] and learning speed ; and training stochastic regularizers in support of cross-domain robustness [ 33 ] . Most of these applications share the property that meta-parameters impact validation loss only indirectly through their effect on model parameters , and so computing validation loss gradients with respect to meta-parameters usually leads to the need to compute second-order derivatives , and store longer computational graphs for backpropagation . This eventually becomes a bottleneck to execution time , and – more severely – to scaling the size of the underlying models , given the practical limitation of GPU memory . There has been steady progress in the development of diverse practical algorithms for computing validation loss with respect to meta-parameters [ 20 , 19 , 21 ] . Nevertheless they mostly share some form of the aforementioned limitations . In particular , the majority of recent successful practical applications [ 31 , 33 , 14 , 2 , 5 , 17 , 30 ] essentially use some variant of the T1 − T2 algorithm [ 20 ] to estimate the gradient ∂ℓV∂λ of validation loss w.r.t . hyperparameters . This approach computes the gradient online at each step of updating the base model θ , and estimates it as ∂ℓV∂λ ≈ ∂ℓV ∂θ ∂2ℓT ∂θ∂λ , for training loss ℓT . As with many alternative estimators , this requires second-order derivatives , and extending the computational graph . Besides the additional computation cost , this limits the size of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . base model that can be used in a given GPU , since the memory cost of meta-learning is now multiple times the size of vanilla backpropagation . This in turn prevents the application of meta-learning to problems where large state-of-the-art model architectures are required . To address this issue , we draw inspiration from evolutionary optimization methods [ 27 ] to develop EvoGrad , a meta-gradient algorithm that requires no higher-order derivatives and as such is significantly faster and lighter than the standard approaches . In particular , we take the novel view of estimating meta-gradients via a putative inner-loop evolutionary update to the base model . As this requires no gradients itself , the meta-gradient can then be computed using first-order gradients alone , and without extending the computational graph – leading to efficient hyperparameter updates . Meanwhile for efficient and accurate base model learning , the real inner-loop update can separately be carried out by conventional gradient descent . Our EvoGrad is a general meta-optimizer applicable to many meta-learning applications , among which we choose three to demonstrate its impact : the LFT model [ 33 ] observes that a properly tuned stochastic regularizer can significantly improve cross-domain few-shot learning performance . We show that by training those regularizer parameters with EvoGrad , rather than the standard secondorder approach , we can obtain the same improvement in accuracy with significant reduction in time and memory cost . This allows us to scale LFT from the original ResNet10 to ResNet34 within a 12GB GPU . Second , the Meta-Weight-Net ( MWN ) [ 31 ] model deals with label noise by meta-learning an auxiliary network that re-weights instance-wise losses to down-weight noisy instances and improve validation loss . We also show that EvoGrad can replicate MWN results with significant cost savings . Third , we demonstrate the benefits of EvoGrad on an application from NLP , in addition to the ones from computer vision : low-resource cross-lingual learning using MetaXL approach [ 35 ] . To summarize , our main contributions are : ( 1 ) We introduce EvoGrad , a novel method for gradientbased meta-learning and hyperparameter optimization that is simple to implement and efficient in time and memory requirements . ( 2 ) We evaluate EvoGrad on a variety of illustrative and substantial meta-learning problems , where we demonstrate significant compute and memory benefits compared to standard second-order approaches . ( 3 ) In particular , we illustrate that EvoGrad allows us to scale meta-learning to bigger models than was previously possible on a given GPU size , thus bringing meta-learning closer to the state-of-the-art frontier of real applications . We provide source code for EvoGrad at https : //github.com/ondrejbohdal/evograd . 2 Related work . Gradient-based meta-learning solves a bilevel optimization problem where validation loss is optimized with respect to the meta-knowledge by backpropagating through the update of the model on training data and with meta-knowledge . The meta-knowledge updates form an outer loop , around an inner loop of base model updates . The inner loop can run for one [ 20 ] , few [ 29 , 21 ] , or many [ 19 ] steps within each outer-loop iteration . Meta-knowledge can take many forms , for example , it can be an initialization of the model weights [ 7 ] , feature-wise transformation layers [ 33 ] , regularization to improve domain generalization [ 2 ] or even a synthetic training set [ 34 , 5 ] . Most substantial practical applications use a one or few-step inner loop for efficiency . More recently , several methods [ 19 , 25 ] have utilized Implicit Function Theorem ( IFT ) to develop new gradient-based meta-learners . These methods use multiple inner-loop steps without the need to backpropagate through whole inner loop , which significantly improves memory efficiency over methods that need keep track of the whole inner loop training process . However , IFT methods assume the model has converged in the inner loop . This makes them unsuited for the majority of practical applications above where training the inner loop to convergence for each hypergradient step is infeasible . Furthermore , the hypergradient is still more costly compared to one-step T1 − T2 method . The costs come from the associated overhead with approximating an inverse Hessian of the training data with respect to the model parameters . Note that the Hessian itself does not need to be stored due to the mechanics of reverse-mode differentiation [ 8 , 3 ] . However , this does not eliminate the remaining calculations which still require higher-order gradients that result in backpropagation via longer graphs due to additional gradient nodes . For these reasons , we focus comparison on the more widely used T1 − T2 strategy which is oriented at single-step inner loops similar to EvoGrad . Theoretically it is also possible to use hypernetworks [ 18 ] to find good hyperparameters in a first-order way . Hypernetworks take hyperparameters as inputs and generate model parameters . However , the approach is not commonly used , likely due to the difficulty of generating well-performing model parameters . We provide experimental results to support this hypothesis in the supplementary material . Meta-learning can be categorized into several groups , depending on the type of meta-knowledge and also if the model is trained from scratch as part of the inner loop [ 9 ] . Offline meta-learning approaches train a model from scratch per each update of the meta-knowledge , while online metalearning approaches train the model and meta-knowledge jointly . As a result , offline meta-learning is extremely expensive [ 6 , 37 ] when scaled beyond few-shot learning problems where only a few iterations are sufficient for training [ 7 , 1 , 15 ] . Therefore most larger-scale problems [ 16 , 10 , 33 , 35 ] use online learning in practice , and this is where we focus our contribution . The meta-knowledge to learn can take different forms . A particular dichotomy is between the special case where the meta-knowledge corresponds to the base model itself , in the form of an initialization ; and the more general cases where it does not . The former initialization meta-learning has been popularized by MAML [ 7 ] , and is widely used in few-shot learning . This can be solved relatively efficiently , for example using a first-order approximation of MAML [ 7 ] , Reptile [ 23 ] or minibatch proximal update [ 36 ] . On the other hand , there are vastly more cases [ 9 ] where the meta-knowledge is different from the model itself , such as LFT ’ s stochastic regularizer to improve cross-domain generalization [ 33 ] , MWN ’ s instance-wise loss weighting network for label noise robustness [ 31 ] , a label generation network to improve self-supervised generalization [ 17 ] , a Feature-Critic loss to improve domain generalization [ 14 ] and many others . In this more general case , most applications rely on a T1 − T2-like algorithm , as the efficient approximations specific to MAML do not apply . The ability to significantly improve the efficiency of gradient-based meta-learning would have a large impact as methods like these would directly benefit from it in runtime and energy consumption . More crucially , they could scale to bigger and more state-of-the-art neural network architectures . 3 Methods . 3.1 Background : meta-learning as bilevel optimization . We aim to solve a bilevel optimization problem where our goal is to find hyperparameters λ that minimize the validation loss ℓV of the model parametrized by θ and trained with loss ℓT and λ : λ∗ = argmin λ ℓ∗V ( λ ) , where ℓ ∗ V ( λ ) = ℓV ( λ , θ ∗ ( λ ) ) and θ∗ ( λ ) = argmin θ ℓT ( λ , θ ) . ( 1 ) In order to meta-learn the value of λ using gradient-based methods , we need to calculate the hypergradient ∂ℓV∂λ . We can expand its calculation as follows : ∂ℓ∗V ( λ ) ∂λ = ∂ℓV ( λ , θ ∗ ( λ ) ) ∂λ + ∂ℓV ( λ , θ ∗ ( λ ) ) ∂θ∗ ( λ ) ∂θ∗ ( λ ) ∂λ . ( 2 ) In meta-learning and hyperparameter optimization more broadly , the direct term ∂ℓV ( λ , θ ∗ ( λ ) ) ∂λ is typically zero because the hyperparameter does not directly influence the value of the validation loss – it influences it via the impact on the model weights θ . However , the model weights θ are themselves trained using gradient optimization , which gives rise to higher-order derivatives . We propose a variation on this step where the update of the model weights is inspired by evolutionary methods , allowing us to eliminate the need for higher-order derivatives . We consider the setting where the hypergradient of hyperparameter λ is estimated online [ 20 ] together with updating the base model θ , as this is the most widely used setting in substantial practical applications [ 31 , 33 , 14 , 2 , 5 , 30 , 16 , 35 ] . 3.2 The EvoGrad update . Given the current model parameters θ ∈ RM , hyperparameters λ ∈ RN , training loss ℓT and validation loss ℓV , we aim to estimate ∂ℓV∂λ for efficient gradient-based hyperparameter learning . The key idea is – solely for the purpose of hypergradient estimation – to consider a simple evolutionary rather than gradient-based inner-loop step on θ . Evolutionary inner step First , we sample random perturbations ϵ ∈ RM ∼ N ( 0 , σI ) , and apply them to θ . Sampling K perturbations , we can create a population of K variants { θk } Kk=1 of the current model as θk = θ + ϵk . We can now compute the training losses { ℓk } Kk=1 for each of the K models , ℓk = f ( DT |θk , λ ) using the current minibatch DT drawn from the training set . Given these loss values , we can calculate the weights ( sometimes called fitness ) of the population of candidate models as w1 , w2 , . . . , wK = softmax ( [ −ℓ1 , −ℓ2 , . . . , −ℓK ] /τ ) , ( 3 ) where τ is a temperature parameter that rescales the losses to control the scale of weight variability . Given the weights { wk } Kk=1 , we complete the current step of evolutionary learning by updating the model parameters via the affine combination θ∗ = w1θ1 + w2θ2 + · · ·+ wKθK . ( 4 ) Computing the hypergradient We now evaluate the updated model θ∗ for a minibatch from the validation set DV and take gradient of the validation loss ℓV = f ( DV |θ∗ ) w.r.t . the hyperparameter : ∂ℓV ∂λ = ∂f ( DV |θ∗ ) ∂λ ( 5 ) One can easily verify that the computation in Eq . 5 does not involve second-order gradients as no first-order gradients were used in the inner loop . This is in contrast to the typical approach [ 20 , 21 ] of applying gradient-based updates in the inner loop and differentiating through it ( in either forward-mode or reverse-mode ) , or even applying the implicit function theorem ( IFT ) [ 19 ] , all of which trigger higher-order gradients and an extended computation graph . Algorithm flow In practice we follow the flow of T1−T2 [ 20 ] used by many substantive applications [ 33 , 31 , 2 , 16 , 35 ] . We take alternating steps on θ using the exact gradient ∂ℓT∂θ , and on λ using the hypergradient ∂ℓV∂λ , which in EvoGrad is estimated as in Eq . 5 .

Summary: The authors introduce EvoGrad, a method for approximating second order gradients for use in Meta-Learning applications. The method approximates the inner loop of optimization with "evolutionary" update. The authors show their method is more computationally efficient than the T1-T2 method, yields good approximations to the second order gradients and is able to converge in the toy setting, and achieves better results than MetaWeightNet, a second order method, on a cross-domain few-shot classification task.