Role of two learning rates in convergence of model-agnostic meta-learning

Model-agnostic meta-learning ( MAML ) is known as a powerful meta-learning method . However , MAML is notorious for being hard to train because of the existence of two learning rates . Therefore , in this paper , we derived a sufficinent condition of inner learning rate α and meta-learning rate β for a simplified MAML to locally converge to local minima from any point in the vicinity of the local minima . We find that the upper bound of β depends on α . Moreover , we show that the threshold of β increases as α approaches its own upper bound . This result is verified by experiments on various few-shot tasks and architectures ; specifically , we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network . Based on this outcome , we present a guideline for determining the learning rates : first , search for the largest possible α ; next , tune β based on the chosen value of α . 1 INTRODUCTION . A pillar of human intelligence is the ability to learn and adapt to unseen tasks quickly and based on only a limited quantity of data . Although machine learning has achieved remarkable results , many recent models require massive quantities of data and are designed for solving particular tasks . Meta-learning , one of the ways of tackling this problem , tries to develop a model that can adapt to new tasks quickly by learning to learn new concepts from few data points ( Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) . Among meta-learning algorithms , model-agnostic meta-learning ( MAML ) , a gradient-based metalearning method proposed by Finn et al . ( 2017 ) , has recently been extensively studied . For example , MAML is used for continual learning ( Finn et al. , 2019 ; Jerfel et al. , 2019 ; Spigler , 2019 ; Al-Shedivat et al. , 2018 ) , reinforcement learning ( Finn et al. , 2017 ; Al-Shedivat et al. , 2018 ; Gupta et al. , 2018 ; Deleu & Bengio , 2018 ; Liu & Theodorou , 2019 ) and probablistic inference ( Finn et al. , 2018 ; Yoon et al. , 2018 ; Grant et al. , 2018 ) . The reason why MAML is widely used is because MAML is simple but efficient and applicable to a wide range of tasks independent of model architecture and the loss function . However , MAML is notorious for being hard to train ( Antoniou et al. , 2019 ) . One of the reasons why training MAML is hard is the existence of two learning rates in MAML : the inner learning rate α and meta-learning rate β . A learning rate is known to be one of the most important parameters , and tuning this parameter may be challenging even if the simple gradient descent ( GD ) method is used . Nevertheless , we do not yet know the relationship between these two learning rates and have little guidance on how to tune them . Hence , guidelines for choosing these parameters are urgently needed . In this paper , we investigate the MAML algorithm and propose a guideline for selecting the learning rates . First , in Section 2 we briefly explain by using an approximation how MAML can be regarded as optimization with the negative gradient penalty . Because the gradient norm is related to the shape of the loss surface , a bias towards a larger gradient norm can make training unstable . Next , based on the approximation explained in Section 2 , in Section 3 , we derive a sufficinent condition of α and β for a simplified MAML to locally converge to local minima from any point in the neighborhood of the local minima . Furthermore , by removing a constraint , we derive a sufficient condition for local convergence with fewer simplifications as well . We find that the upper bound βc of meta-learning rate depends on inner learning rate α . In particular , βc of α ≈ αc is larger than that of α = 0 , where αc is the upper bound of α . This is verified by experiments in Section 5 . These results imply a guideline for selecting the learning rates : first , search for the largest possible α ; next , tune β . 2 MAML AS OPTIMIZATION WITH NEGATIVE GRADIENT PENALTY . 2.1 MAML . The goal of MAML is to find a representation that can rapidly adapt to new tasks with a small quantity of data . In other words , MAML performs optimization for parameters θ ∈ Rd that the optimizer can use to quickly reach the optimal parameter θ∗τ for task τ with few data points . To this end , MAML takes the following steps to update θ . First , it samples a batch of tasks from task distribution P ( τ ) and updates θ for each task τ with stochastic gradient descent ( SGD ) . Although MAML allows multiple-step being taken to update θ , we will consider the case only one step being taken for simplicity . The update equation is as follows : θ′τ = θ − α∇θLτ ( θ ) , ( 1 ) where α is a step size referred to as the inner learning rate , Lτ ( θ ) is the loss of τ , and ∇θLτ ( θ ) is an estimate of the true gradient . The data used for this update is called training data . Next , MAML resamples data , referred to as test data , from each τ and computes the loss at the updated parameters θ′τ , obtaining Lτ ( θ ′ τ ) for each task . Finally , to determine θ that can be adapted to θ ′ τ for all tasks , θ is updated with the gradient of a sum of loss values Lτ ( θ′τ ) over all tasks . In other words , θ ← θ − β∇θ ∑ τ∼P ( τ ) Lτ ( θ ′ τ ) , ( 2 ) where β is the learning rate called the meta-learning rate and∇θ ∑ τ∼P ( τ ) Lτ ( θ ′ τ ) is an estimate of the true gradient by using the test data . Though learning rates α and β can be tuned during training or different for each task in practice , we will think them as fixed scalar hyperparameters . 2.2 NEGATIVE GRADIENT PENALTY . Unless otherwise noted , we will consider the case of only one step being made per update , and the data are not resampled to compute the loss for updating θ . The case of multiple steps and that of training data and test data being separated are considered in Appendix A . The gradient of the loss at θ′τ is gτ ( θ ′ τ ) = ∇θLτ ( θ′τ ) = ∇θθ′τ ∂Lτ∂θ′τ , where g ( · ) is the gradient of L ( · ) with respect to θ . If α is small , we can assume that I ∂Lτ∂θ′τ = gτ ( θ ) ; this seems to hold since α is usually small ( Finn et al. , 2017 ) . Then , ∇θLτ ( θ′τ ) = ∇θθ′τ ∂Lτ ∂θ′τ = ( I − α∇2θLτ ) ∂Lτ ∂θ′τ ( 3 ) ≈ gτ ( θ ) − αHτ ( θ ) gτ ( θ ) . ( 4 ) The result but not the procedure with the approximation is the same as that with the well-known firstorder approximation as long as data are not resampled and only one step being taken . The first order approximation has been mentioned by Finn et al . ( 2017 ) and extensively studied by Nichol et al . ( 2018 ) and Fallah et al . ( 2019 ) . It is known that the error induced by the first-order approximation does not degrade the performance so much in practice ( Finn et al. , 2017 ) . Also , Fallah et al . ( 2019 ) theoretically proved that the first-order approximation of MAML does not affect convergence result when each task is similar to each other or α is small enough . For simplicity , we will assume that only one task is considered during training , omitting task index τ . Therefore , instead of ∑ τ∼P ( τ ) Lτ ( θ ′ τ ) , we will consider L ( θ ′ ) as the loss of the simplified MAML . Since the MAML loss is just a sum of task-specific loss , extension to the case of multiple tasks being considered is straightforward , as provided in Appendix A.1.1 . Because ∇θ ( g ( θ ) ⊤g ( θ ) ) = 2H ( θ ) g ( θ ) , if we define L̃ ( θ ) = L ( θ′ ) , L̃ ( θ ) ≈ L ( θ ) − α 2 g ( θ ) ⊤g ( θ ) . ( 5 ) The above means that the simplified MAML can be regarded as optimization with the negative gradient penalty . We will analyze this simplified MAML loss in Section 3 . It can also be interpreted as a Taylor series expansion of the simplified MAML loss for the first-order term , up to scale : L̃ ( θ ) = L ( θ − α∇θL ( θ ) ) ( 6 ) ≈ L ( θ ) − α∇θL ( θ ) ⊤∇θL ( θ ) ( Taylor series expansion ) = L ( θ ) − αg ( θ ) ⊤g ( θ ) . ( 7 ) The fact that the simplified MAML is optimization with the negative gradient penalty is worth keeping in mind . Because the goal of gradient-based optimization is to find a point where the gradient is zero , a bias that favors a larger gradient is highly likely to make training unstable ; this can be a cause of instability of MAML ( Antoniou et al. , 2019 ) . In fact , as shown in Fig . 1 , the gradient norm becomes larger during training , as do the gradient inner products , as Guiroy et al . ( 2019 ) observed . 3 LEARNING RATE FOR LOCAL CONVERGENCE . In this section , we will derive a sufficient condition of learning rate α and β for local convergence from any point in the vicinity of the local minima . To this end , we will assume that only one step is taken for update and training data and test data are not distinguished as we did in Section 2 . Also , we do not consider SGD but steepest GD to derive the condition . In other words , same training data are assumed to be used continuously for updating parameters during training . First , we will consider the case of only single task being considered . Next , we will consider the case of multiple tasks being considered . 3.1 SINGLE TASK . 3.1.1 CONDITION FOR INNER LEARNING RATE α First , we derive the sufficient condition of learning rate α . To this end , we will consider the sufficinet condition that a fixed point is a local minimum . Taking the Taylor series for the second-order term at a fixed point θ∗ , the simplified MAML loss is L̃ ( θ ) ≈ L̃ ( θ∗ ) + 1 2 ( θ − θ∗ ) ⊤H̃ ( θ − θ∗ ) . ( 8 ) where H̃ = H − α ( Tg +H2 ) is the Hessian matrix of L̃ ( θ ) at θ∗ and g = ∇θL ( θ∗ ) ∈ Rd , H = ∇2θL ( θ∗ ) ∈ Rd×d , and T = ∇3θL ( θ∗ ) ∈ Rd×d×d . The calculation of H̃ is presented in Appendix B . We calculated the magnitudes of Tg and H2 numerically and observed that Tg was much smaller than H2 in practice . Hence , we will ignore Tg while deriving the condition and will thus assume that H̃ = H − αH2 . Further details are provided in Appendix C , and the case of Tg being considered is provided in Appendix D. Since PΛH̃P ⊤ = P [ ΛH − αΛ2H ] P⊤ where ΛH̃ is a diagonal matrix with entries that are eigenvalues of H̃ and P is a matrix with rows that are eigenvectors of H̃ , the sufficinet condition of α for θ∗ to be a local minimum is ∀i , λ ( H̃ ) i = λ ( H ) i − αλ ( H ) 2i > 0 ( 9 ) ⇒ ∀i , α < 1 λ ( H ) i . ( 10 ) Note that λ ( A ) i represents the ith eigenvalue of matrix A . Hence , the sufficient condition of α for θ∗ to be a local minimum is ∀i , α < 1 λ ( H ) i . ( 11 ) Therefore , αc is the inverse of the largest eigenvalue of H . 3.1.2 CONDITION FOR META-LEARNING RATE β Next , we derive the sufficient condition of meta-learning rate β for the simplified MAML to locally converge to the local minima discussed above from any point in the vicinity of the local minimum . This is an extension of research of LeCun et al . ( 1998 ) . Since PP⊤ = I , the simplified MAML loss can be written as L̃ ( θ ) ≈ L̃ ( θ∗ ) + 1 2 ( ( θ − θ∗ ) ⊤P ) P⊤H̃P ( P⊤ ( θ − θ∗ ) ) . ( 12 ) By using the simplified loss defined in Eq . 8 , the update equation of the parameter θ with GD is θ ( t+ 1 ) = θ ( t ) − β∇θL̃ ( θ ) ( 13 ) = θ ( t ) − βH̃ ( θ − θ∗ ) ( 14 ) where t ( t = 0 , ... , M ) is iteration and M is the total number of iterations . Hence , θ ( t+ 1 ) − θ∗ = ( I − βH̃ ) ( θ ( t ) − θ∗ ) . If we denote P⊤ ( θ − θ∗ ) by v , the simplified MAML loss in Eq . 12 is L̃ ( v ) ≈ L̃ ( 0 ) + 12v ⊤ΛH̃v . Because the gradient of L̃ ( v ) for v is ∇vL̃ ( v ) = ΛH̃v , the update equation of v is v ( t+ 1 ) = v ( t ) − βΛH̃v ( t ) = ( I − βΛH̃ ) v ( t ) , ( 15 ) where v ( t ) is the value of v during iteration t. Assuming that Eq . 11 holds , the sufficinet condition of β is as follows : for all i , |1− βλ ( H − αH2 ) i| = |1− β ( λ ( H ) i − αλ ( H ) 2i ) | < 1 ( 16 ) ⇒ −1 + β ( λ ( H ) i − αλ ( H ) 2i ) < 1 ( ∵ λ ( H ) i − αλ ( H ) 2i > 0 holds because of Eq . 11 ) ( 17 ) ⇒ β < 2 λ ( H ) i − αλ ( H ) 2i . ( 18 ) Consequently , the sufficient condition for the simplified MAML to locally converge to local minima from any point in the vicinity of the local minima is as follows : ∀i , α < 1 λ ( H ) i ∧ β < 2 λ ( H ) i − αλ ( H ) 2i . ( 19 ) Vanilla GD with learning rate β corresponds to MAML if α = 0 . In this case , β < 2λmax is the condition of β , where λmax is the largest eigenvalue of H , because 2/λmax is smaller than any other 2/λi ( LeCun et al. , 1998 ) . Though this holds for the simplified MAML as well , this is not the case if α is close to αc . The reason is that βc diverges as α approaches 1λ ( H ) i , or αc as Eq . 18 indicates . Hence , for the simplified MAML we must consider not only the largest but also other eigenvalues and in particular , the second-largest eigenvalue . In short , unlike when vanilla GD is employed , βc depends on α in the case of the simplified MAML , and βc is expected to be larger if α is close to αc , as shown in Fig . 2 . This finding is validated by experiments presented in Section 5 .

Summary: The authors study a method to help tuning the two learning rates used in the MAML training algorithm. First, they derive a necessary condition for the convergence of the gradient descent in the single task setting. The condition relies on the eigenvalues of the Hessian of the task loss. This condition is reminiscent of convergence criteria for gradient descent on quadratic objectives, and the authors make the interesting observation that the criteria for the exterior learning rate beta depends on the interior learning rate alpha.