Decoupling Adaptation from Modeling with Meta-Optimizers for Meta Learning

1 INTRODUCTION . Meta-learning or learning to learn is an appealing notion due to its potential in addressing important challenges when applying machine learning to real-world problems . In particular , learning from prior tasks but being able to to adapt quickly to new tasks improves learning efficiency , model robustness , etc . A promising set of techiques , Model-Agnostic Meta-Learning ( Finn et al. , 2017 ) or MAML , and its variants , have received a lot of interest ( Nichol et al. , 2018 ; Lee & Choi , 2018 ; Grant et al. , 2018 ) . However , despite several efforts , understanding of how MAML works , either theoretically or in practice , has been lacking ( Finn & Levine , 2018 ; Fallah et al. , 2019 ) . For a model that meta-learns , its parameters need to encode not only the common knowledge extracted from the tasks it has seen , which form a task-general inductive bias , but also the capability to adapt to new test tasks ( similar to those it has seen ) with task-specific knowledge . This begs the question : how are these two sets of capabilities represented in a single model and how do they work together ? In the case of deep learning models , one natural hypothesis is that while knowledge is represented distributedly in parameters , they can be localized – for instance , lower layers encode task-general inductive bias and the higher layers encode adaptable task-specific inductive bias . This hypothesis is consistent with one of deep learning ’ s advantages in learning representations ( or feature extractors ) using its bottom layers . Then we must ask , in order for a deep learning model to meta-learn , does it need more depth than it needs for solving the target tasks ? In other words , is having a large capacity to encode knowledge that is unnecessary post-adaptation the price one has to pay in order to be adaptable ? Is there a way to have a smaller ( say , less deep ) meta-learnable model which still adapts well ? This question is of both scientific interest and practical importance – a smaller model has a smaller ( memory ) footprint , faster inference and consumes less resources . In this work , through empirical studies on both synthetic datasets and benchmarks used in the literature , we investigate these questions by analyzing how well different learning models can meta-learn and adapt . We choose to focus on MAML due to its popularity . Our observations suggest depth is indeed necessary for meta-learning , despite the tasks being solvable using a shallower model . Thus , applying MAML to shallower models does not result in successful meta-learning models that can adapt well . Moreover , our studies also show that higher layers are responsible more for adapting to new tasks while the lower layers are responsible for learning task-general features . Our findings prompt us to propose a new method for meta-learning . The new approach introduces a meta-optimizer which learns to guide the ( parameter ) optimization process of a small model . The small model is used for solving the tasks while the optimizer bears the burden of extracting the knowledge of how to adapt . Empirical results show that despite using smaller models , the proposed algorithm with small models attains similar performance to larger models which use MAML to meta-learn and adapt . We note that a recent and concurrent work to ours addresses questions in this line of inquiry ( Raghu et al. , 2019 ) . They reach similar conclusions through different analysis and likewise , they propose a different approach for improving MAML . We believe our work is complementary to theirs . 2 RELATED WORK . Meta-learning , or learning-to-learn , is a vibrant research area with a long and rich history , lying at the intersection of psychology ( Maudsley , 1980 ; Biggs , 1985 ) , neuroscience ( Hasselmo & Bower , 1993 ) , and computer science ( Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) Of particular interest to this manuscript is the line of work concerned with optimization-based meta-learning ( OBML ) algorithms in the few-shot regime , of which MAML is a particular instance . ( Finn & Levine , 2018 ; Finn et al. , 2017 ; Finn , 2018 ) Since its inception , MAML has been widely applied to tackle the few-shot learning challenge , in domains such as computer vision ( Lee et al. , 2019 ) , natural language processing ( Gu et al. , 2018 ) , and robotics ( Nagabandi et al. , 2018 ) . It is also the basis of extensions for continual learning ( Finn et al. , 2019 ) , single- and multi-agent reinforcement learning ( Rothfuss et al. , 2018 ; Al-Shedivat et al. , 2017 ) , objective learning ( Chebotar et al. , 2019 ) , transfer learning ( Kirsch et al. , 2019 ) , and domain adaptation ( Li et al. , 2018 ) . Due to its generality , the adaptation procedure MAML introduces – which is the focus of our analysis – has recently been branded as Generalized Inner Loop MetaLearning . ( Grefenstette et al. , 2019 ) While popular in practical applications , relatively few works have analysed the convergence and modelling properties of those algorithms . Finn & Levine ( 2018 ) showed that , when combined with deep architectures , OBML is able to approximate arbitrary meta-learning schemes . Fallah et al . ( 2019 ) recently provided convergence guarantees for MAML to approximate first-order stationary points for non-convex loss surfaces , under some assumptions on the availability and distribution of the data . Other analyses ( empirical or theoretical ) have attempted to explain the generalization ability of OBML ( Guiroy et al. , 2019 ; Nichol et al. , 2018 ) , the bias induced by restricting the number of adaptation steps ( Wu et al. , 2018 ) , or the effect of higher-order terms in the meta-gradient estimation ( Foerster et al. , 2018 ; Rothfuss et al. , 2019 ) Closely related to our proposed methods are works attempting to improve the adaptation mechanisms of OBML . Meta-SGD ( Li et al. , 2017 ) meta-learns per-parameter learning rates , while Alpha MAML ( Behl et al. , 2019 ) adapts those learning rates during adaptation via gradient-descent . MetaCurvature ( Park & Oliva , 2019 ) propose to learn a Kronecker-factored pre-conditioning matrix to compute fast-adaptation updates . Their resulting algorithm is a special case of one of our methods , where the linear transformation is not updated during adaptation . Another way of constructing preconditioning matrices is to explicitly decompose all weight matrices of the model in two separate components , as done in T-Nets ( Lee & Choi , 2018 ) . The first component is only updated via the evaluation loss , while the second is also updated during fast-adaptation . Warped Gradient Descent ( Flennerhag et al. , 2019 ) further extends T-Nets by allowing both components to be non-linear functions . Instead of directly working with gradients , ( Chebotar et al. , 2019 ; Xu et al. , 2018 ) suggest to directly learn a loss function which is differentiated during fast-adaptation and results in faster and better learning . Additionally , meta-optimizers have also been used for meta-descent ( Sutton , 1981 ; Jacobs , 1988 ; Sutton , 1992b ; a ; Schraudolph , 1999 ) . They can be learned during a pre-training phase ( Andrychowicz et al. , 2016 ; Li & Malik , 2017a ; b ; Metz et al. , 2019b ; a ; c ; Wichrowska et al. , 2017 ) or online ( Kearney et al. , 2018 ; Jacobsen et al. , 2019 ; Ravi & Larochelle , 2017 ) . Our work differentiates from the the above by diagnosing and attempting to address the entanglement between modelling and adaptation in the meta-learning regime . We uncover a failure mode of MAML with linear and smaller models , and propose an effective solution in the form of expressive meta-optimizers . 3 ANALYSIS OF MAML . 3.1 BACKGROUND . In MAML and its many variants ( Lee & Choi , 2018 ; Nichol et al. , 2018 ; Li et al. , 2017 ) , we have a model whose parameters are denoted by θ . We would like to optimize θ such that the resulting model can adapt to new and unseeen tasks fast . To this end , we are given a set of ( meta ) training tasks , indexed by τ . For each such task , we associate with a loss Lτ ( θ ) . Distinctively , MAML minimizes the expected task loss after an adaptation phase , consisting of a few steps of gradient descent from the model ’ s current parameters . Since we do not have access to the target tasks to which we wish to adapt to , we use the expected loss over the training tasks , LMETA = E τ∼p ( τ ) [ Lτ ( θ − α∇Lτ ( θ ) ) ] ( 1 ) where the expectation is taken with respect to the distribution of the training tasks . α is the learning rate for the adaptation phase . The right-hand-side uses only one step gradient descent such that the aim is to adapt fast : in one step , we would like to reduce the loss as much as possible . In practice , a few more steps are often used . 3.2 ANALYSIS . Shallow models can be hard to meta-learn Many intuitive explanations for why MAML works exist . One appealing suggestion is that the minimizer of the meta-learning loss LMETA is chosen in such a way that it provides a very good initialization for the adaptation phase ; however , if the model is shallow such that Lτ is convex in its parameters , then any initialization that is good for fast adapting to one subset of tasks could be bad for another subset of tasks since all the tasks have precisely one global minimizer and those minimizers can be arbitrarily far from each other . When the test tasks are distributed similar to the training tasks , the ideal initialization point has to be the “ mean ” of the minimizers of the training tasks — the precise definition of the mean is not important , as we will see below . We illustrate a surprising challenge by studying MAML on a synthetic dataset and the Omniglot task ( Lake et al. , 2015 ) . Specifically , for the former study , we construct a set of binary classification tasks by first randomly sampling datapoints w ∈ R100 from a standard Gaussian and use each of them to define a linear decision boundary of a binary classification task . We assume the boundaries pass through the origin and we sample training , validation and testing samples by randomly sampling data points from both sides of the decision boundaries . By construction , a linear model such as logistic regression is sufficient to achieve very high accuracy on any task . But can MAML learn a logistic regression model from a subset of training tasks that is able to adapt quickly to the test tasks ? Note that due to the random sampling of the training tasks , the average of the minimizers ( ie , the samples from the Gaussian distribution ) is the origin . Likewise , for a set of test tasks randomly sampled the same way , the origin provides the best initialization by not favoring any particular task . Figure 1 reports the 1-step post-adaptation accuracy on the test tasks for the meta-learned logistic regression model . Surprisingly , the model fails to perform better than chance . Despite the simplicity of the tasks , logistic regression models are unable to find the origin as an initialization that adapts quickly to a set of test tasks that are similar to the training tasks . Figure 1 also reports how being deep can drastically change the behavior of MAML . There , we add a 4-layer linear network ( LinNet ) to the logistic regression model ( before the sigmoid activation ) . Note that while the model has the same representational capacity as a linear logistic regression , it is overparameterized and has many local optimizers . As such , MAML can train this model such that its 1-step adaptation accuracy reaches 92 % on average . We observe the same phenomena on meta-learning with MAML on the Omniglot dataset ( details of the dataset are given in the Section 5 ) . The shallow logistic regression model achieves 45 % accuracy on average ( for the 5-way classification tasks ) after 2-step adaptation from the meta-learned initialization . However , with a linear network , the adapted model achieves significantly higher accuracy – 70 % on average , while having the same modeling capacity as the logistic regression . In summary , these experiments suggest that even for tasks that are solvable with shallow models , a model needs to have enough depth in order to be meta-learnable and to adapt . We postpone the description of our experiments on nonlinear models to section 5 , where we also show that having sufficient depth is crucial for models to be meta-learnable , even when the tasks require fewer layers . A natural question arises : if being deep is so important for meta-learning on even very simple tasks , what different roles , if any , do different layers of a deep network play ? Depth enables task-general feature learning and fast adaptation We hypothesize that for deep models meta-learned with MAML , lower layers learn task-invariant features while higher layers are responsible for fast-adaptation . To examine this claim , we meta-train a model consisting of four convolutional layers ( C1 - C4 ) and a final fully-connected layer ( FC ) on Omniglot ( Lake et al. , 2015 ) and CIFAR-FS ( Bertinetto et al. , 2019 ) . ( Experimental setups are detailed in Section 5 . ) Once the model has finished meta-training , we perform a layer-wise ablation to study each layer ’ s effect on adaptation . In particular , we iterate over each layer and perform two sets of experiments . In the first , we freeze the weights of the layer such that it does not get updated during fast-adaptation – we call it freezing only this . In the second experiment , we freeze all layers but the layer such that this layer is updated during fast-adaptation – we call it adapting only this . The left two plots in Figure 2 report the average accuracy over 100 testing tasks from both datasets . We observe that freezing only the first few lower layers ( C1-C3 ) does not cause noticeable degradation to the post-adaptation accuracy . In fact , as long as the last convolutional layer ( C4 ) is not frozen , post-adaptation accuracy remains unaffected . This indicate that C1-C3 provide information that is task-invariant , while C4 is crucial for adaptation . This does not mean FC is not important — since adapting C4 requires gradients passing through the FC layer , it can not be arbitrary . In fact , in the rightmost plot of the figure , C1-C3 are held fixed during adaptation . While C4 and FC are allowed to adapt , and FC is perturbed with noise . When the noise is strong , the performance degrades significantly . Thus , we conclude that both C4 and FC play important roles in the mechanism for fast adaptation . We note that the recent work by Raghu et al . ( 2019 ) concurently reached similar conclusions on the mini-ImageNet dataset , using feature similarity-based analyses of the model ’ s representations . These observations highlight a fundamental issue : the property of being meta-learnable entails more model capacity than being learnable for a specific task . Thus , MAML can fail on models that lack the capacity to encode both task-general features and adaptation information , even when the models themselves are powerful enough to perform well on each of the tasks used for the meta-learning procedure . For example , with linear models ( e.g . logistic regression ) , the parameters are forced to overlap and serve both modelling and adaptation purposes . However , as soon as the models are overparameterized , the extra layers enable meta-learnability . In section 5 , we show that this observation also applies to nonlinear models where MAML-trained models quickly lose their performance when the number of layers is reduced .

Summary: This paper presents an experimental study of gradient based meta learning models and most notably MAML. The results suggest that modeling and adaptation are happening on different parts of the network leading to an inefficient use of the model capacity which explains the poor performance of MAML on linear (or small networks) models. To tackle this issue they proposed a kronecker factorization of the meta optimizer.