Meta-Learning without Memorization

1 INTRODUCTION . The ability to learn new concepts and skills with small amounts of data is a critical aspect of intelligence that many machine learning systems lack . Meta-learning ( Schmidhuber , 1987 ) has emerged as a promising approach for enabling systems to quickly learn new tasks by building upon experience from previous related tasks ( Thrun & Pratt , 2012 ; Koch et al. , 2015 ; Santoro et al. , 2016 ; Ravi & Larochelle , 2016 ; Finn et al. , 2017 ) . Meta-learning accomplishes this by explicitly optimizing for few-shot generalization across a set of meta-training tasks . The meta-learner is trained such that , after being presented with a small task training set , it can accurately make predictions on test datapoints for that meta-training task . While these methods have shown promising results , current methods require careful design of the meta-training tasks to prevent a subtle form of task overfitting , distinct from standard overfitting in supervised learning . If the task can be accurately inferred from the test input alone , then the task training data can be ignored while still achieving low meta-training loss . In effect , the model will collapse to one that makes zero-shot decisions . This presents an opportunity for overfitting where the meta-learner generalizes on meta-training tasks , but fails to adapt when presented with training data from novel tasks . We call this form of overfitting the memorization problem in meta-learning because the meta-learner memorizes a function that solves all of the meta-training tasks , rather than learning to adapt . Existing meta-learning algorithms implicitly resolve this problem by carefully designing the metatraining tasks such that no single model can solve all tasks zero-shot ; we call tasks constructed in this Implementation and examples available here : https : //github.com/google-research/ google-research/tree/master/meta_learning_without_memorization . way mutually-exclusive . For example , forN -way classification , each task consists of examples from N randomly sampled classes . The N classes are labeled from 1 to N , and critically , for each task , we randomize the assignment of classes to labels { 1 , 2 , . . . , N } ( visualized in Appendix Figure 3 ) . This ensures that the task-specific class-to-label assignment can not be inferred from a test input alone . However , the mutually-exclusive tasks requirement places a substantial burden on the user to cleverly design the meta-training setup ( e.g. , by shuffling labels or omitting goal information ) . While shuffling labels provides a reasonable mechanism to force tasks to be mutually-exclusive with standard few-shot image classification datasets such as MiniImageNet ( Ravi & Larochelle , 2016 ) , this solution can not be applied to all domains where we would like to utilize meta-learning . For example , consider meta-learning a pose predictor that can adapt to different objects : even if N different objects are used for meta-training , a powerful model can simply learn to ignore the training set for each task , and directly learn to predict the pose of each of the N objects . However , such a model would not be able to adapt to new objects at meta-test time . The primary contributions of this work are : 1 ) to identify and formalize the memorization problem in meta-learning , and 2 ) to propose a meta-regularizer ( MR ) using information theory as a general approach for mitigating this problem without placing restrictions on the task distribution . We formally differentiate the meta-learning memorization problem from overfitting problem in conventional supervised learning , and empirically show that naı̈ve applications of standard regularization techniques do not solve the memorization problem in meta-learning . The key insight of our metaregularization approach is that the model acquired when memorizing tasks is more complex than the model that results from task-specific adaptation because the memorization model is a single model that simultaneously performs well on all tasks . It needs to contain all information in its weights needed to do well on test points without looking at training points . Therefore we would expect the information content of the weights of a memorization model to be larger , and hence the model should be more complex . As a result , we propose an objective that regularizes the information complexity of the meta-learned function class ( motivated by Alemi et al . ( 2016 ) ; Achille & Soatto ( 2018 ) ) . Furthermore , we show that meta-regularization in MAML can be rigorously motivated by a PAC-Bayes bound on generalization . In a series of experiments on non-mutually-exclusive task distributions entailing both few-shot regression and classification , we find that memorization poses a significant challenge for both gradient-based ( Finn et al. , 2017 ) and contextual ( Garnelo et al. , 2018a ) meta-learning methods , resulting in near random performance on test tasks in some cases . Our meta-regularization approach enables both of these methods to achieve efficient adaptation and generalization , leading to substantial performance gains across the board on non-mutually-exclusive tasks . 2 PRELIMINARIES . We focus on the standard supervised meta-learning problem ( see , e.g. , Finn et al . ( 2017 ) ) . Briefly , we assume tasks Ti are sampled from a task distribution p ( T ) . During meta-training , for each task , we observe a set of training data Di = ( xi , yi ) and a set of test data D∗i = ( x∗i , y∗i ) with xi = ( xi1 , . . . , xiK ) , yi = ( yi1 , . . . , yiK ) sampled from p ( x , y|Ti ) , and similarly forD∗i . We denote the entire meta-training set asM = { Di , D∗i } Ni=1 . The goal of meta-training is to learn a model for a new task T by leveraging what is learned during meta-training and a small amount of training data for the new task D. We use θ to denote the meta-parameters learned during meta-training and use φ to denote the task-specific parameters that are computed based on the task training data . Following Grant et al . ( 2018 ) ; Gordon et al . ( 2018 ) , given a meta-training set M , we consider meta-learning algorithms that maximize conditional likelihood q ( ŷ∗ = y∗|x∗ , θ , D ) , which is composed of three distributions : q ( θ|M ) that summarizes meta-training data into a distribution on metaparameters , q ( φ|D , θ ) that summarizes the per-task training set into a distribution on task-specific parameters , and q ( ŷ∗|x∗ , φ , θ ) that is the predictive distribution . These distributions are learned to minimize − 1N ∑ i Eq ( θ|M ) q ( φ|Di , θ ) [ 1 K ∑ ( x∗ , y∗ ) ∈D∗i log q ( ŷ∗ = y∗|x∗ , φ , θ ) ] . ( 1 ) For example , in MAML ( Finn et al. , 2017 ) , θ and φ are the weights of a predictor network , q ( θ|M ) is a delta function learned over the meta-training data , q ( φ|D , θ ) is a delta function centered at a point defined by gradient optimization , and φ parameterizes the predictor network q ( ŷ∗|x∗ , φ ) ( Grant et al. , 2018 ) . In particular , to determine the task-specific parameters φ , the task training data D and θ are used in the predictor model φ = θ + αK ∑ ( x , y ) ∈D∇θ log q ( y|x , φ = θ ) . Another family of meta-learning algorithms are contextual methods ( Santoro et al. , 2016 ) , such as conditional neural processes ( CNP ) ( Garnelo et al. , 2018b ; a ) . CNP instead defines q ( φ|D , θ ) as a mapping from D to a summary statistic φ ( parameterized by θ ) . In particular , φ = aθ ◦ hθ ( D ) is the output of an aggregator aθ ( · ) applied to features hθ ( D ) extracted from the task training data . Then θ parameterizes a predictor network that takes φ and x∗ as input and produces a predictive distribution q ( ŷ∗|x∗ , φ , θ ) . In the following sections , we describe a common pitfall for a variety of meta-learning algorithms , including MAML and CNP , and a general meta-regularization approach to prevent this pitfall . 3 THE MEMORIZATION PROBLEM IN META-LEARNING . The ideal meta-learning algorithm will learn in such a way that generalizes to novel tasks . However , we find that unless tasks are carefully designed , current meta-learning algorithms can overfit to the tasks and end up ignoring the task training data ( i.e. , either q ( φ|D , θ ) does not depend on D or q ( ŷ∗|x∗ , φ , θ ) does not depend on φ , as shown in Figure 1 ) , which can lead to poor generalization . This memorization phenomenon is best understood through examples . Consider a 3D object pose prediction problem ( illustrated in Figure 1 ) , where each object has a fixed canonical pose . The ( x , y ) pairs for the task are 2D grey-scale images of the rotated object ( x ) and the rotation angle relative to the fixed canonical pose for that object ( y ) . In the most extreme case , for an unseen object , the task is impossible without using D because the canonical pose for the unseen object is unknown . The number of objects in the meta-training dataset is small , so it is straightforward for a single network to memorize the canonical pose for each training object and to infer the object from the input image ( i.e. , task overfitting ) , thus achieving a low training error without using D. However , by construction , this solution will necessarily have poor generalization to test tasks with unseen objects . As another example , imagine an automated medical prescription system that suggests medication prescriptions to doctors based on patient symptoms and the patient ’ s previous record of prescription responses ( i.e. , medical history ) for adaptation . In the meta-learning framework , each patient represents a separate task . Here , the symptoms and prescriptions have a close relationship , so we can not assign random prescriptions to symptoms , in contrast to the classification tasks where we can randomly shuffle the labels to create mutually-exclusiveness . For this non-mutually-exclusive task distribution , a standard meta-learning system can memorize the patients ’ identity information in the training , leading it to ignore the medical history and only utilize the symptoms combined with the memorized information . As a result , it may issue highly accurate prescriptions on the meta-training set , but fail to adapt to new patients effectively . While such a system would achieve a baseline level of accuracy for new patients , it would be no better than a standard supervised learning method applied to the pooled data . We formally define ( complete ) memorization as : Definition 1 ( Complete Meta-Learning Memorization ) . Complete memorization in meta-learning is when the learned model ignores the task training data such that I ( ŷ∗ ; D|x∗ , θ ) = 0 ( i.e. , q ( ŷ∗|x∗ , θ , D ) = q ( ŷ∗|x∗ , θ ) = ED′|x∗ [ q ( ŷ∗|x∗ , θ , D′ ) ] ) . Memorization describes an issue with overfitting the meta-training tasks , but it does not preclude the network from generalizing to unseen ( x , y ) pairs on the tasks similar to the training tasks . Memorization becomes an undesired problem for generalization to new tasks when I ( y∗ ; D|x∗ ) I ( ŷ∗ ; D|x∗ , θ ) ( i.e. , the task training data is necessary to achieve good performance , even with exact inference under the data generating distribution , to make accurate predictions ) . A model with the memorization problem may generalize to new datapoints in training tasks but can not generalize to novel tasks , which distinguishes it from typical overfitting in supervised learning . In practice , we find that MAML and CNP frequently converge to this memorization solution ( Table 2 ) . For MAML , memorization can occur when a particular setting of θ that does not adapt to the task training data can achieve comparable meta-training error to a solution that adapts θ . For example , if a setting of θ can solve all of the meta-training tasks ( i.e. , for all ( x , y ) in D and D∗ the predictive error is close to zero ) , the optimization may converge to a stationary point of the MAML objective where minimal adaptation occurs based on the task training set ( i.e. , φ ≈ θ ) . For a novel task where it is necessary to use the task training data , MAML can in principle still leverage the task training data because the adaptation step is based on gradient descent . However , in practice , the poor initialization of θ can affect the model ’ s ability to generalize from a small mount of data . For CNP , memorization can occur when the predictive distribution network q ( ŷ∗|x∗ , φ , θ ) can achieve low training error without using the task training summary statistics φ . On a novel task , the network is not trained to use φ , so it is unable to use the information extracted from the task training set to effectively generalize . In some problem domains , the memorization problem can be avoided by carefully constructing the tasks . For example , for N -way classification , each task consists of examples from N randomly sampled classes . If the classes are assigned to a random permutation ofN for each task , this ensures that the task-specific class-to-label assignment can not be inferred from the test inputs alone . As a result , a model that ignores the task training data can not achieve low training error , preventing convergence to the memorization problem . We refer to tasks constructed in this way as mutuallyexclusive . However , the mutually-exclusive tasks requirement places a substantial burden on the user to cleverly design the meta-training setup ( e.g. , by shuffling labels or omitting goal information ) and can not be applied to all domains where we would like to utilize meta-learning .

Summary: This paper illustrates, identifies, and formally defines a memorization problem in meta-learning -- the model can simply memorize meta-training tasks and ignore meta-training train sets. The paper proposes to optimize the mutual information between testing predictions and the training data (given input and meta model), and upper bound it by imposing a information bottleneck between output and input+model. Unlike related work, this paper specifically is able to generalize to meta-test even when the meta-train dataset is not made confusing enough (i.e. even when model can learn well from test data in meta-train alone), making it applicable to use cases where it is hard to make the dataset confusing.