Minimizing Memorization in Meta-learning: A Causal Perspective

1 INTRODUCTION . Recently , there has been renewed interest in meta-learning which empowers more human-like machines that suffice to learn a wide range of tasks with minimal supervision ( Bengio et al. , 1991 ; Thrun & Pratt , 2012 ; Finn et al. , 2017 ; Raghu et al. , 2020 ) . While metric-based meta-learning algorithms ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) only solve few-shot classification problems , we focus on gradient-based meta-learning algorithms in this work that are more flexible ( Finn et al. , 2017 ; Li et al. , 2017 ) . Gradient-based meta-learning algorithms formulate the meta-knowledge as the initialization for a base learner and learn the initialization by a bi-level optimization procedure during the meta-training phase . Concretely , the initialization is adapted to each meta-training task by its support set , while the performance of the adapted model on its query set in turn serves as feedback to update the initialization . This bi-level optimization scheme , though designed to learn a well-generalized initialization , runs a high risk of inducing a sufficiently expressive initialization that memorizes all meta-training tasks . This kind of overfitting is named memorization overfitting ( Yin et al. , 2020 ) , where the initialization solves the query set even without relying on the support set for adaptation . As a consequence , such an initialization poorly generalizes to meta-testing tasks . As suggested in Yin et al . ( 2020 ) , the more non-mutually exclusive meta-training tasks are and the more powerful the model initialization is , the higher risk of memorization arises . To combat the memorization overfitting , Yin et al . ( 2020 ) proposed to regularize the capacity of the initialization , and task augmentation strategies have been recently explored in ( Rajendran et al. , 2020 ; Yao et al. , 2021 ) . Despite the effectiveness of the three algorithms , understanding their benefits rigorously within a unified analytic tool remains a mystery . To bridge the gap , we develop a causal perspective on metalearning , as illustrated by the causal graph in Figure 1 . We argue that the universal label space of the base learner turns to be a confounder causing a spurious correlation between the initializations learned in different steps of meta-training . Such a spurious correlation biases the meta-knowledge that should be only updated by the performance of task-specific models . Fortunately , the causal graph in Figure 1b offers valuable insights into how to minimize memorization via deconfounder approaches . In particular , we have demonstrated the deconfounding role of both lines of existing works : 1 ) regularizer-based methods directly weaken the correlation between the initialization metatrained in the last step ( i.e. , θ′ ) and the task-specific model Φ during meta-training , though the limited flexibility of the initialization in this case still promotes spurious relations ; 2 ) augmentation-based methods take different mapping functions from task labels to the universal label space for various tasks , but the performance highly depends on the independence between mapping functions . Drawing upon the causal perspective , we put forward a new direction of deconfounder approaches by applying the causal inference principle of front-door adjustment . We propose two easy implementations of this principle , which are to sample multiple stratification of the initialization by Dropout and to predict the label as well as the bin that the label belongs to , respectively . Take the backbone of MAML ( Finn et al. , 2017 ) as an example . We name the two deconfounder approaches as MAML-Dropout and MAML-Bins , respectively . The main contributions of our paper are as follows . ( 1 ) We , for the first time , develop a causal perspective of meta-learning and shed light on the memorization overfitting with causality in Section 2.2 . ( 2 ) We place existing methods into the proposed causal framework and adequately demonstrate how they alleviate the memorization overfitting in Section 2.3 and Section 2.4 . ( 3 ) We propose a new deconfounder approach following the principle of front-door adjustment in Section 2.4 and two methods that implement the approach in Section 3 . ( 4 ) We showcase that our methods remain compatible with off-the-shelf meta-learning algorithms and consistently improve their performance . 2 PROBLEM FORMULATION . 2.1 META-LEARNING AND THE OVERFITTING PROBLEM . Meta-learning learns the model initialization θ from a series of tasks Ti sampled from a specific task distribution p ( T ) . All tasks in p ( T ) share some common features , so that starting from the initialization θ a new task sampled from the same task distribution can be quickly learned with a resulting task-specific model φ . The tasks used to learn the initialization are considered as metatraining tasks Dtrain , while novel tasks are meta-testing tasks Dtest . Each i-th task Ti consists of a support set si = { ( xsi , j , ysi , j ) } Ksi j=1 and a query set qi = { ( xqi , j , y q i , j ) } Kqi j=1 , where ( x , y ) denote the features and the label of a sample , Ksi and K q i denote the number of support and query samples . Gradient-based meta-learning formulate learning such a initialization θ as a bi-level optimization problem ( See Figure 1a ) . During inner-loop optimization , the adapted model φi for the i-th task is initialized from θ and updated by its support set si . In outer-loop optimization , the initial- ization θ is optimized according to performances of adapted models on query sets , i.e. , by losses between label yqi , j and prediction ŷ q i , j . Following ( Grant et al. , 2018 ; Gordon et al. , 2019 ; Yin et al. , 2020 ) , we formulate the objective of meta-learning as maximizing the conditional likelihood pφ ( ŷ q|xq , θ , s ) , where the inner-loop optimization learns the conditional distribution of task-specific models p ( φ|θ , s ) and the outer-loop optimizes the distribution of θ p ( θ|Dtrain ) . Consequently , the objective for inner-loop optimization ( i.e. , task objective ) is L ( φi ) = 1 Ks Ks∑ j=1 L ( fφi , θ ( xsi , j ) , ysi , j ) , and the objective for outer-loop optimization ( i.e. , meta-objective ) is L ( θ ) = 1 N N∑ i=1 Ep ( φi|θ , si ) [ 1 Kqi Kqi∑ j=1 L ( fφi , θ ( x q i , j ) , y q i , j ) ] . Take the algorithm of MAML ( Finn et al. , 2017 ) as a concrete example . During the outer-loop update , MAML optimizes the delta function p ( θ|Dtrain ) on meta-training tasks to learn the initialization θ of a base learner f ; in the inner-loop update , p ( φ|θ , s ) is also a point estimation by gradient optimization , i.e. , φi = θ − α∇θ 1Ks ∑Ks j=1 L ( fθ ( xsi , j ) , ysi , j ) . Finally , we predict for a query sample by the adapted model , i.e. , p ( ŷqi , j |x q i , j , φi ) = fφi ( x q i , j ) ( Grant et al. , 2018 ; Yin et al. , 2020 ) . In meta-learning , there are two types of overfitting problems : 1 ) memorization overfitting , which happens when the meta-knowledge memorizes all query sets in meta-training tasks even without adapting on the support sets , and 2 ) learner overfitting , which happens when meta-knowledge is only effective on meta-training tasks and fails to generalize to meta-testing tasks ( Yin et al. , 2020 ; Rajendran et al. , 2020 ) . In this paper , we focus on the former . Regularizer-based and augmentation-based methods have been proposed to combat the memorization overfitting , but how to systematically understand the benefits of these methods within a unified analytic tool is still a mystery . 2.2 A CAUSAL VIEW OF META-LEARNING . Firstly , we introduce causality and causal graph , which are main theories supporting our work . Then , we show the causal graph for gradient-based meta-learning and formulate the memorization overfitting ( Yin et al. , 2020 ; Rajendran et al. , 2020 ) in a causal view . Besides , we explain the reason why existing methods alleviate the memorization in various degrees via our causal graph . Lastly , we propose a deconfounding principle with frontdoor adjustment . Causation and Causal Graph . Causation describes causal relationship among variables instead of correlation . Causal graph ( Pearl et al. , 2016 ) addresses causality problems with a directed acyclic graph G = < V , E > , where a node Vi ∈ V denotes a variable and a directed edge Vi → Vj ∈ E denotes that variable Vi is a direct cause of Vj . Revisit of meta-learning : a causal view . During the meta-training phase , given N meta-training tasks , we define si and qi to be the support set and the query set of task i , respectively . Then , in meta-training data Dtrain , we have all support sets S as { si } Ni=1 and all query sets Q as { qi } Ni=1 . Support sets S and query sets Q consist of randomly drawn examples that are i.i.d . Thus , given meta-training tasks , S andQ are independent . Given query sets of the training set , the input variables XQ = { xqi } Ni=1 is determined , so Q→ XQ . It is obvious that query sets Q has a causal effect on labels of query sets ( i.e. , Q→ Y ) . According to the workflow of meta-learning shown in Figure 1a , we can easily find the causal links of inner-loop optimization S → Φ and outer-loop optimization Φ→ ŶQ→ θ← Y as shown in Figure 1b . In Figure 1b , let θ′ denote the meta-knowledge learned from last step . It is trained in the same way as θ ( the meta-knowledge learned from current step ) ; therefore , there is also a causal link from Y to θ′ ( i.e. , Y → θ′ ) . We omit the connection from the predictions in the last step since the causal effect can be merged into the effect from Φ→ ŶQ ( See Appendix A.1 ) . The connection θ′ → Φ denotes that meta-knowledge θ′ obtained in last update has a causal effect on task-specific models Φ since Φ is always trained by leveraging meta-knowledge θ′ as initialization . Finally , we can obtain the causal graph of meta-learning as shown in Figure 1b . In Figure 1b , the key idea of meta-learning is that by utilizing the past meta-knowledge θ′ as initialization , one can optimize the task-specific model Φ with new support sets S for a more general meta-knowledge θ . But it ignores the confounder Y ( in causal path ŶQ ← Φ ← θ′ ← Y → θ ) which influences both the meta-knowledge in the past step and the current step , leading to spurious correlation between θ′ and θ . This spurious correlation biased by the confounder Y makes the taskspecific model especially challenging to be sufficiently adapted by the support set S , thereby putting the initialization at high risk of memorization . We would highlight the difference of labels Y in meta-learning from those in conventional machine learning . In meta-learning , despite the universal label space , the same label varies from task to task in semantic meanings . For example , the label of 0 may indicate “ dog ” in one task and represent “ cat ” in another . Thus , Y is not only affected by query sets Q , but also by a hidden variable ( i.e. , how to map a task label to the universal label space for various tasks ) . The hidden variable can be denoted as a unobserved exogenous variable and omitted in Figure 1b . Deconfounded meta-learning . In meta-learning , the meta-knowledge θ is learned by p ( θ|θ′ , S , Q ) in each step , although the correlation between θ′ and θ would be spurious since the causal path ŶQ ← Φ← θ′ ← Y → θ shown in Figure 1b demonstrates Y is a confounder of path θ′ → · · · → θ ( see proof in A.2 ) . Given Dtrain = ( S , Q ) and omitting the intermediate nodes , we simplify the causal graph with three nodes { Y , θ′ , θ } as shown in Figure 2a . Y opens the backdoor path from θ and θ′ . However , the backdoor adjustment is not applicable to to cut the causal relationship between Y and θ′ because the edges Y → θ and Y → θ′ in the causal graph would be changed simultaneously since they perform exactly the same roles in meta-learning . Despite this , we propose two kinds of deconfounded methods applying to MAML ( Finn et al. , 2017 ) —one is inspired by some recent works ( Rajendran et al. , 2020 ; Yin et al. , 2020 ; Yao et al. , 2021 ; Tseng et al. , 2020 ) ; the other is based on front-door criterion . These methods are introduced in Section 2.3 and Section 2.4 .

Summary: This paper studies the memorization overfitting problem in meta-learning from a causal perspective. It provides explanations to current solutions and proposes to update the meta parameters by maximizing the interventional distribution $p(\theta | do(\theta'), S, Q)$. It provides an identification strategy for the interventional distribution by the front door criterion.