Year: 2021

Abstract:
Meta-learning has emerged as a potent paradigm for quick learning of few-shot tasks, by leveraging the meta-knowledge learned from meta-training tasks. Wellgeneralized meta-knowledge that facilitates fast adaptation in each task is preferred; however, recent evidence suggests the undesirable memorization effect where the meta-knowledge simply memorizing all meta-training tasks discourages task-specific adaptation and poorly generalizes. There have been several solutions to mitigating the effect, including both regularizer-based and augmentation-based methods, while a systematic understanding of these methods in a single framework is still lacking. In this paper, we offer a novel causal perspective of meta-learning. Through the lens of causality, we conclude the universal label space as a confounder to be the causing factor of memorization and frame the two lines of prevailing methods as different deconfounder approaches. Remarkably, derived from the causal inference principle of front-door adjustment, we propose two frustratingly easy but effective deconfounder algorithms, i.e., sampling multiple versions of the meta-knowledge via Dropout and grouping the meta-knowledge into multiple bins. The proposed causal perspective not only brings in the two deconfounder algorithms that surpass previous works in four benchmark datasets towards combating memorization, but also opens a promising direction for meta-learning.

heading: 1 INTRODUCTION
text: Recently, there has been renewed interest in meta-learning which empowers more human-like machines that suffice to learn a wide range of tasks with minimal supervision (Bengio et al., 1991; Thrun & Pratt, 2012; Finn et al., 2017; Raghu et al., 2020). While metric-based meta-learning algorithms (Vinyals et al., 2016; Snell et al., 2017) only solve few-shot classification problems, we focus on gradient-based meta-learning algorithms in this work that are more flexible (Finn et al., 2017; Li et al., 2017). Gradient-based meta-learning algorithms formulate the meta-knowledge as the initialization for a base learner and learn the initialization by a bi-level optimization procedure during the meta-training phase. Concretely, the initialization is adapted to each meta-training task by its support set, while the performance of the adapted model on its query set in turn serves as feedback to update the initialization.
This bi-level optimization scheme, though designed to learn a well-generalized initialization, runs a high risk of inducing a sufficiently expressive initialization that memorizes all meta-training tasks. This kind of overfitting is named memorization overfitting (Yin et al., 2020), where the initialization solves the query set even without relying on the support set for adaptation. As a consequence, such an initialization poorly generalizes to meta-testing tasks. As suggested in Yin et al. (2020), the more non-mutually exclusive meta-training tasks are and the more powerful the model initialization is, the higher risk of memorization arises. To combat the memorization overfitting, Yin et al. (2020) proposed to regularize the capacity of the initialization, and task augmentation strategies have been recently explored in (Rajendran et al., 2020; Yao et al., 2021).
Despite the effectiveness of the three algorithms, understanding their benefits rigorously within a unified analytic tool remains a mystery. To bridge the gap, we develop a causal perspective on metalearning, as illustrated by the causal graph in Figure 1. We argue that the universal label space of the base learner turns to be a confounder causing a spurious correlation between the initializations learned in different steps of meta-training. Such a spurious correlation biases the meta-knowledge
that should be only updated by the performance of task-specific models. Fortunately, the causal graph in Figure 1b offers valuable insights into how to minimize memorization via deconfounder approaches. In particular, we have demonstrated the deconfounding role of both lines of existing works: 1) regularizer-based methods directly weaken the correlation between the initialization metatrained in the last step (i.e.,θ′) and the task-specific model Φ during meta-training, though the limited flexibility of the initialization in this case still promotes spurious relations; 2) augmentation-based methods take different mapping functions from task labels to the universal label space for various tasks, but the performance highly depends on the independence between mapping functions.
Drawing upon the causal perspective, we put forward a new direction of deconfounder approaches by applying the causal inference principle of front-door adjustment. We propose two easy implementations of this principle, which are to sample multiple stratification of the initialization by Dropout and to predict the label as well as the bin that the label belongs to, respectively. Take the backbone of MAML (Finn et al., 2017) as an example. We name the two deconfounder approaches as MAML-Dropout and MAML-Bins, respectively.
The main contributions of our paper are as follows. (1) We, for the first time, develop a causal perspective of meta-learning and shed light on the memorization overfitting with causality in Section 2.2. (2) We place existing methods into the proposed causal framework and adequately demonstrate how they alleviate the memorization overfitting in Section 2.3 and Section 2.4. (3) We propose a new deconfounder approach following the principle of front-door adjustment in Section 2.4 and two methods that implement the approach in Section 3. (4) We showcase that our methods remain compatible with off-the-shelf meta-learning algorithms and consistently improve their performance.

heading: 2 PROBLEM FORMULATION

heading: 2.1 META-LEARNING AND THE OVERFITTING PROBLEM
text: Meta-learning learns the model initialization θ from a series of tasks Ti sampled from a specific task distribution p(T ). All tasks in p(T ) share some common features, so that starting from the initialization θ a new task sampled from the same task distribution can be quickly learned with a resulting task-specific model φ. The tasks used to learn the initialization are considered as metatraining tasks Dtrain, while novel tasks are meta-testing tasks Dtest. Each i-th task Ti consists of a support set si = {(xsi,j , ysi,j)} Ksi j=1 and a query set qi = {(xqi,j , y q i,j)} Kqi j=1
, where (x, y) denote the features and the label of a sample, Ksi and K q i denote the number of support and query samples.
Gradient-based meta-learning formulate learning such a initialization θ as a bi-level optimization problem (See Figure 1a). During inner-loop optimization, the adapted model φi for the i-th task is initialized from θ and updated by its support set si. In outer-loop optimization, the initial-
ization θ is optimized according to performances of adapted models on query sets, i.e.,by losses between label yqi,j and prediction ŷ q i,j . Following (Grant et al., 2018; Gordon et al., 2019; Yin et al., 2020), we formulate the objective of meta-learning as maximizing the conditional likelihood pφ(ŷ
q|xq, θ, s), where the inner-loop optimization learns the conditional distribution of task-specific models p(φ|θ, s) and the outer-loop optimizes the distribution of θ p(θ|Dtrain). Consequently, the objective for inner-loop optimization (i.e., task objective) is
L(φi) = 1
Ks Ks∑ j=1 L(fφi,θ(xsi,j), ysi,j),
and the objective for outer-loop optimization (i.e., meta-objective) is
L(θ) = 1 N N∑ i=1 Ep(φi|θ,si)[ 1 Kqi Kqi∑ j=1 L(fφi,θ(x q i,j), y q i,j)].
Take the algorithm of MAML (Finn et al., 2017) as a concrete example. During the outer-loop update, MAML optimizes the delta function p(θ|Dtrain) on meta-training tasks to learn the initialization θ of a base learner f ; in the inner-loop update, p(φ|θ, s) is also a point estimation by gradient optimization, i.e.,φi = θ − α∇θ 1Ks ∑Ks j=1 L(fθ(xsi,j), ysi,j). Finally, we predict for a query sample by the adapted model, i.e., p(ŷqi,j |x q i,j , φi) = fφi(x q i,j) (Grant et al., 2018; Yin et al., 2020).
In meta-learning, there are two types of overfitting problems: 1) memorization overfitting, which happens when the meta-knowledge memorizes all query sets in meta-training tasks even without adapting on the support sets, and 2) learner overfitting, which happens when meta-knowledge is only effective on meta-training tasks and fails to generalize to meta-testing tasks (Yin et al., 2020; Rajendran et al., 2020). In this paper, we focus on the former. Regularizer-based and augmentation-based methods have been proposed to combat the memorization overfitting, but how to systematically understand the benefits of these methods within a unified analytic tool is still a mystery.

heading: 2.2 A CAUSAL VIEW OF META-LEARNING
text: Firstly, we introduce causality and causal graph, which are main theories supporting our work. Then, we show the causal graph for gradient-based meta-learning and formulate the memorization overfitting (Yin et al., 2020; Rajendran et al., 2020) in a causal view. Besides, we explain the reason why existing methods alleviate the memorization in various degrees via our causal graph. Lastly, we propose a deconfounding principle with frontdoor adjustment.
Causation and Causal Graph. Causation describes causal relationship among variables instead of correlation. Causal graph (Pearl et al., 2016) addresses causality problems with a directed acyclic graph G =< V,E >, where a node Vi ∈ V denotes a variable and a directed edge Vi → Vj ∈ E denotes that variable Vi is a direct cause of Vj . Revisit of meta-learning: a causal view. During the meta-training phase, given N meta-training tasks, we define si and qi to be the support set and the query set of task i, respectively. Then, in meta-training data Dtrain, we have all support sets S as {si}Ni=1 and all query sets Q as {qi}Ni=1. Support sets S and query sets Q consist of randomly drawn examples that are i.i.d. Thus, given meta-training tasks, S andQ are independent. Given query sets of the training set, the input variables XQ = {xqi}Ni=1 is determined, so Q→ XQ. It is obvious that query sets Q has a causal effect on labels of query sets (i.e.,Q→ Y ). According to the workflow of meta-learning shown in Figure 1a, we can easily find the causal links of inner-loop optimization S → Φ and outer-loop optimization Φ→ ŶQ→ θ← Y as shown in Figure 1b. In Figure 1b, let θ′ denote the meta-knowledge learned from last step. It is trained in the same way as θ (the meta-knowledge learned from current step); therefore, there is also a causal link from Y to θ′ (i.e., Y → θ′). We omit the connection from the predictions in the last step since the causal effect can be merged into the effect from Φ→ ŶQ (See Appendix A.1). The connection θ′ → Φ denotes that meta-knowledge θ′ obtained in last update has a causal effect on task-specific models Φ since Φ is always trained by leveraging meta-knowledge θ′ as initialization. Finally, we can obtain the causal graph of meta-learning as shown in Figure 1b.
In Figure 1b, the key idea of meta-learning is that by utilizing the past meta-knowledge θ′ as initialization, one can optimize the task-specific model Φ with new support sets S for a more general meta-knowledge θ. But it ignores the confounder Y (in causal path ŶQ ← Φ ← θ′ ← Y → θ) which influences both the meta-knowledge in the past step and the current step, leading to spurious correlation between θ′ and θ. This spurious correlation biased by the confounder Y makes the taskspecific model especially challenging to be sufficiently adapted by the support set S, thereby putting the initialization at high risk of memorization. We would highlight the difference of labels Y in meta-learning from those in conventional machine learning. In meta-learning, despite the universal label space, the same label varies from task to task in semantic meanings. For example, the label of 0 may indicate “dog” in one task and represent “cat” in another. Thus, Y is not only affected by query sets Q, but also by a hidden variable (i.e., how to map a task label to the universal label space for various tasks). The hidden variable can be denoted as a unobserved exogenous variable and omitted in Figure 1b. Deconfounded meta-learning. In meta-learning, the meta-knowledge θ is learned by p(θ|θ′, S,Q) in each step, although the correlation between θ′ and θ would be spurious since the causal path ŶQ ← Φ← θ′ ← Y → θ shown in Figure 1b demonstrates Y is a confounder of path θ′ → · · · → θ (see proof in A.2). Given Dtrain = (S,Q) and omitting the intermediate nodes, we simplify the causal graph with three nodes {Y, θ′, θ} as shown in Figure 2a. Y opens the backdoor path from θ and θ′. However, the backdoor adjustment is not applicable to to cut the causal relationship between Y and θ′ because the edges Y → θ and Y → θ′ in the causal graph would be changed simultaneously since they perform exactly the same roles in meta-learning. Despite this, we propose two kinds of deconfounded methods applying to MAML (Finn et al., 2017) —one is inspired by some recent works (Rajendran et al., 2020; Yin et al., 2020; Yao et al., 2021; Tseng et al., 2020); the other is based on front-door criterion. These methods are introduced in Section 2.3 and Section 2.4.

heading: 2.3 DECONFOUNDED META-KNOWLEDGE
text: One possible solution to break the connection from Y to θ′ is to use different label mapping functions in different steps of meta-training as shown in Figure 2b. Y ′ ← Q → Y denotes two kinds of meta label representation of query sets’ labels. In this fork structure, Y and Y ′ are independent, conditional on query setsQ. The backdoor path from Y to θ′ is closed. Thus, there is no confounder in the new causal graph and the model can learn p(θ|θ′, S,Q) directly. Under this view, Meta-augmentation (Rajendran et al., 2020) and MetaMix (Yao et al., 2021) can be considered as two deconfounded meta-knowledge models. Both these two methods randomize the labels of query sets to prevent the memorization. As spurious correlations are reduced, these methods achieve better performance than original MAML. Meta-augmentation applies a CEIncreasing augmentation(Rajendran et al., 2020) in each step which changes the labels of the same task. MetaMix generates fake data with manifold mixup (Verma et al., 2019) and channel shuffle. As a result, in different steps, meta-knowledge θ is even optimized on a different label space.
In fact, even these two methods reduce spurious correlations, they have different performances on same tasks. This phenomenon is due to only a partial of the correlation is blocked by conditioning on Q. The augmentation function sampled from a random space still confounds the model. To be specific, Y ′ and Y are not independent.
Another possible way to deconfound meta-knowledge is to constrain the meta-knowledge θ′ to weaken the correlation between Y and θ′. Meta-regularization on weights (Yin et al., 2020) ap-
plies this method. Meta-regularization limits the meta-knowledge by a meta-regularized objective to avoid memorization of meta-training tasks. However, naive regularization weakens the usability of fast adaptation in the inner-loop as Y → θ is limited simultaneously. As a result, regularizerbased method suffers from a trade-off of effectiveness and generalization. Besides, the weakened correlation between θ′ and Φ still confounds the model.

heading: 2.4 DECONFOUNDED META-LEARNING MODEL
text: With considering the mediator Φ in the path from θ′ to θ, we can also simplify the causal path ŶQ ← Φ ← θ′ ← Y → θ in Figure 1b with four nodes {Y, θ′,Φ, θ} as shown in Figure 2c. Then, the other way block the backdoor path from θ′ to θ is to disconnect the path from meta-knowledge θ′ to Φ via frontdoor adjustment. We propose a novel way to deconfound meta-learning model in Figure 2d, and propose to calculate p(θ|do(Φ), Q) instead of p(θ|Φ, Q), which enables the model eliminating the confounder Y , i.e.,p(θ|do(θ′), S,Q). This is so called “frontdoor adjustment”, which is proved in Appendix A.2. The deconfounded meta-learning model is
p(θ|do(θ′), S,Q) = ∑ Φ p(Φ|θ′, S)p(θ|do(Φ), Q)
= ∑ Φ p(Φ|θ′, S) ∑ θ′i p(θ|Φ, θ′i, Q)p(θ′i) = ∑ θ′i p(θ|Φ, θ′i, Q)p(θ′i), (1)
where p(Φ|θ′, S) = 1 in delta function. In Eq.(1), we stratify the confounded past meta-knowledge θ′, i.e.,θ′ = {θ′i}, where θ′i is a stratum of θ′. p(θ|Φ, θ′i, Q) denotes optimizing θ grouped by θ′i. Thus, Φ is grouped in the same way. We propose two implementations of MAML to stratify θ′ in Section 3. After frontdoor adjustment, we break the frontdoor path from θ′ to Φ. Therefore, the model would not memorize the query-set of meta-training tasks.

heading: 3 TWO METHODS TO DECONFOUND MAML

heading: 3.1 MAML-DROPOUT
text: Our first idea is inspired from MC-dropout (Gal & Ghahramani, 2016). We split θ′ into different parts by dropout, i.e.,
p(θ|do(Φ), Q) = ∫ p(θ|Φ, θ′i, Q)p(θ′i)dθ′i ≈ 1
M M∑ i=1 p(θ|Φ, θ′i, Q) = 1 M M∑ i=1 p(θ|Φ, θ′, Q, zi),
(2) whereM is sample times, θ′i indicates a combination of θ
′ and zi, which is a set of dropout variables sampled from Bernoulli distribution. θ′ is independent with zi.
Different from DropGrad (Tseng et al., 2020), we add dropout layers in forward network only on query sets during meta-training. We adopt multi-step optimization in inner-loop (Antoniou et al., 2019) to update almost all meta-knowledge on a training step, which avoids limiting the model’s flexibility. In each training step, a batch of meta-training tasks are used to optimize the model, so empirically, we sample different parts of θ′ through Monte Carlo method for different tasks in a batch to approximate results of Eq.(2). The meta-training objective of MAML-Dropout is
L(θ) = − 1 N N∑ i=1 T∑ t=0 vt Kqi Kqi∑ j=1 L(fφi,t,zi,t(x q j), y q j ),
s.t. φi,t = { θ′, if t = 0 φi,t−1 − α∇φi,t−1 1Ksi ∑Ksi j=1 L(fφi,t−1(xs), ys)), else ,
(3)
where N is the number of meta-training tasks, T is the number of inner-loop steps, vt denotes the importance weight of the target set loss at step t (Antoniou et al., 2019), Kqi denotes the number of query samples in the i-th task, learned weights φi,t and random variable zi,t parameterizes the adaptive model. The gradients of the dropped part in φi,t are set to zero.
In the meta-testing phase, we remove all dropout layers. All meta-knowledge guides the model to learn a new task without regularization and spurious correlation.

heading: 3.2 MAML-BINS
text: As meta-knowledge in MAML extracts powerful features (Raghu et al., 2020), we propose another method to stratify θ′ through the linear combination of features. In this situation, we add an auxiliary task to classify training data to several bins covering all training data points and dividing features into finite groups. So we have Eq.(1) as:
p(θ|do(Φ), Q) = 1 M ∑ M p(θ|Φ, θ′m, Q) = 1 M ∑ M p(θ|Φ, θ′, Q,Wm), (4)
where θ′m = Wm × θ′ is a linear combination of features determined by a bin m. We define an N -way M -bin task as an N -way classification with M bins. We propose an auxiliary task as a M -class classification problem to assign data to different bins. We detail the auxiliary task in Appendix C. We cluster features of all training data to K bins with a pretrained classifier and set the cluster id of the data point as its auxiliary task label, i.e.,b. During the inner-loop step, the model learns the main task and the auxiliary classification, i.e.,the combination of a group of meta-knowledge. The output of model is O = fφ(x), where O ∈ RM×N . The prediction of bins is p(b̂|x, φ) = 1N ∑N j=1O T j and the prediction of classification is p(ŷ|x, φ) = 1M ∑M i=1Oi, where O T j is the j-th column vector ofO andOi is the i-th row vector ofO, which is parameterized byWi×φ. Therefore, we have the outer-loop objective is:
L(θ) = 1 NT NT∑ i=1 Ep(φi|θ,si){ 1 Kqi Kqi∑ j=1 [L(ŷqj , y q j ) + λL(b̂ q j , b q j)]}, (5)
where λ is the weight of bin-classification loss.
As for a regression task, we split the value range of training data into several intervals as bins and set the interval id of the data point as its auxiliary task label. The training process is like a “1-way M -bin classification”, and its objective is same as Eq.(5). A M -bin task reshapes meta-knowledge θ′ to M stratifications, and frontdoor adjustment is implemented in this way.
MAML-Dropout only adds dropout layers and MAML-Bins only adds an additional objective in the outer-loop optimization, which are easy to apply and do not incur additional computation overhead. We combine MAML-Dropout and MAML-Bins in Algorithm 1 and discuss more in Appendix B.

heading: 4 RELATED WORK
text: Gradient-based meta-learning methods (Finn et al., 2017; Raghu et al., 2020; Grant et al., 2018; Li et al., 2017; Lee & Choi, 2018) learn a model initialization as meta-knowledge and fast adapt to new tasks with the initialization. Because these methods are model agnostic, they are wildly implemented in many research areas, e.g.,few-shot learning, reinforcement learning and transferlearning. However, the learned initialization tends to overfit the meta-training tasks, especially the query set of meta-training tasks. Yin et al. (2020) and Rajendran et al. (2020) firstly formulated the meta-overfitting problem. Various methods were proposed to solve the overfitting problem in gradient-based methods. The most common way is using the standard regularization techniques, such as adding adaptive noise (Lee et al., 2019), limiting trainable parameters (Yin et al., 2020; Oh et al., 2021; Zintgraf et al., 2019), and enforcing the similarity of different tasks (Jamal & Qi, 2019).A regularizer-based method, DropGrad (Tseng et al., 2020), applies dropout to support sets but not to the query set. The common positive effect of these methods is weakening the spurious correlation. But, they still use confounded past meta-knowledge which limits the flexibility of meta-knowledge as described in Section 2.3. Recently, Yao et al. (2021) and Rajendran et al. (2020) proposed task-augmentation methods to solve the overfitting problem. Both these methods partially block the backdoor path and achieve outstanding performance than regularization algorithms.
Our work solves the confounder of meta-learning through causal reasoning, especially causal graph and do-calculus (Pearl, 2009; Pearl et al., 2016). Some recent works (de Haan et al., 2019; Zhang et al., 2020; Kocaoglu et al., 2018; Yang et al., 2020; Kurutach et al., 2018; Qi et al., 2020; Nair et al., 2019; Mahajan et al., 2019; Nauta et al., 2019) have shown that causal reasoning helps deep learning models to mine causal relation instead of correlation; meanwhile, the deep model’s powerful representation ability is beneficial to causal models dealing with high-dimensional data. Most relevant
to ours are Bengio et al. (2020) and Yue et al. (2020) which combined meta-learning and causality. The goal of Bengio et al. (2020) is to leverage a meta-learning objective to discover causal structures for fast transfer learning, which solves a completely different research problem from ours. Yue et al. (2020) proposed IFSL to deconfound the pre-trained knowledge during meta-testing, which cannot handle the memorization issue arised during meta-training. Our work mainly focuses on the meta-training process and solves memorization overfitting, which is crucial in meta-learning.

heading: 5 EXPERIMENT
text: We compare our methods with the state-of-the-art solution of memorization overfitting —MetaMix (Yao et al., 2021). We evaluate the performance on different backbones, such as MAML (Finn et al., 2017), ANIL (Raghu et al., 2020), MetaSGD (Li et al., 2017), and T-NET (Lee & Choi, 2018) (together with MetaMix in Appendix E.4), to show the compatibility of our methods. In addition, the ablation study and the analysis of hyperparameters show the robustness of our methods.
5.1 SINUSOID REGRESSION
Table 1: Performance (MSE ± 95% confidence interval) of sinusoid regression problem.
Model 5-shot 10-shot
IFSL 0.59± 0.15 0.15± 0.04 DropGrad 0.57± 0.15 0.14± 0.07 MR-MAML 0.57± 0.11 0.10± 0.02 Meta-Aug 0.53± 0.10 0.10± 0.02 ANIL 0.54± 0.10 0.10± 0.02 ANIL-MetaMix 0.51± 0.10 0.08± 0.02 ANIL-ours 0.49± 0.10 0.08± 0.02 MAML 0.59± 0.12 0.16± 0.06 MAML-MetaMix 0.47± 0.10 0.08± 0.02 MAML-ours 0.45± 0.08 0.06± 0.01 MetaSGD 0.56± 0.11 0.14± 0.04 MetaSGD-MetaMix 0.46± 0.10 0.07± 0.02 MetaSGD-ours 0.43± 0.07 0.04± 0.01 T-Net 0.54± 0.11 0.11± 0.03 T-Net-MetaMix 0.49± 0.10 0.08± 0.02 T-Net-ours 0.47± 0.09 0.07± 0.02
First, we evaluate the performance on a toy sinusoid regression problem. We construct a more challenging problem to further corroborate the superiority of our methods. The data for each task is created in forms of A · sinw · x+ b + , with A ∈ [0.1, 5.0], w ∈ [0.5, 2.0] and b ∈ [0, 2π]. Gaussian observation noise with µ = 0 and = 0.3 is added to each data point sampled from the target task. The regression results are computed by a two-layer Multilayer Perceptron. Implementation of our methods (MAMLDropout+MAML-Bins) in this experiment uses 5 bins and 0.3 dropout rate. Please kindly refer to Appendix D.1 for more details of the experimental setup. We report the mean squared error (MSE) as the evaluation criterion.
According to Table 1, comparing with some basic gradient-based meta-learning algorithms: IFSL (Yue et al., 2020) only focuses on the meta testing phase, so, it cannot improve the performance in a non-mutually-exclusive task; MR-MAML (Yin et al., 2020) achieves a minor improvement, which accords with our analysis in Section 2.3. Meta-Augmentation’s (Rajendran et al., 2020) error is larger than MetaMix (Yao et al., 2021) because mapped labels in different steps generated by MetaMix is more random and independent. Our methods bring a huge improvement applied in different baselines.
We also evaluate our method separately, as shown in Figure 3b. We find that both methods have positive effects and using them together achieves the best performance. Besides, we explore how the number of bins and the dropout rate influence results (see Figure 5b and Figure 5d in Appendix E.1). As a result, the dropout rate should be in [0.1, 0.3] because a low dropout rate stratifies θ′ in similar ways while a high dropout rate limits the generalization of meta-knowledge. For the same reason, the optimal number of bins M is also in a range of [4, 10].

heading: 5.2 DRUG ACTIVITY PREDICTION
text: Following Yao et al. (2021), we apply our methods to the drug activity prediction task (Martin et al., 2019). The task set contains 4276 assays (i.e.,tasks). In each task, we need to predict activities of several compounds on a specific target protein; whereas there are only a few labeled data in the support set. We split tasks into meta-training tasks, meta-validation tasks and meta-testing tasks in the same way as Yao et al. (2021). Other details of datasets and settings are given in Appendix D.3.
We also evaluate the square of Pearson coefficientR2 between the prediction and the ground-truth in each task (Martin et al., 2019; Yao et al., 2021), rather than the mean squared error as an evaluation metric because values of these data are noisy and the coefficient is more meaningful. As the same reason, MAML-Bins brings additional noise, so we only apply MAML-Dropout with 0.1 dropout rate in this experiment. Results of our method are reported in Table 2. In four different groups, our method (MAML-Dropout) is capable of improving the performance on all three backbones.
5.3 POSE PREDICTION
Table 3: Performance (MSE ± 95% confidence interval) of pose prediction.
Model 10-shot 15-shot
Weight Decay 2.772± 0.259 2.307± 0.226 CAVIA 3.021± 0.248 2.397± 0.191 Meta-dropout 3.236± 0.257 2.425± 0.209 Meta-Aug 2.553± 0.265 2.152± 0.227 MR-MAML 2.907± 0.255 2.276± 0.169 IFSL 3.186± 0.256 2.482± 0.231 TAML 2.785± 0.261 2.196± 0.163 ANIL 6.746± 0.416 6.513± 0.384 ANIL-MetaMix 6.354± 0.393 6.112± 0.381 ANIL-ours 6.289± 0.416 6.064± 0.397 MAML 3.098± 0.242 2.413± 0.177 MAML-MetaMix 2.438± 0.196 2.003± 0.147 MAML-ours 2.396± 0.209 1.931± 0.134 MetaSGD 2.803± 0.239 2.331± 0.182 MetaSGD-MetaMix 2.390± 0.191 1.952± 0.154 MetaSGD-ours 2.369± 0.204 1.926± 0.112 T-Net 2.835± 0.189 2.609± 0.213 T-Net-MetaMix 2.563± 0.201 2.418± 0.182 T-Net-ours 2.487± 0.212 2.402± 0.178
We also evaluate another regression task created from Pascal 3D data (Xiang et al., 2014). Following Yin et al. (2020), we randomly select 50 objects for meta-training and the other 15 objects for meta-testing. Same as the past works (Yin et al., 2020), we use a base model with a threeconvolution-block encoder and a fourconvolution-block decoder. Implementation of our methods (MAMLDropout+MAML-Bins) in this experiment uses 5 bins and 0.2 dropout rate. Detailed settings are described in Appendix D.2.
In Table 3, we evaluate more algorithms in this experiment. We find it is difficult for regularizer-based methods to overcome memorization overfitting, especially under 10-shot setting. If there are only few samples in the support set, model is hard to adapt to a specific task and tends to memorize the query set in the meta trainging phase. In this task, our methods’ performance exceeds MetaMix again, which highlights the deconfounding ability of our methods. In Figure 3a, applying our methods separately, the performance of model still achieves a significant advancement.

heading: 5.4 IMAGE CLASSIFICATION
text: We also study the memorization overfitting in a few-shot image classification problem with two benchmarks, Omniglot (Lake et al., 2011) and MiniImagenet (Vinyals et al., 2016). Following Yin et al. (2020); Rajendran et al. (2020), these experiments are under a non-mutually-exclusive setting. “non-mutually-exclusive N -way K-shot classification” means each class is assigned with an unchangeable label from 1 to N in different tasks and training steps. Each task contains N classes labeled from 1 to N . This setting aggravates the memorization overfitting according to the causality described in Section 2.3 and highlights the power of deconfounding. We use a four-block convolutional network, which is the same as the model of Yao et al. (2021) and suffer from less metaoverfitting than the deeper network used in (Yin et al., 2020; Rajendran et al., 2020). We evaluate
different meta-learning backbones and compare them with our methods (MAML-Dropout+MAMLBins) using 5 bins and dropout rate 0.1. Detailed settings are described in Appendix D.4.
We report our results in Table 4. Under a non-mutually-exclusive setting, our method significantly boosts gradient-based methods; even outperforms MetaMix, which proves that our methods have a better deconfounding ability. Besides, under the same setting, we investigate the influence of different hyperparameters, including different numbers of bins and dropout rates, on classification tasks. As shown in Figure 5a and Figure 5c in Appendix E.1, different hyperparameters improve the performance robustly. The ablation study on image classification is reported in Figure 3c and Figure 3d. Combining two implements of frontdoor adjustment has the best performance on these tasks. To show the effectiveness of our proposed methods, we compare pre-inner-update accuracy and meta-testing post-inner-update accuracy during meta-training under the Omniglot 20-way, 1- shot setting as shown in Table 5 in Appendix E.2. Additionally, we conduct the experiments on the mutually-exclusive setting of MiniImageNet in Appendix E.3.

heading: 6 CONCLUSION
text: In this paper, we rethink memorization overfitting from a causal perspective and construct a causal graph for gradient-based meta-learning. Under this causal graph, we identify the root cause of the memorization problem as a spurious correlation in meta-learning. Drawing upon our causal graph, we not only illustrate how existing methods solve the memorization problem but also propose a novel causal intervention principle to debias the spurious correlation. Two implementations of the proposed principle have demonstrated their effectiveness and compatibility in four benchmark datasets. More importantly, we believe that this causal perspective opens a new door to improving meta-learning.

heading: REPRODUCIBILITY STATEMENT
text: We have conducted experiments under the setting in Section 5 and reported the results in Section 5 and Appendix E. We have provided dataset details in Appendix D and implementation details along with hyperparameter settings in Appendix D. We will release the code upon acceptance.

heading: A DETAILED PROOF

heading: A.1 PROOF OF MERGED CAUSAL RELATION
text: We merge the causal relation XQ → Ŷ ′Q → θ′ → Φ→ ŶQ to XQ → Ŷ ′Q. We have:
p(XQ, ŶQ, Ŷ ′ Q,Φ, θ ′) = p(ŶQ|XQ,Φ)p(Φ|θ′)p(θ′|Ŷ ′Q)p(Ŷ ′Q|XQ)p(XQ).
Given θ′, then,
p(XQ, ŶQ,Φ|θ′ = θ∗) = ∑ Ŷ ′Q p(ŶQ|XQ,Φ)p(Φ|θ′ = θ∗)p(Ŷ ′Q|XQ)p(XQ)
= p(ŶQ|XQ,Φ)p(Φ|θ′ = θ∗)p(XQ), (A.1)
whose graph is the same as Figure 1b.

heading: A.2 THE CAUSAL EFFECT
text: Causal effect rule (Pearl et al., 2016) Given a causal graph G in which PA is a set of parent nodes of X , the causal effect of X on Y is given by
p(Y = y|do(X = x)) = ∑ z p(Y = y|X = x, PA = z)p(PA = z), (A.2)
where z ranges over all the combinations of values that the variables in PA can take.
If a variable Z has no effect on Y , then we have p(Y = y|do(X = x)) = ∑ z p(Y = y|X = x, Z = z)p(Z = z)
= ∑ z p(Y = y|X = x)p(Z = z)
= p(Y = y|X = x).
(A.3)
In this case, the correlation between X and Y is the causal effect of X on Y . However, if a variable Z has a effect on Y , then
p(Y = y|X = x) = ∑ z p(Y = y|X = x, Z = z)p(Z = z|X = x)
6= ∑ z p(Y = y|X = x, Z = z)p(Z = z), (A.4)
so the correlation between X and Y is different from the causal effect. In this case, Z open the backdoor path of X and Y , which causes a spurious correlation.
In Figure 1b, there is no backdoor path between {S, θ} and {Q, θ}, but Y open the backdoor path between {θ′, θ}. Therefore, the causal effect of {θ′, S,Q} on θ is
p(θ|do(θ′, S,Q)) = p(θ|do(θ′), S,Q) (A.5)
Frontdoor adjustment We apply frontdoor adjustment (Pearl et al., 2016) to calculate p(θ|do(θ′), S,Q). Firstly, according to the causal graph Figure 1b, we have
p(Φ|θ′, S) = P (Φ|do(θ′), S),
and,
p(θ|do(Φ), Q) = ∑ θ′i p(θ|Φ, θ′i, Q)p(θ′i)
Then, the frontdoor adjustment for meta-learning is p(θ|do(θ′), S,Q) = ∑ Φ p(Φ|do(θ′), S)p(θ|do(Φ), Q)
= ∑ Φ p(Φ|θ′, S)p(θ|do(Φ), Q)
= ∑ Φ p(Φ|θ′, S) ∑ θ′i p(θ|Φ, θ′i, Q)p(θ′i)
= ∑ θ′i p(θ|Φ, θ′i, Q)p(θ′i)
(A.6)
Complete causal graphs Complete causal graphs of two kinds of deconfounding methods mentioned in Section 2.3 and Section 2.4 are shown in Figure 4. Augmentation-based methods randomize the labels of query sets, i.e.,, Y ′ ← Q → Y . The frontdoor adjustment breaks the link θ′ → Φ. According to causal graphs, both these two kinds of methods solve the problem that Y is a confounder in meta-learning.

heading: B DETAILED ALGORITHM

heading: B.1 PSEUDO-CODES
text: We show the pseudo-codes of meta-training for MAML-Bins together with MAML-Dropout in Alg. 1.

heading: B.2 DISCUSSION OF OUR TWO METHODS
text: The two proposed methods sample stratification and deconfound in different manners: different stratums in MAML-Dropout are θ′s dropping different parts of features, while different stratums in MAML-Bins are different combinations of existing features represented by θ′. They are complementary and mutually reinforcing, as evidenced in Figure 3. In general, MAML-Dropout tends to have more stratums than MAML-Bins, accounting for its better performance.

heading: C AUXILIARY CLASSIFICATION TASK
text: To assign the images into different groups, we propose a novel method to train the feature extractor and groups the output of network with a standard clustering algorithm, kmeans. Thus, our method has two procedure: training stage and clustering stage.
Training stage. We train a feature extractor fθ (parametrized by the network parameters θ) and the classifier C(·|W ) (parametrized by the weight matrix W ∈ Rd×c) from scratch by minimizing
Algorithm 1 Meta-training Process of MAML-Dropout and MAML-Bins Require: Task distribution p(T ); Learning rate α, β; A pretrained bin classifier C; Number of
inner-loop steps t; Auxiliary classification loss weight λ; Dropout rate r. Randomly initialize parameter θ0 while not coverage do
Sample a batch of tasks {Ti}ni=1 for all Ti do
Sample support set si = {(xsi,j .ysi,j)} Ksi j=1 and query set qi = {(x q i,j .y q i,j)} Kqi j=1 from Ti Classify the query set by C and match bin labels B to query set qi = {(xqi,j .y q i,j), b q i,j} Kqi j=1 Compute the task-specific parameter φi = φi,t on the support set using Eq.(3) Sample dropout masks zi ∼ Bernoulli(r) for model fφi Compute the output of the model fφi with dropout masks zi, i.e.,Oi = fφi,zi(X q i ) Compute the model prediction Ŷ qi with the mean of Oi’s column vectors and the bin prediction B̂qi with the mean of Oi’s row vectors Compute the the loss as L(Ŷ qi , Y q i ) + λL(B̂ q i , B q i )
end for Update θ0 = θ0 − βn ∑n i=1∇θ0 [L(Ŷ q i , Y q i ) + λL(B̂ q i , B q i )]
end while
a standard cross-entropy classification loss Lpred using the training examples in the base classes xi ∈ X . Here, we denote the dimension of the encoded feature as d and the number of output classes as c. The classifier C(.|W ) consists of a linear layer WTθ (xi) followed by a softmax function σ. Note that the training procedure in this model does not involve sampling mini-batches of classes and data points (episode) as in typical meta-learning algorithms.
Clustering stage. To assign the images into different groups, we fix the pre-trained network parameter θ in our feature extractor fθ, and cluster the output of the network by a standard clustering algorithm, k-means. k-means takes the representation fθ(x) as input, and clusters them into k distinct groups based on a geometric criterion. More precisely, it jointly learns a d× k centroid matrix C and the cluster assignments yn of each image n by solving the following problem:
min C∈Rd×k
1
N N∑ n=1 min yn∈{0,1}k ‖fθ(xn)− Cyn‖22 such that y>n 1k = 1. (A.1)
Solving this problem provides a set of optimal assignments (y∗n)n≤N and a centroid matrix C ∗. These assignments are then used as pseudo-labels of bins; we make no use of the centroid matrix.

heading: D DETAILED EXPERIMENTAL SETUP

heading: D.1 SINUSOID REGRESSION
text: To set up a toy sinusoid regression problem that is non-mutually-exclusive, we create data for each task in the following way: The data for each task is created in forms of A · sinw · x+ b + , with A ∈ [0.1, 5.0], w ∈ [0.5, 2.0] and b ∈ [0, 2π]. At the test time, we expand the range of the tasks by randomly sampling the data-generating A uniformly from [0.1, 5], w from [0.5, 2.0], b from [0, 2π] and use a one-hot vector for each (A, b), w as input to the network. The meta-training tasks are a proper subset of the meta-testing tasks. We set the number of bins to be 5, the dropout rate to be 0.3 and the weight of auxiliary task to be 1 in these tasks.

heading: D.2 POSE PREDICTION
text: To preprocess the pose prediction tasks, we follow (Yin et al., 2020) to preprocess the pose tasks1. There are 50 and 15 categories in the meta-training and meta-testing, respectively, where each category contains 100 gray images in the size of 128× 128.
1code link: https://github.com/google-research/google-research/tree/master/meta learning without memorization/pose data
Following Yin et al. (2020), in pose prediction task, the base model is comprised of a fixed encoder with three convolutional blocks and an adapted decoder with four convolutional blocks. Each convolutional block is composed of a convolutional layer, a batch normalization layer and a ReLU activation layer. We set the number of bins to be 5, the dropout rate to be 0.2 and the weight of auxiliary task to be 0.6 in these tasks.

heading: D.3 DRUG ACTIVATY PREDICTION
text: This task comes from a public dose-response activity assay dataset from ChEMBL 2 and preprocessed by Martin et al. (2019). The training compounds in support sets and the testing compounds in query sets are separated by Martin et al. (2019) and the split of the meta-training, meta-validation and meta-testing tasks are as same as Yao et al. (2021).
The base model of drug activity prediction is a two-layer Multilayer Perceptron(MLP) neural network with 500 neurons in each layer. Each fully connected layer is followed by a batch normalization layer and leaky ReLU activation. In either meta-training or meta-testing, the number of inner-loop adaptation steps equals to 10. During meta-training, the task batch size, the outer-loop learning rate, the inner-loop learning rate are set to 8, 0.001 and 0.01. The meta-training process altogether runs for 50 epochs while 60 epochs using Dropout, each of which includes 500 iterations. Dropout rate is set to be 0.1. In order to prevent the influence of noise data, we use a query-set-mixup strategy as Yao et al. (2021), i.e.,we apply manifold mixup on query set for all experiments in this task.
D.4 IMAGE CLASSIFICATION
In image classification, for non-mutually exclusive setting in 5-way miniImagenet, 64 meta-training classes are split to 5 sets, where 4 sets have 13 classes and the rest one has 12 classes. For each set, a fixed class label is assigned to each class within this set, which remains unchanged across different tasks. During meta-training, we randomly select one class from each set and take all the five selected classes to construct a task, which ensures that each class consistently has one label across tasks. In our experiments, we list the classes within each set as follows.
• Set 1: n07584110, n04243546, n03888605, n03017168, n04251144, n02108551, n02795169, n03400231, n03476684, n04435653, n02120079, n01910747, n03062245
• Set 2: n03347037, n04509417, n03854065, n02108089, n04067472, n04596742, n01558993, n04612504, n02966193, n07697537, n01843383, n03838899, n02113712
• Set 3: n04604644, n02105505, n02108915, n03924679, n01704323, n09246464, n04389033, n03337140, n06794110, n04258138, n02747177, n13054560, n04443257
• Set 4: n13133613, n01770081, n02606052, n02687172, n02101006, n03676483, n04296562, n02165456, n04515003, n01749939, n02111277, n02823428, n01532829
• Set 5: n02091831, n07747607, n03998194, n02089867, n02074367, n02457408, n04275548, n03220513, n03527444, n03908618, n03207743, n03047690
A similar process is applied to Omniglot, where 1200 meta-training classes are randomly split into 20 sets with 60 classes in each set. For all datasets, we utilize the classical convolutional neural network with 4 convolutional blocks as the base model (Finn et al., 2017; Snell et al., 2017). We set the number of bins to be 5, the dropout rate to be 0.1 and the weight of auxiliary task to be 0.2 in these tasks.
The image sizes of Omniglot and MiniImagenet are set to be 28×28×1 and 84×84×3, respectively.

heading: E ADDITIONAL EXPERIMENT RESULTS

heading: E.1 HYPERPARAMETER SENSITIVITY
text: The hyperparameters in our experiments are determined according to the performance on a holdout set of meta-validation tasks. Besides, we analyze the influence of different numbers of bins for
2https://www.ebi.ac.uk/chembl
MAML-Bins and different dropout rate for MAML-Dropout. The results show the robustness of our methods against different hyperparameters.

heading: E.2 OVERFITTING ANALYSIS
text: We compare the shallow and deeper base model under the Omniglot 20-way 1-shot setting in Table 5. As for MAML, the memorization overfitting on the deep model is more serious, which really hurts the testing performance. Our methods solves the memorization problem in meta-knowledge achieves a better performance.

heading: E.3 RESULTS UNDER MUTUALLY-EXCLUSIVE SETTING
text: In Table 6, we report the results under the standard mutually-exclusive setting on MiniImagenet. Label shuffling is introduced to construct meta-training tasks under the mutually-exclusive setting, which significantly reduces the memorization overfitting. However, applying the proposed methods on this setting still achieves comparable and even better performance than original MAML, which further demonstrates the effectiveness of our proposed methods.

heading: E.4 RESULTS TOGETHER WITH METAMIX
text: We apply our methods together with MetaMix to Omniglot, MiniImagenet and sinusoid regression. The results in the Table 7 and Table 8 show further and big improvement of the combination compared to using MetaMix only.

Review summaries:
This paper aims to battle memorization overfitting problem in meta learning using causality. The authors view the process of meta-learning through the lens of causality which produces several causal graphs i.e. Fig. 1(b) for the general one and Fig. 2 (a-d) for special cases. Based on the causal graphs, the authors use 'front-door' adjustment and propose two algorithms MAML Dropout and MAML Bins. 

This paper studies the memorization overfitting problem in meta-learning from a causal perspective. It provides explanations to current solutions and proposes to update the meta parameters by maximizing the interventional distribution $p(\theta | do(\theta'), S, Q)$. It provides an identification strategy for the interventional distribution by the front door criterion.

The paper addresses the undesirable memorization problem in the context of gradient-based meta-learning, where the meta-learning knowledge obtained as a result of memorizing all meta-training tasks yields poor generalization and is detrimental to task specific adaptation. While previous work has focused on regularization-based and augmentation-based solutions to prevent the memorization of all meta-training tasks, the authors argue these methods and their benefits are still not well understood and propose a causal perspective of meta-learning in a unified causal framework. In this context the authors identify the universal label space of the base model as the confounding factor of memorization causing spurious correlations between initializations learnt in different meta-training steps. Informed by this analysis, the authors demonstrate why existing meta-learning overfitting solutions work, and propose two deconfounder approaches to address the issue of memorization, namely i) sampling multiple versions of the meta-knowledge via dropout (MAML-Dropout), and ii) grouping the meta-knowledge into bins (MAML-Bins).   

This paper studies the problem of memorization overfitting in meta-learning and proposes to construct a causal graph for gradient-based meta-learning. From a perspective of causal graph, this paper demonstrates how existing methods solve the memorization problem. Further, it proposes a novel causal intervention principle to debias the spurious correlation. Specifically, two implementations of the proposed principle are conducted, i.e., by sampling multiple versions of the meta-knowledge via Dropout and grouping the meta-knowledge into multiple bins. Experiments on four benchmark datasets demonstrates the effectiveness and compatibility of proposed algorithm.