Title: LEARNING TO BALANCE: BAYESIAN META-LEARNING FOR IMBALANCED AND OUT-OF-DISTRIBUTION TASKS

Abstract: While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the meta-knowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the meta-knowledge or task-specific learning. We formulate this objective into a Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on multiple realistic task-and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework.
or we may naively approximate by taking the expectation inside for computational efficiency:
In the experimental section, we show that Eq. ( 8) largely outperforms Eq. ( 9).

1. INTRODUCTION:

Despite the success of deep learning in many real-world tasks such as visual recognition and machine translation, such good performances are achievable at the availability of large training data, and many fail to generalize well in small data regimes. To overcome this limitation of conventional deep learning, recently, researchers have explored meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) approaches, whose goal is to learn a model that generalizes well over distribution of tasks, rather than instances from a single task, in order to utilize the obtained meta-knowledge across tasks to compensate for the lack of training data for each task.

However, so far, most existing meta-learning approaches (Santoro et al., 2016; Vinyals et al., 2016; Snell et al., 2017; Ravi & Larochelle, 2017; Finn et al., 2017; Li et al., 2017) have only targeted an artificial scenario where all tasks participating in the multi-class classification problem have equal number of training instances per class. Yet, this is a highly restrictive setting, as in real-world scenarios, tasks that arrive at the model may have different training instances (task imbalance), and within each task, the number of training instances per class may largely vary (class imbalance). Moreover, the new task may come from a distribution that is different from the task distribution the model has been trained on (out-of-distribution task) (See (a) of Figure 1 ).

Under such a realistic setting, the meta-knowledge may have a varying degree of utility to each task. Tasks with small number of training data, or close to the tasks trained in meta-training step may want to rely mostly on meta-knowledge obtained over other tasks, whereas tasks that are out-of-distribution or come with more number of training data may obtain better solutions when trained in a task-specific manner. Furthermore, for multi-class classification, we may want to treat the learning for each class differently to handle class imbalance. Thus, to optimally leverage meta-learning under various imbalances, it would be beneficial for the model to task-and class-adaptively decide how much to use from the meta-learner, and how much to learn specifically for each task and class.

To this end, we propose a novel Bayesian meta-learning framework, which we refer to as Bayesian Task-Adaptive Meta-Learning (Bayesian TAML), that learns variables to adaptively balance the effect of meta-and task-specific learning. Specifically, we first obtain set-representations for each task, which are learned to convey useful statistics about the task or class distribution, such as mean, variance, and cardinality (the number of elements in the set), and then learn the distribution of three balancing variables a function of the set: 1) task-dependent learning rate multiplier, which decides how far away to deviate from the meta-knowledge, when performing task-specific learning. Tasks with higher shots could benefit from taking gradient steps afar, while tasks with few shots may need to stay close to the initial parameter. 2) class-dependent learning rate, which decides how much information to use from each class, to automatically handle class imbalance where the number of instances per class can largely vary. 3) task-dependent modulator for initial model parameter, which modifies the shared initialization for each task, such that each task can decide how much and what to use from the shared initial model parameter and what to ignore based on its set representation. This is especially useful when handling out-of-distribution task, which may need to ignore some of the meta-knowledge.

We validate our model on CIFAR-FS and miniImageNet dataset, as well as a new dataset that consists of heterogeneous datasets, under a scenario where every class in each episode can have any number of shots, that leads to task and class imbalance, and where the dataset at meta-test time is different from that of meta-training time. The experimental results show that our Bayesian TAML significantly improves the performance over the existing approaches under these realistic scenarios. Further analysis of each component reveals that the improvement is due to the effectiveness of the balancing terms for handling task and class imbalance, and out-of-distribution tasks.

To summarize, our contribution in this work is threefold:

• We consider a novel problem of meta-learning under a realistic task distribution, where the number of instances across classes and tasks could largely vary, or the unseen task at the meta-test time is largely different from the seen tasks.

• For effective meta-learning with such imbalances, we propose a Bayesian task-adaptive meta-learning (Bayesian TAML) framework that can adaptively adjust the effect of the meta-learner and the task-specific learner, differently for each task and class.

• We validate our model on realistic imbalanced few-shot classification tasks with a varying number of shots per task and class and show that it significantly outperforms existing meta-learning models.

2. RELATED WORK:

Meta-learning Meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998 ) is an approach to learn a model to generalize over a distribution of task. The approaches in general can be categorized into either memory-based, metric-based, and optimization-based methods. A memory-based approach (Santoro et al., 2016) learns to store correct instance and label into the same memory slot and retrieve it later, in a task-generic manner. Metric-based approaches learn a shared metric space (Vinyals et al., 2016; Snell et al., 2017) . Snell et al. (2017) defines the distance between the instance and the class prototype, such that the instances are closer to their correct prototypes than to others. As for optimization-based meta-learning, MAML (Finn et al., 2017) learns a shared initialization parameter that is optimal for any tasks within few gradient steps from the initial parameter. Meta-SGD (Li et al., 2017) improves upon MAML by learning the learning rate differently for each parameter. For effective learning of a meta-learner, meta-learning approaches adopt the episodic training strategy (Vinyals et al., 2016) which trains and evaluates a model over a large number of tasks, which are called meta-training and meta-test phase, respectively. However, existing approaches only consider an artificial scenario which samples the classification of classes with exactly the same number of training instances, both within each episode and across episodes. On the other hand, we consider a more challenging scenario where the number of shots per class and task could vary at each episode, and that the task given at meta-test time could be an out-of-distribution task.

Task-adaptive meta-learning The goal of learning a single meta-learner that works well for all tasks may be overly ambitious and leads to suboptimal performances for each task. Thus recent approaches adopt task-adaptively modified meta-learning models. Oreshkin et al. (2018) proposed to learn the temperature scaling parameter to work with the optimal similarity metric. Qiao et al. (2018) also suggested a model that generates task-specific parameters for the network layers, but it only trains with many-shot classes, and implicitly expects generalization to few-shot cases. Rusu et al. (2019) proposed a network type task-specific parameter producer, and Lee & Choi (2018) proposed to differentiate the network weights into task-shared and task-specific weights. Our model also aims to obtain task-specific parameter for each task, but is rather focused on learning how to balance between the meta-learning and task-/class-specific learning.

Probabilistic meta-learning Recently, a probabilistic version of MAML has been proposed (Finn et al., 2018) , where they interpret a task-specific gradient update as a posterior inference process under variational inference framework. Kim et al. (2018) proposed Bayesian MAML with a similar motivation but with a stein variational inference framework and chaser loss. Gordon et al. (2019) proposed a probabilistic meta-learning framework where the parameter for a novel task is rapidly estimated under decision theoretic framework, given a set representation of a task. The motivation behind these works is to represent the inherent uncertainty in few-shot classification tasks. Our model also uses Bayesian modeling, but it focuses on leveraging the uncertainties of the meta-learner and the gradient-direction in order to balance between meta-and task-or class-specific learning.

3. LEARNING TO BALANCE:

We first introduce notations and briefly recap the model-agnostic meta-learning (MAML) by Finn et al. (2017) . Suppose a task distribution p(τ ) that randomly generates task τ consisting of a training set D τ = {X τ , Y τ } and a test set Dτ = { Xτ , Ỹτ }. Then, the goal of MAML is to meta-learn the initial model parameter θ as a meta-knowledge to generalize over the task distribution p(τ ), such that we can easily obtain the task-specific predictor θ τ in a single (or a few) gradient step from the initial θ. Toward this goal, MAML optimizes the following gradient-based meta-learning objective:

EQUATION

where α denotes stepsize and L denotes empirical loss such as negative log-likelihood of observations. Note that by meta-learning the initial point θ, the task-specific predictor θ τ = θ -α∇ θ L(θ; D τ ) can minimize the test loss L(•; Dτ ) even with D τ consisting of only a few samples. We can easily extend the Eq. ( 1), such that we obtain θ τ with more than one inner-gradient steps from the initial θ.

However, the existing MAML framework has the following limitations that prevent the model from efficiently solving real-world problems involving class/task imbalance and out-of-distribution tasks.

1. Class imbalance. MAML does not provide any framework to handle class imbalance within each task. Therefore, classes with large number of training instances (head classes) may dominate the task-specific learning during the inner-gradient steps, yielding low performance on classes with fewer shots (tail classes). 2. Task imbalance. The model has a fixed number of inner-gradient steps and stepsize α across all tasks, which prevents the model from adaptively deciding how much to resort to the meta-knowledge or how much to learn from the given dataset, depending on the number of the training examples per task.

3. Out-of-distribution tasks. The model assumes that the initial model parameter θ will be equally useful for the unseen tasks, but for unseen tasks that are significantly different from the previously seen tasks, the initial parameter may be less useful.

3.1. TASK-ADAPTIVE META-LEARNING (TAML):

As shown in Figure 1 for the concepts, we introduce three balancing variables ω τ , γ τ , z τ to tackle each problem mentioned above. How to generate these variables will be described in Section 4. Also, see the experimental section for how to generate the realistic tasks with class and task imbalance.

Tackling class imbalance. To handle class imbalance, we vary the learning rate of class-specific gradient for each inner-optimization step. Specifically, for class c = 1, . . . , C, we introduce a set of class-specific scalars

ω τ = (ω τ 1 , . . . , ω τ C ) ∈ [0, 1] C , which are multiplied to each of the class-specific gradients ∇ θ L(θ; D τ 1 ), . . . , ∇ θ L(θ; D τ C )

, where D τ c is the set of instances and labels for class c. We expect ω τ c to be large for tail-classes with small number of training instances, such that they could be considered more in the inner-optimization steps. We generate ω τ with Softmax function, and denote the input to the function as ωτ .

Tackling task imbalance. To control whether the model parameter for the current task stays close to the initial parameter or deviate far from it, we introduce task-dependent learning-rate multipliers

γ τ = (γ τ 1 , . . . , γ τ L ) ∈ [0, ∞) L

, such that for each layer l = 1, . . . , L, the learning rate becomes γ τ 1 α, γ τ 2 α, . . . , γ τ L α1 . We expect γ τ to be large for large tasks, such that they rely more on taskspecific updates, while small tasks use small γ τ to benefit more from the meta-knowledge. To amplify the step-size difference between large and small tasks, we generate γ τ with an exponential function, and denote the input to the function as γτ .

Tackling out-of-distribution tasks. Finally, we introduce z τ which modulates the initial parameter θ for each task. We expect z τ to learn to relocate the initial θ to a new starting point, such that out-of-distribution (OOD) tasks can deviate much from the shared initialization θ if the current initialization is suboptimal for the given task. Specifically, we use z τ = 1 + zτ for the channel of the convolutional network weights and z τ = zτ for the biases, which modify the initial parameter as follows: θ 0 ← θ • z τ for the weights and θ 0 ← θ + z τ for the biases, where θ 0 denotes the new initialization modulated by z τ . We denote this operation as θ * z τ , which is similar to task-dependent modulation of batch normalization parameters (Oreshkin et al., 2018; Requeima et al., 2019) .

A unified framework. Finally, we assemble all these components together into a single unified framework. With a slight abuse of notations, the update rule can be written as follows:

EQUATION

EQUATION

where α is a multi-dimensional global learning rate vector that is learned (Li et al., 2017) , and the multiplication operator • is appropriately defined. The last step θ K corresponds to the task-specific predictor θ τ .

3.2. BAYESIAN TASK-ADAPTIVE META-LEARNING:

We now introduce the variational inference framework for the input of the three balancing variables, that we previously denote as ωτ , γτ , zτ . Bayesian modeling allows to incorporate randomness in the posterior of those variables. In MAML framework, such randomness generates the ensemble of diverse task-specific predictors, which allows to effectively exploit the information in the given dataset D τ . Bayesian modeling also allows to find more robust latent structures of those balancing variables (See Figure 7 ). The goal of learning for each task τ is to maximize the conditional loglikelihood of the joint dataset Dτ and D τ : log p( Ỹτ , Y τ | Xτ , X τ ; θ). However, solving it involves the true posterior p(φ τ |D τ , Dτ ), which is intractable. Thus, we resort to amortized variational inference with a tractable form of approximate posterior q(φ τ |D τ , Dτ ; ψ) parameterized by ψ. We let the three variables share the same inference network pipeline to minimize the computational cost. Further, similarly to Ravi & Beatson (2019) , we drop the dependency on the test dataset Dτ for the approximate posterior, in order to make the two different pipelines consistent; one for meta-training where we observe the whole test dataset, and the other for meta-testing where the test labels are unknown. The form of our approximate posterior is now q(φ τ |D τ ; ψ).

It greatly simplifies the inference framework, while ensuring to have a valid lower bound of the log evidence. Also, considering that performing the inner-gradient steps with the training dataset D τ automatically maximizes the training log-likelihood in MAML framework, we slightly modify the objective so that the expected log-likelihood term only involves the test examples with the appropriate scaling factor. The resultant form of the approximated lower bound that suits for our meta-learning purpose is as follows:

L τ θ,ψ = N τ + M τ M τ Mτ m=1 E q(φ τ |D τ ;ψ) log p(ỹ τ m |x τ m , φ τ ; θ) -KL[q(φ τ |D τ ; ψ) p(φ τ )]. (5)

We assume q(φ τ |D τ ; ψ) fully factorizes for each variable and also for each dimension as well:

EQUATION

where we assume that each single dimension of q(φ τ |D τ ; ψ) follows univariate Gaussian having trainable mean and variance. We also let each dimension of prior p(φ τ ) factorize into N (0, 1), such that the KL-divergence can have the especially simple closed form (Kingma & Welling, 2014).

The final form of the meta-training minimization objective with Monte-Carlo (MC) approximation for the expectation in Eq. ( 5) is as follows:

EQUATION

where φ τ s ∼ q(φ τ |D τ ; ψ). We implicitly assume the reparameterization trick for φ τ to obtain stable and unbiased gradient estimate w.r.t. ψ (Kingma & Welling, 2014). We set the MC sample size to S = 1 for meta-training for computational efficiency. When meta-testing, we perform MC approximation with sufficiently large sample size (e.g. S = 10): 

p(ỹ τ

4.1. DATASET ENCODING:

The main challenge in modeling our variational distribution q(φ τ |D τ ; ψ) is to decide how to refine the training dataset D τ into informative representation, which is not trivial. This inference network should capture all the necessary statistical information to recognize any imbalances in the dataset D τ . Mean-pooling (Edwards & Storkey, 2017) or sum-pooling (Zaheer et al., 2017) is frequently used as a practical set-encoder, where each instance in the set is transformed by the shared nonlinearity, and then averaged or summed together to generate a single vector summarizing the set, followed by an additional nonlinearity. However, for the classification dataset D τ which is the set of (class) sets, those non-hierarchical pooling methods will perform poorly as they ignore the label information. Therefore, we use a two-layer hierarchical set encoder which first encodes each class as a set of samples and then encodes the set as the set of classes (see the Appendix B for the justification).

However, there is an additional limitation of mean pooling when using it to describe a task: it does not recognize the number of elements in the set2 . This could be a critical limitation in recognizing the imbalances in the given task. Therefore we explicitly input the number of elements into the set encoder. Yet, the set cardinality alone is insufficient in capturing the distribution of the dataset. Suppose that we have a set containing replications of a single instance. Then, although the set has only limited information, the set encoding cannot recognize it and will overestimate the information.

To prevent this, we encode the variance of the set as well.

Based on this intuition, we define the set encoder StatisticsPooling(•) that generates the concatenation of the set statistics such as mean, variance, and cardinality. We use this encoder to first encode each class, and then the whole set of classes as follows:

v τ = StatisticsPooling {NN 2 (s c )} C c=1 , s c = StatisticsPooling {NN 1 (x)} x∈X τ c

for classes c = 1, . . . , C. X τ c is the collection of class c examples in task τ . NN 1 and NN 2 are some neural networks parameterized by ψ. The vector v τ finally summarizes the entire dataset D τ for few-shot classification. Note that the class-specific balancing variables ωτ 1 , . . . , ωτ C are generated from s 1 , . . . , s C , and other task-specific balancing variables γτ and zτ are generated from v τ with a few additional layers (See Figure 3 ).

5. EXPERIMENTS:

We next validate our method on multiple benchmark datasets with more realistic task distribution.

Datasets We validate our method on the following benchmark datasets. CIFAR-FS: This dataset (Bertinetto et al., 2019) is a variant of CIFAR-100 dataset that consists of 100 general object categories. Each class comes with 600 examples, each of which is a color image that contains 32 × 32 pixels. We split the dataset into 64/16/20 classes for training/validation/test. SVHN: This dataset (Netzer et al., 2011) is frequently used as an OOD dataset for CIFAR-10 and CIFAR-100. It consists of 26, 032 color images of 32 × 32 pixels, from 10 digits classes. miniImageNet: This dataset (Vinyals et al., 2016) is a subset of the ImageNet dataset (Russakovsky et al., 2015) . It consists of total 100 classes, each of which has 600 images resized into 84 × 84. We split the dataset into subsets containing 64/16/20 classes for training/validation/test. CUB: This dataset contains 11, 788 images of 200 bird species. We resize the images into 84 × 84. We split the dataset into Table 1 : Any-shot classification results. For each model, we run 3 independent trials and jointly test them over total 9, 000 = 3 × 3, 000 episodes. We report mean accuracies and 95% confidence intervals. Realistic task distribution To define realistic task distribution p(τ ), we first randomly sample C = 5 classes from the whole set of classes. Then with 0.5 probability, we sample the set size N c ∼ Unif(1, 50) independently for each class c = 1, . . . , C, in order to simulate class imbalance.

Meta:

On the other hand, with the other 0.5 probability, we again sample the set size N c ∼ Unif(1, 50) but apply the same single sample to all the classes, in order to simulate task imbalance. We set the number of test (or query) examples to 15 for each class.

Experimental setup We use conventional 4-block convolutional neural networks with 32 channels (Finn et al., 2017) . All the baselines and our model become transductive through batch normalization at meta-test time, following Finn et al. (2017) . We perform early-stopping for all the baselines and our model with meta-validation set performance. We set the number of inner-gradient steps to 5 for meta-training and 10 for meta-testing, for all the models that take inner-gradient steps. See the Appendix A for more details about the experimental setup.3 

5.1. MAIN RESULTS:

Any-shot classification Table 1 shows the results under the realistic task distribution with task and class imbalance. We first observe that Meta-SGD and MT-net outperforms MAML. They precondition the inner-gradients with diagonal or block-diagonal matrix (Flennerhag et al., 2020) , which seems to provide extra flexibility to handle task and class imbalance. ABML, one of the recent Bayesian meta-learning models, largely underperforms MAML in this setting, mainly because each taskspecific predictor is excessively regularized by the prior distribution. However, our graphical model in Figure 2 does not directly impose a prior distribution on task-specific predictors, but only on the balancing variables φ. Prototypical Networks perform relatively well on in-distribution (ID) tasks but not on out-of-distribution (OOD) tasks. This is because metric-based models do not take the gradient steps for OOD tasks, such that the novel information in the dataset cannot be effectively incorporated into each task-specific predictor. Proto-MAML has been proposed to take the advantage of both metric-based and gradient-based approach (Triantafillou et al., 2020) , but it does not outperform Prototypical Networks in our experiments. Finally, we observe that our Bayesian TAML significantly outperforms all the baselines on both ID and OOD datasets. Bayesian TAML performs especially well on OOD datasets, because when the given task largely differs from the meta-knowledge, the model is able to deviate far from the meta-knowledge; however this is difficult for the other baselines.

Multi-dataset any-shot classification:

We further test our model under a more challenging setting where tasks could come from a highly heterogeneous dataset (Triantafillou et al., 2020) . We meta-train the models with Aircrafts, Quickdraw and VGG-Flower dataset, and meta-test with the three datasets plus two additional datasets for OOD tasks -Traffic Signs and Fashion-MNIST. We see from Table 2 that Bayesian TAML also largely outperforms baselines in this setting. rototypical Networks perform relatively well for this multi-dataset classification due to their ability of learning a flexible metric space. Proto-MAML brings only marginal improvements on Prototypical Networks since it takes additional gradient steps without considering task dependency. On the other hand, Bayesian TAML effectively combines the advantage of both metric-based and gradient-based approaches by task-dependently modulating the gradient steps to handle task and class imbalance as well. See the Appendix A for the detailed experimental setup.

5.2. EFFECTIVENESS OF THE BALANCING VARIABLES:

We now validate the effectiveness of the three balancing variables to clearly show their effectiveness.

z τ for handling distributional shift. z τ modulates the initial model parameter θ, and decides what and how much to use from the meta-knowledge θ based on the relatedness between the initial θ and the task at hand. We validate the effectiveness of this balancing variable by examining the performance of the baseline methods and our models using the datasets (SVHN, CUB) that are highly heterogeneous from the meta-training datasets (CIFAR-FS, miniImageNet). The results in Table 3 show that our model with only z τ can effectively handle these out-of-distribution (OOD) tasks. We observe in Figure 4 that z τ actually relocates the initial θ far from the initial parameter for these OOD tasks given at meta-test time, with larger displacements for highly heterogeneous tasks (Figure 4, right) . This allows the model to either stick to, or deviate from the meta-knowledge based on the similarity between the tasks given at the meta-training and meta-test time. Moreover, the last two rows of Table 3 show that the MC approximation in Eq. ( 8) largely outperforms the naive approximation in Eq. ( 9), which suggests that z τ learns very large variance. We conjecture that the role of such random initialization in MAML framework is to increase the effective learning rate for OOD tasks (See Appendix D for the discussion). γ τ and z τ for handling task imbalance. We then examine how the two balancing variables, γ τ and z τ handle inter-task imbalance where tasks used for meta-learning have largely different number of instances. Figure 5 (a) shows the performance of the baseline models and our models with each of the balancing variables, when the number of instances per task varies from 5 to 2000. We observe that our Bayesian z-TAML and Bayesian γ-TAML largely outperform Meta-SGD, especially by large degree when the number of instances per task is large. Figure 5 (b) shows that the effectiveness of z τ largely depends on the number of MC samples used in Eq. ( 8), demonstrating the importance of incorporating uncertainties in random initializations for handling task imbalance. We further observe from Figure 5 (c) that the task-dependent learning rate multiplier γ τ rapidly grows as the number of instances per task increases. This agrees with our intuition that larger tasks should take larger inner-gradient steps, to learn more from the given task rather than resorting to meta-knowledge. ω τ for handling class imbalance. ω τ rescales the class-specific gradients to handle class imbalance where the number of instances per class (i.e. shot) largely varies. Table 4 shows the results under the varying degree of class imbalance across the task distribution. We observe that Bayesian ω-TAML outperforms Meta-SGD, especially by larger degree under higher class imbalance. Notably, Bayesian ω-TAML outperforms a heuristic balancing method which divides each class-specific gradient by the cardinality of each class set (Meta-SGD with 1/N ). The accuracy improvements over Meta-SGD in Figure 6 demonstrate that this heuristic balancing method overly emphasizes the tail classes with few training instances, thereby deteriorating the performance on classes with sufficiently large number of instances. On the other hand, our Bayesian ω-TAML learns the appropriate balancing variables which allow to obtain large performance gains on all classes. 

5.3. MORE ABLATION STUDIES:

Effectiveness of Bayesian modeling We further demonstrate the effectiveness of Bayesian modeling by comparing it with the deterministic version of our model (Deterministic TAML), where three balancing variables are no longer stochastic. We apply 2 regularization on the variables with coefficients that are equivalent to the KL-divergence in Eq. ( 5). The results in Table 5 show that the Bayesian TAML significantly outperforms its deterministic counterpart, especially with larger gap on the OOD task (SVHN). We also see from the last two rows of the table that MC approximation in Eq. ( 8) is more beneficial for the OOD tasks than for the ID tasks (See Appendix D for the discussion).

Figure 7 further shows that the balancing variable γ τ for handling task imbalance, more sensitively reacts on Bayesian TAML than on Deterministic TAML, which suggests that Bayesian modeling amplifies the effect of the balancing variables. Dataset encoding Finally, we perform an ablation study of the proposed task encoding network. The results in Table 6 show that the proposed hierarchical encoding scheme for classification dataset, with set cardinality and variance is significantly more effective than simple mean-pooling methods (Zaheer et al., 2017; Edwards & Storkey, 2017) 4 .

6. CONCLUSION:

We propose Bayesian TAML that learns to balance the effect of meta-learning and task-adaptive learning, to consider meta-learning under a more realistic task distribution where each task and class can have varying number of instances. Specifically, we encode the dataset for each task into hierarchical representations, and use it to modulate the original parameter, learning rate, and the classspecific gradients. We use a Bayesian framework to infer the posterior of these balancing variables, and propose an effective variational inference framework to solve for them. Our model outperforms existing meta-learning methods when validated on imbalanced few-shot classification tasks. Further analysis of each balancing variable shows that each variable effectively handles task imbalance, class imbalance, and out-of-distribution tasks. We believe that our work makes a meaningful step toward application of meta-learning to real-world problems.

Traffic Signs: This dataset (Houben et al., 2013) Based on the previous justification of DeepSets (Zaheer et al., 2017) , we can easily justify the Set-of-Sets structure proposed in the main paper as well, in terms of the two-level permutation invariance properties required for any classification dataset. The main theorem of DeepSets is:

Theorem 1. A function f operating on a set X ∈ X is a valid set function (i.e. permutation invariant), iff it can be decomposed as f (X) = ρ 2 ( x∈X ρ 1 (x)), where ρ 1 and ρ 2 are appropriate nonfcities.

See (Zaheer et al., 2017) for the proof. Here we apply the same argument twice as follows. Inserting s c = g({x c,1 , . . . , x c,N }) into the expression of f , we arrive at the following valid composite function operating on a set of sets:

EQUATION

Let F denote the composite of f and (multiple) g and let NN 2 denote the composite of ρ 1 and ρ 4 . Further define NN 1 := ρ 3 and NN 3 := ρ 2 . Then, we have While our framework does not place any restriction on selecting the statistics for set encoding, we perform further ablation study on the effectiveness of higher-order statistics for our specific experimental setting. We see from Table 7 that the higherorder statistics such as element-wise sample skewness and kurtosis improve the performance given sample mean and diagonal covariance. However, the set cardinality seems more effective than those statistics as it could be the most direct and relevant criteria for detecting imbalances in a set. In this section, we analyze the effect of the variance of z τ for solving OOD tasks, that randomizes the MAML initialization parameter θ. Define E q(z τ |D τ ;ψ) [θ 0 ] = θ * E q [z τ ], the initialization parameter modulated by the posterior mean of z τ . Then, we measure the 2 distance from E q [θ 0 ] to the two different versions of the final task-specific parameter θ τ after taking 10 gradient steps, in order to see how the posterior variance of z τ affects the final task-specific parameter θ τ as a function of z τ :

F

• θ τ (E q [z τ ]]): Task-specific predictor θ τ obtained without the variance of z τ , such that the expectation is taken before the inner-gradient steps. • E z [θ τ (z τ )]: Task-specific predictor θ τ obtained with the variance in the random initialization, such that the expectation is taken outside of the gradient steps. We evaluate the expectation with MC approximation, having the ensemble of the diverse task-specific predictor samples θ τ 1 , . . . , θ τ S (we use S = 50).

Figure 8 suggests that the role of the random initialization is to increase the effective learning rate for the OOD tasks. We see from the left bar graph that if we do not consider variance in the initialization (θ τ (E q [z τ ])), the OOD tasks deviate relatively less than the ID tasks (4.61 vs. 5.18), although it should deviate much considering the distributional discrepancy. On the other hand, if we incorporate the random initialization to obtain task-specific parameter (E q [θ τ (z τ )]), OOD tasks can deviate further from the initialization (4.61 → 5.11). It directly results in the performance gain because the task-specific learner can exploit more of the information in the OOD tasks.

:

We initially exponentially decayed the learning rate γ ∈ [0, 1], but found it suboptimal as it prevents the learning rate from growing for many-shot tasks.

We also found out that sum-pooling is very unstable in encoding set of variable size.

Code is available at https://github.com/haebeom-lee/l2b

We also found out that using set cardinality is more effective than utilizing higher-order statistics such as skewness or kurtosis (See the Appendix C for further ablation study).

Figures and Tables:

Figure 1: Concept. (a) To handle task imbalance, class imbalance and out-of-distribution (OOD) tasks, we introduce task-specific balancing variables γ τ , ω τ , and z τ . (b) With those variables, we learn to balance between the meta-knowledge θ and task-specific update to handle imbalances and distributional shift.

Figure 2: Graphical model. (a) Generative process. (b) Inference.

Figure 3: Inference Network. The proposed dataset encoder captures the instance-wise and class-wise statistics hierarchically, from which we infer three different balancing variables.

Figure 4: T-SNE visualization of θ and θ * E[z τ ].

Figure 5: Ablation study on task imbalance (Meta-training: CIFAR-FS, Meta-test: SVHN).

Figure 6: E[ω τ ] and accuracy improvements over Meta-SGD.

Figure8: 2 distance between the initialization and the task-specific parameters, under different treatment of the expectation over q(z τ |D τ ; ψ). We use Bayesian z-TAML and evaluate with CIFAR-FS / SVHN 50-shot tasks.

Multi-dataset any-shot classification results.

Table content: <table><tr><td/><td>-training</td><td/><td/><td colspan="2">CIFAR-FS</td><td>miniImageNet</td></tr><tr><td colspan="2">Meta-test</td><td/><td colspan="2">CIFAR-FS</td><td>SVHN</td><td>miniImageNet</td><td>CUB</td></tr><tr><td colspan="2">MAML (Finn et al., 2017)</td><td/><td colspan="3">71.55±0.23 45.17±0.22</td><td>66.64±0.22</td><td>65.77±0.24</td></tr><tr><td colspan="2">Meta-SGD (Li et al., 2017)</td><td/><td colspan="3">72.71±0.21 46.45±0.24</td><td>69.95±0.20</td><td>65.94±0.22</td></tr><tr><td colspan="2">MT-net (Lee &amp; Choi, 2018)</td><td/><td colspan="3">72.30±0.22 49.17±0.21</td><td>67.63±0.23</td><td>66.09±0.23</td></tr><tr><td colspan="2">ABML (Ravi &amp; Beatson, 2019)</td><td/><td colspan="3">67.24±0.24 36.52±0.17</td><td>56.91±0.19</td><td>57.88±0.20</td></tr><tr><td colspan="6">Prototypical Networks (Snell et al., 2017) 73.24±0.20 42.91±0.18</td><td>69.11±0.19</td><td>60.80±0.19</td></tr><tr><td colspan="3">Proto-MAML (Triantafillou et al., 2020)</td><td colspan="3">71.80±0.21 40.16±0.17</td><td>68.96±0.18</td><td>61.77±0.19</td></tr><tr><td colspan="2">Bayesian TAML</td><td/><td colspan="3">75.15±0.20 51.87±0.23</td><td>71.46±0.19</td><td>71.71±0.21</td></tr><tr><td>Meta-training</td><td/><td/><td colspan="4">Aircraft, QuickDraw, and VGG-Flower</td></tr><tr><td>Meta-test</td><td>Aircraft</td><td colspan="5">QuickDraw VGG-Flower Traffic Signs Fashion-MNIST</td></tr><tr><td>MAML</td><td>48.60±0.17</td><td colspan="2">69.02±0.18</td><td colspan="2">60.38±0.16</td><td>51.96±0.22</td><td>63.10±0.15</td></tr><tr><td>Meta-SGD</td><td>49.71±0.17</td><td colspan="2">70.26±0.16</td><td colspan="2">59.41±0.27</td><td>52.07±0.35</td><td>62.71±0.25</td></tr><tr><td>MT-net</td><td>51.68±0.17</td><td colspan="2">68.78±0.18</td><td colspan="2">64.20±0.16</td><td>56.36±0.23</td><td>62.86±0.15</td></tr><tr><td colspan="2">Prototypical Networks 50.63±0.16</td><td colspan="2">72.31±0.17</td><td colspan="2">65.52±0.15</td><td>49.93±0.18</td><td>64.26±0.13</td></tr><tr><td>Proto-MAML</td><td>51.15±0.17</td><td colspan="2">69.84±0.18</td><td colspan="2">65.24±0.17</td><td>53.93±0.20</td><td>63.72±0.15</td></tr><tr><td>Bayesian TAML</td><td>54.43±0.16</td><td colspan="2">72.03±0.16</td><td colspan="2">67.72±0.16</td><td>64.81±0.21</td><td>68.94±0.13</td></tr></table>

Ablation study on distributional shift.

Table content: <table/>

Ablation study on class imbalance.

Table content: <table><tr><td>Meta-training / Meta-test</td><td colspan="3">Number of instances per class</td></tr><tr><td>CIFAR-FS / CIFAR-FS</td><td>10</td><td>5 -25</td><td>1 -50</td></tr><tr><td>MAML</td><td colspan="3">73.60±0.19 71.15±0.19 67.43±0.22</td></tr><tr><td>Meta-SGD</td><td colspan="3">73.25±0.19 72.68±0.19 71.61±0.19</td></tr><tr><td>Meta-SGD with 1/N</td><td colspan="3">71.33±0.19 72.43±0.19 72.23±0.19</td></tr><tr><td>Bayesian ω-TAML</td><td colspan="3">73.44±0.18 73.20±0.18 72.86±0.19</td></tr></table>

Classification performance of Bayesian and Deterministic TAML on seen and unseen dataset.

Table content: <table><tr><td>Meta-training: CIFAR-FS</td><td>CIFAR-FS</td><td>SVHN</td><td/><td>3</td><td/><td colspan="2">Det-Conv2 Det-Conv3</td><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>2.5</td><td/><td colspan="2">Det-Conv4</td><td/><td/><td/></tr><tr><td>MAML Meta-SGD Deterministic TAML</td><td>70.19±0.23 72.71±0.21 73.82±0.21</td><td>41.81±0.19 46.45±0.24 46.78±0.24</td><td>Gamma</td><td>1.5 2</td><td/><td colspan="2">Bayes-Conv2 Bayes-Conv3 Bayes-Conv4</td><td/><td/><td/></tr><tr><td>Bayesian TAML (Naive approx.)</td><td>73.52±0.20</td><td>47.15±0.20</td><td/><td>1</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Bayesian TAML (MC approx.)</td><td>75.15±0.20</td><td>51.87±0.23</td><td/><td>0.5</td><td>5</td><td>25</td><td>50</td><td>100</td><td>150</td><td>200</td><td>250</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="5">Number of instances per task</td></tr><tr><td/><td/><td/><td colspan="9">Figure 7: E[γ τ ] vs. Bayesianness.</td></tr><tr><td/><td/><td/><td colspan="9">Meta-training / Meta-test Hierarchical encoding CIFAR-FS / CIFAR-FS × √</td></tr><tr><td/><td/><td/><td colspan="3">Mean</td><td/><td/><td colspan="4">73.84±0.21 73.69±0.21</td></tr><tr><td/><td/><td/><td colspan="3">Mean + N</td><td/><td/><td colspan="4">73.17±0.21 74.88±0.20</td></tr><tr><td/><td/><td/><td colspan="4">Mean + Var. + N</td><td/><td colspan="4">73.93±0.21 75.15±0.20</td></tr><tr><td/><td/><td colspan="10">Table 6: Ablation study on dataset encoding</td></tr><tr><td/><td/><td colspan="7">schemes. N: Set cardinality.</td><td/><td/></tr></table>

has only test set consisting of 43 classes. Each class has 900 examples. We use the test set of Fashion-MNIST (Xiao et al., 2017) for meta-testing. This dataset has 10 classes with 1000 examples per class. Statistics Pooling for each class c = 1, . . . , C. fc layer with 4-dim (across the statistics) and ReLU Shared encoder NN 2 : s 1 , . . . , s C → v τ fc layer with 128-dim and ReLU fc layer with 32-dim Statistics Pooling over all the class representations fc layer with 4-dim (across the statistics) and ReLU

Table content: <table><tr><td>Fashion-MNIST: A.4 INFERENCE NETWORK ARCHITECTURE</td></tr><tr><td>We describe the network architecture of the inference network that takes a classification dataset as an</td></tr><tr><td>input and generates three balancing variables as output.</td></tr><tr><td>Shared encoder NN 1 : X τ 1 , . . . , X τ C → s 1 , . . . , x C</td></tr><tr><td>3 × 3 Conv2d with 10-dim and ReLU</td></tr><tr><td>2 × 2 Max pool</td></tr><tr><td>3 × 3 Conv2d with 10-dim and ReLU</td></tr><tr><td>2 × 2 Max pool</td></tr><tr><td>fc layer with 64-dim</td></tr><tr><td>Generate ω τ : s τ 1 , . . . , s τ C → (µ τ ω1 , σ τ ω1 ), . . . , (µ τ ω C , σ τ ω C )</td></tr><tr><td>fc layer with 64-dim and ReLU</td></tr><tr><td>fc layer to generate (µ τ ωc , σ τ ωc ) for each class c = 1, . . . , C.</td></tr><tr><td>Generate γ τ : v τ → (µ τ γ1 , σ τ γ1 ), . . . , (µ τ γ L , σ τ γ L )</td></tr><tr><td>fc layer with 64-dim and ReLU</td></tr><tr><td>fc layer to generate (µ τ γ l , σ τ γ</td></tr></table>

with appropriate nonfcities ρ 1 and ρ 2 . 2. A function g operating on a set of examples {x c,1 , . . . , x c,N } is a valid set function (i.e. permutation invariant w.r.t. the order of {x c,1 , . . . , x c,N }) iff it can be decomposed as g({x c,1 , . . . , x c,N }) = ρ 4 ( N i=1 ρ 3 (x c,i )) with appropriate nonfcities ρ 3 and ρ 4 .

Table content: <table/>

{x 1,1 , . . . , x 1,N }, . . . , {x C,1 , . . . , x C,N } = NN 3 is the number of classes and N is the number of examples per class. See Section A.4 for the actual encoder structure. C ABLATION STUDY ON HIGHER-ORDER STATISTICS Var. + Skew. 73.66±0.21 50.33±0.23 Mean + Var. + Kurt. 73.47±0.21 50.27±0.23 Mean + Var. + N 75.15±0.20 51.87±0.23 Further ablation study on dataset encoding schemes. N: Set cardinality.

Table content: <table><tr><td>C</td><td>N</td><td/></tr><tr><td>NN 2</td><td>NN 1 (x c,i )</td><td>(11)</td></tr><tr><td>c=1</td><td>i=1</td><td/></tr></table>

Peer review (iclr2020):
Summary ======== This paper introduces a mechanism for gradient-based meta-learning models for few-shot classification to be able to adapt to diverse tasks that are imbalanced and heterogeneous. In particular, each encountered task may have varying numbers of shots (task imbalance) and even within each task, different classes may have different numbers of shots (class imbalance). Further, test tasks might come from a different distribution than the training tasks. They propose to handle this scenario by introducing three new types of variables which control different facets of the degree and type of task adaptation, allowing to decide how much to reuse meta-learned knowledge versus new knowledge acquired from the training set of the given task. Specifically, their newly introduced variables are: 1) the factor for learning rate decay (a scalar) for the inner-loop task adaptation optimization which allows to not deviate too much from the global initialization when insufficient data is available, 2) a class-specific learning rate (one scalar per class) that allows to tune more for under-represented classes of the training set, 3) a set of weights on the global initialization (one scalar per dimension) that can down-weigh each component if it’s not useful for the task at hand (e.g. if test tasks have significantly different statistics than training tasks did). The values of these variables are predicted based on the training set of the task: the support set is encoded via a hierarchical variant of a set encoding (where pooling is done using higher order statistics too instead of simply averaging). The resulting encoded support set is the input to the network that produces the values for the three sets of variables discussed above. Each new variable is treated in a Bayesian fashion: a prior is defined over it (Normal(0,1)), which is updated by conditioning on the training set to form a posterior for each given task. Specifically, each of the above variables is represented by a Gaussian whose mean and variance are the learnable parameters that are produced by the network described above. Experimentally, this method outperforms others on a setting of imbalanced tasks (the shot is sampled uniformly at random from a designated range). The gain over other methods is large in particular when evaluated on out-of-distribution tasks (coming from a different dataset) and when the imbalance is large. Comments (in decreasing order of importance) ======================================== A) The Bayesian framework helps because it offers an elegant way to use a prior. In the deterministic version, was any effort made to resemble the effect of that prior? For example, one can define a regularizer that penalizes behaviors that ignore the meta-knowledge too much (e.g. too large values for gamma, or for the class-specific learning rates etc). Albeit more ‘hacky’, if these regularization coefficients are tuned properly, they might result in a similar effect to that of having a prior. A fair comparison to the deterministic variant should include this. B) I think that gamma and z can be merged into a single set of parameters? In particular, imagine a per-dimension-of-	heta learning rate. This would then be large for a dimension when there is a larger need for adapting that dimension of 	heta. In the case of large training sets, this can be large for all dimensions, recovering the behavior of a large gamma. For the case of diverse datasets, this would behave as the current z (updates a lot the dimensions of 	heta that are irrelevant for the given task due to the dataset shift). C) Meta-Dataset (https://arxiv.org/abs/1903.03096) is a recent benchmark for few-shot classification that introduces both of what is referred to here as task imbalance and class imbalance and also is comprised of heterogeneous datasets and evaluates performance on some held-out datasets too. The current state-of-the-art on it (as far as I know) is CNAPs [1] which employs a flexible adaptation mechanism on a per-task basis but is fully amortized (performs no gradient-based adaptation to each task) and makes no explicit effort to tackle imbalanced tasks as is done here. I’m curious how this method would compete in that setup. It definitely seems to be a strong candidate for that benchmark! Less important ============= D) which dataset is used in Tables 4 and 5? I assume it’s Omniglot (due to the numbers being in the 90s) but it would be good to say this explicitly. E) In section 5.2, expressions such as x5 and x15 are used to characterize the degree of imbalance of a task. How exactly are these computed? Does x5 mean that the largest shot is 5 times larger than the smallest shot? It would be good to explicitly state this. F) In the Related Work section, in the Meta-learning paragraph there is a sentence that’s not accurate: “Metric-based approaches learn a shared metric space [...] such that the instances are closer to their correct prototypes than to others”. This sentence does not describe all metric based approaches. It describes Prototypical Networks (Snell et al) but not, for example, Matching Networks (Vinyals et al) nor many others that like Matching Networks perform example-based comparisons and don’t aggregate a class’ examples into a prototype. In a nutshell =========== I think this work is a useful contribution for moving towards a more realistic setting in few-shot classification. It captures some desiderata of models that can operate in more realistic settings and outperforms previous models in those scenarios. My comments above are mostly suggesting improvements and clarifications but I am inclined to recommend acceptance of this paper. References ========= [1] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.