Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks
Authors: Hae Beom Lee
Hayeon Lee
Donghyun Na
Saehoon Kim
Minseop Park
Eunho Yang
Sung Ju Hwang
Venue:  (2020)
Keywords: meta-learning
few-shot learning
Bayesian neural network
variational inference
learning to learn
imbalanced and out-of-distribution tasks for few-shot learning

Abstract: While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the meta-knowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the meta-knowledge or task-specific learning. We formulate this objective into a Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on two realistic task- and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework. 

Published as a conference paper at ICLR 2020
LEARNING TO BALANCE: BAYESIAN META-LEARNING FOR IMBALANCED AND OUT-OF-DISTRIBUTIO TASKS
Hae Beom Lec'" Hayeon Lee Donghyun Na²; Sachoon Kim², Minseop Park2, Eunho Yang Sung Ju Hwang!.3 KAIST" TmaxData², AITRICS South Korea
(haebeom. lee, hayeon eunhoy, sjhwang82 (aa.ngg/2kkaiait..... Sanghyun_na@tmane co. kr, (shkim, mike_seop petcoppeaaitiitc..om com
ABSTRACT
While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that the number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the lnowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the knoalkno or task-specific learning. We formulate this objective into Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on multiple realistic task- and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework.
INTRODUCTION
Despite the success of deep learning in many real-world tasks such as visual recognition and machine translation, such good performances are achievable at the availability of large training data, and many fail to generalize well in small data regimes. To overcome this limitation of conventional deep learning, recently, researchers have explored meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) approaches, whose goal is to learn model that generalizes well over distribution of tasks, rather than instances from a single task, in order to utilize the obtained meta- neta-knowledge across tasks compensate for the lack of training data for each task.
However, so far, most existing meta-learning approaches (Santoro et al., 2016; Vinyals et al., 201 6; Snell al., 2017; Ravi & Larochelle, 2017; Finn et al., 2017; Li al., 2017) have only targeted an artificial scenario where all tasks participating in the multi-class classification problem have equal number of training instances per class. Yet, this is a highly restrictive setting, in real-world scenarios, tasks that arrive at the model may have different training instances (task imbalance), and within each task, the number of training instances per class may largely vary (class imbalance). Moreover, the new task may come from distribution that is different from the task distribution the model has been trained on (out-of-distre distt---dit task) (See (a) of Figure 1).
Under such realistic setting, the neta-knowledge may have a varying degree of utility to each task. Tasks with small number of training data, or close to the tasks trained in meta- tetta-trining step may want to rely mostly on meta-Knowledge obtained over other tasks, whereas tasks that outtof-distribution or come with more number of training data may obtain better solutions when trained a task-specific
"Equal contribution

---

Published as a conference paper at ICLR 2020
(a) Realistic Datasets
(b) Learning to Balance
Meto Training
Meto-Test
Tall class
0
g
Small task
Head class
Large task
Class Imbal.
DOD
0 +-
VoL.
Figure 1: Concept. (a) To handle task imbalance, class imbalance and out-of-distribution (OOD) tasks, introduce task specific balancing variables " .w", and z" (b) With those variables, we learn to balance between the mntt-knnowedde and task-specific update to handle imbalances and distributional shift.
manner. Furthermore, for multi-class classification, we may want to treat the learning for each class differently to handle class imbalance. Thus, to optimally leverage meta-learning under various imbalances, it would be beneficial for the model to task- and class-adaptively decide how much to use from the meta-learner, and how much to learn specifically for each task and class.
To this end, we propose novel Bayesian meta-learning framework, which we refer to Bayesian Task-Adaptive Meta-Learning (Bayesian TAML), that learns variables to adaptively balance the effect of meta- and task-specific learning. Specifically, task, which are learned to convey useful statistics about the task or class distribution, such as mean, variance, and cardinality (the number of elements in the set), and then learn the distribution of three balancing variables a function of the set: 1) task-dependent learning rale multiplier, which decides how far away deviate from the meta-knowledge, when performing task-specific learning. Tasks with higher shots could benefit from taking gradient steps afar, while tasks with few shots may need stay close to the initial parameter. 2) class-dependent learning rate, which decides how much information to use from each class, to automatically handle class imbalance where the number of instances per class can largely vary. 3) task task- mask-dependent modulator for initial model parameter, which modifies the shared initialization for each task, such that each task can decide how much and what to use from the shared initial model parameter and what to ignore based on its set representation. This is especially useful when handling out-of-distee -i-oof--ittibutton task, which may need to ignore some of the meta-knowledge.
first obtain set-representations for each
We validate our model on CIFAR-FS and minilmageNet dataset, well as a new dataset that consists of heterogeneous datasets, under a scenario where every class in each episode can have any number of shots, that leads to task and class imbalance, and where the dataset at meta-test time is different from that of meta- tetta-traiinng time. The experimental results show that our Bayesian TAML significantly improves the performance analysis of each component reveals that the improvement is due to the effectiveness of the balancing terms for handling task and class imbalance, and cidoo--distriuttinn tasks.
the existing approaches under these realistic scenarios. Further
To summarize, our contribution in this work is threefold:
We consider novel problem of meta-learning under a realistic task distribution, where the number of instances across classes and tasks could largely vary, or the unseen task the meta-test time is largely different from the seen tasks.
For effective meta-learning with such imbalances, we propose a Bayesian task-adaptive meta-learning (Bayesian TAML) framework that can adaptively adjust the effect of the meta-learner and the task-specific learner, differently for each task and class.
We validate our model on realistic imbalanced few-shot classification tasks with a varying number of shots per task and class and show that significantly outperforms existing meta-learning models.
RELATED WORK
Meta-learning Meta-learning (Schmidhuber, 1987; Thrun & Pratt, 1998) is approach to learn a model to generalize over a distribution of task. The approaches general can be categorized methods. A memory- ap- proach (Santoro et al., 2016) learns to store correct instance and label into the same memory slot and retrieve it later, in task-generic manner. Metric-based approaches learn shared metric
into either memory-bate based, metric- based, and optimization-bant
2

---

Published as a conference paper at ICLR 2020
space (Vinyals et al., 2016; Snell et al., 2017). Snell et al. (2017) defines the distance between the instance and the class prototype, such that the instances are closer to their correct prototypes than to others. As for optimization- based meta- meta-learning, MAML (Finn et al., 2017) learns shared initial- ization parameter that is optimal for any tasks within few gradient steps from the initial parameter. Meta-SGD (Li et al., 2017) improves upon MAML by learning the learning rate differently for each parameter. For effective learning of a meta-learner, meta-learning approaches adopt the episodic training strategy (Vinyals et al., 2016) which trains and evaluates a model over a large number of tasks, which are called meta- training and meta-test phase, respectively. However, existing approaches only consider an artificial scenario which samples the classification of classes with exactly the same number of training instances, both within each episode and across episodes. On the other hand, we consider a more challenging scenario where the number of shots per class and task could vary at each episode, and that the task given at meta-test time could be an out-of-distributtinn task.
Task-adaptive meta-learning The goal of learning single meta-learner that works well for all tasks may be overly ambitious and leads to suboptimal performances for each task. Thus recent approaches adopt aisk-adaptively modified meta-learning models. Oreshkin al. (2018) proposed to learn the temperature scaling parameter to work with the optimal similarity metric. Qiao et al. (2018) also suggested a model that generates task-specific parameters for the network layers, but it only trains with many-shot classes, and implicitly expects generalization to few-shot cases. Rusu et al. (2019) proposed network type task-specific parameter producer, and Lee & Choi (2018) proposed to differentiate the network weights into task-shared and task-specific weights. Our model also aims to obtain task-specific parameter for each task, but is rather focused on learning how to balance between the meta-learning and task-/class-speciffe learning.
Probabilistic meta-learning Recently, a probabilistic version MAML has been proposed (Finn et al., 2018), where they interpret task-specific gradient update as a posterior inference process under variational inference framework. Kim et al. (2018) proposed Bayesian MAML with a similar motivation but with a stein variational inference framework and chaser loss. Gordon et al. (2019) proposed a probabilistic meta-learning framework where the parameter for a novel task is rapidly estimated under decision theoretic framework, given set representation of a task. The motivation behind these works is to represent the inherent uncertainty in few-shot classification tasks. Our model also uses Bayesian modeling, but it focuses on leveraging the uncertainties of the meta-learner and the gradient-direction in order to balance between meta- and task- or class-specific learning.
3 LEARNING TO BALANCE
We first introduce notations and briefly recap the model-agnostic meta-lcarning (MAML) by Finn et al. (2017). Suppose a task distribution p(T) that randomly generates task consisting of a training set D (X*, Y"} and a test set Dr (X", Yr). Then, the goal of MAML is meta-learn the initial model parameter 0 as a meta-knowledge to generalize over the task distribution P(T), such that we can easily obtain the task-specific predictor 0J in a single (or a few) gradient step from the initial 0. Toward this goal, MAML optimizes the following gradient-base meta-learning objective:
min
Σ T(P(())
L(O C(0-aVo (o
where a denotes stepsize and L denotes empirical loss such as negative log-likelihood of observations. Note that by meta-learning the initial point 0 the task-specific predictor 0 = 0 - a VoC(O D can minimize the test loss L(:; DT) even with D consisting of only few samples. We can easily extend the Eq. (1), such that we obtain 0 with more than one inner-gradient steps from the initial A.
However, the existing MAML framework has the following limitations that prevent the model from efficiently solving real-world problems involving class/task imbalance and
tasks.
1. Class imbalance. MAML does not provide any framework to handle class imbalance within each task. Therefore, classes with large number of training instances (head classes) may dominate the task-spec learning during the inner-gradient steps, yielding low performance on classes with fewer shots (tail classes).
2. Task imbalance. The model has a fixed number of inner mendietr steps and stepsize a across all tasks, which prevents the model from adaptively deciding how much to resort to
3

---

Published as a conference paper at ICLR 2020
the keetllnoowedge or how much to learn from the given dataset, depending the number of the training examples per task.
3. Out-of-distribution tasks. The model assumes that the initial model parameter e will be equally useful for the unseen tasks, but for unseen tasks that are significantly different from the previously seen tasks, the initial parameter may be less useful.
3.1 TASK-ADAPTIVE META- (TAML)
As shown in Figure/ for the concepts, introduce three balancing variables w", y",, z" tackle each problem mentioned above. How to generate these variables will be described in Section 4. Also, see the experimental section for how generate the realistic tasks with class and task imbalance.
Tackling class imbalance. To handle class imbalance, we vary the learning rate of class-specific gradient for each inne-ootimization step. Specifically, for class c = I...., C, we introduce a set of class-specifie scalars w (wT..... ...., w E [0, 1Jc. which are multiplied to each of the class-specific gradients V (o Di).... VoC(o; DC), where D. is the set of instances and labels for class c. We expect W. to be large for tail-classes with small number of training instances, such that they could be considered more in the inner-optimization steps. We generate w with Softmax function, and denote the input to the function a
Tackling task imbalance. To control whether the model parameter for the current task stays close to the initial parameter or deviate far from it, we introduce task-dependent learning-rate multipliers Y (.i...., ................................ 7F.) ....)..,777)) E [0, oc) such that for each layer / = 1..... L, the learning rate becomes aa, aa..... Laa. We expect y to be large for large tasks, such that they rely more on task- specific updates, while small tasks use small y to benefit more from the neta-kno To amplify the step-size difference between large and small tasks, generate y with exponential function, and denote the input to the function as 5
Tackling cittrf-distributton tasks. Finally, we introduce zt which modulates the initial parameter 0 for each task. We expect z" to learn to relocate the initial O to a new starting point, such that out-of- -od--ii-iisitittionnn (OOD) tasks can deviate much from the shared initialization 0 if the current initialization is suboptimal for the given task. Specifically, we use 2 = 1 + 2 for the channel of the convolutional network weights and z" = z for the biases, which modify the initial parameter as follows: 0, + O z for the weights and Og + 0 + z for the biases, where O, denotes the new initialization modulated by z² We denote this operation as x zz, which is similar to task task- deepedent modulation of batch normalization parameters (Oreshkin al., 2018; Requeima al., 2019).
A unified framework. Finally, we assemble all these components together into a single unified framework. With a slight abuse of notations, the update rule can be written as follows:
0g * zz,
(2)
02 Oa-1 --11--
T" o
a W Vo. C(On-ii for k = I,..., K
(3)
where is a multi-dimennsonal global learning rate vector that is learned (Li et al., 2017), and the multiplication operator o is appropriately defined. The last step 0g corresponds to the tasc-specific predictor 0r.
3.2 BAYESIAN TASK-A ADAPTIVE META-LEARNING
We now introduce the variational inference framework for the input of the three balancing variables, that we previously denote as W5, 5", 2². Bayesian modeling allows to incorporate randomness in the posterior of those variables. In MAML framework, such randomness generates the ensemble of diverse task-specific predictors, which allows to effectively exploit the information in the given dataset Dr. Bayesian modeling also allows to find more robust latent structures of those balancing variables (See Figure 7).
We initially exponentially decayed the learning rate 7 E [0, 1), but found it suboptimal as it prevents the learning rate from growing for many-shot tasks.
4

---

Published as a conference paper at ICLR 2020
Firstly, define X (9% Mr- m=1 for test. Let Φ denote the collection of "" 4 and 2' for uncluttered notation. Then, inspired by Gordon et al. (2019) and Finn et al. (2018), we define the generative process for meta- learning framework as follows for each task (See Figure 2):
{xx,, J and Y = {yn)mm for training, and X {x
Mr /xm/m= m- and Y
N. II ply " 0)
P x X 0) p(Φ))
II m==
ply PIyMI & x-:
(4)
for the complete data likelihood. Note that the deterministic is shared across all the tasks.
VARIATIONAL INFERENCE
The goal of learning for each task is to maximize the conditional log- likelihood of the joint dataset DDr and 'D:: log Y Y (=
However, solving it involves the true posterior p( T (D=,D") Dr), which intractable. Thus, we resort to amortized variational inference with tractable form of approximate posterior ( [ [DT, D"; +b) parameter- ized by vb. We let the three variables share the same inference network pipeline minimize the computational cost. Further, similarly to Ravi & Beatson (2019), we drop the dependency on the test dataset Dr for the approximate posterior, in order to make the two different pipelines consistent; one for meta-training where observe the whole test dataset, and the other for meta- veta-testing where the test labels are unknown. The form of our approximate posterior is now q( DD; +b). It greatly simplifies the inference framework, while ensuring to have valid lower bound of the log evidence. Also, considering that performing the inner- amer-gradient steps with the training dataset D automatically maximizes the training log-likelihooe in MAML framework, slightly modify the objective so that the expected log- log-likelihood term only involves the test examples with the appropriate scaling factor. The resultant form of the approximated lower bound that suits for our meta- leaa-lea purpose is as follows:
i
M.
Mr
(h)
Figure 2: Graphical model.
(a) Generative process. (b) In- ference.
Nr + Mr
BolorIDrsss logg(P? 8 o²:
(5)
M+
We assume q( ;"; bb) fully factorizes for each variable and also for each dimension as well:
O( +b) II ( : 2b) II ( DT :b)) II 9 (=T
where we assume that each single dimension of q trainable mean and variance. We also let each dimension of prior p( (²) factorize into ( 1), such that the KL-divergence can have the especially simple closed form (Kingma & Welling, 2014).
v) [D=;
follows univariate Gaussian having
The final form of the meta veta-ttainning minimization objective with Monte-Carlo (MC) approximation for the expectation Eq. (5) is as follows:
M. M
min 0.
logr( 03:0) N. M, (T
m=
where Φं a( (';;)b) We implicitly assume the reparameterization trick for of obtain stable and unbiased gradient estimate w.r.t. vb (Kingma & Welling, 2014). We set the MC sample size S = 1 for meta-training for computational efficiency. When meta-testing, we perform MC approximation with sufficiently large sample size (e.g. S = 10):
p(TT1&;:0) E.[p(ST Φ5:
Σ EMORR
~ q< (8)
we may naively approximate by taking the expectation inside for computational efficiency: Eylp( " 0) ~ p(yT By( 0)-
In the experimental section, show that Eq. (8) largely outperforms Eq. (9).
5

---

Published as a conference paper at ICLR 2020
Global
y
cardialllt
mean
centrniity
Statisties pooling
Statistics pooling
1x3 oone
com
NN
NNJ
NN.
NN2
conv
0
3x3 comy
Task- specific
X
D train
Dr
test
Figure 3: Inference Network. The proposed dataset encoder captures the instance-wise and class-wise statistics hicrarchically, from which infer three different balancing variables.
4.1 DATASET ENCODING
The main challenge in modeling our variational distribution g( [D;; (6) is to decide how to refine the training dataset D into informative representation, which is not trivial. This inference network should capture all the necessary statistical information to recognize any imbalances in the dataset DD Mean-pon coan-pooling (Edwards & Storkey, 2017) or sum-pooling (Zaheer et al., 2017) is frequently used practical set-encoder, where each instance in the set is transformed by the shared nonlinearity, and then averaged or summed together to generate a single vector summarizing the set, followed by an additional nonlinearity. However, for the classification dataset D which the set of (class) sets, those non non-hierarchical pooling methods will perform poorly as they ignore the label information. Therefore, we use a two-layer hierarchical set encoder which first encodes each class as set of samples and then encodes the set as the set of classes (see the Appendix B for the justification).
However, there is an additional limitation mean pooling when using it to describe a task: it does not recognize the number of elements in the set". This could be a critical limitation in recognizing the imbalances the given task. Therefore we explicitly input the number of elements into set encoder. Yet, the set cardinality alone is insufficient in capturing the distribution of the dataset. Suppose that we have a set containing replications only limited information, the set encoding cannot recognize and will overestimate the information. To prevent this, we encode the variance of the set as well.
single instance. Then, although the set has
Based this intuition, define the set encoder StatisticsPoollnng() ))) that generates the concatena- tion of the set statistics such as mean, variance, and cardinality. We use this encoder to first encode each class, and then the whole set of classes as follows:
v Statistics ((NNa (S:)]=_))))) s. StatisticsPooling (INN1(x))xEX)
for classes 1,..., C. X is the collection of class eexamples in task NN₃ and NN; are some neural networks parameterized by sb. The vector v finally summarizes the entire dataset Dr for few-shot classification. Note that the class-specific balancing variables wf,..., , are generated from Sj,.... Sc. and other task-specific balancing variables 5" and z are generated from v with few additional layers (See Figure 3).
5 EXPERIMENTS
We next validate our method on multiple benchmark datasets with more realistic task distribution.
Datasets We validate our method on the following benchmark datasets. CIFAR-FS: This dataset (Bertinetto al., 2019) is a variant of CIFAR-100 dataset that consists of 100 general object categories. Each class comes with 600 examples, each of which is a color image that contains 32 32 pixels. We split the dataset into 64/ 64/16/20 classes for rraiiingvalidationnnes SVHN: This dataset (Netzer et al., 201 1) is frequently used an OOD dataset for CIFAR- 10 and CIFAR- 100. consists of 26, 032 color images 32 x 32 pixels, from 10 digits classes. minilmageNet: This dataset (Vinyals et al., 2016) is a subset of the ImageNet dataset (Russakovsky et al., 2015). It consists total 100 classes, each of which has 600 images resized into 84 x 84. We split the dataset into subsets containing 64/ 16/20 classes for training/ valiaiioovaliiaiion/tsss. CUB: This dataset contains 11, 788 images of 200 bird species. We resize the images into 84 x 84. We split the dataset into
We also found out that sum-pooling is very unstable in encoding of variable size.

---

Published as a conference paper at ICLR 2020
Table I: Any-shot classification results. For each model, we run independent trials and jointl test them over total 9, 000 3 x 3, 000 episodes. We report mean accuracies and 95% confidence intervals.
Meta-traiee -eaannra
CIFAR-FS
minilmageNet
Meta-test
CIFAR-FS
SVHN
minilmageNet
CUB
MAML (Finn et al., 2017)
71.55
45.17
66. 06.64
65.77=0024
Meta-SGD (Li al., 2017)
72.71
46.45..024
69.95
65,94.10.22
MT-net (Lee & Choi, 2018)
72.30
49.17:021
67.63
66.09.10.3
ABML (Ravi & Beatson, 2019)
67.24:10.4
36.52:10.17
1:.11
57.88
Prototypical Networks (Snell et al., 2017)
23.24
42.91
69. 1 10.19
60.80
Proto-MAMI. (Triantafillou al., 2020)
71.80=0021
40.1642017
68.961..18
61.77=0.19
Bayesian TAML
75.15 0.20
51.87 1.22
7..46
71.71 1..71
Table 2: Multi-dataset any-shot classification results.
Meta-training
Aircraft, QuickDraw, and VGG-Flower
Meta-test
Aircraft
QuickDraw
VGG-Flower
Traffic Signs
Pashion-MNIST
MAML.
48.60
69.022m.18
60.382n16
51.96112
63. 10.10
Meta-SGI
49.7
10.26=0.16
59.41
52.07
62.71
MT-net
51.68++0.17
68.78
64.20=#016
56.36
62.86=0.15
Prototypical Networks
50.63+
72.31 0.11
65.52+0.15
49.93
64.26=
Proto-MAML
51.15+
69.84
65.24
53,93+000
63.72+0.15
Bayesian TAML.
54.43.2016
72.0320.16
67.72=0.16
1428
68.94=0.13
100/50/50 classes for rraiiin/validationneesss Since this dataset is fine-grained, we regard it as an out-of-distribution dataset of the coarse-grained minilmageNet dataset.
Realistic task distribution To define realistic task distribution p(r), we first randomly sample C' = 5 classes from the whole set of classes. Then with 0.5 probability, we sample the set size No Unif(1,50) independently for each class c 1, ...... C, in order to simulate class imbalance. On the other hand, with the other 0.5 probability, we again sample the set size N₂ ~ Unif(1, 50) but apply the same single sample to all the classes, in order simulate task imbalance. We set the number of test (or query) examples to 15 for each class.
Experimental setup We use conventional 4-block convolutional neural networks with 32 chan- nels (Finn et al., 2017). All the baselines and our model become transductive through batch normal- ization at meta-test time, following Finn et al. (2017). We perform early-stopping for all the baselines and our model with meta-validation for meta-training and 10 for meta-testing, for all the models that take inner- menr-grrdient steps. See the Appendix A for more details about the experimental setup.
set performance. We set the number of inner-gradies steps to 5
5.1
MAIN RESULTS
Any-shot classification Table shows the results under the realistic task distribution with task and class imbalance. We first observe that Meta-SGD and MT--et outperforms MAML. They precondition the inner- ameerggradeents with diagonal or block-diagonal matrix (Flennerhag al., 2020), which seems to provide extra flexibility to handle task and class imbalance. ABML, one of the recent Bayesian meta-learning models, largely underperforms MAML this setting, mainly because each task- specific predictor is excessively regularized by the prior distribution. However, our graphical model in Figure 2 does not directly impose a prior distribution on task losk-spccific predictors, but only on the balaneing variables Φb. Prototypical Networks perform relatively well on in-distribution (ID) tasks but not on out-of -ittr--diittibttion (OOD) tasks. This is because metric-based models do not take the gradient steps for OOD tasks, such that the novel information in the dataset cannot be effectively incorporated into each task-specific predictor. Proto-MAML has been proposed to take the advantage of both netric-based and gradient-based approach (Triantafillou et al., 2020), but it does not outperform Prototypical Networks in our experiments. Finally, we observe that our Bayesian TAML significantly outperforms all the baselines on both ID and OOD datasets. Bayesian TAML performs especially well on OOD datasets, because when the given task largely differs from the meta- neta-knowledge, the model is able to deviate far from the neta-knowledge; however this difficult for the other baselines.
Multi-dataset any-shot classification We further test our model under a more challenging setting where tasks could come from highly heterogeneous dataset (Triantafillou et al., 2020). We meta-
Code available https://////c http//gtthubhcoeo hwtp:/cccc/cccccoocoeeeeee 1ee/12b
7

---

Published as a conference paper at ICLR 2020
train the models with Aircrafts, Quickdraw and VGG-Flower dataset, and meta-test with the three datasets plus two additional datasets for OOD tasks Traffic Signs and Fashion-MNIST. We see from Table 2 that Bayesian TAML also largely outperforms baselines in this setting. rototypical Networks perform relatively well for this multi-dataset classification due to their ability of learning a flexible metric space. Proto-MAML brings only marginal improvements takes additional gradient steps without considering task dependency. On the other hand, Bayesian TAML effectively combines the advantage of both metric-based and gradient-based approaches by lusk-dependently modulating the gradient steps to handle task and class imbalance as well. See the Appendix A for the detailed experimental setup.
Prototypical Networks since it
5.2 EFFECTIVENESS OF THE BALANCING VARIABLES
We now validate the effectiveness of the three balancing variables to clearly show their effectiveness.
z" for handling distributional shift. z modulates the initial model parameter 0, and decides what and how much to use from the meta-knowledge A based the relatedness between the initial and the task at hand. We validate the effectiveness of this balancing variable by examining the performance of the baseline methods and our models using the datasets (SVHN, CUB) that highly heterogeneous from the meta- meta-training datasets (CIFAR-FS, minilmageNet).. The results in Table show that our model with only z" can effectively handle these -fitt-i-iittiittionn (OOD) tasks. We observe in Figure 4 that z" actually relocates the initial 0 far from the initial parameter for these OOD tasks given at meta-test time, with larger displacements for highly heterogeneous tasks (Figure 4, right). This allows the model to either stick to, or deviate from the meta-knowledge based on the similarity between the tasks given at the meta-training and meta-test time. Moreover, the last two rows of Table 3 show that the MC approximation in Eq. (8) largely outperforms the naive approximation in Eq. (9), which suggests that z learns very large variance. We conjecture that the role of such random initialization in MAML framework is increase the effective learning rate for OOD tasks (See Appendix D for the discussion).
Meta-training
CIFAR-FS
minilmageNet
Meta-test
SVHN
CUB
MAML
45.17
65.77.0.1
Meta-SGD
46.45
65.9410.2
Axcri Cu
B. z- z-TAML (naive approx.)
1.20
67.90
vo2i Flower
merimmaeeett CUB
B. z z-TAML (MC approx.)
52.29 1-224
69.11
Teshhmn-eiLi iiba Thets
Intial Theta
Table 3: Ablation study on distributional shift.
Figure 4: T-SNE visualization of and 0 + E'z") ".""
y and z for handling task imbalance. We then examine how the two balancing variables, y and z" handle inter-task imbalance where tasks used for meta-learning have largely different number of instances. Figure 5(a) shows the performance of the baseline models and our models with each of the balancing variables, when the number of instances per task varies from 5 to 2000. We observe that our Bayesian z- 2-TAML. and Bayesian 7 T-TAM largely outperform Meta-SGD. especially by large degree when the number of instances per task is large. Figure 5(b) shows that the effectiveness of x" largely depends on the number of MC samples used in Eq. (8), demonstrating the importance of incorporating uncertainties random initializations for handling task imbalance. We further observe from Figure 5(c) that the lask-dependent learning rate multiplier Y rapidly grows the number of instances per task increases. This agrees with our intuition that larger tasks should take larger inner-gradient steps, to learn more from the given task rather than resorting
knowlegee.e
-TMM-TAMI
TAMLL
74
10
Conw1
65
Accuracy (98)
-ee-Mtta 5G0 -G- MAML
62
55
56
Gamia
6
Conw3
50
Conod
JAML (5r3)
44
TS
38
Edrapolation
Extrapolation
32
Extrapolation
26
5
100
250
500
1000
2000
25
100
250
500
1000 2000
25
1000
250
500
1000 2000
Number instances task
Number of instances task
Number instances
(a) Task size vs. Acc.
(b) MC sample size Acc.
(c) Task size vs. E(y")
Figure 5: Ablation study on task imbalance (Meta-training: CIFAR-FS, Meta-test: SVHN).
8

---

Published as a conference paper at ICLR 2020
w for handling class imbalance. w rescales the class-specifi gradients to handle class imbalance where the number of instances per class (i.e. shot) largely varies. Table 4 shows the results under the varying degree of class imbalance across the task distribution. We observe that Bayesian w-TAML outperforms Meta-SGD, especially by larger degree under higher class imbalance. Notably, Bayesian w w-TAMI. outperforms heuristic balancing method which divides each class-specific gradient by the cardinality of each class set (Meta-SGD with 1/N). The accuracy improvements over Meta-SGD in Figure 6 demonstrate that this heuristic balancing method overly emphasizes the tail classes with few training instances, thereby deteriorating the performance on classes with sufficiently large number of instances. balancing variables which allow to obtain large performance gains on all classes.
On the other hand, our Bayesian w- w-TAML learns the appropriate
0.5
Accuracy Imp mmprvvmmntt
GZ2M-55 with 1/N 1e-AMI
Meta-training/ / Meta-test
Number of instances per class
V. Mets Mets SGD (%)
CIFAR-FS/ / CIFAR-FS
10
2-
1- 50
03
Omera
MAML
73.60
71.15
67.43=0022
Meta-SGD
73.25
72.68
20.11
0.
Mcta-SGD with 1/N
71.33+0019
72.43++019
72.23+00.9
1-5
6-10
11-20
21"300
41
40 %0
Bayesian w- TAML
73.44
73.20 72.86=0.19
Number instances per class
Table 4: Ablation study on class imbalance.
Figure 6:
[[(="]
and accuracy improvements
over Meta-SGD.
5.3 MORE ABLATION STUDIES
Effectiveness of Bayesian modeling We further demonstrate the effectiveness of Bayesian mod- eling by comparing it with the deterministic version of our model (Deterministic TAML), where three balancing variables are no longer stochastic. We apply l regularization on the variables with coefficients that are equivalent to the Ki-diiergence in Eq. (5). The results in Table show that the Bayesian TAML significantly outperforms its deterministic counterpart, especially with larger gap on the OOD task (SVHN). We also see from the last two rows of the table that MC approximation in Eq. (8) is more beneficial for the OOD tasks than for the ID tasks (See Appendix D for the discussion). Figure further shows that the balancing variable y" for handling task imbalance, more sensitively reacts Bayesian TAML than Deterministic TAML, which suggests that Bayesian modeling amplifies the effect of the balancing variables.
Gel-Cundt
Meta-training: CIFAR-FS
CIFAR-FS
SVHN
-Ott-Cord
25
-Ott-Cormd
MAML
20.19
41.81
Bayos Conv2
Bayes Conva
Meta-SGD
72.71
46.45
Bayes Comid
Deterministic TAML
73.82:20:1
46.78:10.24
Bayesian TAML (Naive approx.)
73.52:10.20
47.15
Bayesian TAML (MC upprox.)
75.15
51.87 4-223
05
Table 5: Classification performance of Baycsian and Determin- istic TAML on seen and unseen dataset.
instances per
Figure 7: Ely"] vs. Bayesianness.
Dataset encoding Finally, we perform an ablation study of the proposed task encoding network. The results in Table 6 show that the proposed hierarchi- cal encoding scheme for classification dataset, with set cardinality and variance is significantly more ef- fective than simple mean-pooling methods (Zaheer al., 2017; Edwards & Storkey, 2017)
Meta-training / Meta-test CIFAR-FS CIFAR-FS
Hierarchical encoding
x
V
Mean
73.84
73.69=02
Mcan N
17.17
24.88
Mean + Var. + N
73.93+021
75.15
Table 6: Ablation study on dataset encoding schemes. N: Set cardinality.
6 CONCLUSION
We propose Bayesian TAML that learns to balance the effect of meta-learning and task-adaptive learning, to consider meta-learning under more realistic task distribution where each task and class can have varying number of instances. Specifically, we encode the dataset for each task into
We also found out that using set cardinality is more effective than utilizing higher-order statistics such as skewness kurtosis (See the Appendix C for further ablation study).
9

---

Published as a conference paper at ICLR 2020
hierarchical representations, and use it to modulate the original parameter, learning rate, and the class- specific gradients. We use a Bayesian framework to infer the posterior of these balancing variables, and propose effective variational inference framework to solve for them. Our model outperforms existing meta-learning methods when validated on imbalanced few-shot classification tasks. Further analysis of each balancing variable shows that each variable effectively handles task imbalance, class imbalance, and out-of-distribution tasks. We believe that our work makes meaningful step toward application of meta-learning to real-world problems.
Acknowledgements This work supported by Google AT Focused Research Award, Samsung Research Funding Center of Samsung Electronics (No. SRFC-IT1502
the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018R) AA5A 1059921), and the Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govermeen(MMIIT) (No.2019. 0-00075, and Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In ICLR, 2019.
Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NeurIPS, 2018.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu. Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leariig with warped gradient descent. ICLR, 2020.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. In ICLR, 2019.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark, In The 2013 international joint conference neural networks (IJCNN,, pp. 1-8. IEEE, 2013.
Takashi Kawashima Jongmin Kim Jonas Jongejan, Henry Rowley and Nick Fox-Gieg. The quick, draw! a.i. experiment. 2016. URL hwtp:hhth///. qu ckdraw wi thgoog e com.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning In NewrIPS, 2018.
Diederik Kingma and Max Welling. Auto encoding variational bayes. In ICLR, 2014.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In JCML, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint arXiv:1707.0983: 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1306.533,, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEFE, 2008
10

---

Published as a conference paper at ICLR 2020
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In CVPR, 2018.
Sachin Ravi and Alex Beatson, Amortized bayesian meta-learning. ICLR, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR, 2017.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurlPS, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22) 252, 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmente neural networks. In ICML., 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, on learning how to learn: the meta-meta.... hook. PhD thesis, Technische Universität München, 1987.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. ICLR, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In NIPS, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 178.07178..07747, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In NIPS, 2017.

---

Published as a conference paper at ICLR 2020
A EXPERIMENTAL SETUP
A.1 BASELINES
We describe the baseline models and our task-adaptive learning to balance model.
1) MAML. The Model-Agmostic Meta-Learning (MAML) model by Finn et al. (2017), which aims to learn the global initial model parameter, from which can take few gradient steps to get task-specific predictors.
2) ABML. This model interprets MAML under hierarchical Bayesian framework, but they propose to share and amortize the inference rules across both global initial parameters as well as the task-specific parameters.
3) MT-net. A gradient-based meta-learning model proposed by Lee & Choi (2018). The model obtains task-specific parameter only w.r.t. subset of the whole dimension (M-net), followed by linear transformation to learn metric space (T-net).
4) Meta-SGD. A base MAML with the learnable learning-rate vector (without any restriction on sign) sementt-wisely multiplied to each step inner-gradient (Li et al., 2017).
5) Prototypical Networks. A metric-based few-shot classification model proposed by Snell al. (2017). The model learns a metric space based on Euclidean distance between class prototypes and query embeddings.
6) Proto-MAML. A variant of MAML (Triantafillou et al., 2020) that replaces the initialization of the final fully- cunnecco layer matrix with the equivalent one of the Prototypical Networks (Snell et al., 2017). This model combines the advantage of both metric-based and gradient-based approach. We set a to 0.0005 for any-shot classification and 0.01 for multi-dataset experiments.
7) Bayesian TAMI. Our model that can adaptively balance between meta- and task-specific learners for each task and class.
A.2 ANY-SHOT CLASSIFICATION.
We describe more detailed experimental settings for any-shot classification. For MAML, ABML and MT-NET, we set the inner-gradient stepsize a to 0.5 for CIFAR-FS SOLFS//SHN, SVHN, and 0.1 for minilmageNet / CUB, after searching the range a E {0.01, 0.05, 0.1, 0.53.
CIFAR-FS and SVHN: We meta-train all models for total 50K iterations with meta-batch size set to 4. The outer learning rate is set to 0.001 for all the baselines and our models.
minilmageNet and CUB: We meta-train all models for total 80K iterations with meta-batch size set to The outer learning rate is 0.0001 for all the baselines and our models.
A.3 MULTI-DATASET CLASSIFICATION
For multi-dataset classification, construct subset of the whole collection of the Meta- Meta-Dataset (Tri- antafillou et al., 2020). We resize the images into 32 x 32 pixels. For each task, We uniformly select one dataset among Aircrafts, Quickdraw and VGG-Flower and randomly sample 10 classes from the dataset. Then we sample instances from each class with the number of instances per class ranging from to 50. The number of test instances is equally set to 15 for all classes. At meta-test time, use the three datasets plus two more out-of-distribution datasets Traffic Signs and Fashion Bashion-MNIST. For MAML, ABML and MT-NET, we set the inner-gradien stepsize a to 0.5. We set the number of classes for each task to 10, meta-batch size to 3, meta-training iterations to 60K, and outer learning rate to 0.001 for all models.
Aireraft: We split this dataset (Maji et al., 2013) into 70/15/15 classes for meta- train- ing/validation/test with 100 examples for each class.
Quick Draw: We split this dataset (Jonas Jongejan & Fox-Gieg, 2016) into 241/52/52 /52 classes for meta- traininn/validdaiionness with randomly sampled 200 examples for each class.
VGG Flower: This dataset (Nilsback Zisserman, 2008) contains 40 between 258 images for each class and we split this dataset into 71/16/1 15 classes for meta- training/validatioonnsss
12

---

Published as a conference paper at ICLR 2020
Traffic Signs: This dataset (Houben et al., 2013) has only test set consisting of 43 classes. Each class has 900 examples.
Fashion-MNIST:: We use the test set of Fashion-MNIST (Xiao et al., 2017) for meta-festing. This dataset has 10 classes with 1000 examples per class.
A.4 INFERENCE NETWORK ARCHITECTURE
We describe the network architecture the inference network that takes classification dataset as an input and generates three balancing variables as output.
Shared encoder NN, X ..... X $ xc
3 x 3 Comv2d with 10-dim and ReL.U
2 X 2 Max pool
3 X 3 Conv2d with 10-dim and ReLU
2 X 2 Max pool
fc layer with 64-dim
Statistics Pooling for each class I....,C C.
Sc layer with 4-dim (across the statistics) and ReLU
Shared encoder NN₂ $1, . SC 4
layer with 128-dim and ReLU fc layer with 32-dim
Statistics Pooling over all the class representations Sc layer with 4-dim (across the statistics) and ReLU
Generate w : s....., S - (1h) oZ,).... (MWw owo)
layer with 64-dim and ReLU fc layer to generate (KT,, oo,) for each class I,...,C. C.
Generate y": + (MT, ........ (P52, FTc)
fc layer with 64-dim and ReLU fc layer to generate (Ki oxt) for each layer = 1, 1,.... L...,. L.
Generate z": - 11, T
fc layer with 64-dim and ReLU fc layer to generate (m o7,) for the output channels
B JUSTIFICATION FOR SET-OF-SETS STRUCTURE.
Based on the previous justification of DeepSets (Zaheer et al., 2017), we can easily justify the Set-of- Sets structure proposed in the main paper well, in terms of the two-level permutation invariance properties required for any classification dataset. The main theorem of DeepSets is:
Theorem I. A function f operating iff it can decomposed as f(X) = p2 p2 P1 (x)), where pi and p2 are appropriate nonfcities.
set X E X is valid set function (i.e. permutation invariant),
See (Zaheer et al., 2017) for the proof. Here we apply the same argument twice as follows.
1. A function f operating
set of representations /s1,...,sc) (we assume each s. is an
output from a shared function g) a valid set function (i.e. permutation invariant w.r.t. the order of fs....., ,c)) }), iff it can be decomposed as f([s)....,Sc) = p2 ( (s.)) with appropriate nonfeities a and p2.
13

---

Published as a conference paper at ICLR 2020
2. A function 9 operating set of examples (Xe,1..... Xe.N ) is valid set function (i.e. permutation invariant w.r.t. the order of (xe,,....., Xx.N )) iff it can decomposed as g( (x(xx,11. 11x,,........................... Ne, ps ( (xx.i)) with appropriate nonfcities P3 and p4.
Inserting S. g g( Xe,N }) into the expression of f, we arrive at the following valid compos- ite function operating set of sets:
C Eer
f (ls (19 (.
x = P2
P1
P1 Σ (xC..)
(10)
Let F denote the composite of and (multiple) and let NN2 denote the composite of #1 and pa- Further define NN1 == P3 and NN3 == P2. Then, we have
c NN₂
N ΣnN (Xxx)) )) (11)
((xx..)....,... xI,N) (xc,)...., xx,N)x NN3
'NN,
where C is the number of classes and N is the number of examples per class. See Section A.4 for the actual encoder structure.
C ABLATION STUDY ON HIGHER-ORDER STATISTICS
While our framework does not place any restriction on selecting the statistics for set encoding, we per- form further ablation study the effectiveness of
Meta-trainine
Meta-test
CIFAR-FS
CIFAR-FS
SVHN
Mean + Var.
73.37
49.81
higher-order statistics for our specific experimen-
Mean + Var. Skew.
73.661021
50.331a21
tal setting. We see from Table 7 that the higher- order statistics such as element-wise sample skew- and kurtosis improve the performance given sample mean and diagonal covariance. However, the set cardinality seems more effective than those statistics as it could be the most direct and relevant criteria for detecting imbalances in a set.
Mean Var. + Kurt.
73.47=021
50.27402
Mean Var. +
75.15
51.87
Table 7: Further ablation study dataset encoding schemes. N: Set cardinality.
D ANALYSIS ON RANDOM INITIALIZATION FOR SOLVING OUT-OF-DISTRRBUTION TASKS
In this section, we analyze the effect of the variance zf for solving OOD tasks, that randomizes the MAML initialization parameter Define Ex JD' : [0,] 0 E.g [z'], the initial- ization parameter modulated by the posterior mean of z² Then, we measure the & distance from [0] to the two different versions of the final task-specific parameter O after taking 10 gradient steps, in order to see how the posterior variance of z affects the final task task-specific parameter O as a function of z
Lg Distance from Eg[0g]
5.40
5.24
5.20
5.18
4.61
4.50
4,40
Task-specific predictor 0J obtained with- out the variance of z², such that the expectation is taken before the inner-gradient steps.
o (EE,[z" [x]]))
4.20
(F,,,(()))
E,(0"(x'))
ID OOD
Figure 8: l2 distance between the ini- tialization and the task-specific param- eters, under different treatment expectation over q(z" (b); We use Bayesian z z-TAML and evaluate with CIFAR-FS SVHN 50-shot tasks.
[__ (x") )]: Task-specific predictor 0r obtained with the variance in the random initialization, such that the expectation is taken outside of the gradient steps. We evaluate the expectation with MC approximation, hav- ing the ensemble of the diverse task-specific predictor samples o7.... ..... 0 (we use S 50).
Figure 8 suggests that the role of the random initialization is increase the effective learning rate for the OOD tasks. We see from the left bar graph that we do not consider variance in the initialization (0 (Z,(22])) the OOD tasks deviate relatively less than the ID tasks (4.61 vs. 5818), although it.
14

---

Published as a conference paper at ICLR 2020
should deviate much considering the distributional discrepancy. On the other hand, if we incorporate the random initialization to obtain task-specific parameter (E,(0" (z )), OOD tasks can deviate further from the initialization (4.61 - 5.11). It directly results in the performance gain because the
task-specific learner can exploit more of the information in the OOD tasks.
15


Reviews:
[{"review_id": "rJep4Itotr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThis paper proposed to improve existing meta learning algorithms in the presence of task imbalance, class imbalance, and out-of-distribution tasks. Starting from the model-agnostic meta-learning (MAML) algorithm (Finn et al. 2017), to tackle task imbalance, where the number of training examples of varies across different tasks, a task-dependent learning rate decaying factor was learned to be large for large tasks and small for small tasks. In this way, the small task can benefit more from the meta-knowledge and the large task can benefit more from task-specific training. To tackle class imbalance, a class-specific scaling factor was applied to the class-specific gradient. The scaling factor was large for small class and small for large class so that different classes can be treated equally. To tackle the out-of-distribution tasks, a task-dependent variables was learned to emphasize meta-knowledge for the test task similar to training tasks. Additional model parameters are learned through variational inference. Experimental results on benchmark datasets demonstrate the proposed approach outperformed its competing alternatives. Analysis of each component confirm they work as expected.\n\nComments\n---------------\nThis paper is well motivated and clearly written. The empirical evaluation also support major claims in the paper.\n\nCan the author provide more details on the inference of the model? In the likelihood term in Eq. (7), the task specific parameters \\theta^{\\tau} was parameterized by Eq. (3), which contains K iterative gradient updates. How was the gradient w.r.t. \\theta was computed in this setting?\n\nThe task-specific learning rate decaying factor was constrained to be between 0 and 1 using the function f(). The class-specific scaling factor made use of the SoftPlus() function, for the same purpose of scaling learning rate, why do these two different options of functions were applied?\n\nFor the scaling vector of the initial parameters g(z^{\\tau}), for its zero entries, the initialization of the corresponding entries in task-specific parameter \\theta would be zero. Would it be better to apply a linear interpolation between \\theta and a randomly-initialized vector in Eq (2)?\n\nEdits after reading the author's rebuttal\n==================================\nThe author's reply well addressed my questions. After reading other reviewers' positive comments and the author's thorough reply, I decide to increase my rating to 8: Accept. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lLvaCTYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nThis paper introduces a mechanism for gradient-based meta-learning models for few-shot classification to be able to adapt to diverse tasks that are imbalanced and heterogeneous. In particular, each encountered task may have varying numbers of shots (task imbalance) and even within each task, different classes may have different numbers of shots (class imbalance). Further, test tasks might come from a different distribution than the training tasks. They propose to handle this scenario by introducing three new types of variables which control different facets of the degree and type of task adaptation, allowing to decide how much to reuse meta-learned knowledge versus new knowledge acquired from the training set of the given task.\n\nSpecifically, their newly introduced variables are: 1) the factor for learning rate decay (a scalar) for the inner-loop task adaptation optimization which allows to not deviate too much from the global initialization when insufficient data is available, 2) a class-specific learning rate (one scalar per class) that allows to tune more for under-represented classes of the training set, 3) a set of weights on the global initialization (one scalar per dimension) that can down-weigh each component if it’s not useful for the task at hand (e.g. if test tasks have significantly different statistics than training tasks did). \n\nThe values of these variables are predicted based on the training set of the task: the support set is encoded via a hierarchical variant of a set encoding (where pooling is done using higher order statistics too instead of simply averaging). The resulting encoded support set is the input to the network that produces the values for the three sets of variables discussed above. Each new variable is treated in a Bayesian fashion: a prior is defined over it (Normal(0,1)), which is updated by conditioning on the training set to form a posterior for each given task. Specifically, each of the above variables is represented by a Gaussian whose mean and variance are the learnable parameters that are produced by the network described above. \n\nExperimentally, this method outperforms others on a setting of imbalanced tasks (the shot is sampled uniformly at random from a designated range). The gain over other methods is large in particular when evaluated on out-of-distribution tasks (coming from a different dataset) and when the imbalance is large.\n\nComments (in decreasing order of importance)\n========================================\nA) The Bayesian framework helps because it offers an elegant way to use a prior. In the deterministic version, was any effort made to resemble the effect of that prior? For example, one can define a regularizer that penalizes behaviors that ignore the meta-knowledge too much (e.g. too large values for \\gamma, or for the class-specific learning rates etc). Albeit more ‘hacky’, if these regularization coefficients are tuned properly, they might result in a similar effect to that of having a prior. A fair comparison to the deterministic variant should include this.\n\nB) I think that \\gamma and z can be merged into a single set of parameters? In particular, imagine a per-dimension-of-\\theta learning rate. This would then be large for a dimension when there is a larger need for adapting that dimension of \\theta. In the case of large training sets, this can be large for all dimensions, recovering the behavior of a large \\gamma. For the case of diverse datasets, this would behave as the current z (updates a lot the dimensions of \\theta that are irrelevant for the given task due to the dataset shift).\n\nC) Meta-Dataset (https://arxiv.org/abs/1903.03096) is a recent benchmark for few-shot classification that introduces both of what is referred to here as task imbalance and class imbalance and also is comprised of heterogeneous datasets and evaluates performance on some held-out datasets too. The current state-of-the-art on it (as far as I know) is CNAPs [1] which employs a flexible adaptation mechanism on a per-task basis but is fully amortized (performs no gradient-based adaptation to each task) and makes no explicit effort to tackle imbalanced tasks as is done here. I’m curious how this method would compete in that setup. It definitely seems to be a strong candidate for that benchmark!\n\nLess important\n=============\nD) which dataset is used in Tables 4 and 5? I assume it’s Omniglot (due to the numbers being in the 90s) but it would be good to say this explicitly.\nE) In section 5.2, expressions such as x5 and x15 are used to characterize the degree of imbalance of a task. How exactly are these computed? Does x5 mean that the largest shot is 5 times larger than the smallest shot? It would be good to explicitly state this.\nF) In the Related Work section, in the Meta-learning paragraph there is a sentence that’s not accurate: “Metric-based approaches learn a shared metric space [...] such that the instances are closer to their correct prototypes than to others”. This sentence does not describe all metric based approaches. It describes Prototypical Networks (Snell et al) but not, for example, Matching Networks (Vinyals et al) nor many others that like Matching Networks perform example-based comparisons and don’t aggregate a class’ examples into a prototype.\n\nIn a nutshell\n===========\nI think this work is a useful contribution for moving towards a more realistic setting in few-shot classification. It captures some desiderata of models that can operate in more realistic settings and outperforms previous models in those scenarios. My comments above are mostly suggesting improvements and clarifications but I am inclined to recommend acceptance of this paper.\n\nReferences\n=========\n[1] Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes. Requeima et al. NeurIPS 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HygSWGg2tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a Bayesian approach for meta learning in settings were the tasks might be OOD or have imbalanced class distribution. The proposed approach has 3 task-specific balancing variables with a prior and an inference network. Using an amortized inference scheme, the model unifies the meta-learning objective loss with the lower bound of probabilistic model marginal likelihood. \n\nThe paper is well-written and well-motivated. I only have some minor comments and questions:\n- Can you add the standard errors for the results in Section 5.2 (maybe at least in the Appendix)? \n- Specifically it would be interesting to see if the results for analyzing the class imbalance variable are statistically significant; specially in light of the recent work on the effects of importance weighting in DL (see “What is the Effect of Importance Weighting in Deep Learning?” by Byrd and Lipton) which essentially question the value of importance weighting for handling class imbalance in various DL settings. \n- For the experiments in Section 5.1 can you also report the values for the three task-specific balancing variables? (Maybe in the Appendix).\n\nMinor:\n“obtains significantly improves over” -> \"significantly improves over\"\n\nOverall, I found the paper interesting and practically useful, although I believe some additions to the empirical evaluation can improve the impact of the paper. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the paper presents a compelling method that addresses an important problem. This paper should clearly be accepted, and I would suggest for it to be considered for an oral presentation.

I would encourage the authors to take into account the reviewers' suggestions (many of which were already addressed in the rebuttal period) and my own suggestion.

The main suggestion I would have in regard to improving the paper is to position it a bit more carefully in regard to prior work on Bayesian meta-learning. This is an active research field, with quite a number of papers. There are two that are especially close to the VI method that the authors are proposing: Gordon et al. and Finn et al. (2018). For example, the graphical model in Figure 2 looks nearly identical to the ones presented in these two prior papers, as does the variational inference procedure. There is nothing wrong with that, but it would be appropriate for the authors to discuss this prior work a bit more diligently -- currently the relationship to these prior works is not at all apparent from their discussion in the related work section. A more appropriate way to present this would be to begin Section 3.2 by stating that this framework follows prior work -- there is nothing wrong with building on prior work, and the significant and important contribution of this paper is no way diminished by being up-front about which parts are inspired by previous papers.
Decision: Accept (Talk)