Hierarchical Gaussian Mixture based Task Generative Model for Robust Meta-Learning
Authors: Yizhou Zhang
Jingchao Ni
Wei Cheng
Zhengzhang Chen
Liang Tong
Haifeng Chen
Venue: Submitted to ICLR 2023 (2023)

Abstract: Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task embeddings, fits mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization algorithm. Extensive experiments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.

Under review as a conference paper at ICLR 2023
HIERARCHICAL GAUSSIAN MIXTURE BASED TASK GENERATIVE MODEL FOR ROBUST
Anonymous authors
Paper under double-blind review
ABSTRACT
Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in real- ity. most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs practical solution: (1) the various sources of tasks may compose a multi- component mixture distribution, and (2) novel tasks may come from a distribu- tion that is unseen during mutta-triining In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task em- beddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation Expectation-Mxximizaul -aximization algorithm. Extensive ex- periments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.
INTRODUCTION
Training models in small data regimes is of fundamental importance. It demands a model's ability to quickly adapt to new environments and tasks. To compensate for the lack of training data for each task, meta-lcarning (a.ka. learning to Iearn) has become an essential paradigm for model training by generalizing neta-knowledge across tasks (Snell et al., 2017; Finn et al., 2017). While most existing meta-learning approaches were built upon an assumption that all training/testing tasks are sampled from the same distribution, more realistic scenario should accommodate training tasks that lie in a mixture of distributions, and testing tasks that may belong to or deviate from the learned distributions. For example, in recent medical research, global model is typically trained on the historical medical records of certain set of patients in the database (Shukla & Marlin, 2019; Wu et al., 2021). However, due to the uniqueness of individuals (e.g., gender, age, genetics). patients' data have substantial discrepancy, and the pre-trained model may demonstrate significant demo- graphic or geographical biases when testing on a new patient (Purushotham et al., 2017). This issue can be mitigated by personalized medicine approaches (Chan Ginsburg, 2011: Ni al., 2022) where each patient is regarded as a task, and the pre-trained model is fine-tuned (i.e., personalized) on support set of few records collected in a short period (e.g., a few weeks) from every patient for adaptation. In this case, the training tasks (i.e., patients) could be sampled from a mixture of distributions (e.g., different age groups), and a testing task may or may not belong any of the observed groups. As such, meta-training strategy that is able to fit mixture of task distributions and identify novel tasks is desirable for making meta-learning a practical solution.
One way to tackle the mixture distributions of tasks is to tailor the transferable knowledge to each task by learning a task-specific representation (Oreshkin et al., 2018; Vuorio et al., 2018; Lee & Choi, 2018), but discussed in (Yao et al., 2019a), the over-customized knowledge prevents its generalization among closely related tasks (e.g., tasks from the same distribution). The more recent methods try to balance the generalization and customization of the meta-know keeal-knowledge by promoting local generalization either among a cluster of related tasks (Yao et al., 2019a), or within neighbor- hood of a meta- weta-knowledde graph of tasks (Yao al., 2019b). Neither of them explicitly learns the

---

Under review as a conference paper at ICLR 2023
underlying distribution from which the tasks are generated, rendering them infeasible for detecting novel tasks that are distiiibbibtt.nnn.. However, detecting novel tasks is crucial in high-stake do- mains, such medicine and finance, which provides users (e.g. physicians) confidence whether to trust the results of testing task or not, and facilitates the downstream decision-making
In (Lee et al., 2019a), task-specific tuning variable was introduced to modulate the initial param- eters learned by MAML (Finn et al., 2017), so that the impacts of the meta--nowledde on different tasks are adjusted differently, e.g., novel tasks receive less impact than known tasks do. Whereas, this method focuses on improving model performance on different tasks (either known or novel), but neglects the critical mission of detecting which tasks are novel. In practice, providing an unreliable accuracy on novel task, without differentiating it from other tasks may be meaningless and risky.
Since the aforementioned methods cannot simultancously handle the mixture distribution of tasks and novel tasks, a practical solution is in demand. In this work, we consider tasks as instances, and demonstrate the dual problem of modeling the mixture of task distributions and detecting novel tasks are two sides of the same coin, i.e., density estimation on task instances. To this end, we propose new Hierarchical Gaussian Mixture based Task Generative Model (HTGM) to explicitly model the generative process of task instances. Our contributions are summarized as follows.
For the first time, the widely used empirical process of generating task is theoretically extended to and specified by a hicrarchy of Gaussian mixture (GM) distributions. HTGM generates a task embedding from a task-level GM, and uses it to define the ciskccondtiooee mixture probabilities for a class-level GM, from which samples are drawn, for instantiating the generated task. To allow realistic classes per task, new Gibbs distribution is proposed to underlie the class-level GM.
HTGM is an encoder-agnostie framework, thus is flexible to different domains. It inherits metric- based meta-learning methods, and only introduces a small overhead to an encoder for parameter- izing its distributions, thus is efficient, and enables large-scale backbone networks. The model pa- rameters are learned end-to-end by maximum likelihood estimation via principled Expectation- Maximization (EM) algorithm. The bounds of our likelihood function is theoretically analyzed.
In the experiments, we evaluated HTGM on benchmark image datasets for validating its ability to take advantage of large backbone networks, its effectiveness in modeling the mixture distribution of tasks, and its usefulness in identifying novel tasks. The results demonstrate HTGM outperforms the state-of of-the-art (SOTA) baselines with sigmificant improvements in most cases.
2 RELATED WORK
To the best of our knowledge, this is the first work to explicitly model the generative process of task instances from a mixture of distributions for meta-learning with novel task detection. Meta-lcarning aims to handle the few-shot learning problem, which derives memory-based (Mishra et al., 2018), optimization-bai based (Finn et al., 2017; Li et al., 2017), and metric-based methods (Vinyals et al., 2016; Snell et al., 2017), which often consider an artificial scenario where training/test tasks sampled from the same distribution. To enable more varying tasks, task-adaptive methods facilitates the customization of meta-knowledde by learning task-specific parameters (Rusu et al., 2018; Lee & Choi, 2018). temperature scaling parameters (Oreshkin et al., 2018). and task-specific modulation on model initialization (Vuorio et al., 2018; Yao et al., 2019a;b; Lee et al., 2019a). Among them, there are methods tackling the mixture distribution of tasks by clustering tasks (Yao et al., 2019a) or learning task graphs (Yao et al., 2019b), and method relocating the initial parameters for different tasks so that they use the meta-knowledge differently (Lee et al., 2019a) As discussed before, none of these methods jointly handle the mixture of task distributions and the detection of novel tasks.
Our model is built upon metric-based methods, and learns task embeddings for modeling task dis- tributions. Achille et al. (2019) also proposed to learn embeddings for tasks and introduced meta- learning method, but not for few-shot learning. Its embeddings (rather than episode-wise sampling), and the meta-lcarning framework is for model selection. The model in (Yao al., 2019a) has an augmented encoder for task embedding, but it does not explicilly model task generation, and is not designed for novel task detection (empirical comparison in 4.1).
from a pre-specified set of tasks
Conventional novelty detection aims to identify and reject samples from unseen classes (Cheng & Vasconcelos, 2021). It relates to open-set recognition (Vaze et al., 2022), which aims to simulta- neously identify unknown samples and classify samples from known classes. Out-of-distribution
2

---

Under review as a conference paper at ICLR 2023
(OOD) detection (Liang et al., 2018; Liu et al., 2020) can be seen as special case of novelty de- tection where novel samples are from other problem domains or datasets, thus are considered to be easier to detect than novelties (Cheng & Vasconcelos, 2021). These methods are for large-scale training. In contrast, we want to detect novel tasks, which is a new problem in the small data regime.
Hierarchical Gaussian Mixture (HGM) model has appeared in some traditional works (Goldberger & Roweis, 2005; Olech & Paradowski, 2016; Athey et al., 2019) for hierarchical clustering by ap- plying GM agglomeratively or divisively, which do not pre-train models for meta-learning, and is remarkably different from the topic in this paper. The differences are elaborated in Appendix B.I. Moreover, we discuss the relevant multi-task learning methods with task grouping in Appendix B.2.
HIERARCHICAL GAUSSIAN MIXTURE BASED TASK GENERATIVE MODEL
Meta-learning methods typically use episodic learning strategy, where the meta-training set D consists of batch of episodes. Each episode samples task T from a distribution p(T). Task r has a support set D3 = [(x3, )² for training, and a query set D3 = {(x], v ) for testing, where n, is small number to denote few training samples. In particular, in a commonly used N-way K-shot Q-query task (Vinyals et al., 2016), D. and D' contain N classes, with K and Q samples per class respectively, i.e., n2 = NK and n = NQQ
Let (x;) - y be a base model (* denotes s or q), and [e(:; be the adapted model on D.. The training objective on is to minimize the average test error of the adapted model, i.e., Fx DD (y). i fo (x"; D))). where E(.,)) is a loss function (e.g., cross-entropy loss), and the meta- training process aims find the parameter O that minimizes this error over all episodes in Du, Then, fe is evaluated on every episode of meta-test set D that samples a task from the same distribution P(T). Usually, P(T) a simple distribution (Finn et al., 2017; Lee al., 2019a). In this work, p(r) is generalized to a mixture distribution consisting of multiple components pı (T), .... Pr (T), and test episode may sample a task either in or out of any component of p(T). As such, given the training tasks in DD our goal is to estimate the underlying density of p(r), so that once test task is given, we can (1) identify if it is a novel task, and (2) adapt fe to it with optimal accuracy.
Specifically, the base model Jo can be written as a combination of an encoder go, and a predictor i.c. fo(x;) = he, (go_ (x,) (x)) (Tian et al., 2020). In this work, we focus on metric-based non- parametrie learner, i.e., 0p- Ø (e.g.. prototypical networks (Snell et al., 2017)), not only because metric-based classifiers were confirmed as more effective than probabilistic classifiers for novelty detection (Jeong al., 2021), but also for its better training efficiency that fits large-scale backbone networks than the costly nested-loop training of optimization-banl methods (Tian et al., 2020).
Formally, our goal is to find the model parameter O that maximizes the likelihood of observing task т. In other words, let fo(x;) ea E Rdd be the sample embedding, we want to maximize the likelihood of the joint distribution po (o v the observed data in D, [D;, D4). We consider each task as an instance, with a representation v. d in the embedding space (the method to infer V, is described in Sec. 3.2). To model the unobserved mixture component, we associate every task with latent variable 2, to indicate to which component it belongs. Suppose there are possible components, and let n = Tls +ng be the total number of samples in D,. the log-likelihood to maximize can be written by hierarchically factorizing on yf and marginalizing out v,, and 2,.
((D+;)) n
log (p =
log [pa 7 los (p("))
log [po(e|lui1/ / p(uilv- (.vllw))(vv:ddd..
(I)
[/, p(uilve)) Σ plv. ler (x)))( ]or]
log po(e (ut
where po pe (v]) specifies the probability of sampling et from the yf -th class, p(y, (v,) is the prob- ability of sampling the -th class for task т, and p(v)lz+) indicates the probability of generating a
task from the 2, mixture component. p(zz) is prior the 2, component. Hence, Eq. (1) implies a generative process of task т: 2, - V, + v, + eç. Next, we define each of the afore- mentioned distributions and propose our HTGM method.
3

---

Under review as a conference paper at ICLR 2023
3.1 MODEL SPECIFICATION AND PARAMETERIZATIO
In Eq. (I), the class-conditional distribution pa pa(ee [V ). the task-conditioaal distribution p(y, (V>). and the mixture distribution of tasks defined by (p(v)2z7), [r),)(z>) are not not specified. To make Eq. (1) optimizable, we introduce our HTGM that models the generative process of tasks. Because Dj and D2 follow the same distribution, in the following. we ignore the superscript * for simplicity.
Class-Conee Conditional Distribution. First, similar to (Lee et al., 2018; 2019b), we use Gaussian distri- bution to model the embeddings e,'s in every class. Let yy and Ey. be the mean and variance of the distribution of the y: class, then (e(el)y)) = (ee ). In fact, the samples in all of the classes of task т comprise Gaussian mixture distribution, where p(y:) the mixture probability of the y:-th class. In Eq. (1), p(y:) is factorized to be task-specific, i.e., ()), which resorts to another mixture distribution p(v>) of tasks, and establishes structure of hierarchical mixture.
Cons-Conditional Distribution. A straightforward definition of p(y Vv) )) is the density at pl in Gaussian distribution with v, as the mean, where p is the mean (or prototype) of the y,-th class. However, doing so exposes two problems: (1) the density function of Gaussian distribution is log- concave with one global maximum. Given the mean and variance, maximizing its oo-liikelhood tends to collapse the prototypes n 's of all classes in 7, making them indistinguishable and impair- ing classification; (2) given Vr. this method tends to sample classes with small Dv, (mi,), ). where Dv, (.) measures the Mahalanobis distance between a data point and the Gaussian distribution cen- tered at V,- However, in most of the existing works, classes are often uniformly sampled from a domain without any prior distances (Finn et al., 2017). Fitting the distance function with such "uniform" classes naively leads to an iol-posed learning problem with degenerated solutions. In light (parameterized) density function with at least N
of these issues, seek to define p(ya vV)) global optimums so that can distinguish the N different class prototypes of N-way tasks. The N equal (global) optimums also allow it to fit N classes uniformly sampled from a domain. To this end, let f be the surrogate embedding of the k-th class, we propose a Gibbs distribution π T(rg Vr, w) defined by v, and trainable parameters w with an energy function. Then we write p(y; k(v,) as
exp [- Eu v=) (M = klv+) = x =HilV,, w)
Jng exp Few (
(2)
where Ew (ME v.) = min (1 W W is our energy function, and the denominator in Eq (2) is a normalizing constant (with respect to (e))). a.k.a. the partition function in an energy- based model (EBM) (LeCun et al., 2006). w ( WN are trainable parameters. with W, E Raxdd Given w and Vz, Eq. (2) has N global maximums at μ WIV,, .... u = WNV,- More interpretations of the proposed task-conditional distribution can be found in Appendix B.3.
Mixture Distribution of Tasks. In Eq. (1), the task distribution p(v,) is factorized as mixture of p( p(vr|z, = 1), .... p(V+/2, r). weighted by their respective mixture probability p(zz). Thus we specify p(v-) as Gaussian mixture distribution, and introduce n², and Σ as the mean and variance for each component, i.e., p(v, /2,) = (vvilu', E. ). Then the generation of v, involves two steps: (1) draw latent variable 2, from categorical distribution on jp(2, 1), ..... p(2, r)], which can be Uniform(r). and (2) draw v, from N N(r, Σ (Bishop, 2006).
As such, our HTGM generative process of an N-way K-shot Q-query task T can be summarized as
1. Draw a latent task variable z, ~ Categorical(p((z> = 1)....p( p(z, = =)]))
Draw a task embedding v- ~ NN(',, )
3. For A = 1, 1,..., N:
(a) Draw class prototype un (Ma VT, from the proposed Gibbs distribution in Eq. (2)
(b) For i = L,..., Q:
i. Set y/4 = kt draw a sample embedding e, ~ (eellp )
ii Allocate (e., to the support set D; if < K; else allocate (ei. to the query set DJ
To reduce complexity, we investigate the feasibility of using isotropic Gaussian with tied variance, ie., E = ΣN = I²I, for class distributions, which turned out to be efficient in our experi- ments. Here, I is the identity matrix, is a hyperparameter. Tied variance is also commonly used trick in Gaussian discriminate analysis (GDA) for generative classifiers (Lee et al., 2018). For task distributions, the variances Σ E!.... Σ) can be automatically inferred by our algorithm in Sec. 3.2.
4

---

Under review as a conference paper at ICLR 2023
(a)
Support set (X/
[e
pass-poooing
Class-polin
means
Task Pess-pooling
V,
p(z,)
Task-level GMM
p(v,le,)
Mi....
fe
Class-pooling
x (4S,, Iv. ())
Gibbs distribution
W. Ww
Class means
Query set (x, the
1of122 1 p(eil)))
Training process
Support (N li-
[oft Jems
Training task embeddings
Novelty?
Testing
Closs pooling
Class means
fo
Taok g0ogoooo
Data flow
Wy
Pooling
Adaptation
Sampling
Veteo Wy
Part of training loss
Clessification
Adapted class
Query set
Model parameters db
Figure I:
An illustration of HTGM on its (a) the training process, and (b) the testing process. In (a), (2(3) are the three parts of the training loss in Eq. (3). In (b), the training task embeddings contain the embeddings of all training tasks, i.e. the outputs of the task-pooling in (a).
Finally, substituting po(e, (b)) = (e, II)), Pw (w v,) = π (HE Vr, w) (y: = k), p(v)l=+) N(v-lm', Σi,) and p(2+) Uniform(r) in Eq. (1), whose probabilities are specified and param- eterized, we get our HTGM induced loss GHTGM (D;; 0, w). The class means u task means pl, and variances E are inferred in the E-step of our EM algorithm (details are in Sec. 3.2 and A.4).
3.2 MODEL OPTIMIZATION
is hard to directly optimize (ITGG (Dr; 0, w) because the exact posterior inference is intractable (due the integration over v,). To solve it, resort to variational methods, and introduce approximated posterior 9 (v= which is defined by an inference network é, and implies we want to infer v, from its observed support set D,. The query set DJ is not included because is unavailable during model testing. Then we propose to maximize a lower-bound of Eq. (1), which is derived as (the details are in Appendix A.1)
lITOM (inoM(Dr;0r Dr;0,w) fUTOM ELDO (DA
Σ log wo (e:l)))
(3)
By By-naulv.11g) log + log Σ p(v, |z- )p (=)=+)))((
H (goovv (t))
where H (Vg((v-IDP) = - fy. 2o (v- (D:) log (v, DDv is the entropy function. Similar to VAE (Kingma & Welling, 2013). Eq. (3) estimates the expectation (in the second term) by sampling v, from 4 (v= instead of the integration in Eq. (I), hence facilitates computation. Next, we elaborate on the inference network, the challenges of maximizing Eq. (3), and our workarounds.
Inference Network. Similar to VAE, o (v, (D?) is defined a Gaussian distribution. A where f is the output of the inference network, which approximates ptz, in Step 2 of the generative process, and ō is a hyperparameter for the corresponding variance. As illustrated by Fig. 1(a), the inference network is built upon the base model f'(() with two non-parametric aggregation (i.e. mean pooling) functions, thus & = 0. The first function aggregates class-wise embeddings to prototypes ue, 's, similar to prototypical networks (Snell et al., 2017). Differently, the second aggregates all the reparameterization trick (Kingma & Welling,
prototypes to u².- During model training, 2013) to sample v, from N It is noteworthy that H (go (D:)) in Eq. (3) becomes constant because 22 is constant.
5

---

Under review as a conference paper at ICLR 2023
Challenge 1: Trivial Solution. In Eq. (3), since the first term log pe (e.lv, 2n lle, H 11
(constants are ignored) only penalizes the distance between a sample e, and its own class mean H (i.e., intra-class distances) without considering inter-class relationships, different class means /i u in task т could collide, drawing all sample embeddings to the same spot. To avoid such trivial solution and improve the stability of optimization, we apply negative sampling (Mikolov et al., 2013)
(Do Ue, 0, w) = -oo log Ea [exp (- 2,
[xxp(-222a lle - (3,
where e is a negative sample embedding from any class in the support set, and n is the mean of the positive class. In practice, we found it is beneficial to integrate lacg with our likelihood GITGМ in Eq. (1) during training, i.e. (HTGM - Di= loeg. Correspondingly, from Eq. (3) we have
(D++; w) = no (,)
lmn O, w)
(5)
11
which does not only serve as a robust training loss, but also helps solve the next challenge.
The Partition Function in Eq. (2). The second term Pw (w vy in Eq. (3) involves computing the partition function in Eq. (2) (i..,, the denominator), which is intractable because of the integration over all possible g s. To solve it, we propose upper bound of the partition function J/x. exp E v.)] d S /V22d11dd (the derivation in Appendix A.2), which
Challenge 2:
is a constant with a specific N. By replacing the partition function in Eq. (2) with N /22 we got lower bound of pw (yel v.). which in turn relaxes the lower bound in Eq. (3). The following theorem (the proof is in Appendix A.3) states the tightness of the relaxed bound is controllable.
Theorem 1. Among the N global maximums W ... W N of Eg. (2), let WAV, and Wiv. (1 $ h, & N) be the pair with the smallest Euclidean distance D(W W,,,), we have N 1
lim W.W.v.Wir-)-oo
exp Eu v. d.)
(6)
This theorem indicates the partition function approximates N / when all pairs of the global maximums are far apart. It is noteworthy that during training (i.e., maximizing the likelihood) fit WiV...., v, to the different class prototypes μis ... Sy in N-way tasks. Because Eneg in Eq. (4) tends to maximize the distances between different prototypes through the negative samples, maximizing the joint loss l in Eq. (5) tends to separate W1V, ... W vr, thus tighten the relaxed bound after using N 22d - according to Theorem 1. This is another benefit of negative sampling.
Optimization via pnpeectttioo-Mxxiizzaiiiio to estimate the mixture distribution p(zz). Similar to optimizing Gaussian mixture models, we alternately infer p(z,) and solve the model parameters (0, w through an Exxpecation-Maxxiizzan algorithm. In E-step, we infer p(zr) when fixing model parameters. In M-step, when fixing p(27), {0, w can be efficiently solved by optimizing Eq. (5) with stochastic gradient descent (SGD). The formula to infer p(2,) and the detailed training algorithm of HTGM can be found in Appendix A.4.
In the third term of 4ITGM ELBO in Eq. (3), we need
3.3 MODEL ADAPTATION
Fig. 1(b) illustrates the adaptation process of HTGM. Given a new N-way task 7' from the meta- test set Dir its support set DJ, is fed to the inference network to generate (1) class prototypes μi. 4x (similar to prototypical networks), and (2) distribution 9s IDe.,, from which we draw the average task embedding v, , Recall that the inference network is the base model fo(-) with class-pooling and task- pooling layers, illustrated in Fig. 1(b), and & 0. Then v,r is projected to Wyv..... v, which represent the N optimal choices of class prototypes for task r' learned by the Gibbs distribution in Eq. (2) from the training tasks. They are used to adapt ...... 2 so that the adapted prototypes are drawn towards the closest classes from the mixture component that task 7' belongs to. The adaptation is performed by selecting the closest optimum for each prototype, i.e., n = aye; + (1 - W V. where I* = arg min D D(H Wiv,') using Euclidean distance D(.,)) -) and a is a hyperparameter. Finally. we (1) assess if r' is novelty pre-fitted GMM on the embeddings v,'s of the training
by computing the likelihood of v. tasks in D"r, and (2) perform classification on each sample x in the query set DJ, using the adapted exp(- (DD()o(((())))))))))))) EN exp (PP)((((())))))
prototypes by p(u, j'(x

---

Under review as a conference paper at ICLR 2023
Setting
Model
Bird
Texture
Aireraft
Fungi
Average
TAML
55.77
3..7811.30
1..56=1.37
41.00=1.550
44.28
MAML
53.9411445
31.6611.31
51.37=1.38
1..12=1..6
44.77
Meta-SGD
55.58+1.43
32.3811.32
52.99=1.36
41.74=1.344
45.67
MUMOMAMI.
56.82+1..49
33.81+1.36
13.14+1.39
42.22++.440
46.50
5-way
HSML
60.98+1.50
35.01=11.36
57.38+1.40
1..02++1.39
49.35
1-shot
ARML
62.33+1.47
35.65++1.40
58.56+11.41
44.82++1.38
50.34
ProtoNet
61.54:11227
38.84.11.42
73.42.11.23
46.52.11.42
55.08
MetaOptNett
62.80.11.29
44.30.11.45
68.6411.22
47.04.11.38
55.70
ProtoNet-Aug
65.04.11.29
44.68.11.43
70.4411.32
1..30.11.40
57.37
NCA
1.25
40.98::1.44
68.70+1.26
1..36=1134
54.66
FEATS
62.60+1.31
44.12=1149
68.86#1.28
1..92=+1.34
55,88
HTGM (ours)
70.12:11.28
47.76.11.49
1.25
52.06.11.41
61.37
TAML
69.50:L0.75
45.1T
63.92.10774
50.99110.87
57.88
MAML
68.52410.79
44.56:10.68
1..78
51.85
57.78
Meta-SGD
67.87+10.74
45.49-10.68
66.84-10.70
52.51
58.18
MUMOMAML
70.49
45.8910.69
67.31
53.96:10.82
59.41
5-way
HSML
71.6
48.0810.69
73.49-0.68
56.32:10.80
62.39
5-shot
ARML
73.34:10.70
49.67110.7
74.88-0.64
57.55
63.86
ProtoNet
78.88+0.72
57.93+0075
86.42+0.557
62.52+0079
71.44
MetaOptNet
81.66+0071
61.97+0.78
84.03+0.56
63.80=+0.81
72.87
ProtoNct-Aug
80.62:+0.71
58.30+0.77
87.05+0.53
63.62-+0.81
72.39
NCA
79.16++0.75
58.69+10.76
85.27+0.53
61.68+0.80
71.20
FEATS
78.37100.72
$7.02.0.73
85.55
61.56.10.80
70.63
HTGM (ours)
82.27+00.74
60.67
08.48
65.70:10.79
74.28
Table 1: Results (accuracy commiiddnc
confidence) of the compared methods on Plain-Multi dataset.
4 EXPERIMENTS
In this section, we evaluate HTGM's effectiveness on few-shot classification and novel task detection on benchmark datasets, and compare with SOTA methods.
Datasets. The first is the Plain-Multi benchmark proposed in (Yao et al., 2019a). It includes four finegrained image classification datasets, i.e., CUB-200-2011 (Bird), Deseribable Textures Dataset (Texture), FGVC of Aircraft (Aircraft), and FGVCX-Fung (Fungi). In each episode, task samples classes from one of the four datasets, so that different tasks are from a mixture of the four domains. The second is the Art-Multi benchmark from (Yao et al., 2019b), whose distribution is more complex than Plain-Multi. Similar to (Jerfel et al., 2019), each image in Plain-Multi was applied with two filters, i.e., blur filter and pencil filter, respectively, to simulate changing distribution of few-shot tasks. Afterward, together with the original four datasets, a total of 12 datasets comprise Art-Multi and each task is sampled from one of them. Both benchmarks were divided into the meta-trainin meta-validation, and meta-test sets by following their corresponding papers.
Baselines. We compare HTGM with the most relevant SOTA methods meta-learning, including (1) optimization-bate methods: MAML (Finn et al., 2017) and Meta-SGD (Li et al., 2017) learn globally shared initialization among tasks. MUMOMAML (Vuorio et al., 2018) is task-specific method. TAML (Lee et al., 2019a) handles imbalanced tasks. HSML (Yao et al., 2019a) and ARML (Yao al., 2019b) learn locally shared initial parameters in clusters of tasks and neighborhoods of meta-graphe of tasks, respectively; and (2) Metric- methods: ProtoNet (Snell et al., 2017) learns prototypes with distance- basedd classifier. MetaOptNet (Lee et al., 2019c 9c) uses an SVM classifier with kernel metrics. ProtoNet-Aug (Su et al., 2020), FEATS (Ye et al., 2020) and NCA (Laenen & Bertinetto, 2021) were built upon ProtoNet by augmenting images (e.g., rotation, jigsaw), adding prototype aggregator (e.g.. Transformer), and using contrastive training loss, (instead of prototype- based loss), respectively. The detailed setup of these methods is deferred to Appendix C.I.
Implementation. Following (Tian et al., 2020), optimization-basee baselines use the standard four- block convolutional layers as the base learner, and metric- based methods use ResNet- 12 as the base learner. The output dimension of these networks is 640 (MetaOptNet paper). In our experiments, we observed the optimization- when using ResNet-12 12, indicating their limitation in using large backbone networks. To test them
16000 as in its methods have oft-oo-memory issues

---

Under review as a conference paper at ICLR 2023
Setting
Model
Original
Blur
Pencil
Average
TAML
1.32
40.02:1
3..11=1.34
39.11
MAML
1..70=1.35
4..53+1.38
36.71
39,98
Meta-SGD
1..21
1..36:11.39
37.21
41.26
MUMOMAMI.
45.63+
41.59+
39.24+1.336
42.15
HSMI.
1.39
44.43-+1.34
41.44+11.34
44.60
5-way,I-ae I-shot
ARML
1.36
1..62++.33
39.78+11.34
42.69
ProtoNet
55.23.1.311
51.70.1..42
49.22.11.44
52.05
MetaOpINet
16.10.1..35
12.33.11.43
49.08.11.45
52.50
ProtoNet-Aug
1.36
55.00.1..40
49.73.11.53
54.12
NCA
1.35
10.80+1.49
47.99=1.45
51.64
FEATS
1.33
10.90=
47.96=1.448
51.07
HTGM (ours)
1..1
58.80-1..42
53.23
57.74
TAML
58.5410.73
55.23:10.75
49.23:10.75
54.33
MAML
58.30:10.74
55.71110744
49.59:10.73
54.50
Meta-SGD
57.82+10.72
55.54-10.73
50.24:10.72
54.53
MUMOMAML
58.60:0.755
56.29:10.72
51.15=0.73
55,35
HSML
60.6310073
57.91
53.93:10.72
57.49
5-way,1 -shot
ARML
61.78:10.44
58.73:00.75
55.27=1073
58.59
ProtoNct
71.34+0.73
67.28+0.755
64.32+00.76
67.65
MetaOptNet
72.33++0.72
68.90+0.78
63.89+0.71
68.37
ProtoNct-Aug
72.87=00771
70.50+0.72
63.98+0073
68.78
NCA
72.44+0.72
67.33+0..1
62.98-+0.78
67.58
FEATS
71.9910.71
67.5411.72
63.09.10.76
67.54
HTGM (ours)
7..67=00.70
7..24
65.22=0077
70.37
Table 2: Results (accuracy confidence) of the compared methods on Art-Multi dataset.
on ResNet-12, we followed the ANIL method (Raghu et al., 2020) by pre-training ResNet- 12 via ProtoNet, freezing the encoder, and fine-tuning the last fully-connected layer. In this case, HSML and ARML cannot work properly as they require joint training of the encoder and other layers. The details are in Appendix D.5. For training, Adam optimizer was used. Each batch contains tasks. Each model trained with 20000 episodes. The learning rate of metric-based methods is 1e The learning rates for inner- and outer-loops for optimization-based methods are 1e and 1e 14- The weight decay 1 1 For HTGM, we tasks. The number of mixture components r varies w.r.. different datasets, and was grid-searched within [2, 4, 16, 32). All hyperparameters were set according the meta-validation sets.
1.0, 0.1, 0.5 (0.9) for 1-shot (5-shot)
4.1 EXPERIMENTAL RESULTS
Few-shot classification. Following (Tian et al., 2020), we report the mean accuracy and 95% con- fidence interval of 1000 random tasks with 5-way 1-shot/5-shot, 5/25-query tests. Following (Yao et al., 2019b), we report the accuracy of each domain (Bird, Texture, Aircraft and Fungi) and the overall average accuracy for Plain-Multi, and report the accuracy of each image filtering strategy and the overall average accuracy for Art-Multi.
Table and summarize the results. From the tables, have several observations. First, metric- based methods generally outperform optimization-base methods. This is because of the efficiency of metric-based methods, enabling them to larger backbone network, which is consistent with the results in (Tian et al., 2020). Built upon the metric- based method, HTGM only introduces a few Distribution-relatte parameters and thus has the flexibility to scale with the encoder size. Second, baselines designed for dealing with mixture distributions of tasks, i.e., HSML and ARMI,, outper- form their counterparts without such design, demonstrating the importance to consider mixture task distribution in practice. Finally, HTGM outperforms the SOTA baselines in most cases by large margins, suggesting its effectiveness modeling the generative process of task instances.
Novel task detection. We also evaluate HTGM on the task of detecting novel N-way-K-shh tasks (N = 5, = 1) that are drawn out of the training task distributions. To this end, we train each comapred model in the Original domain in Art-Multi dataset, and test the model on tasks drawn from either Original domain (i.e., known tasks), or (Blur, Pencil} domains (i.e., novel tasks), and evaluate if the model tell whether testing task is known novel.
8

---

Under review as a conference paper at ICLR 2023
(a)
(b) MetaOptNet
(c) ProtoNet-Aug
(d) HTGM
Newi
Rovel TTakk
Nove Teaks
Nornaltee
Horraloed Liabaed
Morraltis
lomaled Lialhees
Figure 2: The frequency of tasks w.r.t. the normalized likelihood for (a) HSML (b) MetaOptNet (c) ProtoNet-Aug (d) HTGM. The x-axis ranges vary as only 95% tasks with top scores were preserved.
For comparison, since none of the baselines detects novel tasks, we adapt them as follows. For metric-based methods, since they use fixed encoder for all training/testing tasks, we averaged the sample embeddings in each task to represent the task. Then a separate GMM model was built upon the training task embeddings, and its likelihood was adapted to score the novelty of testing tasks (some details of setup are in Appendix C.2. However, optimization-based models perform gradient descent on the support set of each task, leading to varying encoders per task. As such, sample not comparable, and cannot obtain task embeddings in the same
embeddings of different tasks way as before. Among them, only HSML has an augmented task-level encoder for task embedding, allowing to include it for comparison. For fair comparison, HTGM also trains GMM on its task embeddings for detecting novel tasks. Moreover, two HTGM variants were included for ablation analysis understand some design choices: (1) HTGM-Gaussian replaces the Gibbs distribution in Eq. (2) with Gaussian distribution; (2) HTGM w/o GMM removes the task-level GM, i.e., the third term in Eq. (3). The classification results of the ablation variants are in Appendix D.4, Following (Cheng & Vasconcelos, 2021; Vaze et al., 2022; Sharma et al., 2021), we report Area Under ROC (AUROC), Average Precision (AP), and Max-Fl for performance evaluation.
Table 3 summarize the results, from which we observe HTGM outperforms all base- lines over all evaluation metrics, indicat-
Model
AUROC
AP
Max-FI
HSML
55.96
37.94
50.17
ing the superior quality of task embeddings learned by our model. The embeddings follow the specified mixture distribution of tasks p(v,) as described in Sec. 3.1. which fits the mixture data well hence allowing detect novel tasks that close to the boundary. Since the baselines learn em- beddings without explicit constraint, they even don't fit the post-hoc GMM very well. Moreover, HTGM outperforms HTGM w/o GMM, which is even worse than some other baselines. This further validates the neces- sity to introduce the regularization of task-level mixture distribution p(vz). Also, the drops of AUROC and AP of HTGM-Gaussianl distribution for the task-conditional distribution in Eq. (2). Similar to (Vaze et al., 2022), in Fig. 2, we visualized the normalized likelihood histogram of known and novel tasks for HSML, MetaOpt- Net (the best baseline), ProtoNet-Aug (the near-best baseline), and HTGM. The figures indicate the likelihoods (i.e., novelty scores) of HTGM are more distinguishable for known and novel tasks than the baselines. We also analyzed the hyperparameters of HTGM, which are in D.I, D.2, D.3.
ProtoNet
65.17
41.51
56.07
MetaOptNet
72.71
63.77
58.33
NCA
66.28
$1.45
52.74
ProtoNet-Aug
72.67
57.93
59.07
FEATS
59.35
42.57
49.31
HTGM w/o GMM
70.24
62.45
57.75
-ruM-Gaussian
74.06
66.18
60.62
HTGM
75.66
68.03
60.51
Table 3: Comparison between HTGM and its variants and the applicable baselines on novel task detection.
Gaussian demonstrate the importance of our unique design of the Gibbs
CONCLUSION
In this paper, propose novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM models the generative process of task instances, and performs maximum likeli- hood estimation to learn task embeddings. which can help adjust prototypes acquired by the feature extractor and thus achieve better performance. Moreover, by explicitly modeling the mixture dis- tribution of tasks in the embedding space, HTGM can effectively detect the tasks that are drawn from distributions unseen in the meta-training stage. The extensive experimental results indicate the advantage of the proposed method on both few-shot classification and novel task detection.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In ICCV, pp. 6430-6439, 2019.
Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as layer in neural networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 136-145. PMLR, 2017.
Thomas L Athey, Benjamin D Pedigo, Tingshan Liu, and Joshua Vogelstein. Autogmm: Auto- matic and hierarchical gaussian mixture modeling in python. arXiv preprint 19Xi::999.00688 2019.
M. Christopher Bishop. Pattern recognition and machine learning. Springer, 2006.
Isaac S Chan and Geoffrey Ginsburg. Personalized medicine: progress and promise. Annual review of genomics and human genetics, 12:217-244, 2011.
Jiacheng Cheng and Nuno Vasconcelos. Learning deep classifiers consistent with fine-grained nov- elty detection. In CVPR, pp. 1664-1673, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In ICML, pp. 1126-1135. PMLR, 2017.
Jacob Goldberger and Sam T Roweis. Hierarchical clustering of mixture model. In NeurlPS, pp- 505-512, 2005.
Laurent Jacob, Jean-philippe Vert, and Francis Bach. Clustered multi-task learning: A convex formulation. In NIPS, 2008.
Minki Jeong, Seokeon Choi, and Changick Kim. Few-shot open-set recognition by transformation consistency, In CVPR, pp. 12566-12575, 2021.
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. NeurlPS, 32, 2019.
Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share multi-task feature learning. In ICML, 2011.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.61 14, 2013.
Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi-task learning. In ICMI., pp. 1723-1730, 2012.
Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning. NewrlPS, 34:24581-24592, 2021.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 100), 2006.
Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distriiutiio tasks. arXiv preprint arXiv: /29/7, 2019a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distributiio samples and adversarial attacks. NeurlPS, 31, 2018.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICML, pp. 3763-3772. PMLR, 2019b.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pp. 10657-10665, 2019c.
Yoonho Lee and Seungjin Choi. Gradient- based meta-learning with learned layerwise metric and subspace. In ICML. pp. 2927-2936. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv:1007.09835 2017.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detec- tion in neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detec- tion. pp. 21 21464-21475, 2020.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In NeurIPS, 2013.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In ICLR, 2018.
Jingchao Ni, Wei Cheng. Zhengzhang Chen, Takayoshi Asakura, Tomoya Soma, Sho Kato, and Haifeng Chen. Superrclas-conditional gaussian mixture model for learning fine-grained embed- dings. In ICLR, 2022.
Lukasz P Olech and Mariusz Paradowski. Hierarchical gaussian mixture model with objects at- tached to terminal and non-terminal dendrogram nodes. In Proceedings of the 9th International Conference on Computer Recognition Systems CORES 2015, pp. 191-201 Springer, 2016.
Boris Oreshkin, Pau Rodríguez López, and Alexandre I Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances neural information processing systems, 31, 2018.
Alexandre Passos, Piyush Rai, Jacques Wainer, and Hal Daumé III. Flexible modeling of latent task structures in multitask learning. In ICMI, pp. 1283-1290, 2012.
Sanjay Purushotham, Wilka Carvalho, Tanachat Nilanon, and Yan Liu. Variational recurrent adver- sarial deep domain adaptation. In ICLR, 2017.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In ICLR, 2020.
Andrei A RUsu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2018.
Ketan Rajshekhar Shahapure and Charles Nicholas. Cluster quality analysis using silhouette score. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 747-748. IEEE, 2020.
Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1441-1451, 2021.
Satya Narayan Shukla and Benjamin Marlin. pled time series. In ICLR, 2019.
networks for irregularly sam-
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few- shot learning? In ECCV, pp. 645-666. Springer, 2020.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding is all you need? ECCV, pp. 266-282. Springer, 2020.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In /CLR, 2022.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NIPS, 29, 2016.
11

---

Under review as a conference paper at ICLR 2023
Risto Vuorio, Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv:/812.07172 2018.
Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In AAAI, volume 35, pp. 651-659, 2021.
Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classifica- tion with dirichlet process priors. Journal of Machine Learning Research, 8(1), 2007.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In ICML, pp. 7045-7054. PMLR, 2019a.
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In ICLR, 2019b.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, pp. 8808-8817, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX FOR DETAILS OF DERIVING HTGM
A. THE LOWER-BOUND OF THE LIKELIHOOD FUNCTION
In this section, we provide the details of the lower-bound in Eq. (3). By introducing the approximated posterior 46 (vr )D the likelihood in Eq. (1) becomes (the superscript * is neglected for clarity)
log (e v.) + Σ log (L, pluivs (ov)l,)))(v))dd,,
((.,,)) 0)
n Σ log (L,, p(u(l=))((v))
co(v- (r.) 2 (v-l1/D)) dvr
log (es
Σ log (f,, (o(v.ID:)) p(uil (v()) dv. im1
log logro(e,111)
2o(v, (DF)
logro(e./u) Σ)
/ (v=
logr(u/v.) log.lv logp(v.) loggolv (D2) DD))dvv. dv,
/. (v(v. [ ( - -fo 2 log
log po +
Σ
log (e:lln)
B (v+
+ - H(go(vvlDD))
log (e:
logg(wivv) log Σ p(v-lz+)p(z-)) H(go(v-IDP)
(7)
where the fourth step uses Jensen's inequality. This completes the derivation of Eq. (3).
A.2 THE UPPER-BOUND BOUND OF THE PARTITION FUNCTION
In Sec. 3.2, we apply an upper bound on the partition function in Eq. (2) for solving the challenging
2. The derivation of the upper bound is follows.
-fe exp min (Ellay /.g.
exp
Fu
v-)] du.
W,v+ [3 dpc
/
max ({exp)
-1Iks w [w)v)118)) ddpe <
/
exp - -1
w,,. 1 (8)
N
Σ
[- I dy 2
exp
#=1
where the last equation is from the multidimensional Gaussian integral. This completes the deriva- tion of the upper bound of the partition function.
A.3 THE PROOF THEBRRM
Proof. Let Bj denote ball in Idd. Its center is at W v and its radius is D(WAv, W/v,)/3. Because WAV, and Wiv, (1 < h,1 < N) is the pair with the smallest Euclidean distance D(WAY,, W.v.), for any pair of balls Bj and Bm we have B, U B,, Ø.
In other words, there no overlap between any pair of balls. Therefore, we compute the integral over the joint of all balls, we have
J SASUN
N Σ
exp B B...
Fas (n v. v=)]
exp [- Few (p(: v:))
m=1 pgEn...
Also, because there no overlap between any pair of balls, for each point μh E Bm have
min
1 W w,v+1()
=-11pg W..v1l
(10)
13

---

Under review as a conference paper at ICLR 2023
Therefore, have the following derivation from Eq. (9).
/
exp
Eiw (1bgg v. vo Σ
m=1 L ugeBm
exp E (M v. du
u Bm
(11)
N
Σ
exp -11 Wm., dud -N
exp [ 1-11KG Wm.v.115dp2
m=1
14 4 Bm
JLEBm
Meanwhile, since U Bm is sub-area of the entire IRd space, we have
exp Eu v-) & / exp E Eu v+) ddde
J Bm
(12)
According to the multidimensional Gaussian integral, we have
D.W. / exp vd = 1ad
(13)
mfEam
Therefore,
lim D.WaY Worr) / exp
E (uE 2 N 2
(14)
Since N 222 is its upper bound, based on the squeeze theorem, we have
lim DIW Woo,) mg exp
Ew v.
(15)
which completes the proof of Theorem 1.
A.4
THE TRAINING ALGORITHM OF HTGM
The training algorithm of HTGM is summarized in Algorithm 1.
B APPENDIX FOR FURTHER DISCUSSION
B. DISCUSSION ABOUT THE RELATIONSHIP BETWEEN HTGM AND HGM MODEL
To the best of our knowledge, the Hierarchical Gaussian Mixture (HGM) model has appeared in the traditional works (Goldberger & Roweis, 2005; Olech Paradowski, 2016; Athey et al., 2019) for hierarchical clustering by applying Gaussian Mixture model agglomeratively or divisively on the input samples. They are unsupervised methods that infer clusters of samples, but do not pre- train embedding models (or parameter initializations) that could be fine-tuned for the adaptation to new tasks in meta-learning. Therefore, these methods are remarkably different from meta-learning methods, and think it is non-trivial problem to adapt the concept of HGM to solve the meta- learning problem. To this end, we need to (1) identify the motivation; and (2) solve the new technical challenges. For (1), found the hierarchical structure of mixture distributions naturally appears when we want to model the generative process of tasks from a mixture of distributions, where each task contains another mixture distribution of classes (as suggested by Eq. (1)). In other words, the motivating point of our method is more on meta-learning than HGM. However, we think drawing such a connection between meta-learning and HGM is novel contribution. For (2), our method different from traditional HGM in (a) its generative process of tasks (Sec. 3.1), which is theoretical extension of the widely used empirical process of generating tasks in meta-learning; (b) its Gibbs- style lask-coniitioaal distribution (Eq. (2)) for fitting uniformly sampled classes; (c) the metric- based and-to-end meta-learnig framework (Fig. 1) (note the traditional HGM is not for learning embeddings); (d) the non-trivial derivation of the optimization algorithm in Sect. 3.2 and Alg. 1; and (e) the novel model adaptation process in Sec. 3.3. Solving the technical challenges in the new generative model another novel contribution of the proposed method.
14

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Hierarchical Gaussian Mixture based Task Generative Model (HTGM)
Input: encoder fo, training dataset D", hyperparameters r, a,
Output: model parameters {0, w)
Pre-train the encoder fe via ProtoNet with augmentations.
Pre-train the energy function in Eq. (2) by maximizing A EE= log + log (2xlv-)
for + to MaxEpoch do
/+ E-step */
V= Ø
for (De = {(x;, v D {(},, v.) in Dataloader(D") do
/+ Load task episode +/
// embeddings of the support set p. Task-Pooling (Class-Pooling( ((e)) -)))
{e]) = {xa
// the mean of go(v-1D))
Sample task embedding v, from go (v. N( 50 V= (v.)
end
// fit GMM to V, where (++) represents the labeling of the v,'s in V /x M-step */
[ar) (u)..... ph Ei.... =
11
12
for (D, = (f) DJ = {xx,, 121) in Dataloaden(D") do /. lead a task episode +/
13
{e = (fo(x)) Lns
// forward pass
14
{e = (fo(x)))
// forward pass
{nf. ...... ) = Class-Pooling( ( )
16
n Task- Task-Pooling ( ( )9)
// the mean of (v(v.
Sample task embedding v. from go go (v- V+) N +
17
for = I,.., N do
7 = on4 (1 a) Wi-v,' where arg min SISN D(u Wiv,))
end
Calculate (((c)) V, (1)) (ui...,, - Hi, Ei..., k.h,,w // calculate the loss in Eg. (5) using Eq. (3) and Eq. (4) 0, = (E, w.w)
21
22
// update model parameters
23
end
24 end
B.2 DISCUSSION ABOUT THE RELATED MULTI-TASK LEARNING METHOODS
The modeling of the clustering/grouping structure of tasks or the mixture of distributions of tasks has been studied in multi-tasking learning (MTL). In (Xue al., 2007: Jacob al., 2008), tasks are assumed to have clustering structure, and the model parameters of the tasks in the same cluster are drawn to each other via optimization on their L2 distances. based regularization framework was proposed for grouping task-specific model parameters, where the tasks in the same group are assumed to lie in the same low dimensional subspace for parameter sharing. The method in (Kumar Daumé III, 2012) also uses the subspace based sharing of task parameters, but allows two tasks from different groups to overlap by having one or more bases in common. The method in (Passos al., 2012) introduces generative model for task-specific model parameters that encourages parameter sharing by modeling the latent mixture distribution of the parameters via the Dirichlet process and Beta process.
(Kang et al., 2011), a subspace
The key difference between these methods and our method HTGM lies in the difference between MTL. and meta-learning. In MTL method, tasks are known priori, iee, the testing tasks are from the set of training tasks, and the model is non-inductive at the task-level (but it is inductive at the sample-level).
HTGM, testing tasks can be disjoint from the set of training tasks, thus the model is inductive at the task- In particular, we aim to allow testing tasks that are not from the distribution of the training tasks by enabling the detection of novel tasks, which an extension of the lask-level inductive model. The second difference lies in the generative process. The method in (Passos et al., 2012) models the generative process of the task-specific model parameters (e.g., the weights in regressor). In contrast, HTGM models the generative process of each task by
15

---

Under review as a conference paper at ICLR 2023
generating the classes in it, and the samples in the classes hierarchically, i.e., the (x, y)'s (in Eq. (1) and Sec. 3.1). In this process, we allow our model to fit uniformly sampled classes given a task (without specifying prior on the distance function on classes) by the proposed Gibbs distribution in Eq. (2). Other remarkable differences to the aforementioned MTI. methods include the inference network (Fig. 1(b)), which allows the inductive inference task embeddings and class prototypes; the optimization algorithm (Sec. 3.2) to our specific loss function in Eq. (3), which is from the likelihood in Eq. (I); and the model adaptation algorithm (Sec. 3.3) for performing predictions in a testing task, and detecting novel tasks. As such, the MTL methods can not be trivially applied to solve our problem.
B.3 FURTHER INTERPRETATION OF THE TASK-CONDITIONAL DISTRIBUTION
The task-conditional class distribution Pa (1: k-v+)) in Eq. (2) is defined through an energy func- j=1) 1)) with trainable parameters w = [W...., wx) ). for allowing uniformly sampled classes per task. The conditional distribution p(y: V.) represents how classes distribute for given task T. The reason for its definition in Eq. (2) is follows. it Gaussian distribution with V- (i.e., task embedding) as the mean, p(y: = k can be interpreted as the density at the representation of the k-th class in this Gaussian distribution. i.e., the density at Mk, which is the mean/surrogate embedding of the k-th class. One problem of this Gaussian pl p(u (v,) that different classes, ie., different My, 's, are not uniformly distributed, contradicting the practice that given dataset (e.g., images), classes are often uniformly sampled for constituting task in the empirical studies. Using a uniformly sampled set of classes to fit the Gaussian distribution p(yslei (2;)) will lead to an ill-posed learning problem, as described in Sec. 3.1. To solve it, we introduced w = [......, WN3 in the energy function E (1t; vz) in Eq. (2). W E Rdxd (1 S j S N) can be interpreted
tion Fw (14;; = min (4 wfv-l12
projecting v, to the j-th space spanned by the basis (i.e., columns) of W,: j- There are N different spaces for j = 1, ...... Thus, the N projected task means Wi,,.... WNV. V are in N different spaces. Fitting the energy function Few (mL; Vz) to N uniformly sampled classes ui.-..14% which tend to be far from each other because they are uniformly random, tends to learn W.... W that project v, N far apart spaces that fit each of the μÿ, ".., 2 by closeness, due to the min-pooling operation. This mitigates the aforementioned ill-posed learning problem.
C APPENDIX FOR IMPLEMENTATION DETAILS
C.1 THE SETUP OF THE COMPARED MODELS
Encoder of Metric-based Meta-Learning, For faimess, for all metric-based methods, includ- ing ProtoNet (Snell et al., 2017), MetaOptNet (Lee al., 2019c), ProtoNet-Aug (Su et al., 2020), FEATS (Ye et al., 2020) and NCA (Laenen & Bertinetto, 2021), following (Tian et al., 2020; Lee et al., 2019c), we apply ResNet-12 12 as the encoder. ResNet-12 has 4 residual blocks, each has convolutional layers with kernel size of 3 x 3. ResNet-12 uses dropblock as regularizer, and its number of filters is (60, 160, 320, 640). For MetaOptNet, following its paper (Lee et al., 2019c), flattened the output of the last convolutional layer to acquire 16000-dimensional feature as the image embedding. For other baselines, following (Tian et al., 2020), we used a global average- pooling layer on the top of the last residual block acquire 640-dimensional feature as the image embedding.
Further Details. Following (Snell al., 2017), ProtoNet, ProtoNet-Aug, and NCA use Adam op- timizer with 31 = 0.9 and 8n = 0.99. We did grid-search for the initial learning rate of the Adam within (le- 2, le 1e *), where 1e was selected, which is the same as the official implementa- tion provided by the authors. For FEATS, we chose transformer as the set-to-set function based on the results reported by (Ye et al., 2020). When pre-training the encoder in FEATS, following its pa- per (Ye et al., 2020), we applied the same setting as ProtoNet, which is to use Adam optimizer with an initial learning rate of le 31 = 0.9 and Bg = 0.99. When training its aggregation function, we grid-searched the initial learning rate in {le- 5E le 15-5) since larger learning rate leads to invalid results on our datasets. The optimal choice is le -4 For MetaOptNet, following its paper (Lee et al., 2019c), we used SGD with Nesterov momentum of 0.9, an initial learning rate of 0.1 and a scheduler to optimize it, and applied the quadratic programming solver OptNet (Amos & Kolter, 2017) for the SVM solution in it.
16

---

Under review as a conference paper at ICLR 2023
C.2 THE DETAILS OF THE SETUP FOR NOVEL TASK DETECTION
In the experiments novel task detection in Sec. 4.1, the number of in-distribution tasks (from the Original domain) in the test set 4000 (1000 per task cluster) and the number of novel tasks (from the Blur and Pencil domains) in the test set is 8000 (4000 for the Blur and 4000 for the Pencil).
D APPENDIX FOR EXPERIMENTAL RESULTS
D.1 ANALYSIS o
Setting of o
Bird
Texture
Aircraft
Fungi
0.1
69.33
46.92
75.20
50.78
0.5
70.00
47.98
75.38
$2.38
1.0 (Ours)
70.12
47.76
75.52
52.06
10.0
69.4
47.28
75.32
51.5
Table 4: Analysis of different
Tabel 4 report the effect of different a on the classification performance (5-way-I-sher classification on Multi-Plain dataset). As shown in the table, although the too low or too high setting of this hyper-parameter will hurt the performance, in general the model is robust toward the setting of a.
D.2 ANALYSIS OF ō
Setting of 0.05
Bird
Texture
Aireraft
Fungi
69.78
48.36
74.36
51.34
0.1rSur))
70.12
47.76
75.52
52.06
0.2
70.02
47.50
75.30
51.74
0.5
69.02
46.66
74.46
51.00
Table 5: Analysis of different
Tabel 5 summarize how different ō influence classification performance (5-way-1-ste 1-shot classification on Multi-Plas Plain dataset). In general, different settings of & will influence the model performance at marginal level, indicating our model's robustness toward this hyper-parameter.
D.3 IMPACT OF GMM COMPONENT NUMBER
Number components r
2
8
16
32
Silhouette score
47.70
57.61
12.76
7.81
6.19
Table 6: Analysis on the number of mixture components
Different choices of the number of mixture components does not significantly influence the model classification performance. However, the clustering quality may vary due to the different numbers of components. Here, we report the Silhouette score (Shahapure 2021) w.r.t. the number in Table 6. From Table 6, we can see that selecting component number close to the ground-truth component number of the distribution can benefit the clustering quality.
Nicholas, 2020; Sharma al.,
D.4 CLASSIFICATION PERFORMANCE
ABLATION VARIANTS
We summarize the classification performance of the Ablation Variants HTGM w/o GMM and HTGM-Gaussian in Table As we can see, our unique designs improve the novel task detection performance without significantly decreasing the classification performance.
17

---

Under review as a conference paper at ICLR 2023
Ablation Variants HTGM w/o GMM
Bird
Texture
Aircraft
Fungi
68.86
48.00
75.74
52.28
69.52
47.3
75.38
51.34
HTGM
70.12
47.76
75.52
52.06
Table 7: Ablation study of different variants of our proposed method.
Setting
Model
Bird
Texture
Aircraft
Fungi
Average
ANIL- MIL-
02.64=0..90
43.86=t0.78
70.03+0.85
48.34:00.89
56.22
5-way-1-shor
ANIL-HSMLL
64.33+0087
43.77++0.79
69.71+0.84
47.75++0.89
56.39
ANIL-ARMI.
65.98+0.87
43.57++0.78
70.28++0.84
48.48
57.08
HTGM (ours)
70.12+1.228
47.76++1.49
7..5
12.06++1.41
61.37
ANIL-MAMI.
7..38+0.73
55.36+40.74
79.78+00.63
59.57+0.79
67.27
5-way-5-shot
ANIL.-HMM
78.18
57.70+00.75
81.32++0.22
59.83++0.81
69.26
ANIL-ARMI.
78.7940.71
57.61+0.773
81.86+00.59
60. 19.8
69.61
HTGM (ours)
82.27.10.74
60.67110.78
88.48
65.70.10.79
74.28
Table 8: More results daccuracy--9555 confidence) of the optimization-based methods.
D.5 ABLATION ANALYSIS OF OPTIMIZATION-BASEE METHODS
Table 8 summarizes the performance of MAML, HSML and ARML trained in ANIL method (Raghu et al., 2020), i.c., pre-trained the ResNet-12 by ProtoNet, froze the encoder, and fine-tuned the last fully-connected layers using MAML, HSML and ARML on Plain-Mult dataset. From Table 8, the performance of ANIL-MAML is better than MAML in Table similar to the observation in (Raghu et al., 2020), indicating the effectiveness of ANIL method. However, ANIL-HSML and ANIL-ARML perform similarly ANIL-MAMI, losing their superiority of modeling the mixture distribution of tasks achieved when implemented without ANIL as in Table (up to 5.6% average improvement). This is because the cluster layer HSML and the graph layer in ARML both affect the embeddings learned through backpropagation. i.e., they were designed for joint training with the encoder. When the encoder is frozen, they cannot work properly. For this reason, be con- sistent with the existing researches (Yao et al., 2019a;b) that demonstrated the difference between HSML/ARML and MAML, we used their original designs in See. 4. Meanwhile, we observe the proposed HTGM outperforms MAML, HSML, and ARML trained in ANIL method, this is because MAML cannot model the mixture distribution of tasks, while HSML and ARML cannot work prop- erly when trained in ANIL method.
D.6 LIMITATIONS OF THE PROPOSED METHOD
Model ProtoNet-Aug 0.99 HTGM
5-way-1 1-sot
5-way-5-shot
74.68:10.45
61.80=0.95
74.55=00.4
Table 9: Comparison of our proposed method with other models on mini-imagenet dataset.
In the case when the task distribution similarly to the general metric-based meta-learning methods, e.g., ProtoNet, which only considers a uni-component distribution. To confirm this, we added an experiment that compares our model with ProtoNet-Aug on Mini-Imagenet (Vinyals et al., 2016), which does not have the same explicit mixture distributions as in the Plain-Multi and Art-Multi datasets in Section 4. The results are summarized in Table 9. From the table, observe method performs comparably to ProtoNet, which validates the aforementioned guess. Meanwhile, together with the results in Table and Table the proposed method could be considered as generalization of the metric based methods to the mixture of task distributions.
a mixture, our model would degenerate and perform
18


Reviews:
[{"review_id": "y4MmO7dZJW", "reviewer": "Reviewer_DtsZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Hierarchical Gaussian Mixture method to parametrize the generation process of tasks in meta-learning.  The proposed model can be used for fitting a mixture of task distributions and evaluating the scoring of testing tasks for evaluating the novelty of the testing task.  Experiments on several datasets demonstrate the effectiveness of the proposed methods. ", "summary_of_the_review": "This paper proposes a Hierarchical Gaussian Mixture method to parametrize the generation process of tasks in meta-learning. The method itself is not new and experiments need to be further improved. ", "strength_and_weaknesses": "**Strengths**:\n\n* The proposed method can handle both a mixture of task distributions and detecting novelty in testing tasks is interesting.\n\n\n**Weakness**:\n\n* The proposed method is an application of Hierarchical Gaussian Mixture to the meta learning setting. The method itself is not novel and is a well-studied model in the machine learning community. \n\n\n* For the novel task detection experiments, it would be better to include more experiments that use Mini-ImageNet as source, and  CIFAR, aircraft, etc as novel tasks datasets to see whether it works well in practice. \n\n\n* The compared methods of optimization-based methods appear difficult to scale to a large backbone, but it is not always the case. For example, you can pre-train the backbone and fix most of layers, and only train the last or last few layers, similar to ANIL [1]\n\nReferences;\n\n[1] Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. ICLR 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OGg1nGcSZ4", "reviewer": "Reviewer_gaxY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an approach for few-shot learning, that provides a way to model not only class distributions, but the distribution of tasks.  This provides two benefits: (i) improved performance on few-shot tasks, and more interestingly, (ii) the ability to determine whether a task (support set of (images, class) pairs) is \"novel\" and too far outside the domains the few-shot learner was trained on.  While other works always evaluate on novel classes, sometimes chosen to come from semantically different subtrees than those in meta-training, this work explicitly models the distribution of tasks themselves using Gibbs distribution with basis {W}, in order to identify out-of-distribution tasks.  This forms a hierarchy of mixture models, describing the tasks, classes and embeddings.  The system is optimized using EM, with SGD in the M step.  It is evaluated on Plain-Multi and Art-Multi datasets, finding improved performance over current few-shot-learning techniques augmented with GMM in-distribution model.\n", "summary_of_the_review": "Overall this is an interesting idea and model, with reasonable effectiveness.  It is somewhat complex, though, and I wonder if it (or some of its descriptions) could be simplified.\n", "strength_and_weaknesses": "Modeling task distribution in this way is an interesting idea, and the results of the system are promising, improving upon appropriate baselines.  The formulation is rather complex, leading to a somewhat complicated system, and whether the improvements are worth the complexity is debatable -- but it's certainly a reasonable trade-off.\n\nI think some of the notation and descriptions could be made simpler (see some suggestions below), which might help (see below comments), but the system is pretty described as it is.  I had some trouble understanding the inference procedure, though:  In sec 3, I didn't understand the process for inferring the task embedding $v_{\\tau'}$.  This section says a single $v$ is sampled and then means are adapted to drawn towards the Wv means; but should this use either the most likely $v$ given class prototypes $\\mu^s$, or a larger sample of $v$ from the distribution?\n\nAlso, another approach might have been to use backpropagation, rather than EM, perhaps combined with Gumbel-softmax on the categorical from a direct weight parameterization.  Have you explored other potentially simpler approaches (I'm not sure if this one would work) that still capture the task distribution model?\n\nOverall this is an interesting idea and model, with reasonable effectiveness.  It is somewhat complex, though, and I wonder if it (or some of its descriptions) could be simplified.\n\n\n\n\nFurther questions and comments:\n\nThe notation is a little involved and can be hard to follow sometimes, e.g. differences between $\\mu$, $\\hat\\mu$, $\\bar\\mu$.  Since all the distributions are MOG or MOG-like, except the top level categorical, maybe it could work to index each level instead, e.g. calling the means $\\mu^l$ where $l$ is the hierarchy level, and similarly for other variables.\n\nIs there some intuition for this form of W parameterization, for how it models a task distribution?  it looks like a dictionary W * coeffs v, is one interpretation?  In the limiting case it might have a blockwise W_i = [0 | w_i1, ..., w_ik | 0], with w_ik being a basis for task i, and coeffs in the corresponding block of v.  The actual system of course is not constrained to this --- but is there a similar way to look at this with subspaces, and are the subspaces \"tied together\" between tasks, so that instead of forming a basis of each class mean they are forming one of task space?\n\neq 2 Gibbs distribution is basically another mixture of gaussians, with means {W_j v}.  Why not use MOG here exactly?\n\nchallenges 1, 2:  To me, the l_neg sampling in challenge 1 looks like the contrastive term that would be supplied by partition function mentioned in challenge 2 --- certainly it's of the same form and is subtracted.  I'm not entirely sure of its exact placement, whether it's a substitute for this or if they are in separate places.  Do they correspond in this way, and if so, would the trivial solution in challenge 1 come up if the partition function weren't replaced with a constant bound?\n\nText in 3.3 uses $\\psi$ referring to Fig 1b, but Fig 1b doesn't mention $\\psi$ anywhere in it.  In general there is a bit of back-and-forth between use of $\\psi$ and $\\theta$, since it's mentioned $\\psi = \\theta$.  Could this be made more consistent?\n\nSec 4.1:  The ProtoNet + GMM baseline is good, simple and very appropriate.  How many GMM components were used, and how large was the in-distribution training sample?  Were these tuned in any way to best performance for this baseline approach?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JTPCdwXR26V", "reviewer": "Reviewer_A867", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). The basic assumption is that task embedding fits mixture distribution of Gaussian. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization algorithm. Extensive experiments on benchmark datasets indicated the effectiveness of the method for both sample classification and novel task detection.", "summary_of_the_review": "This paper proposed a novel algorithm for meta learning. It is based on embedding-based few shot learning and mixture distribution of Gaussian for task generation. The experiments also demonstrated the effectiveness of few-shot learning and novel task detection. Overall the paper is well written.", "strength_and_weaknesses": "Strength\n\nS1. I like the idea. Based on the observation that embedding is the key of few-shot learning, modeling with mixture of Gaussian makes perfect sense and is mathmatically solvable.\n\nS2. The paper is well written and easy to follow.\n\nWeaknesses\n\nW1. Task grouping was extensively studied in the context of multi-task learning, e.g., [R1]. As both such mult-task learning algorithms (with metric-based few-shot algorithm pluggined) and the proposed method address a similar problem, agnostic to feature extractors, I think it is worthy discussing them, especially those with similar mathematical tools, in Related works.\n\n[R1] Flexible Modeling of Latent Task Structures in Multitask Learning. ICML 2012.\n\nW2. Experimental comparisons could be more solid. E.g.,  to compare with baselines also designed for dealing with mixture distributions of tasks, i.e., HSML and ARML, both the backbone and base algorithm are different, making it less convincing that the proposed method is better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject