Hierarchical Gaussian Mixture based Task Generative Model for Robust Meta-Learning
Authors: Yizhou Zhang
Jingchao Ni
Wei Cheng
Zhengzhang Chen
Liang Tong
Haifeng Chen
Yan Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Few-Shot Learning
Meta Learning
Task Representation
TL;DR: We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM).

Abstract: Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task embeddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization (EM) algorithm. Extensive experiments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.

Hierarchical Gaussian Mixture based Task Generative Model for Robust Meta-Learning
Yizhou Zhang" Jingchao Nia+1 Wei Cheng", Zhengzhang Chen', Liang Tong",,
Haifeng Chen², Yan Liu
University of Southern California FAWS Al Labs
NEC Laboratories America "Stellar Cyber Inc.
(zhangyiz, yanliu. yanliu cs @usc. edu; "nijingchao@@mail..oo
(wei zchen, hain,naffng | @nec/ com; com;
*Itong@atellarccher .i
Abstract
Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from distribution that is unseen during meta-tran veta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta- training framework underlain by novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which leamss task embeddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end end-to-end by maximum likelihood estimation (EM) algorithm. Extensive experiments on
via an Ampectation-Maximizaatii benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.
Introduction
Training models in small data regimes is of fundamental importance. It demands model's ability to quickly adapt to new environments and tasks. To compensate for the lack of training data for each task, meta-learning (a..a. learning to learn) has become essential paradigm for model training by generalizing weta-knowledde across tasks [41, 8]. While most existing meta-learning approaches were built upon assumption that all training/testing tasks are sampled from the same distribution, more realistic scenario should accommodate training tasks that lie a mixture of distributions, and testing tasks that may belong to deviate from the learned distributions. For example, in recent medical rescarch, a global model is typically trained on the historical medical records of a certain set of patients in the database [40, 49]. However, due to the uniqueness of individuals (e.g., gender, age, genetics) [43], patients' data have substantial discrepancy. and the pre-trained model may demonstrate significant demographic geographical biases when testing new patient [35, 10]. This issue can be mitigated by personalized medicine approaches [5, 31], where each patient is regarded task, and the pre-trained model fine-tuned (i.e., personalized) records collected in a short period (e.g.. a few weeks) from every patient for adaptation. In this case, the training tasks (i.e., patients) could be sampled from mixture of distributions (e.g., different age
support set of few
TTii work was done primarily at NEC Laboratories America. *Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

groups), and a testing task may or may not belong to any of the observed groups. Similar examples can be found in other applications such as the detection of fake news dissemination [7. 55, 57], where a task is post, whose support set consists of a few profiles of users who have disseminated it in short period. The training posts may be drawn from a mixture of topics, and a testing post may not belong to any of the topics. As such, a meta-training strategy that is able to fit model to a mixture of task distributions and enable the identification of novel tasks during inference time is desirable for making meta-learning practical solution.
One way to tackle the mixture distribution of tasks is tailor the transferable knowledge to each task by learning task-specifer representation [33, 48, 25], but as discussed in [52], the over- customized knowledge prevents its generalization among closely related tasks (e.g., tasks from the same distribution). The more recent methods try to balance the generalization and customization of the knowledee by promoting local generalization either among a cluster of related tasks [52], or within a neighhorhood in a meta- eetta-nowledgge graph of tasks [53]. Neither of them explicitly learns the underlying distribution from which the tasks are generated, rendering them infeasible for detecting novel tasks that are out-of-distribution. However, detecting novel tasks is crucial in high-stake domains, such as medicine and finance, which provides users (e.g. physicians) confidence on whether to trust the results of testing task or not, and facilitates downstream decision-making.
In [21], task specifie tuning variable was introduced to modulate the initial parameters learned by the optimization- based meta-learning method MAML [8]. so that the impacts of the meta-knowledd on different tasks are adjusted differently, e.g., novel tasks are less influenced than the known tasks. Whereas, this method focuses on improving model performance on different tasks (either known or novel), but neglects the critical mission of detecting which tasks are novel. In practice, providing an unreliable accuracy on a novel task, without differentiating it from other tasks may be meaningless.
Since the aforementioned methods cannot simultaneously handle the mixture distribution of tasks and novel tasks, a practical solution is in demand. In this work, we consider tasks as instances, and demonstrate the dual problem of modeling the mixture of task distributions and detecting novel tasks are two sides of the same coin, i.e., density estimation on task instances. To this end, we propose a Hierarchical Gaussian Mixture based Task Generative Model (HTGM) to explicitly model the generative process of task instances. Our contributions are summarized as follows.
We extended the widely used empirical process of generating tasks to theoretical process specified by a hierarchy of Gaussian mixture (GM) distributions. HTGM generates a task embedding from a task-level GM, and uses it to define the cask-conditioned mixture probabilities for class-level GM, from which the data samples are drawn. To allow realistic classes per task, a new Gibbs distribution was proposed to underlie the class-level GM.
HTGM is an mncoder-agnosto framework, thus is flexible to different domains. It inherits metric- based meta-learning methods, and only introduces a small overhead to an encoder for parameterizing its distributions, thus is efficient for large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via a principled EM algorithm. The bounds of our likelihood function were also theoretically analyzed.
In the experiments, we evaluated HTGM on benchmark datasets regarding its scalability to large networks, effectiveness in modeling the mixture distribution of tasks, and usefulness in identifying novel tasks. The results demonstrate HTGM outperforms the state-of tfite-of-thhaaan the-art (SOTA) baselines with significant improvements in most cases.
2 Related Work
To the best of our knowledge, this is the first work to explicitly model the generative process of task instances from a mixture of distributions for meta-learning with novel task detection. Meta-lcarning aims to handle the few-shot learning problem, which derives memory-based [30]. optimization- based [8, 26], and metric-based methods [47, 41], and often considers an artificial scenario where training/test tasks are sampled from the same distribution. To enable more varying tasks, task- adaptive methods facilitate the customization of meta- veeta-knowledge by learning task-specific parameters [37, 25], temperature scaling parameters [33], and task- task-specific modulation on model initialization [48, 52, 53, 21]. Among them, there are methods tackling the mixture distribution of tasks by clustering tasks [52, 13, 14, 17], learning task similarity and graphs [58, 53], and relocating the initial parameters for different tasks that they use the meta-knowledge differently [21]. As discussed
2

---

before, none of these methods jointly handles the mixture of task distributions and the detection of novel tasks in meta-testing stage, A more detailed discussion is in Appendix B.1.
Our model is built upon metric-based methods and learns task embeddings to model task distributions.
Achille et al. [1] proposed learn embeddings for tasks and introduced meta-learning method, but not for few-shot learning. Their embeddings are from a pre-specified set of tasks (rather than episode-wise sampling), and the meta-learning framework is for model selection. The model in [52] has an augmented encoder for task embedding, but it does not model task generation, and is not designed for novel task detection (an empirical comparison is in Sec. 4.1).
The conventional novelty detection aims to identify and reject samples from unseen classes [6]. It relates to open-set recognition [46], which aims simultancously identify unknown samples and classify samples from known classes. Out-of-distribution (OOD) detection [27, 28] can be seen as a special case of novelty detection where novel samples are from other problem domains datasets, thus are considered to be easier to detect than novelties [6]. These methods are for large-scale training. In contrast, we want detect novel tasks, which is new problem in the small data regime.
Hierarchical Gaussian Mixture (HGM) model has appeared in some traditional works [9, 32, 50] for hierarchical clustering by applying GM agglomeratively or divisively, which do not pre-train models for meta-learning, and is remarkably different from the topic in this paper. The differences are elaborated in Appendix B.2. Moreover, the idea of learning groups/clusters of tasks also appeared in some multi-task learning (MTL) models. The key difference between these methods and our method HTGM lies in the difference between MTL and meta-learning. In MTL method, all tasks known a priori, i.e., the testing tasks are from the set of training tasks, and the model is inductive at the sample-level but non-induetive at the task-level. More discussions are in Appendix B.3.
3 Hierarchical Gaussian Mixture based Task Generative Model (HTGM)
3.1 Problem Statement
Meta-learning methods typically use an episodic learning strategy, where the meta-training set Di consists of a batch of episodes. Each episode samples task from distribution p(r). Task has support set DD; = {(x], v for training, and a query set D = {(x], v for testing, where ne a small number to denote a few training samples. In particular, a commonly used N-way K-shot Q-query task [47], De and DJ contain N classes, with K and Q samples per class respectively, ie., n1 = N K and n4 = NQ.
Let (x{) + V be a base model (* denotes s or q). and so(: Dt) be the adapted model on De. The training objective on is to minimize the average test error of the adapted model, ie., (. (v). fe (x); D))). where E(.--) is loss function (e.g., cross-entropy loss), and the meta- training process aims to find the parameter that minimizes this error over all episodes Dur Then, So is evaluated on every episode of meta-test set D that samples task from the same distribution p(T). Usually, p(r) is a simple distribution [8, 21]. In this work, P(T) is generalized to mixture distribution consisting of multiple components pL (T), .... Pr (r(T), and a test episode may sample a task either in or out of any component of p(T). As such, given the training tasks in D", our goal is to estimate the underlying density of p(7). so that once test task is given, we can (1) identify if it is novel task, and (2) adapt fo to it with optimal accuracy. Specifically, the base model fo can be written as combination of encoder ge, and a predictor ho,, i.e., i.e. fo = he, (ge. (x)) )) [44]. In this non- parametric learner, i.e. 0p = Ø (e.g., prototypical networks [41]), not only because baaedd classifiers were confirmed more effective than probabilistic classifiers for novelty detection [12], but also for its better training efficiency that fits large backbone networks than the costly nested-loop training of aptimization-basse methods [44].
work, we focus on a metric-
Formally, our goal is to find the model parameter that maximizes the likelihood of observing a task T. In other words, let fo(x;) e E R be sample embedding, we want to maximize the likelihood of the joint distribution pe (e, v² on the observed data in D,- = (D;, D. We consider each task T as an instance, with a representation v, R in the embedding space (the method to infer v, is described in Sec. 3.3). To model the unobserved mixture component, we associate every task with a latent variable z, to indicate to which component it belongs Suppose there are r possible components, and let ng + ng be the total number of samples in D,, the ligeliild
3

---

to maximize can be written by hierarchically factorizing it on of and marginalizing out v and 27.
log (e: (eeeiv?)) uf = 1 72
R(Dr;0)
log (po 1o (p(())
log [po ( p(uilv-) p(uilv-)p(vv)dd.)
(1)
71
log po(e) (u)) wi) [/, p(uilv=) Σ ply+ p(vrlez))(Er)]
where Po(e,lu? v(') specifies the probability of sampling of from the yf wh class, p(u, v-) is the probability of sampling the y class for task т, and p(V+l2>) indicates the probability of generating task from the z,-th mixture component. p(zy) is a prior on the 2,-th component. Hence, Eq. (1) implies generative process of task T: 2r - V, - y + e7. Next, we define each of the aforementioned distributions and propose our HTGM method.
3.2 Model Specification and Parameterization
In Eq. (1), the crasscconditional distribution po lyt), ) the caskiconnitional distribution p(y" v>). and the mixture distribution of tasks defined by {p(v> (p(vr/==). (<),)(2+)} are not specified. To make Eq. (1) optimizable, we introduce our HTGM that models the generative process of tasks. Because Dj and Dş follow the same distribution, in the following, we ignore the superscript for simplicity.
Class- Conditional Distribution. First, similar to [22, 23], use Gaussian distribution to model the embeddings e, 's in every class, Let py, and Ey, be the mean and variance of the y,-th class, then pe(e()))) = (ee py My In fact, the samples in all of the classes of task T comprise a Gaussian mixture distribution, where p(y. is the mixture probability of the y/-th class. In Eq. (1), p(1)) is factorized be task-specific, i.e., p(yslv,), which resorts another mixture distribution p(v,) of tasks, and establishes structure of hierarchical mixture.
Task-Conditional Distribution. A straightforward definition of p(y vv) is the density at pe in a Gaussian distribution with V, as the mean, where Fe is the mean (or prototype) of the y/-th class. However, doing so exposes two problems: (1) the density function of Gaussian distribution is log-concave with one global maximum. Given the mean and variance, maximizing its log- ligeliielhoodd tends to collapse the prototypes 1 's of all classes in T, making them indistinguishable and impairing classification; (2) given V,, this method tends sample classes with small Dv. (mi,, ). where Dv. (.) measures the Mahalanobis distance between a data point and the Gaussian distribution centered at v,- However, in most of the existing works, classes often uniformly sampled from domain without any prior on distances [8]. Fitting the distance function with such "uniform" classes naively leads to an ill-posed learning problem with degenerated solutions. In light of these issues, we seek to define p(y vv) as a (parameterized) density function with at least N global optimums so that it can distinguish the N different class prototypes of N N-way tasks. The N equal (global) optimums also allow it to fit N classes uniformly sampled from a domain. To this end, let μ be the surrogate embedding of the k-th class, we propose a Gibbs distribution x (14 v,, w) defined by v, and trainable parameters with an energy function. Then we write p(y: = k(v=) as
exp |- Ew (M v.) ng exp Few (1gg v+)]
Pwly: = k(v+) T w)
(2)
where Ew (p v-) = min (flleg W, 1vvi1 is our energy function, and the denominator in Eq (2) is a normalizing constant (with respect to (.).), a.ka the partition function in an energy-based model (EBM) [20]. w = IW,..., WN N } are trainable parameters, with W, E R Given and v.. Eq. (2) has N global maximums at g WiV,, ce = W V. More interpretations of the proposed lask-conditional distribution can be found in Appendix B.4.
Mixture Distribution of Tasks. In Eq. (1), the task distribution p(v,) is factorized mixture of p(vr|z, = 1), ... p(v-lz, = r), weighted by their respective mixture probability p(2+). Thus we specifyy p(v,) as Gaussian mixture distribution. We introduce pi,, and Σ. the mean and variance of each component, i.e., p2v) (V E' ), and let p [p....., pr] be the mixture probabilities, where Pr p(z, r) and be Uniform(r). Then v, is genetrated in two

---

steps: (1) draw latent task variable 2, from a categorical distribution on p, and (2) draw v, from N( "x ) [4]. As such, our HTGM generative process an N N-way K-shot Q-query task T can be summarized as following, and Fig. illustrates the corresponding graphical model:
Draw 2, ~ (ple P-I)
2. Draw task embedding Vo Ni Ex,
3. For = 1,,,,,,. N:
(a) Draw class prototype 14 T ( Vr, w) from the proposed Gibhs distribution in Eq. (2)
(b) For i = 1,.... K Q:
i. Set y/: = k, draw sample e; ~ ((e:lll 14 2 Allocate (e,; v4) to the support set D. if S K; else allocate (et, to the query set D.
To reduce complexity, we investigate the feasibility of us- ing isotropic Gaussian with tied variance, i.e., Σl EN = o II, for class distributions. which turned out to be efficient in our experiments. Here, I an identity matrix, is a hyperparameter. Tied variance is also a commonly used trick in Gaussian discriminate analysis (GDA) for genera- tive classifiers [22, 39]. For task distributions, the variances ,i.,,, E, can be automatically inferred by our algorithm, as elaborated in Sec. 3.3.
Σ
F
V
u
el R
n¹
Figure I: Graphical model of the pro- posed generative process for B tasks with N ways, K shots and Q queries.
Finally, in Eq. (I), substituting po (eselyss) - NNee I²I).
Pa (Milv, = T(H. [), (2,) = NV ln!.. E.,). whose probabilities are specified and parameterized, and letting o = Uniform(r) be a uniform prior, we get our HTGM induced loss (TrGM 0 w) The class means 1e task means μ! and variances Σ inferred in the E-step of our EM algorithm (the details can be found in Sec. 3.3 3.3 and A.6).
3.3 Model Optimization
It is hard to directly optimize fITGM (D,: 0, w) because the exact posterior inference is intractable (due the integration over V-). To solve it, we resort to variational methods, and introduce an approximated posterior (D))), which is defined by an inference network Φ, and implies we want to infer v. from its observed support set D.. The query set Dę is not included because is unavailable during model testing. Then we propose to maximize a lower-bound of Eq. (1), which is (the derivation can be found Appendix A.
CHTGMI (D.;
Σ log (e::l))) + E,, walt,, (D3) ( P
( (u:lv.) log Σ p(v, p(y. (p(2,)))
+ H (go (D)))
(3) where H(go(v=lP:)) = - Jv. 2o (v (o(vv-1D2) (D?) log (v, DD )dv, is the entropy function. Similar to VAE [16], Eq. (3) estimates the expectation (in the second term) by sampling v, from (v (De), instead of the integration in Eq. (1), hence facilitates efficient computation. Next, elaborate on the inference network, the challenges of maximizing Eq. (3), and our workarounds.
Inference Network. Similar to VAE, 4 (v) is defined as Gaussian distribution N N(u Vuu_,,"D) II), where n2, is the output of the inference network, which approximates !o!_ in Step 2 of the generative process, and a hyperparameter for the corresponding variance. As illustrated by Fig. 2(a), the inference network is built upon the base model fo(-) with two parameaettiic aggregation (iee mean pooling) functions, thus &- = O. The first function aggregates class-wise wise embeddings to prototypes " 's similar to prototypical networks [41]. Differently, the second aggregates all prototypes to μ², During model training, we used the reparameterization trick [16] sample v, from N(p,, I².). noteworthy that (v:))))) in Eq. (3) becomes a constant because x² is a constant.
Challenge 1: Trivial Solution. In Eq. (3), since the first term Po. (e: (e:lys)
2 |e;
H 1
(constants were ignored) only penalizes the distance between a sample e; and its own class mean p (i.e,, intra-class distances) without considering inter-class relationships, different class means u
5

---

aa)
Support set (Xi fies
[ef]i
Closs pooling
Class meains MaP
p(a)))
Task-lousl GMM
Task Iaak-poling
Ve
p(v122,)
Mi Mi....H.
(( Vv at)
Gibbs distribution
W Ww
Class means
Query sen (4)
[e ! (()b)))
(A5
Training process
(b)
Support [x/]]=s
[ef]]=
Training embeddings
Novelty?
Testing process
Bujpod SOr
Class means
fø
W W
Pooling
Adeptation
Sampling
W.V
Part training loss
Classfficttion
Adapted means
Query set D..
Model parameters +
Figure 2: An illustration of HTGM on its (a) training process, and (b) testing process. (a), 1 the three parts of the training loss in Eq. (3). In (b), the training task embeddings contain the embeddings of all training tasks, i.e. the outputs the task look-pooling in (a).
y in task 7 could collide, drawing all sample embeddings to the same spot. To avoid such trivial solution and improve the stability of optimization, we apply negative sampling [29]
Lnc DD; yi 0, w) = - log Ee,~D, exp |,, 1 1E )]
(4)
2d
where is negative sample embedding from any class in the support set, and μr is the mean of the positive class. In practice, we found it beneficial to integrate laeg with our likelihood bнTом in Eq. (1) during training, i.e. (HTGM Σi= loog. Correspondingly, from Eq. (3) have
(D,;0.w) (D;;
Σ T1
Lnog (Dr: PiriyssC. O,w)
(5)
which does not only serve as a robust training loss, but also helps solve the next challenge.
The Partition Function in Eq. (2). The second term (W vv) in Eq. (3) involves computing the partition function in Eq. (2) (i.e., the denominator), which is intractable because the integration over all possible μ ss. To solve it, propose upper bound of the partition function
Challenge 2:
Jmg exp - Few (p&; v=) v>) du < N (the derivation in Appendix A.2), which is constant
with specific N. By replacing the partition function in Eq. (2) with N WTT we got a lower bound of " (v:l which in turn relaxes the lower bound in Eq. (3). The following theorem (the proof is in Appendix A.3) states that the tightness of the relaxed bound is controllable.
Theorem 3.1. Among the N global maximums W.V...... WNV, of Eq. (2), let WAV, and Wiv,
(1 S h, < N) be the pair with the smallest Euclidean distance Daa. we have
lim Dai
exp
Faw (p dut NVT
(6)
This theorem indicates the partition function approximates N when all pairs of the global maximums are far apart. is noteworthy that when maximizing the likelihood, we fit WiV,, Wxv, to different class prototypes n .... 1 in N-way tasks. Because 'neg in Eq. (4) tends to maximize the distances between different prototypes through the negative samples, maximizing the joint loss & in Eq. (5) tends to separate W WIV,, V,. ... W v. thus tighten the relaxed bound after using N v according to Theorem 3.1. This is another benefit of negative sampling.
Optimization via popectaaion-Maxiiizaatinn In the third term of 4ITGМI Eq. (3), we need to estimate the mixture distribution p(zz). Similar to optimize Gaussian mixture models, alternately

---

infer the posterior p(=+lv=) (i.e., the mixture membership of v,) and solve the model parameters {0, w through an SuspectationMMxxmmizaa model parameters. In M-step, when fixing p(=r| [v],,,, (0, W) can be efficiently solved by optimizing Eq. (5) with stochastic gradient descent (SGD). The detailed optimization algorithm of HTGM can be found in Appendix A.6.
algorithm. In E-step. we infer p(z+lv=) when fixing
3.4 Model Adaptation
Fig. 2(b) illustrates the adaptation process of HTGM. Given new N-way task r' from the meta-test set D its support set Dj, is fed to the inference network to generate (1) class prototypes μ4. ...... u (similar to prototypical networks), and (2) distribution ( (D ). from which we draw the average task embedding v,- = r Recall that the inference network is the base model fo(-) with class-pooling and task-pooling layers, illustrated in Fig. 2(b), and Φb 0. Then v+r projected to WI''', WN'=> which represent the N optimal choices of class prototypes for task r as learned by the Gibbs distribution in Eq. (2) from the training tasks. They are used to adapt μ4 .-. & that the adapted prototypes are drawn towards the closest classes from the mixture component that task r' belongs to. Specifically, the adaptation is performed by selecting the closest optimum for each prototype 14 (1 S j I N), that is
3 on3 + (1-1))))) (1-a)W where 1" = arg min D(H), Wiv.), I5I5N
and -) )) is the Euclidean distance, is hyperparameter, The following theorem confirms the effectiveness of the this adaptation method (the proof can be found in Appendix A.4).
Theorem 3.2. The adapted prototypes pi, .... n from Eq. (7) maximizes the lower-bound (HтCм-L in Eq. (3) of the likelihood in Eq. (1) when a K+2azá
Theorem 3.2 suggests automatic setup of a, which can also be tuned empirically for optimal value using validation datasets. We evaluated the empirical values of a in our experiments and discussed their relationships with the theoretical values in Appendix A.4.
Finally, we (1) assess if 7' a novelty by computing the likelihood of v, in a pre-fitted GMM on the embeddings v, 's of the training tasks in D", and (2) perform classification on each sample x/ in the query set D9, using the adapted prototypes by
exp ( DD (()()()))))))))))))))))))))) (T))
plu/- = j'(x) Σn exp c (DD((f((x),,.)))) (3))
(8)
which is the posterior probability. The derivation of Eq. (8) is in Appendix A.5.
Experiments
In this section, evaluate HTGM's effectiveness few-shot classification and novel task detection on benchmark datasets, and compare it with SOTA methods.
Datasets. The first dataset is the Plain-Multi benchmark [52]. It includes four fine-grained image (Bird), Describable Textures Dataset (Texture), FGVC of Aircraft (Aircraft), and FGVCX-Fungi (Fungi). In each episode, task samples classes from one of the four datasets, so that different tasks are from a mixture of the four domains. The second dataset is the Art-Multi benchmark [53]. whose distribution is more complex than Plain-Multi. Similar to [13], each image in Plain-Mulie was applied with two filters, i.e., blur filter and pencil filter, respectively, to simulate changing distribution of few-shot tasks. Afterward, together with the original four datasets, a total of 12 datasets comprise the Art-Multi, and each task is sampled from one of them. Both benchmarks were divided into the meta-traie meta-training, meta- veta-vallidttion. and meta-test sets by following their corresponding papers. Moreover, we used the Mini-Imagehh dataset [47] to evaluate the case of uni-component distribution of tasks, which is discussed in Appendix D.6.
classification datasets, i.e.,
Baselines. We compared HTGM with the relevant SOTA methods on meta-learning including (1) optimization-based methods: MAML [8] and Meta-SGD [26] learn globally shared initialization
In this section, 1z (1 S S N) is the j-th class mean of support set, its superseript is omitted for clarity.
7

---

Setting
Model
Bird
Texture
Aireraft
Fungi
Avg.
TAML
1..77=11443
3..78+1..30
18.56++1.37
41.00+11.50
44.28
MAML
13.94=1445
31.66#1.11
51.37=1.38
12.12=#1.36
44.77
Meta-SGD
1..58=1.43
32.38=1.32
52.99:1.36
4..74=1.34
45.67
MUMOMAML
16.82=1.449
33.81
53.1411.39
42.22=::.40
46.50
5-way,
HSMI.
60.98+1.50
35.01+1.36
57.38+1.40
44.02++1.39
49.35
I-shot
ARMI.
62.33+1.47
35.65+1.440
18.56+t..11
44.82++1.38
50.34
ProtoNct
61.54=1.27
38.84+1.42
1..4
46,52+11.42
55.08
MetaOpiNet
62.80
44.30.11.45
68.64.11.29
47.04-11.38
55.70
ProtoNet-Aug
63.04
1..68
70.44.11.32
49.30.11.40
57.37
NCA
1.25
1..98
1..70.11.26
46.36.11.34
54.66
FEATS
1..6
1..1
68.86.11.28
47.92-11.34
55.88
HTGM (ours)
70.12.1..22
17.76.1..49
75.52.1
12.06
61.37
TAML
69.50=0.75
45.11
65.92:10.74
50.99
57.88
MAML
68.52=00.79
44.56=0.68
66. 18:7011
51.85:0.85
57.78
Meta-SGD
67.87=00.74
45.49-00.8
66.84:10.70
52.51
58.18
MUMOMAML
70.49=00.76
45.89-10.69
67.31
53.96-1-.82
59.41
5-way,
HSML
71.68=00.73
08.08
73.49+0068
56.82-+0080
62.39
5-shot
ARML
73.34=0.70
49.67:1067
74.88100.6
57.5
63.86
ProtoNet
78.88=0077
57.93-0.75
86.42:0.57
62.52:
71.44
MetaOptNet
81.66=0.77
61.97-0.78
84.03
63,80
72.87
ProtoNet-Aug
80.62=00.71
58.30=t0777
87.05+00.53
63.62+0081
72.39
NCA
79.16+0.77
58.69+00.6
85.27+00.3
61.68+0..0
71.20
FEATS
78.37-0.72
57.02+0.73
0..5
61.56+0..00
70.63
HTGM (ours)
02.77
60.6710.78
88.48
65.70.10.99
74.28
Table 1: Results (accurac==95%% confidence) of the compared methods on Plain-Multi dataset.
among tasks. MUMOMAML [48] is task-specific method. TAML [21] handles imbalanced tasks. HSML [52] and ARML [53] learn locally shared initial parameters in clusters of tasks and neighborhoods of a meta-graph of tasks, respectively; and (2) Metric-based methods: ProtoNet [41] is the prototypical network. MetaOptNet [24] uses an SVM classifier with kernel metrics. ProtoNet- Aug [42], FEATS [54] and NCA [19] were built upon ProtoNet by augmenting images (e.g.. rotation, jigsaw), adding prototype aggregator (e.g., Transformer), and using contrastive training loss (instead of prototype-bast loss), respectively. The detailed setup of these methods is in Appendix C.I.
Implementation. Following [44], the optimization- convolutional layers the base learner, and the metric-based methods used ResNet-12 12. The output dimension of these networks is 640 (MetaOptNet uses 16000 as in its paper). our experiments, observed the optimization-based methods reported out-of-memory errors when using ResNet- 12, indicating their limitation in using large backbone networks. To test them on ResNet-12, we followed the ANIL method [36] by pre-training ResNet-12 via ProtoNet, freezing the encoder, and fine-tuning the last fully-connected layer. In this case, HSML and ARML cannot model the mixture task distribution properly as they require joint training of the encoder and other layers. The details are in Appendix D.5. For training, Adam optimizer was used. Each batch contains 4 tasks. Each model was trained with 20000 episodes. The learning rate of the metric- based methods was 1e The learning rates for the inner- and outer-loops of the aptiiiization-baan based methods were 1 and 1e-4. The weight decay was 14 For HTGM, we a 1.0, ø = 0.1, a = 0.5 (0.9) for 1-shot (5-shot) tasks. The number of mixture components varies w.r.t. different datasets, and was gracched within [2, 4, 8, 16, 32). All hyperparameters were set using the mela-validation sets.
baselines used the standard four-block
4.1 Experimental Results
Few-shot classification. Following [44], we report the mean accuracy and 95% confidence interval of 1000 random tasks with 5-way 1-shot/5-shot, 5/25-query tests. Following [53], we report the accuracy of each domain and the overall average accuracy for Plain-Multi, and report the accuracy of each image filtering strategy and the overall average accuracy for Art-Multi. Table and 2 summarize the results. From the tables, we have several observations. First, metric-based methods generally methods. This because of the efficiency metric-based methods,
outperform optinization- enabling them to fit larger backbone network, which is consistent with the results in [44]. Built upon the metric-based method, HTGM only introduces a few distribution-relatee parameters and thus has the flexibility to scale with the encoder size. Second, the baselines designed for dealing

---

Setting
Model
Original
Blur
Pencil
Avg.
TAML
1..9
40.02:+1.41
1..1
39.11
MAML
12.70=#1.35
4..53:11.3
36.71=1.37
39.98
Meta-SGD
14.21=11.88
42.36::1.39
37.21=1.39
41.26
MUMOMAML
13.63=11.39
41.5911.38
39.24=1.36
42.15
5-way,
HSML
17.92++1.34
44.43+1.34
41.44=1.34
44.60
1-shot
ARML.
1..68-11.34
42.62++1.34
39.78=1.34
42.69
ProtoNet
55.23+:1.31
51.70+1.42
49.22=11.4
52.05
MetaOptNet
16.10
12.33
1..08-1.45
52.50
ProtoNet-Aug
57.63.11.34
15.00.1
19.7
$4.12
NCA
16.12.11.35
10.80.11.49
47.99-1.45
51.64
FEATS
5..33.1:1.3
50.9011
47.96 1.48
$1.07
HTGM (ours)
1..3
58.80-1.442
53.23 1.48
57.74
TAML
58.5420.73
55.23
49.23=0.75
54.33
MAML
58,30=0.77
55.71+0.74
49.59=0073
54,50
Meta-SGD
57.82:40.72
55.54:10.73
50.24-0.72
54.53
MUMOMAMI
58.60-10.75
56.29.10.72
51.15
55.35
5-way,
HSML
60.63-10.73
57.91
53.93=10722
57.49
1-shot
ARML
61.78:10.74
58.731:.755
55.27=0073
58.59
ProtoNet
71.34=0.73
67.28:10.75
64.32=10.76
67.65
MetaOptNet
72.33=10.72
68.9011.77
63.89=0.71
68.37
ProtoNct-Aug
72.87=0.77
70.50+
63.98=00773
68.78
NCA
72.44=0.72
67.33+0071
62.98-0.77
67.58
FEATS
71.99+00.71
6,.54+0.72
63 0.709
67.54
HTGM (ours)
74.67
71.24.1.73
0.77
70.37
Table 2: Results (accuracy+:95% confidence) of the compared methods on Art-Multi dataset.
with mixture distributions of tasks, i.e., HSML and ARML, outperform their counterparts without such design, demonstrating the importance consider mixture task distribution in practice. Finally, HTGM outperforms the SOTA baselines in most cases by large margins, suggesting effectiveness in modeling the generative process of task instances.
Novel task detection. We also evaluate HTGM on the task of detecting novel N N-way-K-shop tasks (N = 5, K = 1) that are drawn out of the training task distributions. To this end, we train each comapred model the Original domain in Art-Multi dataset, and test the model on tasks drawn from either Original domain (i.e., known tasks), [Blur, Pencil} domains (i.e., novel tasks), and evaluate the model can tell whether a testing task is known novel.
For comparison, since none of the baselines detects novel tasks, adapt them as follows. For metric-based methods, since they use fixed encoder for all training/testing tasks, we averaged the sample embeddings in each task represent the task. Then separate GMM model was built upon the training task embed- dings, and its likelihood was adapted score the novelty of testing tasks (some details of the setup are in Appendix C.2).
Model
AUROC
AP
Max-Fl
HSML
55.96
37.94
50.17
ProtoNet
65.17
41.51
56.07
MetaOptNet
72.71
63.77
58.33
NCA
66.28
51.45
52.74
ProtoNet-Aug
72.67
57.93
59.07
FEATS
59.35
42.57
49.31
HIGM w/o GMM
70.24
62,45
57.75
UITGM-Gaussia
74.06
66.18
60.62
HTGM
75.66
68.03
60.51
However, optimization- bassed models perform gradient descent the support set of each task, leading to varying encoders per task. As such, sample embeddings of different tasks are not comparable, and we cannot obtain task embeddings the same way before. Among them. only HSML has an augmented task-level encoder for task embedding, allowing us to include it for comparison. For a fair comparison, our HTGM also trains GMM on its task embeddings for detecting novel tasks. Moreover, two HTGM variants were included for ablation analysis to understand some design choices: (1) HTGM-Gaussianl replaces the Gibbs distribution in Eq. (2) with Gaussian distribution; (2) HTGM w/o GMM removes the task-level GM, ie.. the third term in Eq. (3). The classification results of the ablation variants are in Appendix D.4. Following [6. 46, 56], we report Area Under ROC (AUROC), Average Precision (AP), and Max-F1. Table summarizes the results, from which observe HTGM outperforms all
Table 3: Comparison between HTGM, its variants and applicable baselines on novel task detection.

---

HSML
MetaOptNet
(c) ProtoNet- Aug
(d) HTGM
ERONUT Tasto Meved Tiaks
BROUT TaRts
Streed Trokk
ond Tasis
Mond Tacb
Figure 3: The frequency of tasks w.r.f. the normalized likelihood for (a) HSML (b) MetaOptNet (c) ProtoNet-Aug (d) HTGM. The x-axis ranges vary as only 95% tasks with top scores were preserved.
baselines over all evaluation metrics, indicating the superior quality of task embeddings learned by our model. The embeddings follow the specified mixture distribution of tasks p(v-) as described in Sec. 3.2, which fits the mixture data well hence allowing to detect novel tasks that are close to the boundary. Since the baselines learn embeddings without explicit constraint, they don't fit the post-hoc GMM well. Moreover, HTGM outperforms HTGM w/o GMM, which is even worse than some other baselines. This further validates the necessity to introduce the regularization of task-level mixture distribution p(v,). Also, the drops of AUROC and AP of JTTGM-Gausae demonstrate the importance of our unique design of the Gibbs distribution for the task-conditional distribution in Eq. (2). Similar to [46], in Fig. 3. we visualized the normalized likelihood histogram of known and novel tasks for HSML, MetaOptNet (the best baseline), ProtoNet-Aug (near-best baseline), and HTGM. To better interpret the figures, calculated the ratio between the non-overlapped area of the two distributions and the cotal-area in Fig. 3 for HSML: 0.1379, MetaOpt: 0.4952, ProtoNet- Aug: 0.4806, and HTGM: 0.5578. As can be seen, the ratio of the non-overlapped area of HTGM is higher than other methods, which indicates the likelihoods (i.e. novelty scores) of HTGM are more distinguishable for known and novel tasks than the baseline methods. We also analyzed the hyperparameters a, o, and r of HTGM in Appendix 1. D.2, and D.3.
5 Conclusion
In this paper, we proposed novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM models the generative process of task instances, and performs maximum likelihood estimation to learn task embeddings, which can help adjust prototypes acquired by the feature extractor and thus achieve better performance. Moreover, by explicitly modeling the distribution of tasks in the embedding space, HTGM can effectively detect the tasks that are drawn from distributions unseen during meta training. The extensive experimental results indicate the advantage of HTGM on both few-shot classification and novel task detection.
Broader Impact and Limitation
Our proposed method enables better novel task detection for meta-leariig, In many areas requiring robust decisions, such as healthcare (as described in Sec. 1) and auto-driving, where the accuracy drop and uncertainty on novel tasks is inevitable, our model doctors and human drivers) for diagnosis and decision-making. However, as a probabilistic machine learning model, HTGM does not guarantee 100% accuracy. Also, it is noteworthy that the entropy of VV,
raise an alarm to users (e.g..
T (v,, w) (in Eq. (2)) is proportional to the partition function (i.e. the denominator in Eq. (2)). Thus, our approximation in Sec. 3.3 that replaces the partition function with its upper bound increases the entropy. leading to increased noise in the inferred class prototypes. As such, in scenarios such as serious disease treatment and auto-driving in complex environments, avoid potential wrong decisions from the model, human intervention is still necessary.
Acknowledgments and Disclosure of Funding
This work was primarily finished during the internship of Yizhou Zhang (the first author) at NEC I Laboratories America. Yizhou Zhang's work after the internship was partially supported by NSF Research Grant IIS-2226087 and the Annenberg Fellowship of the University of Southern California. We sincerely appreciate the comments and suggestions from the anonymous reviewers.
10

---

References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In ICCV, pages 6430-6439, 2019.
[2] Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization layer in neural networks. ICML, volume 70, pages 136-145. PMLR, 2017.
[3] Thomas L Athey, Benjamin D Pedigo, Tingshan Liu, and Joshua T Vogelstein. Autogmm: Auto- matic and hierarchical gaussian mixture modeling in python. arXiv preprint arXiv:1909.022 1909.02688. 2019.
[4] M. Christopher Bishop. Pattern recognition and machine learning. Springer, 2006.
[5] Isaac S Chan and Geoffrey Ginsburg. Personalized medicine: progress and promise. Annual review of genomics and human genetics, 12:217-244, 2011.
[6] Jiacheng Cheng and Nuno Vasconcelos. Learning deep classifiers consistent with fine-grained novelty detection. In CVPR, pages 1664-1673, 2021.
[7] Lu Cheng, Ruocheng Guo, Kai Shu, and Huan Liu. Causal understanding of fake news dissemination social media. In Proceedingy of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 148-157, 2021.
[8] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adapta- tion of deep networks. In ICMI., pages 1136-1135. PMIR, 2017.
[9] Jacob Goldberger and Sam T Roweis. Hierarchical clustering of mixture model. In NeurlPS, pages 505-512, 2005.
[10] Mengdi Huai, Chenglin Miao, Qiuling Suo, Yaliang Li, Jing Gao, and Aidong Zhang. Uncorre- lated patient similarity learning. Proceedings of the 2018 SIAM International Conference on Data Mining, pages 270-278. SIAM, 2018
[11] Laurent Jacob, Jean-philippe Vert, and Francis Bach. Clustered multi-task learning: convex formulation. In NIPS, 2008
[12] Minki Jeong, Seokeon Choi, and Changick Kim. Few-shot open-set recognition by transforma- tion consistency. In CVPR, pages 22566-12575, 2021.
[13] Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. NeurlPS, 32, 2019.
[14] Weisen Jiang, James T. Kwok, and Yu Zhang. Subspace learning for effective meta-learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 10177-10194. PMLR, 2022.
[15] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In ICMI., 2011.
[16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rX6v:131..6114, 2013.
[17] Weihao Kong, Raghav Somani, Zhao Song, Sham M. Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5394-5404. PMLR, 2020.
[18) Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi-task learning. In ICML, pages 1723-1730, 2012.
11

---

[19] Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning. NeurIPS, 14:24281-24592, 2021.
[20] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[21] Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian meta-learning for imbalanced and out-of- distribution tasks. arXiv preprini arXiv: 1201, 2019.
[22] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS, 31, 2018.
[23] Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICML, pages 3763-3772. PMLR, 2019.
[24] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pages 10657-10665, 2019.
[25] Yoonho Lee and Seungjin Choi. tradient-based neta-learning with learned layerwise metric and subspace. In ICML, pages 2927-2936. PMLR, 2018.
[26] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. XrXiv preprint arXiv: 00835, 2017.
[27] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018
[28] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. pages 21 21464-21475, 2020.
[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NewrlPS, 2013.
[30] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018
[31] Jingchao Ni, Wei Cheng, Zhengzhang Chen, Takayoshi Asakura, Tomoya Soma, Sho Kato, and Haifeng Chen. Superclass-condttoona conditionni gaussian mixture model for learning fine-grained embeddings. In ICLR, 2022.
[32] tuuaasz P Olech and Mariusz Paradowski. Hierarchical gaussian mixture model with objects attached to terminal and non-terminal dendrogram nodes. In Proceedings of the 9th International Conference on Computer Recognition Systems CORES 2015, pages 191-201. Springer, 2016.
[33] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. NeurlPS, 31, 2018.
[34] Alexandre Passos, Piyush Rai, Jacques Wainer, and Hal Daumé III. Flexible modeling of latent task structures in multitask learning. In ICMI, pages 1283-1290, 2012.
[35] Sanjay Purushotham, Wilka Carvalho, Tanachat Nilanon, and Yan Liu. Variational recurrent adversarial deep domain adaptation. In ICLR, 2017.
[36] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In ICLR, 2020.
[37] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-lcarning with latent embedding optimization. In ICLR, 2018.
[38] Ketan Rajshekhar Shahapure and Charles Nicholas. Cluster quality analysis using silhouette score. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 747-748. IEEE, 2020.
12

---

[39] Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In SIGKDD, pages 1441-1451, 2021.
pages
[40] Satya Narayan Shukla and Benjamin Marlin. intrpollation-prediciioo networks for irregularly sampled time series. In ICLR, 2019.
[41] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
[42] Long-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-vuupnii improve few-shot learning? In ECCV, pages 645-666. Springer, 2020.
[43] Qiuling Suo, Fenglong Ma, Ye Yuan, Mengdi Huai, Weida Zhong, Jing Gao, and Aidong Zhang. Deep patient similarity learning for personalized healthcare. IEEE transactions on nanobioscience, 17(3):219 227, 2018.
[44] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In ECCV, pages 266-282 Springer, 2020.
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
[46] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set set recognition: A good closed-set classifier all you need. In ICLR, 2022.
[47] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. NIPS, 29, 2016.
[48] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal model- agnostic meta-learning. arXiv preprint arXiv:/8/2.07172 2018.
[49] Yinjun Wu, Jingchao Ni, Wei Cheng. Bo Zong. Dongjin Song. Zhengzhang Chen, Yanchi Liu, Xuchao Zhang. Haifeng Chen, and Susan B Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In AAAI, volume 35, pages 651-659 2021.
[50] Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. arXiv preprint 2rXiv:2305.00000 2023.
[51] Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Leurning Research, 8(1), 2007.
[52] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In ICML, pages 7045-7054. PMLR, 2019.
[53] Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. ICER, 2019.
[54] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, pages 8808-8817, 171,, 2020.
[55] Yizhou Zhang, Defu Cao, and Yan Liu. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. Advances in Neural Information Processing Systems, 35:10643-10655, 2022.
[56] Yizhou Zhang, Karishma Sharma, and Yan Liu. Vigdet: Knowledge informed neural temporal point process for coordination detection on social media. Advances in Neural Information Processing Systems, 34:3218-3231, 2021.
[57] Yizhou Zhang, Karishma Sharma, and Yan Liu. Capturing cross-platform interaction for identifying coordinated accounts of misinformation campaigns. European Conference on Information Retrieval, pages 694-702. Springer, 2023.
13

---

[58]
Pan Zhou, Yingtian Zou, Xiao- Yuan, Jiashi Feng, Caiming Xiong, and Steven C. H. Hoi. Task similarity aware meta learning: theory-inspired improvement on MAML. Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur, editors, Proceedings of the Thirty- Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Learning Research, pages 23-33. AUAT Press, 2021.
14

---

A Appendix for Details of Deriving HTGM
A.1 The lower-hound of the likelihood function
In this section, provide the details of the lower-bound in Eq. (3). By introducing the approximated posterior qs (vr ),), the likelihood Eq. (1) becomes (the superscript * neglected for clarity)
1 Σ log (L,, pluilv. )p(v.) n
((D., 0)
" Σ
log (e le vi) +
log (L, p(uilv-)p(v.) 2o(vv (D?)
log po (e:
Σ
go (D$) dv+))
n
i=1
Σ log (L, (o((vID))) plui v- v-)p(())) dv.
log (e(s/ee))
(+(v= (DP)
1=1
log pog
+
1
(v(v=/D?) logp(ulv.)) + log,(v.) log (D?) dv,
v.
Σ / (v.) logp(u/v.) log log dv, - -/o 2
log (e(e)))))
+
1
n
Σ
log (o.
Σ
B Bv logp(u/v.) logg(v.)) H(go(v,(DDt)
1=1
log (e:l)a)))
By mololv logp(uilv)) log Σ p(v-ll+))(z=)) H (co((v)DD))
(9)
where the fourth step uses Jensen's 's inequality. This completes the derivation of Eq. (3).
A.2 The upper-bound of the partition function
In Sec. 3.3, we apply an upper bound on the partition function in Eq. (2) for solving the challenging
2. The derivation of the upper bound is as follows.
[ exp ( v+)] dn =
exp
min (Al
N
/ max [(exp[- W,v, [1]]] 1111) dui Σ
exp
11
-w,r1BBdpp (10)
nG,
exp |- 11,,, W,v: duc, Nv
where the last equation is from the multidimensional Gaussian integral. This completes the derivation of the upper bound of the partition function.
Note that replacing the partition function with its upper bound increases the noise in the inferred class mean. This is because that the entropy of distribution л (HE |V-, w) (MC, is class mean) in Eq. (2) that:
H(T) =
T(M (w), log (Hy, w)dme
(1e V w) log Zdpr + T(pe VV, En v dr),
(11)
log Z + E.,
where E., is the average energy Z the partition function. From this formula, we see that replacing Z with its upper bound increases the entropy of THE w), the distribution where μ sampled and/or inferred. Since entropy a metric that measures the uncertainty random variable, increased entropy means that estimation to f more uncertain, in other words, more
15

---

noisy. This leads to decreasing accuracy of meta- leamning since the class mean is an important parameter estimate meta-learning. To alleviate this issue, we proposed to include an negative sampling term as in Eq. (4) to increase the distance between different class means so that the noise in the class mean estimation, which is brought by the upper bound approximation of Z, will not seriously influence the accuracy.
A.3 The proof of Theorem 3.1
Proof. Let Bj denote sphere in R.. Its center is W, v and its radius is Dii/2. Because W A and W,,, (1 I h, I S N) the pair with the smallest Euclidean distance Dai for any pair of balls B₁ and Bm. we have By n Bm null (a set with 0 volume in IR.).
other words, there no overlap between any pair of spheres. Therefore, we compute the integral over the joint of all spheres, we have
JAECUAL Bi..
exp (u v+) due Σ
exp Ew Ew v; v:) duf
(12)
EB..
Also, because there is no overlap between any pair of spheres, for each point pa E Bm, we have
min
(1) -
-11m Wm.vll
(13)
Therefore, we have the following derivation from Eq. (12).
J
JageUs Bi
exp [ Ew (u de) =
Σ
exp
Eu v.)) d
(14)
N
exp [ 1 1|
Wm Wmvrll3ddu
exp 11 Wmn Wmv-1lldde
m==1 M Bm
Meanwhile, since U Bm sub-area of the entire Rd space, we have
/
exp Eu Eui due & / exp
M Bm
Em v.)
(15)
1
As Bm is a sphere, we can convert the integral into spherical coordinates. Thus we have:
2V
Dna/2
exp
[lkg Wm +11]]du
exp(- ))))) - dr
My cB
T(T)
V
Dn1/2
exp(
-pp(-22) 2(rdr)
r(
(16)
DR./4
exp(-t)) -
r(
(1. D2g/4)
r(g)
where 7(, is lower incomplete gamma function and [(.) is gamma function (T(x) the limitation of (x,, y) when - +0c). According the definition of lower incomplete gamma function, when D + to0, we have
lim Dat a B
exp
E. (w v.)] Vid
(17)
Therefore,
lim exp Eu Eu (p):: N
(18)
16

---

Since NV is its upper bound, based on the squeeze theorem, we have
lim Dhi 1000
exp
Eu ( v. dui = N Nv
(19)
which completes the proof of Theorem 3.1. Moreover, from Equation 16, we know that the error ratio of the approximation, denoted as AER, can be bounded:
Tv - Sug exp ( v=) dug
7 /4)
7 D2./4)
AER
NVa
I(2) =
( 1)!
(20)
which monotonously decreases to as Dpr increases to +00
A.4 The proof of Theorem 3.2
Before starting the proof, we need to clarify the notations. When optimizing Eq. (3) in meta-training stage, we compute the class prototypes pii.... u with both support sample means {ui:..., 2 and query sample means {ff: S 1a (see Fig. 2). In Sec. 3.4, since during meta-testing stage we have no query samples, for clarity we omitted the notations denote the j-th class mean of support set as pag. In the proof of the Theorem 3.2, we rigorously denote the support sample means of class as 14 and the query sample means of class j as pe.
Proof. Recall the Eq. (3):
Sarom..D.;;...m CHTOMA (D.;O.w) w)
1 n Σ6 log (e + Ev, - (D3) log pw (y:
v.) log Σ p(v, la, (p(2.))
H (v (p)))
(21)
By substituting the sampled v. into it, we can acquire:
log lew + Pog (wilv+) + log Σ p(v-12>)p(==))
bHTCMи (D:;0,,)
(22)
HT (go(v-
Note that not every term in the Eq. (22) (the above likelihood function) contains class means p Only (yw()il==) and (Po (e:l))) are involved with ea.. Thus, optimizing je to maximize PHTGM-L(D; 0, is equivalent maximizing the following objective function L(x)):
M(p = Σ log = j[vv)) - Σ log w (
(23)
Moreover, since every term is involved with at most one individual us.. we can separately optimize each individual 2. We denote the embedding of the i-th sample in class j as e function for each individual μ³ is:
(i) The objective
K
L(NE) = Pu: j]v,, +
log pa w (e, =
i=1
log T(u Vof w) +
log N (1) 021)
(24)
|le - mgll min Wiv, - 29 constant
2d
The maximum likelihood estimation 5 of μ should maximize the above objective function. is equivalent to find solution set 14, g I that can minimize the following function:
1( = 2d Σ
| ())
21 + W[W, - 11
(25)
17

---

We first compute the partial derivative of the above function with respect to pe::
1L(ng.1) 8ug g2d +22 -
1 o2d (K
Σ + - -Wiv,) (26)
On the minimum, the partial derivative should be zero, Thus, pej
K+2021
1
K Σx
e
K+2ozu WiV,
20
for the optimal 1* Let us denote 14 = 1 K Σ e; (the support sample mean of class ÿ), K
K+ ad and = W. 1 then we have:
3 = au
68 + (1 - a)) = 1 + 1 (1-a)(Wiv: -1f))
(1 a)r
(27)
Therefore, we know that the optimal p satisfy the above equation. Because an index between 1 and N, there are only N solutions that satisfy the above equation. For these N solutions, we have:
K
L(
m3lB IVV,, m311
2d
K
= 20
110
0. rr12+ + Wiv+ - 14. -11-111-10 (1--a)r/13
(28)
K
K
+ Σ (o) (1)
11)8) +
10 ( 119918) + a² 1
2
Note that 14,0 R Σ k (f) 0
Thus Σx K (e 11,3)) 0. So have:
1( 2d || 11 + a² 1|1111 2 + const
(29)
Therefore, we can know that the optimal that can minimize L should minimize ||mi which is 1' = arg g D Wiv..) using Euclidean distance ((..).
However, our empirical evaluation shows that sometimes the emperically optimal value of a might be different from theoretically optimal value K422 For the 1-shot learning case, the optimal a we acquired from grid search is around 0.5 (same as theoretically optimal value). However, in the 5-shot learning case, the optimal a we acquired from grid search is around 0.9, slightly larger than 0.84 K4422z)) ). We suggests that it might be because during meta-training
K
replace the likelihood with its evidence lower bound and approximate the partition function with its upper bound, introducing noise into the generative model. Thus, we need to scale its weight down. Therefore, we propose to replace K with a hyperparameter and fineunte it on validation set.
A.5 Derivation of Eq.
We first compute p(y' j x') i.e. the posterior distribution the query sample's label y' conditioned on the sample. Then select the label with the highest posterior probability. Utilizing the estimation of class means, we have the label conditional distribution:
plu. j(,
p(x/V = j')p(a =J)
ply. = j'(x)
(30)
E jxx( En p(x(11 = j)p(1 =
Note that during the meta-testing stage, should equally treat each class i.c., assume that the prior probability of each class should be same, i.e. p(vl- = 1) =...- = p(1/- = N). Thus we have:
ply. = j'(x p(x11y/ j')
exp (-D(Jo F..))
(31)
Cx</l( j) E exp e-p pg))
A.6
The training algorithm of HTGM
The training algorithm of HTGM is summarized in Algorithm 1.
18

---

Algorithm 1: Hierarchical Gaussian Mixture based Task Generative Model (HTGM)
Input: encoder fe. training dataset D", hyperparameters r,
Output: model parameters {0, w
Pre-train the encoder fe via ProtoNet with augmentations.
Pre-train the energy function in Eq. (2) by maximizing Ei Pog + (og
for to MaxEpoch
/* E-step
V
== Ø
for
(D, = {(x,, v.)) D = (x),,99) in Dataloaden(D"") do
/- load a task episode
*/
{e)) ms = {=
embeddings the support set // the mean of co(v-
u = Task- Task-Pooliigg Tooling(Classs Dooling(CassPPoolnn(((C 12 ));
Sample task embedding from (vr [D.) (H Ø
V (v+r)
end
{ar 1 (M) pi, E Ei,.,,,,, Σ.) =
// fit a GMM to V, where {ar
11
represents the labeling of the v, /* M-step
*/
12
for (D, = {(x;, ((x),_))) ME DJ [xx], 8/) in Dataloaden do
/a load a task episode
*/
13
{ = [Ja(
// forward pass
(e [fo(x))
// forward paas
15
{ } Class ClolinPooling((((e,, ))))
16
nh = Task-Pooling( (ui.... }9)
// the mean of go(v. SISN D(u Wiv..)
Sample task embedding v. from go (v- (D.) N( 0²1)
17
18
for I.... N do
3 an (1 W.. where 1" arg gigN
20
end
Calculate (( V, (1)) (ui,..,, pi. Ei.... E,how)); // calculate the loss Eq. (5) using Eq. (3) and Eq. (4)
21
22
0 SG :
// update model parameters
end
24 end
B Appendix for Further Discussion
B.1 Discussion about the novel task discussion and meta-learning
As we discussed in Sec. the best of our knowledge, our proposed method HTGM is the first work that jointly considers the task mixture distribution and novel task detection in meta-testing stage. There based on task embedding [52] or task likelihood [13]. However, they have their own respective drawbacks when handling novel task detection in meta-testing stage. For mask-embedding-bieg based method like [52], it does not explicitly model the task distribution. Instead, it considers how to model the task membership of the learnt clusters. As a result, they can only identify the outlying task clusters rather than individual novel tasks. However, in meta- meta-testing stage, we expect the model method DPMM
works considering how to identify novel task clusters in meta-training stage
to identify each individual novel task and raise alerts. The wosk-liklihood-beee [13] can handle individual novel tasks. However, it is hard for them to simultaneously handle quick detection and adaptation. This because likelihood was built on the entire model parameters, and time consuming computation. It is not big issue for meta-training,
leading to model-depende but will serious limit its application to streaming tasks in meta-testing (e.g., in auto-driving domain) where efficiency is critical for timely alarms of novel tasks.
19

---

B.2 Discussion about the relationship between HTGM and HGM model
To the best of our knowledge, the Hierarchical Gaussian Mixture (HGM) model has appeared in the traditional works [9, 32. 3, 50] for hierarchical clustering by applying Gaussian Mixture model agglomeratively or divisively on the imput samples. They are unsupervised methods that infer clusters of samples, but do not pre-train embedding models (or parameter initializations) that could be fine-tuned for the adaptation to new tasks in meta-learning. Therefore, these methods are remarkably different from meta-learning methods, and we think it is non-trivial problem to adapt the concept of HGM to solve the meta-learning problem. To this end, we need to (1) identify the motivation; and (2) solve the new technical challenges. For (I), the hierarchical structure of mixture distributions naturally appears when we want to model the generative process of tasks from a mixture of distributions, where each task contains another mixture distribution of classes (as suggested by Eq. (I)). In other words, the motivating point of our method is more on meta-learning than HGM. However, drawing such connection between meta-learning and HGM is novel contribution. For (2), our method is different from traditional HGM in (a) its generative process of tasks (Sec. 3.2). which is theoretical extension of the widely used empirical process of generating tasks in meta-learning: (b) its Gibbs-style task-conditions distribution (Eq. (2)) for fitting uniformly sampled classes; (c) the metric-based end-to-end meta-learning framework (Fig. 2) (note the traditional HGM is not for learning embeddings); (d) the non-trivial derivation of the optimization algorithm in Sect. 3.3 and
Alg. I; and (e) the novel model adaptation process in Sec. 3.4. Solving the technical challenges in the new generative model is another novel contribution of the proposed method.
B.3 Discussion about the related multi-task learning methods
The modeling of the clustering/grouping structure of tasks or the mixture of distributions of tasks has been studied in multi-tasking learning (MTL). In [51, 11], tasks are assumed to have a clustering structure, and the model parameters of the tasks in the same cluster are drawn to each other via optimization on their L.2 distances. In [15]. subspace based regularization framework was proposed for grouping task-specific model parameters, where the tasks in the same group are assumed to lie in the same low dimensional subspace for parameter sharing. The method in [18] also uses the subspace based sharing of task parameters, but allows two tasks from different groups to overlap by having one or more bases in common. The method in [34] introduces a generative model for task-specific model parameters that encourages parameter sharing by modeling the latent mixture distribution of the parameters via the Dirichlet process and Beta process.
The key difference between these methods and our method HTGM lies in the difference between MTL and meta-learning. In an MTL method, all tasks are known a priori, ie., the testing tasks are from the set of training tasks, and the model is non- non-inductive at the task-level (but it is inductive at the sample-level). In HTGM, testing tasks can be disjoint from the set of training tasks, thus the model is inductive at the task- In particular, we aim to allow testing tasks that are not from the distribution of the training tasks by enabling the detection of novel tasks, which an extension of the task- inductive model. The second difference lies in the generative process. The method in [34] models the generative process of the task-specific model parameters (e.g., the weights in a regressor). In contrast, HTGM models the generative process of cach task by generating the classes in it, and the samples in the classes hierarchically, Le., the (x, y)'s (in Eq. (1) and Sec. 3.2). In this process, allow our model to fit uniformly sampled classes given task (without specifying a prior on the distance function on classes) by the proposed Gibbs distribution in Eq. (2). Other remarkable differences to the aforementioned MTL methods include the inference network (Fig. 2(b)), which allows the inductive inference on task embeddings and class prototypes; the optimization algorithm (Sec. 3.3) our specific loss function in Eq. (3), which is from the likelihood in Eq. (I); and the model adaptation algorithm (Sec. 3.4) for performing predictions in a testing task, and detecting novel tasks. As such, the MTL methods can not be trivially applied to solve our problem.
B.4 Further interpretation of the task-conditional distribution
The task-conditional class distribution Piu (y: = klv,) in Eq. (2) is defined through an energy function En (Ht;v,) = min (f - w, )= with trainable parameters w = (W,,... W N }. for allowing uniformly sampled classes per task. The conditional distribution p(ysi vV=) represents how classes distribute for a given task r. The reason for its definition Eq. (2) is as follows. If it is a Gaussian distribution with v, (i.e., task embedding) as the mean, p(y: kl/x,) can be interpreted as
i....
20

---

the density at the representation of the k-th class in this Gaussian distribution, i.e., the density at pgg which is the mean/surrogate embedding of the k-th class. One problem of this Gaussian p( p(ys(v=) is that different classes, i.e., different y 's, are not uniformly distributed, contradicting the practice that given a dataset (e.g., images), classes are often uniformly sampled for constituting task in the empirical studies. Using uniformly sampled set of classes to fit the Gaussian distribution p(ye v will lead to an ill-posed learning problem, as described in Sec. 3.2. To solve it, we introduced (WI.... } in the energy function Ew (1 v+) in Eq. (2). W E Rdxd (1 j S N) can be interpreted as projecting V, to the j-th space spanned by the basis (i.e., columns) of W There are N different spaces for j = 1, ..... NN. Thus, the N projected task means W1V,, WN V, are in N different spaces. Fitting the energy function Ew (H V,) to N uniformly sampled classes ug. ..... Sy.. which tend to be far from each other because they are uniformly random, tends to learn W,,..., WN that project V- to N far apart spaces that fit each of the u4;.... f by closeness, due to the min-pooling operation. This mitigates the aforementioned ill-posed learning problem.
C Appendix for Implementation Details
C.1 The setup of the compared models
Encoder of Metric-based Meta-Learning, For fairness, for all metric-based methods, including ProtoNet [41]. MetaOptNet [24], ProtoNet-Aug [42]. FEATS [54] and NCA [19], following [44, 24], we apply ResNet-12 12 as the encoder. ResNet-12 has 4 residual blocks, each has 3 convolutional layers ResNet- 12 uses dropblock as a regularizer, and its number of filters is (60, 160, 320, 640). For MetaOptNet, following its paper [24], we flattened the output of the last convolutional layer to acquire a 16000-dimensional feature as the image embedding. For other baselines, following [44], we used global average average-pooling layer on the top of the last residual block to acquire a 640-dimensional feature as the image embedding.
with a kemel size of
Further Details. Following [41]. ProtoNet, ProtoNet-Aug, and NCA use Adam optimizer with 31 0.9 and 3g 0.99. We did grid-search for the initial learning rate of the Adam within fle-2.1e le le 1e where le was selected, which is the same as the official implementation provided by the authors. For FEATS, we chose transformer as the set-to-se set function based on the results reported by [54]. When pre-training the encoder in FEATS, following its paper [54], we applied the same setting as ProtoNet, which is to use Adam optimizer with an initial learning rate of le-3 31 = 0.9 and 823 = 0.99. When training its aggregation function, we grid-searched the initial learning rate in (1e 5e 1e- since a larger learning rate leads to invalid results on our datasets. The optimal choice is le For MetaOptNet, following its paper [24], we used SGD with
Nesterov momentum of 0.9, an initial learning rate of 0. and scheduler to optimize it, and applied the quadratic programming solver OptNet [2] for the SVM solution in it.
C.2 The details of the setup for novel task detection
In the experiments on novel task detection in Sec. 4.1, the number of in-distribution tasks (from the Original domain) in the test set is 4000 (1000 per task cluster) and the number of novel tasks (from the Blur and Pencil domains) in the test set 8000 (4000 for the Blur and 4000 for the Pencil).
D Appendix for Experimental Results
D.1 Analysis of o
Setting of 2d
Bird
Texture
Aircraft
Fungi
0.1
69.33
46.92
75.20
50.78
0.5
70.00
47.98
75.38
52.38
1.0 (Ours)
70.12
47.76
75.52
52.06
10.0
69.4
47.28
75.32
51.5
Table 4: Analysis of different
21

---

Tabel 4 reports the effect of different on the classification performance (5-way-1-s8) shott classification on Multi-Plain dataset). As shown in the table, although the too low or too high setting of this hyper-parameter will hurt the performance, in general the model is robust toward the setting of o.
Analysis of D.2
Setting of o
Bird
Texture
Aircraft
Fungi
0.05
69.78
48.36
74.36
51.34
(Our)
70.12
47.76
75.52
52.06
0.2
70.02
47.50
75.30
51.74
0.5
69.02
46.66
74.46
51.00
Table 5: Analysis of different
Tabel 5 summarizes how different settings of & influence the classification performance (5-way-1-sho) classification on Plain-Multi dataset). In general, different settings of ō will influence the model performance marginal level, indicating model's 's robustness toward this hyper-parameter.
D.3 Impact GMM component number
Number of components
2
16
32
Silhouette score
47.70
57.61
12.76
7.81
6.19
Table 6: Analysis on the number of mixture components
Different choices of the number of mixture components does not significantly influence the model classification performance. However, the clustering quality may vary due the different numbers of components. Here, we report the Silhouette score [38, 39] on Plain-Multi dataset w.r.f. the number in Table 6. From Table component number of the distribution can benefit the clustering quality.
can see that selecting component number close to the ground-truth
D.4 Classification performance of the ablation variants
Ablation Variants
Bird
Texture
Aircraft
Fungi
HTGM w/o GMM
68.86
48.00
75.74
52.28
69.52
47.3
75.38
51.34
HTGM
70.12
47.76
75.52
52,06
Table 7: Ablation study different variants of our proposed method.
We summarize the classification performance of the two Ablation Variants HTGM w/o GMM and VTGGM-Gaussial in Table 7. As we can see, our unique designs improve the novel task detection performance without significantly decreasing the classification performance.
D.5
Ablation analysis of optimization- based methods
Setting
Model
Bird
Texture
Aireralt
Fungi
Average
ANIL-MAML
00.64
43.86=0.778
70.03+0.85
48.34=0.89
56.22
5-way-1-she
ANIL-HSML
64.33
43.77=t0.79
69.71:0.84
47.75+:0.89
56.39
ANIL-ARML
65.9840.87
43.57+00.78
70.28+00.4
48.48-40.22
57.08
HTGM (ours)
7..1
47.76:11.49
1..5
12.06
61.37
ANIL-MAML
74.3820.73
55.3610..74
79.78
59.5740.79
67.27
5-way-5-shot
NILL-HSML
78.18
57.70+0.75
81.32+0.62
59.8340.81
69.26
ANIL-ARMI.
78.79440.7
57.61+0..73
81.86+00.59
60. 19.891
69.61
HTGM (ours)
82.27-4..44
60.67-40.78
0..52
65.70-
74.28
Table 8: More results 955% confidence) of the corimiizzaiio-bbaeeee methods.
22

---

We selected the two best performed optimization-batt baselines HSML and ARML, and the widely used method MAML for this ablation analysis. Table 8 summarizes the performance of MAML. HSMI and ARMI trained in ANIL, method [36], f.e, we pre-trained the ResNet- 12 by ProtoNet, froze the encoder, and fine-tuned the last fully-connected layers using MAMI., HSMI and ARMI. on Plain-Multi dataset. From Table 8, the performance of ANIL-MAML is better than MAML in Table 1, similar to the observation in [36], indicating the effectiveness of ANIL method. However, ANIL-HSML and ANIL-ARML perform similarly to ANIL-MAML, losing their superiority of modeling the mixture distribution of tasks achieved when implemented without ANIL as in Table 1 (up to 5.6% average improvement.. This is because the clustering layers in HSML and the graph layers in ARML both affect the embeddings learned through backpropagation, f.e., they were designed for joint training with the encoder. When the encoder is frozen, they cannot work properly. For this reason, to be consistent with the existing research [52, 53] that demonstrated the difference between HSMI/ARML. and MAMI,, we used their original designs in Sec. 4. Meanwhile, we observed the proposed HTGM outperforms MAMI,, HSML,, and ARML trained in ANIL method, this is because MAML cannot model the mixture distribution of tasks, while HSML and ARML cannot work properly when trained in ANIL method.
D.6 More results on the Mini-ImageNet dataset
Model
5-way-1-she -sho
5-way-5-shot
ProtoNet-Aug 0.93
74.68:0.45
HTGM (ours)
61980=0.95
74.55=0.45
Table 9: Comparison of the proposed method and ProtoNet-Aug on the Mini-ImageNet dataset.
In the case when the task distribution is not a mixture, our model would degenerate to and perform similarly to the general metric-based meta-learning methods, e.g., ProtoNet, which only considers a uni uni-component distribution. To confirm this, we added an experiment that compares our model with ProtoNet-Aus on Mini- Imteeeeeeeett [47], which does not have the same explicit mixture distributions as in the Plain-Multi and Art-Multi datasets in Section 4. The results are summarized in Table From the table, we observe our method performs comparably to ProtoNet, which validates the aforementioned guess. Meanwhile, together with the results in Table 1 and Table 2, the proposed method could be considered as a generalization of the metric-based methods to the mixture of task distributions.
D.7 Task Embedding Visualization.
We provide visualizations of the task embeddings Iearnt by HTGM. Fig is the visualization of the task embeddings for Few-Shot Classification on Plain-Multi dataset (Sec. 4.1) via t-SNE [45]. Every yellow point corresponds to a task sampled from Aircraft. Every blue point corresponds to a task sampled from Texture. Every red point corre- sponds to task sampled from Fungi. Every green point corresponds to task sampled from Bird. As we can see, in general, different classes of tasks are well clustered in the task embedding space, indicating that HTGM learnt task embeddings that capture the difference of tasks from different groups
Figure 4: Clusters of task embeddings on Mlain-Multi.
D.8 Computational Cost
We report the cost of ProtoNet-Aug. NCA, FEATS and our model because they use the same encoder architecture. We evaluated and trained all of the models RTX 6000 GPU with 24 GB memory.
Training Time: According to the training logs, the training of the ProtoNet-Aug took 10 hours, the training of NCA took 6.5 hours, the training of FEATS took 10.5 hours, and the training of the
23

---

proposed model HTGM took 13 hours. Please note that our algorithm and FEATS require pre-training the encoder with ProtoNet-AAug The 10.5 and 13 hours include the 10 hours pre- training phase. The main cost of our model is not from the energy function, because we have reduced its partition function to a constant using Eq. (6) in Theorem 3.1, whose training cost is negligible.
The higher cost is because (I) our model needs to jointly learn GMM model in every EM step and (2) jointly learning the generative model and classification model takes more learning steps to converge. Given the pre-trained ProtoNet-Aug encoder, the FEATS took about 3000 steps to converge, the proposed model took about 10000 steps converge. other words, the EM training algorithm takes more computational overhead. However, the advantage comes with the training cost is the better classification and novel task detection performance.
Test Time: Because we approximated the partition function of the energy function with a constant upper bound, almost added zero computational cost to the model inference. The test time of NCA, FEATS, ProtoNet, and our model are all around 85-95 seconds for 1000 tasks.
24


Reviews:
[{"review_id": "ChrgWAkUZS", "reviewer": "Reviewer_k7fv", "summary": "In realistic scenario, training tasks and test tasks may come from different distributions. However, most existing meat-learning methods do not take it into account. Even if some do, they do not jointly handle the detection of novel tasks. \n\nThe authors propose a metric-based meta learning model that handles both the mixture of task distributions and detection of novel tasks. The effectiveness of the proposed method compared to previous methods is verified on several benchmark datasets.", "questions": "1. I am a bit confused with Eq.(4)-(5). Here, the objective of the additional negative sampling loss is to make the different class means disperse. To this end, Eq.(4) is used. But, if we minimize $l_{neg}$, wouldn’t it force $e_{j}$ and $\\mu$ to be close to each other? Please correct me if I am wrong.\n2. Could the authors elaborate how to interpret the Figure 2? In my understanding, the more distant two tasks are, the better a model is to distinguish two task groups. If so, measuring the distance between two normalized likelihood histogram would be helpful (treating them as probability distributions if necessary). Intuitively, the distance may be roughly estimated as the difference between the red and blue regions. I do not see the significant difference between the models (except HSML).\n3. Is there any particular reason why EM is used over VI? ", "limitations": "Please refer to the weakness and questions above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is novel to approximate the distribution for tasks. In particular, it aligns with realistic scenarios by modeling it as a mixture of distributions.\n2. The paper is well-written; it clearly explains its gaol with mathematical support, and notations are well-described and consistent. Figures are also helpful to understand the proposed idea.\n3. Although Eq.(3) still cannot be computed due to the normalizing constant in $p_\\omega(y_i|v_\\tau)$, the authors provided the reasonable workaround in Theorem 3.1, which is also an interesting contribution.", "weaknesses": "1. Although $l_{HTGM}$ is well-motivated and clear, $l_{neg}$ is somewhat heuristic. \n2. If my understanding is correct, a large part of computational overhead compared to baseline methods e.g. ProtoNet comes from E-step and GMM. Especially, conducting GMM at each epoch can be quite expensive. But the analysis for computational cost is not provided. Can the authors provide the computational overhead (e.g. in wall-clock time or flops) of the proposed method (and time complexity if possible)?\n3. To demonstrate that the effectiveness of the method on cross domain, the experiments on two datasets the authors used are not sufficient, especially, if we want to validate whether it works in practice or not. It could have been much more plausible if experiments were conducted on a larger scale dataset such as Meta-Dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the constructive feedback. We sincerely appreciate your valuable suggestions and questions. The following are our responses.\n\nQ1.I am a bit confused with Eq.(4)-(5). Here, the objective of the additional negative sampling loss is to make the different class means disperse. To this end, Eq.(4) is used. But, if we minimize, wouldn’t it force and to be close to each other? Please correct me if I am wrong.\n\nResponse: Thank you for the question. In our draft, we are maximizing our objective function (the lower bound of the likelihood in Eq. (3)). Thus, when we add the negative sampling term to the objective function, we are jointly maximizing it together with Eq. (3) (not minimizing it), so that the optimization will force the class means to disperse.\n\nQ2. Could the authors elaborate how to interpret the Figure 2? In my understanding, the more distant two tasks are, the better a model is to distinguish two task groups. If so, measuring the distance between two normalized likelihood histogram would be helpful (treating them as probability distributions if necessary). Intuitively, the distance may be roughly estimated as the difference between the red and blue regions. I do not see the significant difference between the models (except HSML).\n\nResponse: Your understanding is right. To better interpret the figures, we calculated the ratio of non-overlap-area/total-area in the four figures. The higher this value is, the more distant the two distributions are. Here is the result:\n\nHSML:0.1379;\n\nMetaOpt:0.4952;\n\nProtoNet-Aug: 0.4806;\n\nHTGM:0.5578;\n\nAs we can see, the ratio of non-overlap-area of HTGM is larger, which means that the two distributions are more distant. We will include these results in a revised draft.\n\n\nQ3. Is there any particular reason why EM is used over VI?\n\nResponse: If we are understanding correctly, this question is asking why we use EM to train the loss of variational inference instead of using common SGD (e.g. Variational Auto-Encoder uses SGD). The reason is that GMM in our model can not be trivially trained with SGD. This is because GMM has some constraint on its parameters (e.g. its covariated matrix must be semi-positive-definite). Therefore, we apply EM algorithm for VI (variational inference), so that in the E-step, we can use non-SGD algorithm to optimize GMM, and in the M-step, we can optimize other neural network parameters with SGD. Detailed algorithms are included in the Appendix A.4. If using VI incorporated with SGD, the GMM may get invalid parameters.\n\nQ4. (From Weakness 2) Can the authors provide the computational overhead (e.g. in wall-clock time or flops) of the proposed method (and time complexity if possible)?\n\nResponse: Yes. Let us report the cost of ProtoNet-Aug, NCA, FEATS and our model because they use the same encoder architecture. We evaluated and trained all of the models on RTX 6000 GPU with 24 GB memory. We will include the time comparison in the Appendix.\n\nTraining Time: According to the training logs, the training of the ProtoNet-Aug took 10 hours, the training of NCA took 6.5 hours, the training of FEATS took 10.5 hours, and the training of the proposed model HTGM took 13 hours. Please note that our algorithm and FEATS require pre-training the encoder with ProtoNet-Aug. The 10.5 and 13 hours include the 10 hours pre-training phase. The major cost of our model is not from the energy function, because we have reduced its partition function to a constant using Eq. (6) in Theorem 3.1, whose training cost is negligible.\n\nThe higher cost is because (1) our model needs to jointly learn a GMM model in every EM step and (2) jointly learning the generative model and classification model takes more learning steps to converge. Given the pre-trained ProtoNet-Aug encoder, the FEATS took about 3000 steps to converge, the proposed model took about 10000 steps to converge. So we agree with the reviewer that the EM training algorithm takes more computational overhead. However, the advantage comes with the training cost is the better classification and novel task detection performance.\n\nTest Time: Because we approximated the partition function of the energy function with a constant upper bound, we almost added zero computational cost to the model inference. The test time of NCA, FEATS, ProtoNet, and our model are all around 85-95 seconds for 1000 tasks.\n\nQ5. (From Weakness 3) It could have been much more plausible if experiments were conducted on a larger scale dataset such as Meta-Dataset.\n\nResponse: Thank you for this suggestion. We used the current datasets in our experiments because they were regarded as benchmarks and used by other mixture-distribution based meta-learning works (e.g., [47]). Among them, the Plain-Multi dataset consists of four datasets that also exists in Meta-Dataset. The main difference is that this benchmark does not include Mini-ImageNet. Thus, we added the following discussion and an experiment on Mini-ImageNet dataset to the Appendix.\n\n\"In the case when the task distribution is not a mixture, our model would degenerate to and perform similarly to the general metric-based meta-learning methods, e.g., ProtoNet, which only considers a uni-component distribution. To confirm this, we added an experiment that compares our model with ProtoNet-Aug on Mini-ImageNet, which does not have the same explicit mixture distributions as in the Plain-Multi and Art-Multi datasets in Section 4. The results are summarized in the following table. From the table, we observe our method performs genrally better than ProtoNet-Aug, which validates the aforementioned guess. Meanwhile, together with the results in Table 1 and Table 2, the proposed method could be considered as a generalization of the metric-based methods to the mixture of task distributions\"\n\n|Model       | 5-way-1-shot | 5-way-5-shot |\n|------------|--------------|--------------|\n|ProtoNet-Aug|59.40±0.93    |**74.68±0.45**|\n|HTGM        |**61.80±0.95**|74.55±0.45    |\n\n"}, {"review_id": "dXhifF5nRR", "reviewer": "Reviewer_XMuD", "summary": "This paper proposes a Hierarchical Gaussian Mixture method as a means to parameterize the task generation process in meta-learning. The authors suggest that their proposed model can effectively fit a mixture of task distributions and evaluate the scoring of testing tasks. The effectiveness of the proposed methods is demonstrated through experiments conducted on multiple datasets.", "questions": "N/A", "limitations": "The authors have clearly addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is well-written and easy to follow\n\n\n* It is interesting that the proposed method can meta-learn both a mixture of task distributions and detect novelty in testing tasks.", "weaknesses": "*  In this paper, the authors present an application of the Hierarchical Gaussian Mixture model to the meta-learning setting. While the paper provides ideas into utilizing this model for meta-learning, the method itself is not novel and has been extensively studied within the machine learning community.\n\n* The paper's theoretical analysis appears limited and would benefit from a more in-depth analysis. Specifically, it is crucial to provide a comprehensive analysis of the generalization bound associated with the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Here are our feedbacks to the review:\n\nResponse to Weakness 1: In this paper, the authors present an application of the Hierarchical Gaussian Mixture model to the meta-learning setting. While the paper provides ideas into utilizing this model for meta-learning, the method itself is not novel and has been extensively studied within the machine learning community.\n\nResponse: Please let us clarify the difference between our method and the existing works on Hierarchical Gaussian Mixture (HGM) model (which was also discussed in the last paragraph in Section 2 and Appendix B.2).\n\nTo the best of our knowledge, the Hierarchical Gaussian Mixture (HGM) model has appeared in some traditional works [8,30,3] for hierarchical clustering by applying Gaussian Mixture model agglomeratively or divisively on the input samples. They are unsupervised methods that infer clusters of samples, but do not pre-train embedding models (or parameter initializations) that could be fine-tuned for the adaptation to new tasks in meta-learning. Therefore, these methods are remarkably different from meta-learning methods, and we think it is a non-trivial problem to adapt the concept of HGM to solve the meta-learning problem. To this end, we need to (1) identify the motivation; and (2) solve the new technical challenges.\n\nFor (1), we found the hierarchical structure of mixture distributions naturally appears when we want to model the generative process of tasks from a mixture of distributions, where each task contains another mixture distribution of classes (as suggested by Eq. 1). In other words, the motivating point of our method is more on meta-learning than HGM. However, we think drawing such a connection between meta-learning and HGM is a novel contribution.\n\nFor (2), our method is different from traditional HGM in (a) its generative process of tasks (Sec. 3.1), which is a theoretical extension of the widely used empirical process of generating tasks in meta-learning; (b) its Gibbs-style task-conditional distribution (Eq. 2) for fitting uniformly sampled classes; (c) the metric-based end-to-end meta-learning framework (Fig. 1) (note the traditional HGM is not for learning embeddings); (d) the non-trivial derivation of the optimization algorithm in Sect. 3.2 and Alg. 1; and (e) the novel model adaptation process in Sec. 3.3. Solving the technical challenges in the new generative model is another novel contribution of the proposed method.\n\nAs such, we think our work is a new meta-learning method with a mixture task generative model.\n\nResponse to Weakness 2: The paper's theoretical analysis appears limited and would benefit from a more in-depth analysis. Specifically, it is crucial to provide a comprehensive analysis of the generalization bound associated with the proposed methods.\n\nResponse: In this paper, we mainly have provided theoretic contributions from the following perspectives:\n\n1. We extended the widely used empirical process of generating a task to a theoretical process specified by a hierarchy of Gaussian mixture (GM) distributions. HTGM generates a task embedding from a task-level GM, and uses it to define the task-conditioned mixture probabilities for a class-level GM, from which the data samples are drawn for instantiating the generated task.\n\n2. To address the challenge of computing the partition function in the Gibbs distribution in the aforementioned class-level GM (in Eq. (2)), we propose to replace it with a theoretically justified upper bound (in Theorem 3.1).\n\n3. In the global rebuttal (and the uploaded PDF file), we additionally provide a theoretic result indicating that our simple model adaptation strategy (in Section 3.3) is actually an Maximum Likelihood Estimation of the class means.\n\nWe think the above three contributions can help sufficiently justify the novel design in our proposed model. Meanwhile, it is noteworthy that in this work, we are actually handling the generalization gap by detecting novel tasks (that is out-of-distribution) with low likelihood and raises alerts. Therefore, the generalization bound is only meaningful for the in-distribution tasks. As for the general in-distribution cases of non-convex models trained with SGD, [1] gives a good generalization bound, ensuring the effectiveness of the general framework applied in our model. Therefore, in this work, we mainly focus on theoretically justifying the aforementioned unique designs.\n\n[1] Kuzborskij, Ilja, and Christoph Lampert. \"Data-dependent stability of stochastic gradient descent.\" International Conference on Machine Learning. PMLR, 2018.\n"}, {"review_id": "XIypaglBzv", "reviewer": "Reviewer_KXFG", "summary": "This paper studies the task distribution in meta learning and proposes modeling the task in multimodal distributions. To enable efficient modelling and inference, the author develops the hierarchical Gaussian mixture task generative mode and optimizes the meta learning model in expectation maximization. The performance of the method is verified in few-shot image classification tasks.", "questions": "(1) Would you please provide the detailed probabilistic graphical model to explain the relationship between the support, the query, the sample-wise embedding and etc? \n\n(2) Since the experiments were performed on typical benchmarks, would you explain the novel tasks a bit and how to define the discovery of novel tasks?\n\n(3) Are task distributions the same for the meta training and meta testing, or do out-of-distribution cases exist?", "limitations": "See the weakness section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is written clearly, and the modelling of task distribution has great significance in the literature.\n2. It shows improved performance in adopted benchmarks over existing metrics based methods.", "weaknesses": "(1) Probabilistic relations and graphical models. \n\nIt is necessary to explain why the generative process Eq. (1) is valid in metrics-based meta learning. In practice, meta learning seldom considers the sample embeddings and the task embedding is of interest. Hence, the conditional distribution $\\ln\\int p(D_{\\tau}^{q}| v_{\\tau})p(v_{\\tau}\\vert D_{\\tau}^{s})dv_{\\tau}$ is mostly commonly used as the objective to maximize. However, in Eq.(1), the conditional dependencies on the support dataset are neglected. My suggestion is that this needs to be reflected in Eq.(1) directly.\n\n(2) Model optimization. \n\nIn Line201-202, it says, “the query set is not included because it is unavailable during model testing”. This does not hold in Bayesian meta learning. Like that in conditional variational autoencoder or neural processes, the $q_{\\phi}(v_{\\tau}|D_{\\tau}^{s})$ is called the approximate prior and the approximate posterior $q_{\\phi}(v_{\\tau}|D_{\\tau}^{s}, D_{\\tau}^{q})$ is used in meta training for more effective inference (though the query set is unavailable in meta testing). Hence, I doubt the way of inference in this work since the conditional prior is most important for meta learning probabilistic models. Especially, the way to infer $e_i$ throughout the manuscript, e.g., $p(e_i|y_i)$ in Eq.(3)-(7), the one-hot information can be quite limited. However, the input $x_i$ is not included in the inference.\n\n(3) Mixture of task distributions. \n\nThe task embeddings lie in a mixture of Gaussian manifold is a reasonable assumption. For the task2vec work, the embedding of the task is visualized and analyzed a bit. In this work, the focus is on task distribution, so it is necessary to include this part in experimental analysis, for example, to check the grouping of latent structures in the task space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for helping us refine the work. Before answering the questions, please allow us to clarify our problem setting.\nIn this work, we aim to develop a robust meta-learning model for mixture task distribution such that when a meta-testing task (e.g. few-shot classification) comes, the model can:\n1. identify whether the task comes from the same mixture distribution of meta-training set (i.e., a known task), or a different distribution (i.e., a novel task). \n2. if the meta-testing task is a known task, adapt the model to this task with the task-level information for boosted performance.\nThe motivation is that in areas requiring robust decisions, such as healthcare, where the accuracy drop on novel tasks is inevitable, a robust and ethically safe choice is to raise an alarm to the users (e.g. doctors) for diagnosis and decision-making.\n\nResponse to Weakness (1): Please let us validate our design from two prespectives.\n\n1. The relation to the commonly used $\\ln\\int p(D^q_{\\tau}|v_{tau})p(v_{tau}|D^s_{\\tau})d_{v_{\\tau}}$\nAs in the aforementioned problem setting, our work is trying to handle both novel task detection and meta-learning. The commonly used $\\ln\\int p(D^q_{\\tau}|v_{\\tau})p(v_{\\tau}|D^s_{\\tau})d_{v_{\\tau}}=p(D^q_{\\tau}|D^s_{\\tau})$ in existing works only considers the how to infer the query set information (e.g. labels) given support set (i.e. meta-learning). Therefore, we propose to handle novel task detection by modeling the likelihood of a task $p(D^q_{\\tau},D^s_{\\tau})$ (Eq. (1)), so that we can use a low likelihood score to detect novel tasks in meta-testing. Also, note $p(D^q_{\\tau},D^s_{\\tau})=p(D^q_{\\tau}|D^s_{\\tau})p(D^s_{\\tau})$. Thus, our training objective function also contained the information of the commonly used likelihood function. \n\n2. The relation between $p(D_{\\tau}|v_{\\tau})$ and the sample embeddings. In our work, we modeled $p(D^q_{\\tau}|v_{\\tau})$ as (Eq. 1):\n$$\\log p(D_{\\tau}=\\\\{(x_i,y_i)\\\\}|v_{\\tau})=\\sum_i^n \\log p(x_i|y_i)p(y_i|v_{\\tau})$$\nThis is because we assume the samples in the task are i.i.d., so we can factorize the distribution to each sample's probability density. We modeled the sample probability density with their sample embeddings and class mean:\n$$p(x_i|y_i) = \\frac{\\exp(||f_{\\theta}(x_i),y_i||^2_2)}{\\sigma^2I} =\\mathcal{N}(e_i|\\mu_{y_i}^c,\\sigma^2I)$$\nThis operation is common in representation learning. For example, in [1], the authors applied similar definition, whose difference from us is that they applied inner product rather than Euclidean distance. In Eq. 1 of our draft, we directly wrote the $p(x_i|y_i)$ as $p(e_i|y_i)$ to emphasize that this probability can be efficiently computed in embedding space. \n\nResponse to Weakness (2): The difference between our model and the example in the review (conditional VAE and Neural Process) is as follows. In our case, during the meta-testing, the infered task embedding $v_{\\tau}$ is fed to the learnt GMM model to compute the prior likelihood $p(v_{\\tau})$. The likelihood value is used to detect novel tasks. For effective detection, we need to ensure the infered $v_{\\tau}$ of in-distribution tasks in meta-testing to have the same distribution as the infered $v_{\\tau}$ of meta-training. In the meta-testing stage, since the query set is not available for novel task detection, we can only infer the $v_{\\tau}$ via $q(v_{\\tau}|D^s_{\\tau})$. If we use $q(v_{\\tau}|D^s_{\\tau}, D^q_{\\tau})$ in meta-training for more powerful inference, there could be a distribution discrepancy between the in-distribution tasks $v_{\\tau}$ in the meta-training set and meta-testing set. Specifically, the variance of $q(v_{\\tau}|D^s_{\\tau})$ may be larger than $q(v_{\\tau}|D^s_{\\tau}, D^q_{\\tau})$ as the former is estimated on fewer samples.\n\nResponse to Weakness (3): We included the visualization of task embedding clusters in the common rebuttal PDF file.\n\nQ1.Detailed probabilistic graphical model?\n \nWe have drawn the probalistic graph for both generation and inference in the commom rebuttal PDF file (Fig. 3). \n\nQ2.Explain the novel tasks and how to define the discovery of novel tasks?\n\nIn typical benchmarks of meta-learning, we evaluate models by sampling a meta-testing dataset, where each data instance in this dataset is a task containing a support set and a query set that are both sampled from the same distribution of the meta-training dataset. The accuracy of meta-learning algorithms usually refers to the average performance of the model on all meta-testing tasks. In our work, a novel task, as we explained in line 22-25 and line 312-316, is a task whose support samples and a query samples are not drawn from the same distribution of the meta-training dataset. In our experiment of novel task detection, as we explained in line 312-316, we applied two typical benchmarks, Art-Muli and Plain-Multi, whose data samples (images) are drawn from different distributions. We then trained meta-learning models only on the Plain-Multi dataset. After that, during meta-testing stage, we mix the meta-testing tasks drawn from Art-Muli and Plain-Multi and then evaluate the models' ability to detect the meta-testing tasks drawn from Art-Muli (i.e. novel tasks).\n\nQ3. Do out-of-distribution cases exist?\n\nIf we understand it correctly, this question is about the settings in our two experiments.\nIn the few-shot classification experiment, since some baselines cannot handle out-of-distribution cases, for fair comparison, we only evaluated the accuracy of our model and baseline models on in-distribution tasks. In this case, out-of-distribution tasks do not exist in the meta-testing set. In the novel task detection experiment, we compared our model with the baselines that can detect out-of-distribution tasks. In this case, we mixed the in-distribution tasks and out-of-distribution tasks and evaluated the ability of the models in detecting the out-of-distribution tasks.\n\n[1] Jian Tang, et al. LINE: Large-scale Information Network Embedding. WWW 2015"}, {"review_id": "Ly0bIsnV48", "reviewer": "Reviewer_dA8b", "summary": "The authors propose a novel probabilistic meta-learning algorithm called HTGM\nwhich models the full hierarchical sampling procedure in few-shot classification\nin order to perform prediction and novel task detection. To learn the parameters\nthey employ the EM procedure together with variational inference. Experimental\nresults show that they perform best among competitors in few-shot classification\nprediction and novel task detection.", "questions": "# Partition Function\nCould you explain what the potential limitations of disregarding the partition\nfunction is? Intuitively it feels like the partition function would have a\nregularizatory effect and disregarding it may lead to some overfitting. Would be\ninterested to hear what the authors think the effect of being able to optimizing\nthis part of the objective would be.\n\n# Tricks\n- Tying $\\theta$ and $\\phi$: Is there are reason for letting the recognition\n  network $q$ be equal to $f$? Is the tying of the parameters done due to\n  convenience and that it empirically was found to work well or is there some other reason?\n- Negative sampling and partition function: Show empirical results that the\n  quantity $D\\_{hl}$ goes to infinity over the course of training.\n- Model adaptation: Is it possible to put the adaptation step on a probabilistic\n  footing? Right now it seems like it's pretty ad-hoc, can we for example trust\n  $p(y'\\_i = j' | x\\_i')$ more as a probabilistic model compared to using any\n  standard (non-probabilistic) metric learning few-shot algorithm which produces\n  an embedding $f_{\\theta}(x)$? One way to show this would be to show that $p$\n  here is more well-callibrated as opposed to using the backbone trained by\n  proto-net or some other algorithm.\n\n# Non-standard experimental dataset\nMeta-dataset is the standard modern community benchmark for few-shot\nmulti-domain classification. I'd be interested to see how the algorithm fares on\nthis benchmark since their probabilistic model explicitly models the\nmulti-domain aspect of few-shot learning for which the meta-dataset seems like a\ngood fit.\n", "limitations": "No, certain limitations should be expanded upon (see Questions). Societal impact is discussed correctly.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality & significance: The authors propose a probabilistic model of the full few-shot\n  classification process which allows for both adaptation (prediction) and\n  novelty detection. Additionally it can be added on top of other metric-learning few-shot learning algorithms out of the box.\n- Quality: The probabilistic model seem reasonable and they discuss problems and\n  solutions of design choices made. Experiments for both prediction and novelty-detection is convincing (but see the weakness section).\n- Clarity: The paper is well-written in general.\n", "weaknesses": "- Partition function: The authors show that the partition function is upper bounded and show asymptotic tightness of this bound for the minimal distance $D_{hl}$, but I'm still not fully convinced by this and would like to know more how this impacts modelling and \"what we lose\" in disregarding the partition function.\n- Probabilistic modelling and \"tricks\": The pipeline has several tricks which may help performance but weakens the interpretation of the model as inferring a correct probabilistic distribution.\n- Non-standard multi-domain experiments: The datasets of Plain-Multi and\n  Art-Multi are non-standard. Ideally additional experiments on for example\n  meta-dataset would be beneficial.\n- Typos:\n  1. Line 172: I think $\\mu_k^c$ should be $\\mu_1^c$ and $\\mu_N^c$?\n  2. Line 314: comapred -> compared\n  3. Line 341: they even don't fit -> they don't even fit\n  4. Table 2: I think the second entry in the \"Setting\" column should be \"5-way, 5-shot\"\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the constructive feedback. Here are our responses.\n\nQ1. Limitations of disregarding the partition function\n\nResponse: The main issue of replacing the partition function with its upper bound is that it increases the noise in the inferred class mean. Note that the entropy of distribution $\\pi(\\mu_k^c|v_{\\tau},w)$ ($\\mu_k^c$ is a class mean) in Eq. (2) is that:\n$$H(\\pi)=-\\int\\pi(\\mu_k^c|v_{\\tau},w)\\log\\pi(\\mu_k^c|v_{\\tau},w)d\\mu_k^c =\\log Z+\\bar{E_w}$$\nwhere $\\bar{E_w}$ is the average energy. Therefore replacing $Z$ with its upper bound increases the entropy of $\\pi(\\mu_k^c|v_{\\tau},w)$, leading to more noise in $\\mu_k^c$. This leads to a decreasing accuracy of meta-learning. To alleviate this issue, we proposed to include an negative sampling term as in Eq. (4) to increase the class mean distance to reduce the influence of the noise\n\nQ2.1 Tying parameters.\n\nResponse: The purpose of the inference network $\\phi$ in $q_{\\phi}$ is to infer $v_{\\tau}$ from the support set $D_{\\tau}^{s}$ in the embedding space, where the input $D_{\\tau}^{s}$ is the same as an encoding network $f$. Thus we built the inference network $\\phi$ upon $f$ (with two aggregation functions) to minimize parameter use for avoiding overcomplicated model. As in Fig 1, the two aggregation functions first aggregates the sample embedings in the support set $D_{\\tau}^{s}$ to class embeddings, then aggregates the class embeddings to the task embedding $v_{\\tau}$. This is similar to VAE where the inference network is an encoder, but in our case, the inference network coincides with the encoder (a.k.a., the base model) in the metric-based meta-learning.\n\nOn the other hand, we think it is unnecessary to define two encoders, one for metric-based meta-learning, i.e., classifying the sample embedings, another for the inference network, i.e., infer task embeddings. This is because the task embedding is inferred from the class embeddings, thus classifying sample embeddings is inevitable in the inference network, making the former encoder a duplicate. Hence, it is intuitive to build $\\phi$ in $q_{\\phi}$ upon $f$.\n\nHowever, it is possible to add more parameters on $f$ to form $\\phi$, such as using parameteric aggregation functions than non-parametric ones, as the framework is flexible. In this work, we used the simplest model structure as it was found suffiently effective to demonstrate the usefulness of the proposed model.\n\nQ2.2 Empirical results of quantity $D_{hl}$.\n\nResponse: We tested the average $D_{hl}$ of our model on different training steps. And with $D_{hl}$, we can estimate the error ratio of our approximation to the partition function. This is because in our model, the error ratio $\\frac{N\\sqrt{2^{d-1}\\pi^d}-Z}{N\\sqrt{2^{d-1}\\pi^d}}$ is bounded by $er(D_{hl})=\\frac{\\gamma(\\frac{d}{2},D_{hl}^2/4)}{(\\frac{d}{2}-1)!}$. We calculate them for each average $D_{hl}$ and include them in the Fig. 2 in the common rebuttal. As we can see, as training goes by, the error ratio of our approximation monotonically decreases. At step 7000, where the error ratio is 0.25, the model acquires the best validation accuracy, which means that 0.25 is enough for the model to handle the tasks.\n\nQ2.3 Model adaptation with probablistic explanation.\n\nResponse: Yes, this model adaptation has a probablistic explanation consisting of two steps.\n\nFirst we learn class means on support set: When a meta-testing episode comes, for each class $y_j$, we first encode its support samples to acquire its support sample embeddings \n$e^{(j)}_1,e^{(j)}_2,...,e^{(j)}_k$. \n\nThen we infer the task embedding $v_{\\tau}$ via $q_{\\Phi}(v_{\\tau}|D^s_{\\tau})$ (in line 208). After that, we freeze other parameters and learn class embedding $\\bar{\\mu}_j^c$ for each class $y_j$ \nvia maximum likelihood estimation. According to line 184-187, the likelihood function that we observe the support set of each class $y_j$ should be \n$$L(\\bar{\\mu}_j^c)=\\log p({y}_j|v{\\tau})\\prod_1^k p({e}{i}^{(j)}|{y}j))=\\log \\pi(\\bar{\\mu}^ck|v{\\tau},w)\\prod_1^k\\mathcal{N}(e^{(j)}_i|\\bar{\\mu}^c,\\Sigma)$$\n\ni.e. $p(y_j|v_{\\tau})$ is the likelihood of step (a) in line 184, and $\\prod_1^k p(e^{(j)}i|yj))$ is the likelihood of step (b) in line 185-186 (the inner loop). Our theoretic analysis in the common rebuttal shows the adaptation step $\\bar{\\mu}_j=\\alpha{\\mu}^c + (1-\\alpha)W{l}v{\\tau}$\nmaximizes the above likelihood function. \n\nHowever, our empirical evaluation shows that sometimes the emperically optimal value of $\\alpha$ might be different from theoretically optimal value $\\frac{k}{k+2\\sigma^{2d}}$ (See the common rebuttal). We suggests that it might be because the noise in partition function approximation (see the answer to Q1). Therefore, we treat $\\alpha$ as a hyperparameter to fineunte on validation set.\n\nSecond, we classify the query sample $x'$: We first compute $p(y'=j'|x')$ i.e. the posterior distribution of the query sample's label $y'$ conditioned on the sample. Then we select the label with the highest probability. Given the learnt class means, we compute the conditional distribution of label:\n\\begin{equation}\n    p(y'_i=j'|x'_i) = \\frac{p(y'_i=j',x'_i)}{\\sum_j^Np(y'_i=j,x'_i)} =\\frac{p(x'_i|y'_i=j')p(y'_i=j')}{\\sum_j^Np(x'_i|y'_i=j)p(y'_i=j)}\n\\end{equation}\nDuring the meta-testing stage, we should equally treat each class, i.e. assuming $p(y'_i=1)=...=p(y'_i=N)$. So we have:\n\\begin{equation}\n   p(y'_i=j'|x'_i) = \\frac{\\exp(||f{\\theta}(x'i)-\\bar{\\mu}{j'}^c||}{\\sum_j^N\\exp(||f{\\theta}(x'i)-\\bar{\\mu}j^c||}\n\\end{equation}\n\nQ3. Non-standard experiment dataset\n\nResponse: We used the current datasets in our experiments because they were regarded as benchmarks and used by other mixture-distribution based meta-learning works (e.g., [47]). Among them, the Plain-Multi dataset consists of four datasets that also exists in Meta-Dataset. The main difference is that this benchmark does not include Mini-ImageNet. Thus, we provided an experiment on Mini-ImageNet dataset in our Appendix D.6\n\n"}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate your valuable comments that help us refine our work. If you have more questions or concerns about our response or current draft, please let us know. We are happy to discuss with you. We have uploaded a PDF file that contains figure and addtitional theoretic analysis for reference.

Best regards,

Authors


Meta-review: The four reviews of this paper all leaned towards acceptance but not strongly, with two borderline accept and two weak accept recommendations; none of the reviewers wished to champion the paper.

Several positive aspects were appreciated:
+ One reviewer considered the modeling of tasks as a mixture of distributions as novel (k7fv)
+ The probabilistic model was considered reasonable (dA8b)
+ Meta-learning both the mixture of task distributions and detecting novelty of testing tasks was appreciated (XMuD) 
+ The workaround for computing eq. 3 was appreciated (k7fv)
+ Experiments were considered convincing (dA8b)
+ The improved performance over existing methods was appreciated (KXFG)
+ The paper was considered well written (k7fv,KXFG,dA8b) and easy to follow (XMuD)

However, concerns were also raised:
- There was concern of novelty: while the paper applies the hierarchical Gaussian mixture model to meta-learning, the method itself is not novel (XMuD). Authors provided some argumentation towards novelties in a rebuttal.
- There was concern about validity of the generative process (KXFG). Reviewers provided argumentation in a rebuttal.
- Somewhat similarly, there was concern that various tricks of the pipeline might weaken the interpretation that the model infers a correct distribution (dA8b)
- One reviewer was not convinced by the upper bound of the partition function and wished more discussion of what is lost by disregarding the partition function (dA8b); authors provided discussion in a rebuttal.
- There was a desire to check the groupings of tasks in a latent space (KXFG); authors provided an embedding visualization in a rebuttal.
- A deteled graphical model of the relationships between variables such as the support, query, and sample-wise embedding was desired. (KXFG); authors provided a graph drawing in a rebuttal.
- There was criticism of the notion that the query set is not included as it is unavailable during model testing; a reviewer argued that does not hold in Bayesian meta-learning (KXFG)
- The theoretical analysis was considered limited, and an analysis of the generalization bound was desired (XMuD)
- The loss function with negative sampling l_neg was considered somewhat heuristic (k7fv)
- Lack of a computational cost analysis was criticized (k7fv); authors provided some computation time results in a rebuttal.
- The two data sets were considered non-standard and experiments on additional data such as a meta-dataset were desired (dA8b)
- The experiments on two data sets were considered overly limited, and experiments on larger scale datasets were desired (k7fv); author provided one additional data set (Mini-ImageNet) in a rebuttal 
After discussion, reviewers felt the rebuttals addressed some of the concerns but some concerns such as how the method would work on datasets like meta-dataset remained. 

Overall the paper feels borderline due to the amount of concerns that were raised, but reviewers mostly seemed to appreciate the rebuttals. Meta-learning is likely a rising need, as ever larger existing models become available whereas data gathered in individual scenarios/tasks may remain sparse; thus new probabilistic solutions to meta-learning and novel task detection could be valuable to the community. Although the approach is an adaptation of hierarchical Gaussian mixture modeling, authors have argued it contains various methodological novelties, and showing that it can yield competitive and outperforming results in meta-learning is useful. Therefore, it may be worth giving the work a chance; if the points raised in rebuttals are adequately incorporated into the paper, the work may be suitable for presentation at NeurIPS.
Decision: Accept (poster)