Title: Hierarchical Gaussian Mixture based Task Generative Model for Robust Meta-Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Few-Shot Learning

Meta Learning

Task Representation
Abstract: Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task embeddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization (EM) algorithm. Extensive experiments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.

Hierarchical Gaussian Mixture based Task Generative Model for Robust Meta-Learning

Abstract

Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a metatraining framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task embeddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization (EM) algorithm. Extensive experiments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.

1 Introduction

Training models in small data regimes is of fundamental importance. It demands a model's ability to quickly adapt to new environments and tasks. To compensate for the lack of training data for each task, meta-learning ( a.k.a. learning to learn) has become an essential paradigm for model training by generalizing meta-knowledge across tasks [41, 8]. While most existing meta-learning approaches were built upon an assumption that all training/testing tasks are sampled from the same distribution, a more realistic scenario should accommodate training tasks that lie in a mixture of distributions, and testing tasks that may belong to or deviate from the learned distributions. For example, in recent medical research, a global model is typically trained on the historical medical records of a certain set of patients in the database [40, 49]. However, due to the uniqueness of individuals ( e.g. , gender, age, genetics) [43], patients' data have a substantial discrepancy, and the pre-trained model may demonstrate significant demographic or geographical biases when testing on a new patient [35, 10]. This issue can be mitigated by personalized medicine approaches [5, 31], where each patient is regarded as a task, and the pre-trained model is fine-tuned ( i.e. , personalized) on a support set of a few records collected in a short period ( e.g. , a few weeks) from every patient for adaptation. In this case, the training tasks ( i.e., patients) could be sampled from a mixture of distributions ( e.g. , different age

groups), and a testing task may or may not belong to any of the observed groups. Similar examples can be found in other applications such as the detection of fake news dissemination [7, 55, 57], where a task is a post, whose support set consists of a few profiles of users who have disseminated it in a short period. The training posts may be drawn from a mixture of topics, and a testing post may not belong to any of the topics. As such, a meta-training strategy that is able to fit a model to a mixture of task distributions and enable the identification of novel tasks during inference time is desirable for making meta-learning a practical solution.

One way to tackle the mixture distribution of tasks is to tailor the transferable knowledge to each task by learning a task-specific representation [33, 48, 25], but as discussed in [52], the overcustomized knowledge prevents its generalization among closely related tasks ( e.g. , tasks from the same distribution). The more recent methods try to balance the generalization and customization of the meta-knowledge by promoting local generalization either among a cluster of related tasks [52], or within a neighborhood in a meta-knowledge graph of tasks [53]. Neither of them explicitly learns the underlying distribution from which the tasks are generated, rendering them infeasible for detecting novel tasks that are out-of-distribution. However, detecting novel tasks is crucial in high-stake domains, such as medicine and finance, which provides users ( e.g. , physicians) confidence on whether to trust the results of a testing task or not, and facilitates downstream decision-making.

In [21], a task-specific tuning variable was introduced to modulate the initial parameters learned by the optimization-based meta-learning method MAML [8], so that the impacts of the meta-knowledge on different tasks are adjusted differently, e.g. , novel tasks are less influenced than the known tasks. Whereas, this method focuses on improving model performance on different tasks (either known or novel), but neglects the critical mission of detecting which tasks are novel. In practice, providing an unreliable accuracy on a novel task, without differentiating it from other tasks may be meaningless.

Since the aforementioned methods cannot simultaneously handle the mixture distribution of tasks and novel tasks, a practical solution is in demand. In this work, we consider tasks as instances, and demonstrate the dual problem of modeling the mixture of task distributions and detecting novel tasks are two sides of the same coin, i.e. , density estimation on task instances. To this end, we propose a Hierarchical Gaussian Mixture based Task Generative Model (HTGM) to explicitly model the generative process of task instances. Our contributions are summarized as follows.

We extended the widely used empirical process of generating tasks to a theoretical process specified by a hierarchy of Gaussian mixture (GM) distributions. HTGM generates a task embedding from a task-level GM, and uses it to define the task-conditioned mixture probabilities for a class-level GM, from which the data samples are drawn. To allow realistic classes per task, a new Gibbs distribution was proposed to underlie the class-level GM.

HTGM is an encoder-agnostic framework, thus is flexible to different domains. It inherits metricbased meta-learning methods, and only introduces a small overhead to an encoder for parameterizing its distributions, thus is efficient for large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via a principled EM algorithm. The bounds of our likelihood function were also theoretically analyzed.

In the experiments, we evaluated HTGM on benchmark datasets regarding its scalability to large networks, effectiveness in modeling the mixture distribution of tasks, and usefulness in identifying novel tasks. The results demonstrate HTGM outperforms the state-of-the-art (SOTA) baselines with significant improvements in most cases.

2 Related Work

To the best of our knowledge, this is the first work to explicitly model the generative process of task instances from a mixture of distributions for meta-learning with novel task detection. Meta-learning aims to handle the few-shot learning problem, which derives memory-based [30], optimizationbased [8, 26], and metric-based methods [47, 41], and often considers an artificial scenario where training/test tasks are sampled from the same distribution. To enable more varying tasks, taskadaptive methods facilitate the customization of meta-knowledge by learning task-specific parameters [37, 25], temperature scaling parameters [33], and task-specific modulation on model initialization [48, 52, 53, 21]. Among them, there are methods tackling the mixture distribution of tasks by clustering tasks [52, 13, 14, 17], learning task similarity and graphs [58, 53], and relocating the initial parameters for different tasks so that they use the meta-knowledge differently [21]. As discussed

before, none of these methods jointly handles the mixture of task distributions and the detection of novel tasks in meta-testing stage. A more detailed discussion is in Appendix B.1.

Our model is built upon metric-based methods and learns task embeddings to model task distributions. Achille et al. [1] proposed to learn embeddings for tasks and introduced a meta-learning method, but not for few-shot learning. Their embeddings are from a pre-specified set of tasks (rather than episode-wise sampling), and the meta-learning framework is for model selection. The model in [52] has an augmented encoder for task embedding, but it does not model task generation, and is not designed for novel task detection (an empirical comparison is in Sec. 4.1).

The conventional novelty detection aims to identify and reject samples from unseen classes [6]. It relates to open-set recognition [46], which aims to simultaneously identify unknown samples and classify samples from known classes. Out-of-distribution (OOD) detection [27, 28] can be seen as a special case of novelty detection where novel samples are from other problem domains or datasets, thus are considered to be easier to detect than novelties [6]. These methods are for large-scale training. In contrast, we want to detect novel tasks, which is a new problem in the small data regime.

Hierarchical Gaussian Mixture (HGM) model has appeared in some traditional works [9, 32, 3, 50] for hierarchical clustering by applying GM agglomeratively or divisively, which do not pre-train models for meta-learning, and is remarkably different from the topic in this paper. The differences are elaborated in Appendix B.2. Moreover, the idea of learning groups/clusters of tasks also appeared in some multi-task learning (MTL) models. The key difference between these methods and our method HTGM lies in the difference between MTL and meta-learning. In an MTL method, all tasks are known a priori , i.e. , the testing tasks are from the set of training tasks, and the model is inductive at the sample-level but non-inductive at the task-level. More discussions are in Appendix B.3.

3 Hierarchical Gaussian Mixture based Task Generative Model (HTGM)

3.1 Problem Statement

Meta-learning methods typically use an episodic learning strategy, where the meta-training set D tr consists of a batch of episodes. Each episode samples a task τ from a distribution p ( τ ) . Task τ has a support set D s τ = { ( x s i , y s i ) } n s i =1 for training, and a query set D q τ = { ( x q i , y q i ) } n q i =1 for testing, where n s is a small number to denote a few training samples. In particular, in a commonly used N -way K -shot Q -query task [47], D s τ and D q τ contain N classes, with K and Q samples per class respectively, i.e. , n s = NK and n q = NQ .

Let f θ ( x ∗ i ) → y ∗ i be a base model ( ∗ denotes s or q ), and f θ ( · ; D s τ ) be the adapted model on D s τ . The training objective on τ is to minimize the average test error of the adapted model, i.e. , E ( x q i ,y q i ) ∈D q τ ℓ ( y q i , f θ ( x q i ; D s τ )) , where ℓ ( · , · ) is a loss function ( e.g. , cross-entropy loss), and the metatraining process aims to find the parameter θ that minimizes this error over all episodes in D tr . Then, f θ is evaluated on every episode of a meta-test set D te that samples a task from the same distribution p ( τ ) . Usually, p ( τ ) is a simple distribution [8, 21]. In this work, p ( τ ) is generalized to a mixture distribution consisting of multiple components p 1 ( τ ) , ..., p r ( τ ) , and a test episode may sample a task either in or out of any component of p ( τ ) . As such, given the training tasks in D tr , our goal is to estimate the underlying density of p ( τ ) , so that once a test task is given, we can (1) identify if it is a novel task, and (2) adapt f θ to it with optimal accuracy. Specifically, the base model f θ can be written as a combination of an encoder g θ e and a predictor h θ p , i.e. , f θ ( x ∗ i ) = h θ p ( g θ e ( x ∗ i )) [44]. In this work, we focus on a metric-based non-parametric learner, i.e. , θ p = ∅ ( e.g. , prototypical networks [41]), not only because metric-based classifiers were confirmed as more effective than probabilistic classifiers for novelty detection [12], but also for its better training efficiency that fits large backbone networks than the costly nested-loop training of optimization-based methods [44].

Formally, our goal is to find the model parameter θ that maximizes the likelihood of observing a task τ . In other words, let f θ ( x ∗ i ) = e ∗ i ∈ R d be sample embedding, we want to maximize the likelihood of the joint distribution p θ ( e ∗ i , y ∗ i ) on the observed data in D τ = {D s τ , D q τ } . We consider each task τ as an instance, with a representation v τ ∈ R d in the embedding space (the method to infer v τ is described in Sec. 3.3). To model the unobserved mixture component, we associate every task with a latent variable z τ to indicate to which component it belongs. Suppose there are r possible components, and let n = n s + n q be the total number of samples in D τ , the log-likelihood

to maximize can be written by hierarchically factorizing it on y ∗ i and marginalizing out v τ and z τ .

where p θ ( e ∗ i | y ∗ i ) specifies the probability of sampling e ∗ i from the y ∗ i -th class, p ( y ∗ i | v τ ) is the probability of sampling the y ∗ i -th class for task τ , and p ( v τ | z τ ) indicates the probability of generating a task τ from the z τ -th mixture component. p ( z τ ) is a prior on the z τ -th component. Hence, Eq. (1) implies a generative process of task τ : z τ → v τ → y ∗ i → e ∗ i . Next, we define each of the aforementioned distributions and propose our HTGM method.

3.2 Model Specification and Parameterization

In Eq. (1), the class-conditional distribution p θ ( e ∗ i | y ∗ i ) , the task-conditional distribution p ( y ∗ i | v τ ) , and the mixture distribution of tasks defined by { p ( v τ | z τ ) , p ( z τ ) } are not specified. To make Eq. (1) optimizable, we introduce our HTGM that models the generative process of tasks. Because D s τ and D q τ follow the same distribution, in the following, we ignore the superscript ∗ for simplicity.

Class-Conditional Distribution. First, similar to [22, 23], we use Gaussian distribution to model the embeddings e i 's in every class. Let µ c y i and Σ c y i be the mean and variance of the y i -th class, then p θ ( e i | y i ) = N ( e i | µ c y i , Σ c y i ) . In fact, the samples in all of the classes of task τ comprise a Gaussian mixture distribution, where p ( y i ) is the mixture probability of the y i -th class. In Eq. (1), p ( y i ) is factorized to be task-specific, i.e. , p ( y i | v τ ) , which resorts to another mixture distribution p ( v τ ) of tasks, and establishes a structure of hierarchical mixture.

Task-Conditional Distribution. A straightforward definition of p ( y i | v τ ) is the density at µ c y i in a Gaussian distribution with v τ as the mean, where µ c y i is the mean (or prototype) of the y i -th class. However, doing so exposes two problems: (1) the density function of Gaussian distribution is log-concave with one global maximum. Given the mean and variance, maximizing its log-likelihood tends to collapse the prototypes µ c y i 's of all classes in τ , making them indistinguishable and impairing classification; (2) given v τ , this method tends to sample classes with small D v τ ( µ c y i ) , where D v τ ( · ) measures the Mahalanobis distance between a data point and the Gaussian distribution centered at v τ . However, in most of the existing works, classes are often uniformly sampled from a domain without any prior on distances [8]. Fitting the distance function with such 'uniform' classes naively leads to an ill-posed learning problem with degenerated solutions. In light of these issues, we seek to define p ( y i | v τ ) as a (parameterized) density function with at least N global optimums so that it can distinguish the N different class prototypes of N -way tasks. The N equal (global) optimums also allow it to fit N classes uniformly sampled from a domain. To this end, let µ c k be the surrogate embedding of the k -th class, we propose a Gibbs distribution π ( µ c k | v τ , ω ) defined by v τ and trainable parameters ω with an energy function. Then we write p ( y i = k | v τ ) as

where E ω ( µ c k ; v τ ) = min ( {|| µ c k -W j v τ || 2 2 } N j =1 ) is our energy function, and the denominator in Eq (2) is a normalizing constant (with respect to µ c k ), a.k.a. the partition function in an energy-based model (EBM) [20]. ω = { W 1 , ..., W N } are trainable parameters, with W i ∈ R d × d . Given ω and v τ , Eq. (2) has N global maximums at µ c k = W 1 v τ , ..., µ c k = W N v τ . More interpretations of the proposed task-conditional distribution can be found in Appendix B.4.

Mixture Distribution of Tasks. In Eq. (1), the task distribution p ( v τ ) is factorized as a mixture of p ( v τ | z τ = 1) , ..., p ( v τ | z τ = r ) , weighted by their respective mixture probability p ( z τ ) . Thus we specify p ( v τ ) as a Gaussian mixture distribution. We introduce µ t z τ and Σ t z τ as the mean and variance of each component, i.e. , p ( v τ | z τ ) = N ( v τ | µ t z τ , Σ t z τ ) , and let ρ = [ ρ 1 , ..., ρ r ] be the mixture probabilities, where ρ r = p ( z τ = r ) and ρ can be Uniform ( r ) . Then v τ is genetrated in two

steps: (1) draw a latent task variable z τ from a categorical distribution on ρ , and (2) draw v τ from N ( µ t z τ , Σ t z τ ) [4]. As such, our HTGM generative process of an N -way K -shot Q -query task τ can be summarized as following, and Fig. 1 illustrates the corresponding graphical model:

Draw z τ ∼ Categorical ([ ρ 1 , ..., ρ r ])

Draw a task embedding v τ ∼ N ( µ t z τ , Σ t z τ )

For k = 1 , ..., N :

(a) Draw a class prototype µ c k ∼ π ( µ c k | v τ , ω ) from the proposed Gibbs distribution in Eq. (2)

(b) For i = 1 , ..., K + Q :

Set y i = k , draw a sample e i ∼ N ( e i | µ c y i , Σ c y i )

Allocate ( e i , y i ) to the support set D s τ if i ≤ K ; else allocate ( e i , y i ) to the query set D q τ

To reduce complexity, we investigate the feasibility of using isotropic Gaussian with tied variance, i.e. , Σ c 1 = ... = Σ c N = σ 2 I , for class distributions, which turned out to be efficient in our experiments. Here, I is an identity matrix, σ is a hyperparameter. Tied variance is also a commonly used trick in Gaussian discriminate analysis (GDA) for generative classifiers [22, 39]. For task distributions, the variances Σ t 1 , ..., Σ t r can be automatically inferred by our algorithm, as elaborated in Sec. 3.3.

Finally, in Eq. (1), substituting p θ ( e i | y i ) = N ( e i | µ c y i , σ 2 I ) , p ω ( y i | v τ ) = π ( µ c y i | v τ , ω ) , p ( v τ | z τ ) = N ( v τ | µ t z τ , Σ t z τ ) , whose probabilities are specified and parameterized, and letting ρ = Uniform ( r ) be a uniform prior, we get our

𝚺

!

𝚺

"

𝛍

!

𝑧

#

𝐞

$

𝝆

𝐯

#

𝛍

%

"

𝐾 + 𝑄

𝑁

𝐵

HTGM induced loss ℓ HTGM ( D τ ; θ , ω ) . The class means µ c y i , task means µ t z τ and variances Σ t z τ are inferred in the E-step of our EM algorithm (the details can be found in Sec. 3.3 and A.6).

3.3 Model Optimization

It is hard to directly optimize ℓ HTGM ( D τ ; θ , ω ) , because the exact posterior inference is intractable (due to the integration over v τ ). To solve it, we resort to variational methods, and introduce an approximated posterior q ϕ ( v τ |D s τ ) , which is defined by an inference network ϕ , and implies we want to infer v τ from its observed support set D s τ . The query set D q τ is not included because it is unavailable during model testing. Then we propose to maximize a lower-bound of Eq. (1), which is (the derivation can be found in Appendix A.1)

where H ( q ϕ ( v τ |D s τ )) = -∫ v τ q ϕ ( v τ |D s τ ) log q ϕ ( v τ |D s τ ) d v τ is the entropy function. Similar to VAE [16], Eq. (3) estimates the expectation (in the second term) by sampling v τ from q ϕ ( v τ |D s τ ) , instead of the integration in Eq. (1), hence facilitates efficient computation. Next, we elaborate on the inference network, the challenges of maximizing Eq. (3), and our workarounds.

Inference Network. Similar to V AE, q ϕ ( v τ |D s τ ) is defined as a Gaussian distribution N ( µ a z τ , ¯ σ 2 I ) , where µ a z τ is the output of the inference network, which approximates µ t z τ in Step 2 of the generative process, and ¯ σ is a hyperparameter for the corresponding variance. As illustrated by Fig. 2(a), the inference network is built upon the base model f θ ( · ) with two non-parametric aggregation ( i.e. , mean pooling) functions, thus ϕ = θ . The first function aggregates class-wise embeddings to prototypes µ c y i 's, similar to prototypical networks [41]. Differently, the second aggregates all prototypes to µ a z τ . During model training, we used the reparameterization trick [16] to sample v τ from N ( µ a z τ , ¯ σ 2 I ) . It is noteworthy that H ( q ϕ ( v τ |D s τ )) in Eq. (3) becomes a constant because ¯ σ 2 is a constant.

Challenge 1: Trivial Solution. In Eq. (3), since the first term log p θ , ω ( e i | y i ) = -1 2 σ 2 d ∥ e i -µ c y i ∥ 2 2 (constants were ignored) only penalizes the distance between a sample e i and its own class mean µ c y i ( i.e. , intra-class distances) without considering inter-class relationships, different class means µ c 1 , ...,

𝜋 𝝁

!

!

'

|𝐯

"

, 𝝎

𝑝 𝐯

"

|𝑧

"

𝑝 𝑧

"

𝐖

#

, … , 𝐖

$

𝐯

"

𝑝 𝐞

%

|𝑦

%

𝐯

"&

𝐖

#

, … , 𝐖

$

𝐖

#

𝐯

"&

, … , 𝐖

$

𝐯

"&

Training task embeddings

Novelty?

Classification

Query set

𝓓

𝝉"

𝒒

(a)

(b)

Query set

{𝐱

&

#

}

&'(

)

!

Support set

{𝐱

&

*

}

&'(

)

"

Class-pooling

𝑓

!

𝑓

!

{𝐞

&

*

}

&'(

)

"

Task-level GMM

Class means

𝝁

(

-

, … , 𝝁

+

-

*

Class means

Adapted class means

Adaptation

Gibbs distribution

3

2

1

{𝐞

&

#

}

&'(

)

!

𝑞

.

𝐯

!

|𝒟

!

*

Support set

{𝐱

&

*

}

&'(

)

"

{𝐞

&

*

}

&'(

)

"

Data flow

Pooling

Sampling

Part of training loss

𝝁

(

/

, … , 𝝁

,

/

Model parameters

𝝓

Task-pooling

Class-pooling

Class-pooling

Task-pooling

Training process

Testing process

Class means

𝝁

(

-

, … , 𝝁

+

-

#

𝑞

.

𝐯

!"

|𝒟

!"

*

𝝁

0

#

1

𝝁

0

#$

1

µ c N in task τ could collide, drawing all sample embeddings to the same spot. To avoid such a trivial solution and improve the stability of optimization, we apply negative sampling [29]

where e j is a negative sample embedding from any class in the support set, and µ c y i is the mean of the positive class. In practice, we found it is beneficial to integrate ℓ neg with our likelihood ℓ HTGM in Eq. (1) during training, i.e. ℓ HTGM + 1 n ∑ n i =1 ℓ neg. Correspondingly, from Eq. (3) we have

which does not only serve as a robust training loss, but also helps solve the next challenge.

Challenge 2: The Partition Function in Eq. (2) . The second term p ω ( y i | v τ ) in Eq. (3) involves computing the partition function in Eq. (2) ( i.e. , the denominator), which is intractable because of the integration over all possible µ c k 's. To solve it, we propose an upper bound of the partition function ∫ µ k exp [ -E ω ( µ c k ; v τ ) ] d µ c k ≤ N √ π d (the derivation is in Appendix A.2), which is a constant with a specific N . By replacing the partition function in Eq. (2) with N √ π d , we got a lower bound of p ω ( y i | v τ ) , which in turn relaxes the lower bound in Eq. (3). The following theorem (the proof is in Appendix A.3) states that the tightness of the relaxed bound is controllable.

Theorem 3.1. Among the N global maximums W 1 v τ , ..., W N v τ of Eq. (2) , let W h v τ and W l v τ ( 1 ≤ h, l ≤ N ) be the pair with the smallest Euclidean distance D hl , we have

This theorem indicates the partition function approximates N √ π d when all pairs of the global maximums are far apart. It is noteworthy that when maximizing the likelihood, we fit W 1 v τ , ..., W N v τ to different class prototypes µ c 1 , ..., µ c N in N -way tasks. Because ℓ neg in Eq. (4) tends to maximize the distances between different prototypes through the negative samples, maximizing the joint loss ℓ in Eq. (5) tends to separate W 1 v τ , ..., W N v τ , thus tighten the relaxed bound after using N √ π d according to Theorem 3.1. This is another benefit of negative sampling.

Optimization via Expectation-Maximization. In the third term of ℓ HTGM-L in Eq. (3), we need to estimate the mixture distribution p ( z τ ) . Similar to optimize Gaussian mixture models, we alternately

infer the posterior p ( z τ | v τ ) ( i.e. , the mixture membership of v τ ) and solve the model parameters { θ , ω } through an Expectation-Maximization algorithm. In E-step, we infer p ( z τ | v τ ) when fixing model parameters. In M-step, when fixing p ( z τ | v τ ) , { θ , ω } can be efficiently solved by optimizing Eq. (5) with stochastic gradient descent (SGD). The detailed optimization algorithm of HTGM can be found in Appendix A.6.

3.4 Model Adaptation

Fig. 2(b) illustrates the adaptation process of HTGM. Given a new N -way task τ ′ from the meta-test set D te , its support set D s τ ′ is fed to the inference network to generate (1) class prototypes µ c 1 , ..., µ c N 2 (similar to prototypical networks), and (2) distribution q ϕ ( v τ ′ |D s τ ′ ) , from which we draw the average task embedding v τ ′ = µ a z τ ′ . Recall that the inference network is the base model f θ ( · ) with class-pooling and task-pooling layers, as illustrated in Fig. 2(b), and ϕ = θ . Then v τ ′ is projected to W 1 v τ ′ , ..., W N v τ ′ which represent the N optimal choices of class prototypes for task τ ′ as learned by the Gibbs distribution in Eq. (2) from the training tasks. They are used to adapt µ c 1 , ..., µ c N so that the adapted prototypes are drawn towards the closest classes from the mixture component that task τ ′ belongs to. Specifically, the adaptation is performed by selecting the closest optimum for each prototype µ c j ( 1 ≤ j ≤ N ), that is

and D ( · , · ) is the Euclidean distance, α is a hyperparameter. The following theorem confirms the effectiveness of the this adaptation method (the proof can be found in Appendix A.4).

Theorem 3.2. The adapted prototypes ¯ µ c 1 , ..., ¯ µ c N from Eq. (7) maximizes the lower-bound ℓ HTGM-L in Eq. (3) of the likelihood in Eq. (1) when α = K K +2 σ 2 d .

Theorem 3.2 suggests an automatic setup of α , which can also be tuned empirically for optimal value using validation datasets. We evaluated the empirical values of α in our experiments and discussed their relationships with the theoretical values in Appendix A.4.

Finally, we (1) assess if τ ′ is a novelty by computing the likelihood of v τ ′ in a pre-fitted GMM on the embeddings v τ 's of the training tasks in D tr , and (2) perform classification on each sample x ′ i in the query set D q τ ′ using the adapted prototypes by

which is the posterior probability. The derivation of Eq. (8) is in Appendix A.5.

4 Experiments

In this section, we evaluate HTGM's effectiveness on few-shot classification and novel task detection on benchmark datasets, and compare it with SOTA methods.

Datasets. The first dataset is the Plain-Multi benchmark [52]. It includes four fine-grained image classification datasets, i.e. , CUB-200-2011 (Bird), Describable Textures Dataset (Texture), FGVC of Aircraft (Aircraft), and FGVCx-Fungi (Fungi). In each episode, a task samples classes from one of the four datasets, so that different tasks are from a mixture of the four domains. The second dataset is the Art-Multi benchmark [53], whose distribution is more complex than Plain-Multi. Similar to [13], each image in Plain-Multi was applied with two filters, i.e. , blur filter and pencil filter, respectively, to simulate a changing distribution of few-shot tasks. Afterward, together with the original four datasets, a total of 12 datasets comprise the Art-Multi, and each task is sampled from one of them. Both benchmarks were divided into the meta-training, meta-validation, and meta-test sets by following their corresponding papers. Moreover, we used the Mini-ImageNet dataset [47] to evaluate the case of uni-component distribution of tasks, which is discussed in Appendix D.6.

Baselines. We compared HTGM with the relevant SOTA methods on meta-learning, including (1) optimization-based methods: MAML [8] and Meta-SGD [26] learn globally shared initialization

among tasks. MUMOMAML [48] is a task-specific method. TAML [21] handles imbalanced tasks. HSML [52] and ARML [53] learn locally shared initial parameters in clusters of tasks and neighborhoods of a meta-graph of tasks, respectively; and (2) Metric-based methods: ProtoNet [41] is the prototypical network. MetaOptNet [24] uses an SVM classifier with kernel metrics. ProtoNet-Aug [42], FEATS [54] and NCA [19] were built upon ProtoNet by augmenting images ( e.g. , rotation, jigsaw), adding prototype aggregator ( e.g. , Transformer), and using contrastive training loss (instead of prototype-based loss), respectively. The detailed setup of these methods is in Appendix C.1.

Implementation. Following [44], the optimization-based baselines used the standard four-block convolutional layers as the base learner, and the metric-based methods used ResNet-12. The output dimension of these networks is 640 (MetaOptNet uses 16000 as in its paper). In our experiments, we observed the optimization-based methods reported out-of-memory errors when using ResNet12, indicating their limitation in using large backbone networks. To test them on ResNet-12, we followed the ANIL method [36] by pre-training ResNet-12 via ProtoNet, freezing the encoder, and fine-tuning the last fully-connected layer. In this case, HSML and ARML cannot model the mixture task distribution properly as they require joint training of the encoder and other layers. The details are in Appendix D.5. For training, Adam optimizer was used. Each batch contains 4 tasks. Each model was trained with 20000 episodes. The learning rate of the metric-based methods was 1 e -3 . The learning rates for the inner- and outer-loops of the optimization-based methods were 1 e -3 and 1 e -4 . The weight decay was 1 e -4 . For HTGM, we set σ = 1 . 0 , ¯ σ = 0 . 1 , α = 0 . 5 ( 0 . 9 ) for 1-shot (5-shot) tasks. The number of mixture components r varies w.r.t. different datasets, and was grid-searched within [2 , 4 , 8 , 16 , 32] . All hyperparameters were set using the meta-validation sets.

4.1 Experimental Results

Few-shot classification. Following [44], we report the mean accuracy and 95% confidence interval of 1000 random tasks with 5-way 1-shot/5-shot, 5/25-query tests. Following [53], we report the accuracy of each domain and the overall average accuracy for Plain-Multi, and report the accuracy of each image filtering strategy and the overall average accuracy for Art-Multi. Table 1 and 2 summarize the results. From the tables, we have several observations. First, metric-based methods generally outperform optimization-based methods. This is because of the efficiency of metric-based methods, enabling them to fit a larger backbone network, which is consistent with the results in [44]. Built upon the metric-based method, HTGM only introduces a few distribution-related parameters and thus has the flexibility to scale with the encoder size. Second, the baselines designed for dealing

with mixture distributions of tasks, i.e. , HSML and ARML, outperform their counterparts without such design, demonstrating the importance to consider mixture task distribution in practice. Finally, HTGM outperforms the SOTA baselines in most cases by large margins, suggesting its effectiveness in modeling the generative process of task instances.

Novel task detection. We also evaluate HTGM on the task of detecting novel N -wayK -shot tasks ( N = 5 , K = 1 ) that are drawn out of the training task distributions. To this end, we train each comapred model in the Original domain in Art-Multi dataset, and test the model on tasks drawn from either Original domain ( i.e. , known tasks), or { Blur, Pencil } domains ( i.e. , novel tasks), and evaluate if the model can tell whether a testing task is known or novel.

For comparison, since none of the baselines detects novel tasks, we adapt them as follows. For metric-based methods, since they use a fixed encoder for all training/testing tasks, we averaged the sample embeddings in each task to represent the task. Then a separate GMM model was built upon the training task embeddings, and its likelihood was adapted to score the novelty of testing tasks (some details of the setup are in Appendix C.2).

However, optimization-based models perform gradient descent on the support set of each task, leading to varying encoders per task. As such, sample embeddings of different tasks are not comparable, and we cannot obtain task

embeddings in the same way as before. Among them, only HSML has an augmented task-level encoder for task embedding, allowing us to include it for comparison. For a fair comparison, our HTGM also trains a GMM on its task embeddings for detecting novel tasks. Moreover, two HTGM variants were included for ablation analysis to understand some design choices: (1) HTGM-Gaussian replaces the Gibbs distribution in Eq. (2) with a Gaussian distribution; (2) HTGM w/o GMM removes the task-level GM, i.e. , the third term in Eq. (3). The classification results of the ablation variants are in Appendix D.4. Following [6, 46, 56], we report Area Under ROC (AUROC), Average Precision (AP), and Max-F1. Table 3 summarizes the results, from which we observe HTGM outperforms all

(a) HSML

(b) MetaOptNet

(c) ProtoNet-Aug

(d) HTGM

baselines over all evaluation metrics, indicating the superior quality of task embeddings learned by our model. The embeddings follow the specified mixture distribution of tasks p ( v τ ) as described in Sec. 3.2, which fits the mixture data well hence allowing to detect novel tasks that are close to the boundary. Since the baselines learn embeddings without explicit constraint, they don't fit the post-hoc GMM well. Moreover, HTGM outperforms HTGM w/o GMM, which is even worse than some other baselines. This further validates the necessity to introduce the regularization of task-level mixture distribution p ( v τ ) . Also, the drops of AUROC and AP of HTGM-Gaussian demonstrate the importance of our unique design of the Gibbs distribution for the task-conditional distribution in Eq. (2). Similar to [46], in Fig. 3, we visualized the normalized likelihood histogram of known and novel tasks for HSML, MetaOptNet (the best baseline), ProtoNet-Aug (near-best baseline), and HTGM. To better interpret the figures, we calculated the ratio between the non-overlapped area of the two distributions and the total-area in Fig. 3 for HSML: 0.1379, MetaOpt: 0.4952, ProtoNetAug: 0.4806, and HTGM: 0.5578. As can be seen, the ratio of the non-overlapped area of HTGM is higher than other methods, which indicates the likelihoods ( i.e. , novelty scores) of HTGM are more distinguishable for known and novel tasks than the baseline methods. We also analyzed the hyperparameters σ , ¯ σ , and r of HTGM in Appendix D.1, D.2, and D.3.

5 Conclusion

In this paper, we proposed a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM models the generative process of task instances, and performs maximum likelihood estimation to learn task embeddings, which can help adjust prototypes acquired by the feature extractor and thus achieve better performance. Moreover, by explicitly modeling the distribution of tasks in the embedding space, HTGM can effectively detect the tasks that are drawn from distributions unseen during meta training. The extensive experimental results indicate the advantage of HTGM on both few-shot classification and novel task detection.

6 Broader Impact and Limitation

Our proposed method enables better novel task detection for meta-learning. In many areas requiring robust decisions, such as healthcare (as described in Sec. 1) and auto-driving, where the accuracy drop and uncertainty on novel tasks is inevitable, our model can raise an alarm to users ( e.g. , doctors and human drivers) for diagnosis and decision-making. However, as a probabilistic machine learning model, HTGM does not guarantee 100% accuracy. Also, it is noteworthy that the entropy of π ( µ c y i | v τ , ω ) (in Eq. (2)) is proportional to the partition function ( i.e. the denominator in Eq. (2)). Thus, our approximation in Sec. 3.3 that replaces the partition function with its upper bound increases the entropy, leading to increased noise in the inferred class prototypes. As such, in scenarios such as serious disease treatment and auto-driving in complex environments, to avoid potential wrong decisions from the model, human intervention is still necessary.

Acknowledgments and Disclosure of Funding

This work was primarily finished during the internship of Yizhou Zhang (the first author) at NEC Laboratories America. Yizhou Zhang's work after the internship was partially supported by NSF Research Grant IIS-2226087 and the Annenberg Fellowship of the University of Southern California. We sincerely appreciate the comments and suggestions from the anonymous reviewers.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for metalearning. In ICCV , pages 6430-6439, 2019.

Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In ICML , volume 70, pages 136-145. PMLR, 2017.

Thomas L Athey, Benjamin D Pedigo, Tingshan Liu, and Joshua T Vogelstein. Autogmm: Automatic and hierarchical gaussian mixture modeling in python. arXiv preprint arXiv:1909.02688 , 2019.

M. Christopher Bishop. Pattern recognition and machine learning . Springer, 2006.

Isaac S Chan and Geoffrey S Ginsburg. Personalized medicine: progress and promise. Annual review of genomics and human genetics , 12:217-244, 2011.

Jiacheng Cheng and Nuno Vasconcelos. Learning deep classifiers consistent with fine-grained novelty detection. In CVPR , pages 1664-1673, 2021.

Lu Cheng, Ruocheng Guo, Kai Shu, and Huan Liu. Causal understanding of fake news dissemination on social media. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 148-157, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , pages 1126-1135. PMLR, 2017.

Jacob Goldberger and Sam T Roweis. Hierarchical clustering of a mixture model. In NeurIPS , pages 505-512, 2005.

Mengdi Huai, Chenglin Miao, Qiuling Suo, Yaliang Li, Jing Gao, and Aidong Zhang. Uncorrelated patient similarity learning. In Proceedings of the 2018 SIAM International Conference on Data Mining , pages 270-278. SIAM, 2018.

Laurent Jacob, Jean-philippe Vert, and Francis Bach. Clustered multi-task learning: A convex formulation. In NIPS , 2008.

Minki Jeong, Seokeon Choi, and Changick Kim. Few-shot open-set recognition by transformation consistency. In CVPR , pages 12566-12575, 2021.

Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. NeurIPS , 32, 2019.

Weisen Jiang, James T. Kwok, and Yu Zhang. Subspace learning for effective meta-learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesv´ ari, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 10177-10194. PMLR, 2022.

Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In ICML , 2011.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Weihao Kong, Raghav Somani, Zhao Song, Sham M. Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 5394-5404. PMLR, 2020.

Abhishek Kumar and Hal Daum´ e III. Learning task grouping and overlap in multi-task learning. In ICML , pages 1723-1730, 2012.

Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning. NeurIPS , 34:24581-24592, 2021.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data , 1(0), 2006.

Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-ofdistribution tasks. arXiv preprint arXiv:1905.12917 , 2019.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS , 31, 2018.

Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICML , pages 3763-3772. PMLR, 2019.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR , pages 10657-10665, 2019.

Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In ICML , pages 2927-2936. PMLR, 2018.

Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 , 2017.

Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR , 2018.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. pages 21464-21475, 2020.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS , 2013.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR , 2018.

Jingchao Ni, Wei Cheng, Zhengzhang Chen, Takayoshi Asakura, Tomoya Soma, Sho Kato, and Haifeng Chen. Superclass-conditional gaussian mixture model for learning fine-grained embeddings. In ICLR , 2022.

Łukasz P Olech and Mariusz Paradowski. Hierarchical gaussian mixture model with objects attached to terminal and non-terminal dendrogram nodes. In Proceedings of the 9th International Conference on Computer Recognition Systems CORES 2015 , pages 191-201. Springer, 2016.

Boris Oreshkin, Pau Rodr´ ıguez L´ opez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. NeurIPS , 31, 2018.

Alexandre Passos, Piyush Rai, Jacques Wainer, and Hal Daum´ e III. Flexible modeling of latent task structures in multitask learning. In ICMl , pages 1283-1290, 2012.

Sanjay Purushotham, Wilka Carvalho, Tanachat Nilanon, and Yan Liu. Variational recurrent adversarial deep domain adaptation. In ICLR , 2017.

Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In ICLR , 2020.

Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR , 2018.

Ketan Rajshekhar Shahapure and Charles Nicholas. Cluster quality analysis using silhouette score. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA) , pages 747-748. IEEE, 2020.

Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In SIGKDD , pages 1441-1451, 2021.

Satya Narayan Shukla and Benjamin Marlin. Interpolation-prediction networks for irregularly sampled time series. In ICLR , 2019.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS , 2017.

Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot learning? In ECCV , pages 645-666. Springer, 2020.

Qiuling Suo, Fenglong Ma, Ye Yuan, Mengdi Huai, Weida Zhong, Jing Gao, and Aidong Zhang. Deep patient similarity learning for personalized healthcare. IEEE transactions on nanobioscience , 17(3):219-227, 2018.

Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In ECCV , pages 266-282. Springer, 2020.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR , 2022.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NIPS , 29, 2016.

Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal modelagnostic meta-learning. arXiv preprint arXiv:1812.07172 , 2018.

Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan B Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In AAAI , volume 35, pages 651-659, 2021.

Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. arXiv preprint arXiv:2305.01068 , 2023.

Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram. Multi-task learning for classification with dirichlet process priors. Journal of Machine Learning Research , 8(1), 2007.

Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In ICML , pages 7045-7054. PMLR, 2019.

Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In ICLR , 2019.

Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR , pages 8808-8817, 2020.

Yizhou Zhang, Defu Cao, and Yan Liu. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. Advances in Neural Information Processing Systems , 35:10643-10655, 2022.

Yizhou Zhang, Karishma Sharma, and Yan Liu. Vigdet: Knowledge informed neural temporal point process for coordination detection on social media. Advances in Neural Information Processing Systems , 34:3218-3231, 2021.

Yizhou Zhang, Karishma Sharma, and Yan Liu. Capturing cross-platform interaction for identifying coordinated accounts of misinformation campaigns. In European Conference on Information Retrieval , pages 694-702. Springer, 2023.

Pan Zhou, Yingtian Zou, Xiao-Tong Yuan, Jiashi Feng, Caiming Xiong, and Steven C. H. Hoi. Task similarity aware meta learning: theory-inspired improvement on MAML. In Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur, editors, Proceedings of the ThirtySeventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021 , volume 161 of Proceedings of Machine Learning Research , pages 23-33. AUAI Press, 2021.

A Appendix for Details of Deriving HTGM

A.1 The lower-bound of the likelihood function

In this section, we provide the details of the lower-bound in Eq. (3). By introducing the approximated posterior q ϕ ( v τ |D s τ ) , the likelihood in Eq. (1) becomes (the superscript ∗ is neglected for clarity)

where the fourth step uses Jensen's inequality. This completes the derivation of Eq. (3).

A.2 The upper-bound of the partition function

In Sec. 3.3, we apply an upper bound on the partition function in Eq. (2) for solving the challenging 2. The derivation of the upper bound is as follows.

where the last equation is from the multidimensional Gaussian integral. This completes the derivation of the upper bound of the partition function.

Note that replacing the partition function with its upper bound increases the noise in the inferred class mean. This is because that the entropy of distribution π ( µ c y i | v τ , ω ) ( µ c y i is a class mean) in Eq. (2) is that:

where ¯ E ω is the average energy and Z is the partition function. From this formula, we see that replacing Z with its upper bound increases the entropy of π ( µ c y i | v τ , ω ) , the distribution where µ c y i is sampled and/or inferred. Since entropy is a metric that measures the uncertainty of a random variable, increased entropy means that our estimation to µ c k is more uncertain, or in other words, more

noisy. This leads to a decreasing accuracy of meta-learning since the class mean µ c y i is an important parameter to estimate in meta-learning. To alleviate this issue, we proposed to include an negative sampling term as in Eq. (4) to increase the distance between different class means so that the noise in the class mean estimation, which is brought by the upper bound approximation of Z , will not seriously influence the accuracy.

A.3 The proof of Theorem 3.1

Proof. Let B j denote a sphere in R d . Its center is at W j v τ and its radius is D hl / 2 . Because W h v τ and W l v τ ( 1 ≤ h, l ≤ N ) is the pair with the smallest Euclidean distance D hl , for any pair of balls B j and B m we have B j ∩ B m is a null set (a set with 0 volume in R d ).

In other words, there is no overlap between any pair of spheres. Therefore, if we compute the integral over the joint of all spheres, we have

Also, because there is no overlap between any pair of spheres, for each point µ c k ∈ B m , we have

Therefore, we have the following derivation from Eq. (12).

Meanwhile, since ⋃ N m =1 B m is a sub-area of the entire R d space, we have

As B m is a sphere, we can convert the integral into spherical coordinates. Thus we have:

where γ ( · , · ) is lower incomplete gamma function and Γ( · ) is gamma function ( Γ( x ) the limitation of γ ( x, y ) when y → + ∞ ). According to the definition of lower incomplete gamma function, when D 2 hl / 4 → + ∞ , we have

Therefore,

Since N √ π d is its upper bound, based on the squeeze theorem, we have

which completes the proof of Theorem 3.1. Moreover, from Equation 16, we know that the error ratio of the approximation, denoted as AER , can be bounded: √

which monotonously decreases to 0 as D hl increases to + ∞

A.4 The proof of Theorem 3.2

Before starting the proof, we need to clarify the notations. When optimizing Eq. (3) in meta-training stage, we compute the class prototypes µ c 1 , ..., µ c N with both support sample means { µ c 1 , ..., µ c N } s and query sample means { µ c 1 , ..., µ c N } q (see Fig. 2). In Sec. 3.4, since during meta-testing stage we have no query samples, for clarity we omitted the notations to denote the j -th class mean of support set as µ c j . In the proof of the Theorem 3.2, we rigorously denote the support sample means of class j as µ c,s j and the query sample means of class j as µ c,q j .

Proof. Recall the Eq. (3):

By substituting the sampled v τ into it, we can acquire:

Note that not every term in the Eq. (22) (the above likelihood function) contains class means µ c . Only { p ω ( y i | v τ ) } and { p θ , ω ( e i | y i ) } are involved with µ c . Thus, optimizing µ c to maximize ℓ HTGM-L ( D τ ; θ , ω ) is equivalent to maximizing the following objective function L ( µ c ) :

Moreover, since every term is involved with at most one individual µ c j , we can separately optimize each individual µ c j . We denote the embedding of the i -th sample in class j as e ( j ) i . The objective function for each individual µ c j is:

The maximum likelihood estimation ¯ µ c j of µ c j should maximize the above objective function. It is equivalent to find a solution set µ c j = ¯ µ c j , l = l ∗ that can minimize the following function:

We first compute the partial derivative of the above function with respect to µ c j :

Onthe minimum, the partial derivative should be zero. Thus, ¯ µ c j = 1 K +2 σ 2 d ∑ K i e ( j ) i + 2 σ 2 d K +2 σ 2 d W l v τ for the optimal l ∗ . Let us denote µ c,s j = 1 K ∑ K i e ( j ) i (the support sample mean of class j ), α = K K +2 σ 2 d and r l = W l v τ -µ c j , then we have:

Therefore, we know that the optimal ¯ µ c j , l satisfy the above equation. Because l is an index between 1 and N , there are only N solutions that satisfy the above equation. For these N solutions, we have:

Note that µ c,s j = 1 K ∑ K i e ( j ) i . Thus ∑ K i ( e ( j ) i -µ c,s j ) = 0

Therefore, we can know that the optimal l that can minimize L should minimize || r l || 2 2 , which is l ∗ = arg min 1 ≤ l ≤ N D ( µ c j , W l v τ ′ ) using Euclidean distance D ( · , · ) .

However, our empirical evaluation shows that sometimes the emperically optimal value of α might be different from theoretically optimal value K K +2 σ 2 d . For the 1-shot learning case, the optimal α we acquired from grid search is around 0.5 (same as theoretically optimal value). However, in the 5-shot learning case, the optimal α we acquired from grid search is around 0.9, slightly larger than 0.84 ( K K +2 σ 2 d ). We suggests that it might be because during meta-training we replace the likelihood with its evidence lower bound and approximate the partition function with its upper bound, introducing noise into the generative model. Thus, we need to scale its weight down. Therefore, we propose to replace K K +2 σ 2 d with a hyperparameter α and fineunte it on validation set.

A.5 Derivation of Eq. (8)

Wefirst compute p ( y ′ = j ′ | x ′ ) i.e. the posterior distribution of the query sample's label y ′ conditioned on the sample. Then we select the label with the highest posterior probability. Utilizing the estimation of class means, we have the label conditional distribution:

Note that during the meta-testing stage, we should equally treat each class i.e., assume that the prior probability of each class should be same, i.e. p ( y ′ i = 1) = ... = p ( y ′ i = N ) . Thus we have:

A.6 The training algorithm of HTGM

The training algorithm of HTGM is summarized in Algorithm 1.

B Appendix for Further Discussion

B.1 Discussion about the novel task discussion and meta-learning

As we discussed in Sec. 2, to the best of our knowledge, our proposed method HTGM is the first work that jointly considers the task mixture distribution and novel task detection in meta-testing stage. There are some works considering how to identify novel task clusters in meta-training stage based on task embedding [52] or task likelihood [13]. However, they have their own respective drawbacks when handling novel task detection in meta-testing stage. For task-embedding-based method like [52], it does not explicitly model the task distribution. Instead, it considers how to model the task membership of the learnt clusters. As a result, they can only identify the outlying task clusters rather than individual novel tasks. However, in meta-testing stage, we expect the model to identify each individual novel task and raise alerts. The task-likelihood-based method DPMM [13] can handle individual novel tasks. However, it is hard for them to simultaneously handle quick detection and adaptation. This is because its likelihood was built on the entire model parameters, leading to model-dependent and time consuming computation. It is not a big issue for meta-training, but will serious limit its application to streaming tasks in meta-testing (e.g., in auto-driving domain) where efficiency is critical for timely alarms of novel tasks.

B.2 Discussion about the relationship between HTGM and HGM model

To the best of our knowledge, the Hierarchical Gaussian Mixture (HGM) model has appeared in the traditional works [9, 32, 3, 50] for hierarchical clustering by applying Gaussian Mixture model agglomeratively or divisively on the input samples. They are unsupervised methods that infer clusters of samples, but do not pre-train embedding models (or parameter initializations) that could be fine-tuned for the adaptation to new tasks in meta-learning. Therefore, these methods are remarkably different from meta-learning methods, and we think it is a non-trivial problem to adapt the concept of HGM to solve the meta-learning problem. To this end, we need to (1) identify the motivation; and (2) solve the new technical challenges. For (1), the hierarchical structure of mixture distributions naturally appears when we want to model the generative process of tasks from a mixture of distributions, where each task contains another mixture distribution of classes (as suggested by Eq. (1)). In other words, the motivating point of our method is more on meta-learning than HGM. However, drawing such a connection between meta-learning and HGM is a novel contribution. For (2), our method is different from traditional HGM in (a) its generative process of tasks (Sec. 3.2), which is a theoretical extension of the widely used empirical process of generating tasks in meta-learning; (b) its Gibbs-style task-conditional distribution (Eq. (2)) for fitting uniformly sampled classes; (c) the metric-based end-to-end meta-learning framework (Fig. 2) (note the traditional HGM is not for learning embeddings); (d) the non-trivial derivation of the optimization algorithm in Sect. 3.3 and Alg. 1; and (e) the novel model adaptation process in Sec. 3.4. Solving the technical challenges in the new generative model is another novel contribution of the proposed method.

B.3 Discussion about the related multi-task learning methods

The modeling of the clustering/grouping structure of tasks or the mixture of distributions of tasks has been studied in multi-tasking learning (MTL). In [51, 11], tasks are assumed to have a clustering structure, and the model parameters of the tasks in the same cluster are drawn to each other via optimization on their L2 distances. In [15], a subspace based regularization framework was proposed for grouping task-specific model parameters, where the tasks in the same group are assumed to lie in the same low dimensional subspace for parameter sharing. The method in [18] also uses the subspace based sharing of task parameters, but allows two tasks from different groups to overlap by having one or more bases in common. The method in [34] introduces a generative model for task-specific model parameters that encourages parameter sharing by modeling the latent mixture distribution of the parameters via the Dirichlet process and Beta process.

The key difference between these methods and our method HTGM lies in the difference between MTL and meta-learning. In an MTL method, all tasks are known a priori , i.e. , the testing tasks are from the set of training tasks, and the model is non-inductive at the task-level (but it is inductive at the sample-level). In HTGM, testing tasks can be disjoint from the set of training tasks, thus the model is inductive at the task-level. In particular, we aim to allow testing tasks that are not from the distribution of the training tasks by enabling the detection of novel tasks, which is an extension of the task-level inductive model. The second difference lies in the generative process. The method in [34] models the generative process of the task-specific model parameters ( e.g. , the weights in a regressor). In contrast, HTGM models the generative process of each task by generating the classes in it, and the samples in the classes hierarchically, i.e. , the ( x , y ) 's (in Eq. (1) and Sec. 3.2). In this process, we allow our model to fit uniformly sampled classes given a task (without specifying a prior on the distance function on classes) by the proposed Gibbs distribution in Eq. (2). Other remarkable differences to the aforementioned MTL methods include the inference network (Fig. 2(b)), which allows the inductive inference on task embeddings and class prototypes; the optimization algorithm (Sec. 3.3) to our specific loss function in Eq. (3), which is from the likelihood in Eq. (1); and the model adaptation algorithm (Sec. 3.4) for performing predictions in a testing task, and detecting novel tasks. As such, the MTL methods can not be trivially applied to solve our problem.

B.4 Further interpretation of the task-conditional distribution

The task-conditional class distribution p ω ( y i = k | v τ ) in Eq. (2) is defined through an energy function E ω ( µ c k ; v τ ) = min( {|| µ c k -W j v τ || 2 2 } N j =1 ) with trainable parameters ω = { W 1 , ..., W N } , for allowing uniformly sampled classes per task. The conditional distribution p ( y i | v τ ) represents how classes distribute for a given task τ . The reason for its definition in Eq. (2) is as follows. If it is a Gaussian distribution with v τ ( i.e. , task embedding) as the mean, p ( y i = k | v τ ) can be interpreted as

the density at the representation of the k -th class in this Gaussian distribution, i.e. , the density at µ k , which is the mean/surrogate embedding of the k -th class. One problem of this Gaussian p ( y i | v τ ) is that different classes, i.e. , different µ y i 's, are not uniformly distributed, contradicting the practice that given a dataset (e.g., images), classes are often uniformly sampled for constituting a task in the empirical studies. Using a uniformly sampled set of classes to fit the Gaussian distribution p ( y i | v τ ) will lead to an ill-posed learning problem, as described in Sec. 3.2. To solve it, we introduced ω = { W 1 , ..., W N } in the energy function E ω ( µ c k ; v τ ) in Eq. (2). W j ∈ R d × d ( 1 ≤ j ≤ N ) can be interpreted as projecting v τ to the j -th space spanned by the basis ( i.e. , columns) of W j . There are N different spaces for j = 1 , ..., N . Thus, the N projected task means W 1 v τ , ..., W N v τ are in N different spaces. Fitting the energy function E ω ( µ c k ; v τ ) to N uniformly sampled classes µ c 1 , ..., µ c N , which tend to be far from each other because they are uniformly random, tends to learn W 1 , ..., W N that project v τ to N far apart spaces that fit each of the µ c 1 , ..., µ c N by closeness, due to the min-pooling operation. This mitigates the aforementioned ill-posed learning problem.

C Appendix for Implementation Details

C.1 The setup of the compared models

Encoder of Metric-based Meta-Learning. For fairness, for all metric-based methods, including ProtoNet [41], MetaOptNet [24], ProtoNet-Aug [42], FEATS [54] and NCA [19], following [44, 24], we apply ResNet-12 as the encoder. ResNet-12 has 4 residual blocks, each has 3 convolutional layers with a kernel size of 3 × 3 . ResNet-12 uses dropblock as a regularizer, and its number of filters is (60, 160, 320, 640). For MetaOptNet, following its paper [24], we flattened the output of the last convolutional layer to acquire a 16000-dimensional feature as the image embedding. For other baselines, following [44], we used a global average-pooling layer on the top of the last residual block to acquire a 640-dimensional feature as the image embedding.

Further Details. Following [41], ProtoNet, ProtoNet-Aug, and NCA use Adam optimizer with β 1 = 0 . 9 and β 2 = 0 . 99 . We did grid-search for the initial learning rate of the Adam within { 1 e -2 , 1 e -3 , 1 e -4 } , where 1 e -3 was selected, which is the same as the official implementation provided by the authors. For FEATS, we chose transformer as the set-to-set function based on the results reported by [54]. When pre-training the encoder in FEATS, following its paper [54], we applied the same setting as ProtoNet, which is to use Adam optimizer with an initial learning rate of 1 e -3 , β 1 = 0 . 9 and β 2 = 0 . 99 . When training its aggregation function, we grid-searched the initial learning rate in { 1 e -4 , 5 e -4 , 1 e -5 } since a larger learning rate leads to invalid results on our datasets. The optimal choice is 1 e -4 . For MetaOptNet, following its paper [24], we used SGD with Nesterov momentum of 0.9, an initial learning rate of 0.1 and a scheduler to optimize it, and applied the quadratic programming solver OptNet [2] for the SVM solution in it.

C.2 The details of the setup for novel task detection

In the experiments on novel task detection in Sec. 4.1, the number of in-distribution tasks (from the Original domain) in the test set is 4000 (1000 per task cluster) and the number of novel tasks (from the Blur and Pencil domains) in the test set is 8000 (4000 for the Blur and 4000 for the Pencil).

D Appendix for Experimental Results

D.1 Analysis of σ

Tabel 4 reports the effect of different σ on the classification performance (5-way-1-shot classification on Multi-Plain dataset). As shown in the table, although the too low or too high setting of this hyper-parameter will hurt the performance, in general the model is robust toward the setting of σ .

D.2 Analysis of ¯ σ

Tabel 5 summarizes how different settings of ¯ σ influence the classification performance (5-way-1-shot classification on Plain-Multi dataset). In general, different settings of ¯ σ will influence the model performance at a marginal level, indicating our model's robustness toward this hyper-parameter.

D.3 Impact of GMM component number

Different choices of the number of mixture components does not significantly influence the model classification performance. However, the clustering quality may vary due to the different numbers of components. Here, we report the Silhouette score [38, 39] on Plain-Multi dataset w.r.t. the number in Table 6. From Table 6, we can see that selecting a component number close to the ground-truth component number of the distribution can benefit the clustering quality.

D.4 Classification performance of the ablation variants

We summarize the classification performance of the two Ablation Variants HTGM w/o GMM and HTGM-Gaussian in Table 7. As we can see, our unique designs improve the novel task detection performance without significantly decreasing the classification performance.

D.5 Ablation analysis of optimization-based methods

We selected the two best performed optimization-based baselines HSML and ARML, and the widely used method MAML for this ablation analysis. Table 8 summarizes the performance of MAML, HSML and ARML trained in ANIL method [36], i.e. , we pre-trained the ResNet-12 by ProtoNet, froze the encoder, and fine-tuned the last fully-connected layers using MAML, HSML and ARML on Plain-Multi dataset. From Table 8, the performance of ANIL-MAML is better than MAML in Table 1, similar to the observation in [36], indicating the effectiveness of ANIL method. However, ANIL-HSML and ANIL-ARML perform similarly to ANIL-MAML, losing their superiority of modeling the mixture distribution of tasks achieved when implemented without ANIL as in Table 1 (up to 5.6% average improvement). This is because the clustering layers in HSML and the graph layers in ARML both affect the embeddings learned through backpropagation, i.e. , they were designed for joint training with the encoder. When the encoder is frozen, they cannot work properly. For this reason, to be consistent with the existing research [52, 53] that demonstrated the difference between HSML/ARML and MAML, we used their original designs in Sec. 4. Meanwhile, we observed the proposed HTGM outperforms MAML, HSML, and ARML trained in ANIL method, this is because MAML cannot model the mixture distribution of tasks, while HSML and ARML cannot work properly when trained in ANIL method.

D.6 More results on the Mini-ImageNet dataset

In the case when the task distribution is not a mixture, our model would degenerate to and perform similarly to the general metric-based meta-learning methods, e.g. , ProtoNet, which only considers a uni-component distribution. To confirm this, we added an experiment that compares our model with ProtoNet-Aug on Mini-ImageNet [47], which does not have the same explicit mixture distributions as in the Plain-Multi and Art-Multi datasets in Section 4. The results are summarized in Table 9. From the table, we observe our method performs comparably to ProtoNet, which validates the aforementioned guess. Meanwhile, together with the results in Table 1 and Table 2, the proposed method could be considered as a generalization of the metric-based methods to the mixture of task distributions.

D.7 Task Embedding Visualization.

We provide visualizations of the task embeddings learnt by HTGM. Fig 4 is the visualization of the task embeddings for Few-Shot Classification on Plain-Multi dataset (Sec. 4.1) via t-SNE [45]. Every yellow point corresponds to a task sampled from Aircraft. Every blue point corresponds to a task sampled from Texture. Every red point corresponds to a task sampled from Fungi. Every green point corresponds to a task sampled from Bird. As we can see, in general, different classes of tasks are well clustered in the task embedding space, indicating that HTGM learnt task embeddings that capture the difference of tasks from different groups.

D.8 Computational Cost

We report the cost of ProtoNet-Aug, NCA, FEATS and our model because they use the same encoder architecture. We evaluated and trained all of the models on RTX 6000 GPU with 24 GB memory.

Training Time: According to the training logs, the training of the ProtoNet-Aug took 10 hours, the training of NCA took 6.5 hours, the training of FEATS took 10.5 hours, and the training of the

proposed model HTGM took 13 hours. Please note that our algorithm and FEATS require pre-training the encoder with ProtoNet-Aug. The 10.5 and 13 hours include the 10 hours pre-training phase. The main cost of our model is not from the energy function, because we have reduced its partition function to a constant using Eq. (6) in Theorem 3.1, whose training cost is negligible.

The higher cost is because (1) our model needs to jointly learn a GMM model in every EM step and (2) jointly learning the generative model and classification model takes more learning steps to converge. Given the pre-trained ProtoNet-Aug encoder, the FEATS took about 3000 steps to converge, the proposed model took about 10000 steps to converge. In other words, the EM training algorithm takes more computational overhead. However, the advantage comes with the training cost is the better classification and novel task detection performance.

Test Time: Because we approximated the partition function of the energy function with a constant upper bound, we almost added zero computational cost to the model inference. The test time of NCA, FEATS, ProtoNet, and our model are all around 85-95 seconds for 1000 tasks.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel probabilistic meta-learning algorithm called HTGM\nwhich models the full hierarchical sampling procedure in few-shot classification\nin order to perform prediction and novel task detection. To learn the parameters\nthey employ the EM procedure together with variational inference. Experimental\nresults show that they perform best among competitors in few-shot classification\nprediction and novel task detection.\n\nSTRENGTHS:\n- Originality & significance: The authors propose a probabilistic model of the full few-shot\n  classification process which allows for both adaptation (prediction) and\n  novelty detection. Additionally it can be added on top of other metric-learning few-shot learning algorithms out of the box.\n- Quality: The probabilistic model seem reasonable and they discuss problems and\n  solutions of design choices made. Experiments for both prediction and novelty-detection is convincing (but see the weakness section).\n- Clarity: The paper is well-written in general.\n\nWEAKNESSES:\n- Partition function: The authors show that the partition function is upper bounded and show asymptotic tightness of this bound for the minimal distance $D_{hl}$, but I'm still not fully convinced by this and would like to know more how this impacts modelling and \"what we lose\" in disregarding the partition function.\n- Probabilistic modelling and \"tricks\": The pipeline has several tricks which may help performance but weakens the interpretation of the model as inferring a correct probabilistic distribution.\n- Non-standard multi-domain experiments: The datasets of Plain-Multi and\n  Art-Multi are non-standard. Ideally additional experiments on for example\n  meta-dataset would be beneficial.\n- Typos:\n  1. Line 172: I think $\\mu_k^c$ should be $\\mu_1^c$ and $\\mu_N^c$?\n  2. Line 314: comapred -> compared\n  3. Line 341: they even don't fit -> they don't even fit\n  4. Table 2: I think the second entry in the \"Setting\" column should be \"5-way, 5-shot\"\n\nQUESTIONS:\n# Partition Function\nCould you explain what the potential limitations of disregarding the partition\nfunction is? Intuitively it feels like the partition function would have a\nregularizatory effect and disregarding it may lead to some overfitting. Would be\ninterested to hear what the authors think the effect of being able to optimizing\nthis part of the objective would be.\n\n# Tricks\n- Tying $\\theta$ and $\\phi$: Is there are reason for letting the recognition\n  network $q$ be equal to $f$? Is the tying of the parameters done due to\n  convenience and that it empirically was found to work well or is there some other reason?\n- Negative sampling and partition function: Show empirical results that the\n  quantity $D\\_{hl}$ goes to infinity over the course of training.\n- Model adaptation: Is it possible to put the adaptation step on a probabilistic\n  footing? Right now it seems like it's pretty ad-hoc, can we for example trust\n  $p(y'\\_i = j' | x\\_i')$ more as a probabilistic model compared to using any\n  standard (non-probabilistic) metric learning few-shot algorithm which produces\n  an embedding $f_{\\theta}(x)$? One way to show this would be to show that $p$\n  here is more well-callibrated as opposed to using the backbone trained by\n  proto-net or some other algorithm.\n\n# Non-standard experimental dataset\nMeta-dataset is the standard modern community benchmark for few-shot\nmulti-domain classification. I'd be interested to see how the algorithm fares on\nthis benchmark since their probabilistic model explicitly models the\nmulti-domain aspect of few-shot learning for which the meta-dataset seems like a\ngood fit.\n\nLIMITATIONS:\nNo, certain limitations should be expanded upon (see Questions). Societal impact is discussed correctly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the task distribution in meta learning and proposes modeling the task in multimodal distributions. To enable efficient modelling and inference, the author develops the hierarchical Gaussian mixture task generative mode and optimizes the meta learning model in expectation maximization. The performance of the method is verified in few-shot image classification tasks.\n\nSTRENGTHS:\n1. This paper is written clearly, and the modelling of task distribution has great significance in the literature.\n2. It shows improved performance in adopted benchmarks over existing metrics based methods.\n\nWEAKNESSES:\n(1) Probabilistic relations and graphical models. \n\nIt is necessary to explain why the generative process Eq. (1) is valid in metrics-based meta learning. In practice, meta learning seldom considers the sample embeddings and the task embedding is of interest. Hence, the conditional distribution $\\ln\\int p(D_{\\tau}^{q}| v_{\\tau})p(v_{\\tau}\\vert D_{\\tau}^{s})dv_{\\tau}$ is mostly commonly used as the objective to maximize. However, in Eq.(1), the conditional dependencies on the support dataset are neglected. My suggestion is that this needs to be reflected in Eq.(1) directly.\n\n(2) Model optimization. \n\nIn Line201-202, it says, “the query set is not included because it is unavailable during model testing”. This does not hold in Bayesian meta learning. Like that in conditional variational autoencoder or neural processes, the $q_{\\phi}(v_{\\tau}|D_{\\tau}^{s})$ is called the approximate prior and the approximate posterior $q_{\\phi}(v_{\\tau}|D_{\\tau}^{s}, D_{\\tau}^{q})$ is used in meta training for more effective inference (though the query set is unavailable in meta testing). Hence, I doubt the way of inference in this work since the conditional prior is most important for meta learning probabilistic models. Especially, the way to infer $e_i$ throughout the manuscript, e.g., $p(e_i|y_i)$ in Eq.(3)-(7), the one-hot information can be quite limited. However, the input $x_i$ is not included in the inference.\n\n(3) Mixture of task distributions. \n\nThe task embeddings lie in a mixture of Gaussian manifold is a reasonable assumption. For the task2vec work, the embedding of the task is visualized and analyzed a bit. In this work, the focus is on task distribution, so it is necessary to include this part in experimental analysis, for example, to check the grouping of latent structures in the task space.\n\nQUESTIONS:\n(1) Would you please provide the detailed probabilistic graphical model to explain the relationship between the support, the query, the sample-wise embedding and etc? \n\n(2) Since the experiments were performed on typical benchmarks, would you explain the novel tasks a bit and how to define the discovery of novel tasks?\n\n(3) Are task distributions the same for the meta training and meta testing, or do out-of-distribution cases exist?\n\nLIMITATIONS:\nSee the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a Hierarchical Gaussian Mixture method as a means to parameterize the task generation process in meta-learning. The authors suggest that their proposed model can effectively fit a mixture of task distributions and evaluate the scoring of testing tasks. The effectiveness of the proposed methods is demonstrated through experiments conducted on multiple datasets.

STRENGTHS:
* This paper is well-written and easy to follow


* It is interesting that the proposed method can meta-learn both a mixture of task distributions and detect novelty in testing tasks.

WEAKNESSES:
*  In this paper, the authors present an application of the Hierarchical Gaussian Mixture model to the meta-learning setting. While the paper provides ideas into utilizing this model for meta-learning, the method itself is not novel and has been extensively studied within the machine learning community.

* The paper's theoretical analysis appears limited and would benefit from a more in-depth analysis. Specifically, it is crucial to provide a comprehensive analysis of the generalization bound associated with the proposed methods.

QUESTIONS:
N/A

LIMITATIONS:
The authors have clearly addressed the limitations and potential negative societal impact of their work
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn realistic scenario, training tasks and test tasks may come from different distributions. However, most existing meat-learning methods do not take it into account. Even if some do, they do not jointly handle the detection of novel tasks. \n\nThe authors propose a metric-based meta learning model that handles both the mixture of task distributions and detection of novel tasks. The effectiveness of the proposed method compared to previous methods is verified on several benchmark datasets.\n\nSTRENGTHS:\n1. It is novel to approximate the distribution for tasks. In particular, it aligns with realistic scenarios by modeling it as a mixture of distributions.\n2. The paper is well-written; it clearly explains its gaol with mathematical support, and notations are well-described and consistent. Figures are also helpful to understand the proposed idea.\n3. Although Eq.(3) still cannot be computed due to the normalizing constant in $p_\\omega(y_i|v_\\tau)$, the authors provided the reasonable workaround in Theorem 3.1, which is also an interesting contribution.\n\nWEAKNESSES:\n1. Although $l_{HTGM}$ is well-motivated and clear, $l_{neg}$ is somewhat heuristic. \n2. If my understanding is correct, a large part of computational overhead compared to baseline methods e.g. ProtoNet comes from E-step and GMM. Especially, conducting GMM at each epoch can be quite expensive. But the analysis for computational cost is not provided. Can the authors provide the computational overhead (e.g. in wall-clock time or flops) of the proposed method (and time complexity if possible)?\n3. To demonstrate that the effectiveness of the method on cross domain, the experiments on two datasets the authors used are not sufficient, especially, if we want to validate whether it works in practice or not. It could have been much more plausible if experiments were conducted on a larger scale dataset such as Meta-Dataset.\n\nQUESTIONS:\n1. I am a bit confused with Eq.(4)-(5). Here, the objective of the additional negative sampling loss is to make the different class means disperse. To this end, Eq.(4) is used. But, if we minimize $l_{neg}$, wouldn’t it force $e_{j}$ and $\\mu$ to be close to each other? Please correct me if I am wrong.\n2. Could the authors elaborate how to interpret the Figure 2? In my understanding, the more distant two tasks are, the better a model is to distinguish two task groups. If so, measuring the distance between two normalized likelihood histogram would be helpful (treating them as probability distributions if necessary). Intuitively, the distance may be roughly estimated as the difference between the red and blue regions. I do not see the significant difference between the models (except HSML).\n3. Is there any particular reason why EM is used over VI?\n\nLIMITATIONS:\nPlease refer to the weakness and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}