SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yizhou Zhang $^{1*}$ , Jingchao Ni $^{2*‡}$ , Wei Cheng $^{3}$ , Zhengzhang Chen $^{3}$ , Liang Tong $^{4*}$ , Haifeng Chen $^{3}$ , Yan Liu $^{1}$
<sup>1</sup>University of Southern California 2AWS AI Labs
$^{3}$ NEC Laboratories America  $^{4}$ Stellar Cyber Inc.
$^{1}\{zhangyiz,yanliu.cs\} @usc.edu;$ $^{2}$ nijingchao@gmail.com;
$^{3}\{$  weicheng,zchen,haifeng $\}$ @nec-labs.com;  $^{4}$ ltong@stellarcyber.ai
Meta-learning enables quick adaptation of machine learning models to new tasks with limited data. While tasks could come from varying distributions in reality, most of the existing meta-learning methods consider both training and testing tasks as from the same uni-component distribution, overlooking two critical needs of a practical solution: (1) the various sources of tasks may compose a multi-component mixture distribution, and (2) novel tasks may come from a distribution that is unseen during meta-training. In this paper, we demonstrate these two challenges can be solved jointly by modeling the density of task instances. We develop a meta-training framework underlain by a novel Hierarchical Gaussian Mixture based Task Generative Model (HTGM). HTGM extends the widely used empirical process of sampling tasks to a theoretical model, which learns task embeddings, fits the mixture distribution of tasks, and enables density-based scoring of novel tasks. The framework is agnostic to the encoder and scales well with large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via an Expectation-Maximization (EM) algorithm. Extensive experiments on benchmark datasets indicate the effectiveness of our method for both sample classification and novel task detection.
Training models in small data regimes is of fundamental importance. It demands a model's ability to quickly adapt to new environments and tasks. To compensate for the lack of training data for each task, meta-learning (a.k.a. learning to learn) has become an essential paradigm for model training by generalizing meta-knowledge across tasks [41, 8]. While most existing meta-learning approaches were built upon an assumption that all training/testing tasks are sampled from the same distribution, a more realistic scenario should accommodate training tasks that lie in a mixture of distributions, and testing tasks that may belong to or deviate from the learned distributions. For example, in recent medical research, a global model is typically trained on the historical medical records of a certain set of patients in the database [40, 49]. However, due to the uniqueness of individuals (e.g., gender, age, genetics) [43], patients' data have a substantial discrepancy, and the pre-trained model may demonstrate significant demographic or geographical biases when testing on a new patient [35, 10]. This issue can be mitigated by personalized medicine approaches [5, 31], where each patient is regarded as a task, and the pre-trained model is fine-tuned (i.e., personalized) on a support set of a few records collected in a short period (e.g., a few weeks) from every patient for adaptation. In this case, the training tasks (i.e., patients) could be sampled from a mixture of distributions (e.g., different age
*This work was done primarily at NEC Laboratories America. ‡Corresponding author.
groups), and a testing task may or may not belong to any of the observed groups. Similar examples can be found in other applications such as the detection of fake news dissemination [7, 55, 57], where a task is a post, whose support set consists of a few profiles of users who have disseminated it in a short period. The training posts may be drawn from a mixture of topics, and a testing post may not belong to any of the topics. As such, a meta-training strategy that is able to fit a model to a mixture of task distributions and enable the identification of novel tasks during inference time is desirable for making meta-learning a practical solution.
One way to tackle the mixture distribution of tasks is to tailor the transferable knowledge to each task by learning a task-specific representation [33, 48, 25], but as discussed in [52], the overcustomized knowledge prevents its generalization among closely related tasks (e.g., tasks from the same distribution). The more recent methods try to balance the generalization and customization of the meta-knowledge by promoting local generalization either among a cluster of related tasks [52], or within a neighborhood in a meta-knowledge graph of tasks [53]. Neither of them explicitly learns the underlying distribution from which the tasks are generated, rendering them infeasible for detecting novel tasks that are out-of-distribution. However, detecting novel tasks is crucial in high-stake domains, such as medicine and finance, which provides users (e.g., physicians) confidence on whether to trust the results of a testing task or not, and facilitates downstream decision-making.
In [21], a task-specific tuning variable was introduced to modulate the initial parameters learned by the optimization-based meta-learning method MAML [8], so that the impacts of the meta-knowledge on different tasks are adjusted differently, e.g., novel tasks are less influenced than the known tasks. Whereas, this method focuses on improving model performance on different tasks (either known or novel), but neglects the critical mission of detecting which tasks are novel. In practice, providing an unreliable accuracy on a novel task, without differentiating it from other tasks may be meaningless.
Since the aforementioned methods cannot simultaneously handle the mixture distribution of tasks and novel tasks, a practical solution is in demand. In this work, we consider tasks as instances, and demonstrate the dual problem of modeling the mixture of task distributions and detecting novel tasks are two sides of the same coin, i.e., density estimation on task instances. To this end, we propose a Hierarchical Gaussian Mixture based Task Generative Model (HTGM) to explicitly model the generative process of task instances. Our contributions are summarized as follows.
- We extended the widely used empirical process of generating tasks to a theoretical process specified by a hierarchy of Gaussian mixture (GM) distributions. HTGM generates a task embedding from a task-level GM, and uses it to define the task-conditioned mixture probabilities for a class-level GM, from which the data samples are drawn. To allow realistic classes per task, a new Gibbs distribution was proposed to underlie the class-level GM.
- HTGM is an encoder-agnostic framework, thus is flexible to different domains. It inherits metric-based meta-learning methods, and only introduces a small overhead to an encoder for parameterizing its distributions, thus is efficient for large backbone networks. The model parameters are learned end-to-end by maximum likelihood estimation via a principled EM algorithm. The bounds of our likelihood function were also theoretically analyzed.
- In the experiments, we evaluated HTGM on benchmark datasets regarding its scalability to large networks, effectiveness in modeling the mixture distribution of tasks, and usefulness in identifying novel tasks. The results demonstrate HTGM outperforms the state-of-the-art (SOTA) baselines with significant improvements in most cases.
To the best of our knowledge, this is the first work to explicitly model the generative process of task instances from a mixture of distributions for meta-learning with novel task detection. Meta-learning aims to handle the few-shot learning problem, which derives memory-based [30], optimization-based [8, 26], and metric-based methods [47, 41], and often considers an artificial scenario where training/test tasks are sampled from the same distribution. To enable more varying tasks, task-adaptive methods facilitate the customization of meta-knowledge by learning task-specific parameters [37, 25], temperature scaling parameters [33], and task-specific modulation on model initialization [48, 52, 53, 21]. Among them, there are methods tackling the mixture distribution of tasks by clustering tasks [52, 13, 14, 17], learning task similarity and graphs [58, 53], and relocating the initial parameters for different tasks so that they use the meta-knowledge differently [21]. As discussed
before, none of these methods jointly handles the mixture of task distributions and the detection of novel tasks in meta-testing stage. A more detailed discussion is in Appendix B.1.
Our model is built upon metric-based methods and learns task embeddings to model task distributions. Achille et al. [1] proposed to learn embeddings for tasks and introduced a meta-learning method, but not for few-shot learning. Their embeddings are from a pre-specified set of tasks (rather than episode-wise sampling), and the meta-learning framework is for model selection. The model in [52] has an augmented encoder for task embedding, but it does not model task generation, and is not designed for novel task detection (an empirical comparison is in Sec. 4.1).
The conventional novelty detection aims to identify and reject samples from unseen classes [6]. It relates to open-set recognition [46], which aims to simultaneously identify unknown samples and classify samples from known classes. Out-of-distribution (OOD) detection [27, 28] can be seen as a special case of novelty detection where novel samples are from other problem domains or datasets, thus are considered to be easier to detect than novelties [6]. These methods are for large-scale training. In contrast, we want to detect novel tasks, which is a new problem in the small data regime.
Hierarchical Gaussian Mixture (HGM) model has appeared in some traditional works [9, 32, 3, 50] for hierarchical clustering by applying GM agglomeratively or divisively, which do not pre-train models for meta-learning, and is remarkably different from the topic in this paper. The differences are elaborated in Appendix B.2. Moreover, the idea of learning groups/clusters of tasks also appeared in some multi-task learning (MTL) models. The key difference between these methods and our method HTGM lies in the difference between MTL and meta-learning. In an MTL method, all tasks are known a priori, i.e., the testing tasks are from the set of training tasks, and the model is inductive at the sample-level but non-inductive at the task-level. More discussions are in Appendix B.3.
3 Hierarchical Gaussian Mixture based Task Generative Model (HTGM)
Meta-learning methods typically use an episodic learning strategy, where the meta-training set  $\mathcal{D}^{\mathrm{tr}}$  consists of a batch of episodes. Each episode samples a task  $\tau$  from a distribution  $p(\tau)$ . Task  $\tau$  has a support set  $\mathcal{D}_{\tau}^{\mathrm{s}} = \{(\mathbf{x}_i^{\mathrm{s}}, y_i^{\mathrm{s}})\}_{i=1}^{n_{\mathrm{s}}}$  for training, and a query set  $\mathcal{D}_{\tau}^{\mathrm{q}} = \{(\mathbf{x}_i^{\mathrm{q}}, y_i^{\mathrm{q}})\}_{i=1}^{n_{\mathrm{q}}}$  for testing, where  $n_{\mathrm{s}}$  is a small number to denote a few training samples. In particular, in a commonly used  $N$ -way  $K$ -shot  $Q$ -query task [47],  $\mathcal{D}_{\tau}^{\mathrm{s}}$  and  $\mathcal{D}_{\tau}^{\mathrm{q}}$  contain  $N$  classes, with  $K$  and  $Q$  samples per class respectively, i.e.,  $n_{\mathrm{s}} = NK$  and  $n_{\mathrm{q}} = NQ$ .
Let  $f_{\theta}(\mathbf{x}_i^*) \to y_i^*$  be a base model ( $*$  denotes s or q), and  $f_{\theta}(\cdot; \mathcal{D}_{\tau}^{\mathrm{s}})$  be the adapted model on  $\mathcal{D}_{\tau}^{\mathrm{s}}$ . The training objective on  $\tau$  is to minimize the average test error of the adapted model, i.e.,  $\mathbb{E}_{(\mathbf{x}_i^{\mathfrak{q}}, y_i^{\mathfrak{q}}) \in \mathcal{D}_{\tau}^{\mathfrak{q}}} \ell(y_i^{\mathfrak{q}}, f_{\theta}(\mathbf{x}_i^{\mathfrak{q}}; \mathcal{D}_{\tau}^{\mathrm{s}}))$ , where  $\ell(\cdot, \cdot)$  is a loss function (e.g., cross-entropy loss), and the meta-training process aims to find the parameter  $\theta$  that minimizes this error over all episodes in  $\mathcal{D}^{\mathrm{tr}}$ . Then,  $f_{\theta}$  is evaluated on every episode of a meta-test set  $\mathcal{D}^{\mathrm{te}}$  that samples a task from the same distribution  $p(\tau)$ . Usually,  $p(\tau)$  is a simple distribution [8, 21]. In this work,  $p(\tau)$  is generalized to a mixture distribution consisting of multiple components  $p_1(\tau), \dots, p_r(\tau)$ , and a test episode may sample a task either in or out of any component of  $p(\tau)$ . As such, given the training tasks in  $\mathcal{D}^{\mathrm{tr}}$ , our goal is to estimate the underlying density of  $p(\tau)$ , so that once a test task is given, we can (1) identify if it is a novel task, and (2) adapt  $f_{\theta}$  to it with optimal accuracy. Specifically, the base model  $f_{\theta}$  can be written as a combination of an encoder  $g_{\theta_e}$  and a predictor  $h_{\theta_p}$ , i.e.,  $f_{\theta}(\mathbf{x}_i^*) = h_{\theta_p}(g_{\theta_e}(\mathbf{x}_i^*))$  [44]. In this work, we focus on a metric-based non-parametric learner, i.e.,  $\pmb{\theta}_p = \varnothing$  (e.g., prototypical networks [41]), not only because metric-based classifiers were confirmed as more effective than probabilistic classifiers for novelty detection [12], but also for its better training efficiency that fits large backbone networks than the costly nested-loop training of optimization-based methods [44].
Formally, our goal is to find the model parameter  $\pmb{\theta}$  that maximizes the likelihood of observing a task  $\tau$ . In other words, let  $f_{\pmb{\theta}}(\mathbf{x}_i^*) = \mathbf{e}_i^* \in \mathbb{R}^d$  be sample embedding, we want to maximize the likelihood of the joint distribution  $p_{\pmb{\theta}}(\mathbf{e}_i^*, y_i^*)$  on the observed data in  $\mathcal{D}_{\tau} = \{\mathcal{D}_{\tau}^{\mathrm{s}}, \mathcal{D}_{\tau}^{\mathrm{q}}\}$ . We consider each task  $\tau$  as an instance, with a representation  $\mathbf{v}_{\tau} \in \mathbb{R}^{d}$  in the embedding space (the method to infer  $\mathbf{v}_{\tau}$  is described in Sec. 3.3). To model the unobserved mixture component, we associate every task with a latent variable  $z_{\tau}$  to indicate to which component it belongs. Suppose there are  $r$  possible components, and let  $n = n_{\mathrm{s}} + n_{\mathrm{q}}$  be the total number of samples in  $\mathcal{D}_{\tau}$ , the log-likelihood
to maximize can be written by hierarchically factorizing it on  $y_{i}^{*}$  and marginalizing out  $\mathbf{v}_{\tau}$  and  $z_{\tau}$ .
$$
\begin{array}{l} \ell (\mathcal {D} _ {\tau}; \pmb {\theta}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \log [ p _ {\pmb {\theta}} (\mathbf {e} _ {i} ^ {*}, y _ {i} ^ {*}) ] = \frac {1}{n} \sum_ {i = 1} ^ {n} \log [ p _ {\pmb {\theta}} (\mathbf {e} _ {i} ^ {*} | y _ {i} ^ {*}) p (y _ {i} ^ {*}) ] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left[ p _ {\boldsymbol {\theta}} \left(\mathbf {e} _ {i} ^ {*} \mid y _ {i} ^ {*}\right) \left[ \int_ {\mathbf {v} _ {\tau}} p \left(y _ {i} ^ {*} \mid \mathbf {v} _ {\tau}\right) p \left(\mathbf {v} _ {\tau}\right) d \mathbf {v} _ {\tau} \right] \right] \tag {1} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left[ p _ {\boldsymbol {\theta}} \left(\mathbf {e} _ {i} ^ {*} | y _ {i} ^ {*}\right) \left[ \int_ {\mathbf {v} _ {\tau}} p \left(y _ {i} ^ {*} | \mathbf {v} _ {\tau}\right) \left[ \sum_ {z _ {\tau} = 1} ^ {r} p \left(\mathbf {v} _ {\tau} | z _ {\tau}\right) p \left(z _ {\tau}\right) \right] d \mathbf {v} _ {\tau} \right] \right] \\ \end{array}
$$
where  $p_{\theta}(\mathbf{e}_i^* | y_i^*)$  specifies the probability of sampling  $\mathbf{e}_i^*$  from the  $y_i^*$ -th class,  $p(y_i^* | \mathbf{v}_{\tau})$  is the probability of sampling the  $y_i^*$ -th class for task  $\tau$ , and  $p(\mathbf{v}_{\tau} | z_{\tau})$  indicates the probability of generating a task  $\tau$  from the  $z_{\tau}$ -th mixture component.  $p(z_{\tau})$  is a prior on the  $z_{\tau}$ -th component. Hence, Eq. (1) implies a generative process of task  $\tau$ :  $z_{\tau} \rightarrow \mathbf{v}_{\tau} \rightarrow y_i^* \rightarrow \mathbf{e}_i^*$ . Next, we define each of the aforementioned distributions and propose our HTGM method.
3.2 Model Specification and Parameterization
In Eq. (1), the class-conditional distribution  $p_{\theta}(\mathbf{e}_i^* | y_i^*)$ , the task-conditional distribution  $p(y_i^* | \mathbf{v}_{\tau})$  and the mixture distribution of tasks defined by  $\{p(\mathbf{v}_{\tau} | z_{\tau}), p(z_{\tau})\}$  are not specified. To make Eq. (1) estimizable, we introduce our HTGM that models the generative process of tasks. Because  $\mathcal{D}_{\tau}^{\mathrm{s}}$  and  $\mathcal{D}_{\tau}^{\mathrm{q}}$  follow the same distribution, in the following, we ignore the superscript * for simplicity.
Class-Conditional Distribution. First, similar to [22, 23], we use Gaussian distribution to model the embeddings  $\mathbf{e}_i$ 's in every class. Let  $\boldsymbol{\mu}_{y_i}^{\mathrm{c}}$  and  $\Sigma_{y_i}^{\mathrm{c}}$  be the mean and variance of the  $y_i$ -th class, then  $p_{\theta}(\mathbf{e}_i|y_i) = \mathcal{N}(\mathbf{e}_i|\boldsymbol{\mu}_{y_i}^{\mathrm{c}},\boldsymbol{\Sigma}_{y_i}^{\mathrm{c}})$ . In fact, the samples in all of the classes of task  $\tau$  comprise a Gaussian mixture distribution, where  $p(y_i)$  is the mixture probability of the  $y_i$ -th class. In Eq. (1),  $p(y_i)$  is factorized to be task-specific, i.e.,  $p(y_i|\mathbf{v}_{\tau})$ , which resorts to another mixture distribution  $p(\mathbf{v}_{\tau})$  of tasks, and establishes a structure of hierarchical mixture.
Task-Conditional Distribution. A straightforward definition of  $p(y_{i}|\mathbf{v}_{\tau})$  is the density at  $\mu_{y_i}^{\mathrm{c}}$  in a Gaussian distribution with  $\mathbf{v}_{\tau}$  as the mean, where  $\pmb{\mu}_{y_i}^{\mathrm{c}}$  is the mean (or prototype) of the  $y_{i}$ -th class. However, doing so exposes two problems: (1) the density function of Gaussian distribution is log-concave with one global maximum. Given the mean and variance, maximizing its log-likelihood tends to collapse the prototypes  $\pmb{\mu}_{y_i}^{\mathrm{c}}$ 's of all classes in  $\tau$ , making them indistinguishable and impairing classification; (2) given  $\mathbf{v}_{\tau}$ , this method tends to sample classes with small  $D_{\mathbf{v}_{\tau}}(\pmb{\mu}_{y_i}^{\mathrm{c}})$ , where  $D_{\mathbf{v}_{\tau}}(\cdot)$  measures the Mahalanobis distance between a data point and the Gaussian distribution centered at  $\mathbf{v}_{\tau}$ . However, in most of the existing works, classes are often uniformly sampled from a domain without any prior on distances [8]. Fitting the distance function with such "uniform" classes naively leads to an ill-posed learning problem with degenerated solutions. In light of these issues, we seek to define  $p(y_{i}|\mathbf{v}_{\tau})$  as a (parameterized) density function with at least  $N$  global optimums so that it can distinguish the  $N$  different class prototypes of  $N$ -way tasks. The  $N$  equal (global) optimums also allow it to fit  $N$  classes uniformly sampled from a domain. To this end, let  $\pmb{\mu}_k^{\mathrm{c}}$  be the surrogate embedding of the  $k$ -th class, we propose a Gibbs distribution  $\pi (\pmb{\mu}_k^{\mathrm{c}}|\mathbf{v}_{\tau},\pmb {\omega})$  defined by  $\mathbf{v}_{\tau}$  and trainable parameters  $\omega$  with an energy function. Then we write  $p(y_{i} = k|\mathbf{v}_{\tau})$  as
$$
p _ {\omega} \left(y _ {i} = k \mid \mathbf {v} _ {\tau}\right) = \pi \left(\boldsymbol {\mu} _ {k} ^ {\mathrm {c}} \mid \mathbf {v} _ {\tau}, \omega\right) = \frac {\exp \left[ - E _ {\omega} \left(\boldsymbol {\mu} _ {k} ^ {\mathrm {c}} ; \mathbf {v} _ {\tau}\right) \right]}{\int_ {\boldsymbol {\mu} _ {k} ^ {\mathrm {c}}} \exp \left[ - E _ {\omega} \left(\boldsymbol {\mu} _ {k} ^ {\mathrm {c}} ; \mathbf {v} _ {\tau}\right) \right]} \tag {2}
$$
where  $E_{\omega}(\pmb{\mu}_{k}^{\mathrm{c}};\mathbf{v}_{\tau}) = \min \left(\left\{\|\pmb{\mu}_{k}^{\mathrm{c}} - \mathbf{W}_{j}\mathbf{v}_{\tau}\|_{2}^{2}\right\}_{j=1}^{N}\right)$  is our energy function, and the denominator in Eq (2) is a normalizing constant (with respect to  $\pmb{\mu}_{k}^{\mathrm{c}}$ ), a.k.a. the partition function in an energy-based model (EBM) [20].  $\pmb{\omega} = \{\mathbf{W}_{1},\dots,\mathbf{W}_{N}\}$  are trainable parameters, with  $\mathbf{W}_i\in \mathbb{R}^{d\times d}$ . Given  $\pmb{\omega}$  and  $\mathbf{v}_{\tau}$ , Eq. (2) has  $N$  global maximums at  $\pmb{\mu}_{k}^{\mathrm{c}} = \mathbf{W}_{1}\mathbf{v}_{\tau}$ , ...,  $\pmb{\mu}_{k}^{\mathrm{c}} = \mathbf{W}_{N}\mathbf{v}_{\tau}$ . More interpretations of the proposed task-conditional distribution can be found in Appendix B.4.
Mixture Distribution of Tasks. In Eq. (1), the task distribution  $p(\mathbf{v}_{\tau}|z_{\tau} = 1)$ , ...,  $p(\mathbf{v}_{\tau}|z_{\tau} = r)$ , weighted by their respective mixture probability  $p(z_{\tau})$ . Thus we specify  $p(\mathbf{v}_{\tau})$  as a Gaussian mixture distribution. We introduce  $\boldsymbol{\mu}_{z_{\tau}}^{\dagger}$  and  $\boldsymbol{\Sigma}_{z_{\tau}}^{\dagger}$  as the mean and variance of each component, i.e.,  $p(\mathbf{v}_{\tau}|z_{\tau}) = \mathcal{N}(\mathbf{v}_{\tau}|\boldsymbol{\mu}_{z_{\tau}}^{\dagger}, \boldsymbol{\Sigma}_{z_{\tau}}^{\dagger})$ , and let  $\rho = [\rho_1, \dots, \rho_r]$  be the mixture probabilities, where  $\rho_r = p(z_{\tau} = r)$  and  $\rho$  can be Uniform(r). Then  $\mathbf{v}_{\tau}$  is generated in two
steps: (1) draw a latent task variable  $z_{\tau}$  from a categorical distribution on  $\pmb{\rho}$ , and (2) draw  $\mathbf{v}_{\tau}$  from  $\mathcal{N}(\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{t}}, \boldsymbol{\Sigma}_{z_{\tau}}^{\mathrm{t}})$  [4]. As such, our HTGM generative process of an  $N$ -way  $K$ -shot  $Q$ -query task  $\tau$  can be summarized as following, and Fig. 1 illustrates the corresponding graphical model:
1. Draw  $z_{\tau} \sim \text{Categorical}([\rho_1, \ldots, \rho_r])$
2. Draw a task embedding  $\mathbf{v}_{\tau} \sim \mathcal{N}(\pmb{\mu}_{z_{\tau}}^{\mathrm{t}}, \pmb{\Sigma}_{z_{\tau}}^{\mathrm{t}})$
3. For  $k = 1,\dots,N$
(a) Draw a class prototype  $\pmb{\mu}_k^{\mathsf{c}}\sim \pi (\pmb{\mu}_k^{\mathsf{c}}|\mathbf{v}_{\tau},\omega)$  from the proposed Gibbs distribution in Eq. (2)
(b) For  $i = 1,\dots ,K + Q$
i. Set  $y_{i} = k$ , draw a sample  $\mathbf{e}_i \sim \mathcal{N}(\mathbf{e}_i|\boldsymbol{\mu}_{y_i}^{\mathrm{c}}, \boldsymbol{\Sigma}_{y_i}^{\mathrm{c}})$
ii. Allocate  $(\mathbf{e}_i, y_i)$  to the support set  $\mathcal{D}_{\tau}^{\mathrm{s}}$  if  $i \leq K$ ; else allocate  $(\mathbf{e}_i, y_i)$  to the query set  $\mathcal{D}_{\tau}^{\mathrm{q}}$
To reduce complexity, we investigate the feasibility of using isotropic Gaussian with tied variance, i.e.,  $\pmb{\Sigma}_{1}^{\mathrm{c}} = \dots = \pmb{\Sigma}_{N}^{\mathrm{c}} = \sigma^{2}\mathbf{I}$ , for class distributions, which turned out to be efficient in our experiments. Here,  $\mathbf{I}$  is an identity matrix,  $\sigma$  is a hyperparameter. Tied variance is also a commonly used trick in Gaussian discriminate analysis (GDA) for generative classifiers [22, 39]. For task distributions, the variances  $\pmb{\Sigma}_{1}^{\mathrm{t}},\dots,\pmb{\Sigma}_{r}^{\mathrm{t}}$  can be automatically inferred by our algorithm, as elaborated in Sec. 3.3.
Finally, in Eq. (1), substituting  $p_{\theta}(\mathbf{e}_i|y_i) = \mathcal{N}(\mathbf{e}_i|\boldsymbol{\mu}_{y_i}^{\mathrm{c}},\sigma^2\mathbf{I})$ $p_{\omega}(y_i|\mathbf{v}_{\tau}) = \pi (\pmb{\mu}_{y_i}^{\mathrm{c}}|\mathbf{v}_{\tau},\pmb {\omega}),p(\mathbf{v}_{\tau}|z_{\tau}) = \mathcal{N}(\mathbf{v}_{\tau}|\pmb{\mu}_{z_{\tau}}^{\mathrm{t}},\pmb{\Sigma}_{z_{\tau}}^{\mathrm{t}})$  whose probabilities are specified and parameterized, and letting  $\rho = \mathrm{Uniform}(r)$  be a uniform prior, we get our
Figure 1: Graphical model of the proposed generative process for  $B$  tasks with  $N$  ways,  $K$  shots and  $Q$  queries.