SlowGuess/NeurIPS_2023_OCR huggingface dataset:
HTGM induced loss  $\ell_{\mathrm{HTGM}}(\mathcal{D}_{\tau};\pmb {\theta},\pmb {\omega})$  . The class means  $\mu_{y_i}^{\mathrm{c}}$  , task means  $\mu_{z_{\tau}}^{\mathrm{t}}$  and variances  $\Sigma_{z_{\tau}}^{\mathrm{t}}$  are inferred in the E-step of our EM algorithm (the details can be found in Sec. 3.3 and A.6).
3.3 Model Optimization
It is hard to directly optimize  $\ell_{\mathrm{HTGM}}(\mathcal{D}_{\tau};\boldsymbol {\theta},\boldsymbol {\omega})$ , because the exact posterior inference is intractable (due to the integration over  $\mathbf{v}_{\tau}$ ). To solve it, we resort to variational methods, and introduce an approximated posterior  $q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathrm{s}})$ , which is defined by an inference network  $\phi$ , and implies we want to infer  $\mathbf{v}_{\tau}$  from its observed support set  $\mathcal{D}_{\tau}^{\mathrm{s}}$ . The query set  $\mathcal{D}_{\tau}^{\mathrm{q}}$  is not included because it is unavailable during model testing. Then we propose to maximize a lower-bound of Eq. (1), which is (the derivation can be found in Appendix A.1)
$$
\begin{array}{l} \ell_ {\mathrm {H T G M - L}} (\mathcal {D} _ {\tau}; \boldsymbol {\theta}, \boldsymbol {\omega}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\log p _ {\boldsymbol {\theta}, \boldsymbol {\omega}} (\mathbf {e} _ {i} | y _ {i}) + \mathbb {E} _ {\mathbf {v} _ {\tau} \sim q _ {\phi} (\mathbf {v} _ {\tau} | \mathcal {D} _ {\tau} ^ {\mathbf {s}})} [ \log p _ {\boldsymbol {\omega}} (y _ {i} | \mathbf {v} _ {\tau}) + \log \sum_ {z _ {\tau} = 1} ^ {r} p (\mathbf {v} _ {\tau} | z _ {\tau}) p (z _ {\tau}) ]\right) \\ + H \left(q _ {\phi} \left(\mathbf {v} _ {\tau} \mid \mathcal {D} _ {\tau} ^ {\mathbf {S}}\right)\right) \\ \end{array}
$$
where  $H(q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathbf{s}})) = -\int_{\mathbf{v}_{\tau}}q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathbf{s}})\log q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathbf{s}})d\mathbf{v}_{\tau}$  is the entropy function. Similar to VAE [16], Eq. (3) estimates the expectation (in the second term) by sampling  $\mathbf{v}_{\tau}$  from  $q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathbf{s}})$ , instead of the integration in Eq. (1), hence facilitates efficient computation. Next, we elaborate on the inference network, the challenges of maximizing Eq. (3), and our workarounds.
Inference Network. Similar to VAE,  $q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathrm{s}})$  is defined as a Gaussian distribution  $\mathcal{N}(\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{a}}, \bar{\sigma}^{2}\mathbf{I})$ , where  $\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{a}}$  is the output of the inference network, which approximates  $\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{t}}$  in Step 2 of the generative process, and  $\bar{\sigma}$  is a hyperparameter for the corresponding variance. As illustrated by Fig. 2(a), the inference network is built upon the base model  $f_{\theta}(\cdot)$  with two non-parametric aggregation (i.e., mean pooling) functions, thus  $\phi = \theta$ . The first function aggregates class-wise embeddings to prototypes  $\boldsymbol{\mu}_{y_i}^{\mathrm{c}}$ 's, similar to prototypical networks [41]. Differently, the second aggregates all prototypes to  $\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{a}}$ . During model training, we used the reparameterization trick [16] to sample  $\mathbf{v}_{\tau}$  from  $\mathcal{N}(\boldsymbol{\mu}_{z_{\tau}}^{\mathrm{a}}, \bar{\sigma}^{2}\mathbf{I})$ . It is noteworthy that  $H(q_{\phi}(\mathbf{v}_{\tau}|\mathcal{D}_{\tau}^{\mathrm{s}}))$  in Eq. (3) becomes a constant because  $\bar{\sigma}^{2}$  is a constant.
Challenge 1: Trivial Solution. In Eq. (3), since the first term  $\log p_{\theta, \omega}(\mathbf{e}_i | y_i) = -\frac{1}{2\sigma^{2d}} \| \mathbf{e}_i - \boldsymbol{\mu}_{y_i}^{\mathrm{c}} \|_2^2$  (constants were ignored) only penalizes the distance between a sample  $\mathbf{e}_i$  and its own class mean  $\boldsymbol{\mu}_{y_i}^{\mathrm{c}}$  (i.e., intra-class distances) without considering inter-class relationships, different class means  $\boldsymbol{\mu}_1^{\mathrm{c}}$ , ...,
Figure 2: An illustration of HTGM on its (a) training process, and (b) testing process. In (a),  $①②③$  are the three parts of the training loss in Eq. (3). In (b), the training task embeddings contain the embeddings of all training tasks, i.e. the outputs of the task-pooling in (a).
$\pmb{\mu}_{N}^{\mathrm{c}}$  in task  $\tau$  could collide, drawing all sample embeddings to the same spot. To avoid such a trivial solution and improve the stability of optimization, we apply negative sampling [29]
$$
\ell_ {\text {n e g}} \left(\mathcal {D} _ {\tau}; y _ {i}, \boldsymbol {\theta}, \boldsymbol {\omega}\right) = - \log \mathbb {E} _ {\mathbf {e} _ {j} \sim \mathcal {D} _ {\tau}} \left[ \exp \left(- \frac {\left\| \mathbf {e} _ {j} - \boldsymbol {\mu} _ {y _ {i}} ^ {\mathrm {c}} \right\| _ {2} ^ {2}}{2 \sigma^ {2 d}}\right) \right] \tag {4}
$$
where  $\mathbf{e}_j$  is a negative sample embedding from any class in the support set, and  $\mu_{y_i}^{\mathrm{c}}$  is the mean of the positive class. In practice, we found it is beneficial to integrate  $\ell_{\mathrm{neg}}$  with our likelihood  $\ell_{\mathrm{HTGM}}$  in Eq. (1) during training, i.e.  $\ell_{\mathrm{HTGM}} + \frac{1}{n}\sum_{i = 1}^{n}\ell_{\mathrm{neg}}$ . Correspondingly, from Eq. (3) we have
$$
\ell \left(\mathcal {D} _ {\tau}; \boldsymbol {\theta}, \boldsymbol {\omega}\right) = \ell_ {\mathrm {H T G M - L}} \left(\mathcal {D} _ {\tau}; \boldsymbol {\theta}, \boldsymbol {\omega}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {\mathrm {n e g}} \left(\mathcal {D} _ {\tau}; y _ {i}, \boldsymbol {\theta}, \boldsymbol {\omega}\right) \tag {5}
$$
which does not only serve as a robust training loss, but also helps solve the next challenge.
Challenge 2: The Partition Function in Eq. (2). The second term  $p_{\omega}(y_i|\mathbf{v}_{\tau})$  in Eq. (3) involves computing the partition function in Eq. (2) (i.e., the denominator), which is intractable because of the integration over all possible  $\pmb{\mu}_k^{\mathrm{c}}$ 's. To solve it, we propose an upper bound of the partition function  $\int_{\pmb{\mu}_k}\exp \left[-E_{\omega}(\pmb{\mu}_k^{\mathrm{c}};\mathbf{v}_{\tau})\right]d\pmb{\mu}_k^{\mathrm{c}}\leq N\sqrt{\pi^d}$  (the derivation is in Appendix A.2), which is a constant with a specific  $N$ . By replacing the partition function in Eq. (2) with  $N\sqrt{\pi^d}$ , we got a lower bound of  $p_{\omega}(y_i|\mathbf{v}_{\tau})$ , which in turn relaxes the lower bound in Eq. (3). The following theorem (the proof is in Appendix A.3) states that the tightness of the relaxed bound is controllable.
Theorem 3.1. Among the  $N$  global maximums  $\mathbf{W}_1\mathbf{v}_{\tau}$ , ...,  $\mathbf{W}_N\mathbf{v}_{\tau}$  of Eq. (2), let  $\mathbf{W}_h\mathbf{v}_{\tau}$  and  $\mathbf{W}_l\mathbf{v}_{\tau}$ $(1 \leq h, l \leq N)$  be the pair with the smallest Euclidean distance  $D_{hl}$ , we have
$$
\lim  _ {D _ {h l} \rightarrow \infty} \int_ {\boldsymbol {\mu} _ {k} ^ {c}} \exp \left[ - E _ {\omega} \left(\boldsymbol {\mu} _ {k} ^ {c}; \mathbf {v} _ {\tau}\right)\right] d \boldsymbol {\mu} _ {k} ^ {c} = N \sqrt {\pi^ {d}} \tag {6}
$$
This theorem indicates the partition function approximates  $N\sqrt{\pi^d}$  when all pairs of the global maximums are far apart. It is noteworthy that when maximizing the likelihood, we fit  $\mathbf{W}_1\mathbf{v}_{\tau}$ , ...,  $\mathbf{W}_N\mathbf{v}_{\tau}$  to different class prototypes  $\pmb{\mu}_1^{\mathrm{c}}$ , ...,  $\pmb{\mu}_N^{\mathrm{c}}$  in  $N$ -way tasks. Because  $\ell_{\mathrm{neg}}$  in Eq. (4) tends to maximize the distances between different prototypes through the negative samples, maximizing the joint loss  $\ell$  in Eq. (5) tends to separate  $\mathbf{W}_1\mathbf{v}_{\tau}$ , ...,  $\mathbf{W}_N\mathbf{v}_{\tau}$ , thus tighten the relaxed bound after using  $N\sqrt{\pi^d}$  according to Theorem 3.1. This is another benefit of negative sampling.
Optimization via Expectation-Maximization. In the third term of  $\ell_{\mathrm{HTGM - L}}$  in Eq. (3), we need to estimate the mixture distribution  $p(z_{\tau})$ . Similar to optimize Gaussian mixture models, we alternately
infer the posterior  $p(z_{\tau}|\mathbf{v}_{\tau})$  (i.e., the mixture membership of  $\mathbf{v}_{\tau}$ ) and solve the model parameters  $\{\theta, \omega\}$  through an Expectation-Maximization algorithm. In E-step, we infer  $p(z_{\tau}|\mathbf{v}_{\tau})$  when fixing model parameters. In M-step, when fixing  $p(z_{\tau}|\mathbf{v}_{\tau})$ ,  $\{\theta, \omega\}$  can be efficiently solved by optimizing Eq. (5) with stochastic gradient descent (SGD). The detailed optimization algorithm of HTGM can be found in Appendix A.6.
3.4 Model Adaptation
Fig. 2(b) illustrates the adaptation process of HTGM. Given a new  $N$ -way task  $\tau'$  from the meta-test set  $\mathcal{D}^{\mathrm{te}}$ , its support set  $\mathcal{D}_{\tau'}^{\mathrm{s}}$  is fed to the inference network to generate (1) class prototypes  $\pmb{\mu}_1^{\mathrm{c}}$ , ...,  $\pmb{\mu}_N^{\mathrm{c},2}$  (similar to prototypical networks), and (2) distribution  $q_{\phi}(\mathbf{v}_{\tau'}|\mathcal{D}_{\tau'}^{\mathrm{s}})$ , from which we draw the average task embedding  $\mathbf{v}_{\tau'} = \pmb{\mu}_{z_{\tau'}}^{\mathrm{a}}$ . Recall that the inference network is the base model  $f_{\theta}(\cdot)$  with class-pooling and task-pooling layers, as illustrated in Fig. 2(b), and  $\phi = \pmb{\theta}$ . Then  $\mathbf{v}_{\tau'}$  is projected to  $\mathbf{W}_1\mathbf{v}_{\tau'}, \dots, \mathbf{W}_N\mathbf{v}_{\tau'}$  which represent the  $N$  optimal choices of class prototypes for task  $\tau'$  as learned by the Gibbs distribution in Eq. (2) from the training tasks. They are used to adapt  $\pmb{\mu}_1^{\mathrm{c}}$ , ...,  $\pmb{\mu}_N^{\mathrm{c}}$  so that the adapted prototypes are drawn towards the closest classes from the mixture component that task  $\tau'$  belongs to. Specifically, the adaptation is performed by selecting the closest optimum for each prototype  $\pmb{\mu}_j^{\mathrm{c}}$  ( $1 \leq j \leq N$ ), that is
$$
\bar {\boldsymbol {\mu}} _ {j} ^ {\mathrm {c}} = \alpha \boldsymbol {\mu} _ {j} ^ {\mathrm {c}} + (1 - \alpha) \mathbf {W} _ {l ^ {*}} \mathbf {v} _ {\tau^ {\prime}}, \quad \text {w h e r e} l ^ {*} = \underset {1 \leq l \leq N} {\arg \min } D \left(\boldsymbol {\mu} _ {j} ^ {\mathrm {c}}, \mathbf {W} _ {l} \mathbf {v} _ {\tau^ {\prime}}\right), \tag {7}
$$
and  $D(\cdot ,\cdot)$  is the Euclidean distance,  $\alpha$  is a hyperparameter. The following theorem confirms the effectiveness of the this adaptation method (the proof can be found in Appendix A.4).
Theorem 3.2. The adapted prototypes  $\bar{\pmb{\mu}}_1^c$ , ...,  $\bar{\pmb{\mu}}_N^c$  from Eq. (7) maximizes the lower-bound  $\ell_{HTGM-L}$  in Eq. (3) of the likelihood in Eq. (1) when  $\alpha = \frac{K}{K + 2\sigma^{2d}}$ .
Theorem 3.2 suggests an automatic setup of  $\alpha$ , which can also be tuned empirically for optimal value using validation datasets. We evaluated the empirical values of  $\alpha$  in our experiments and discussed their relationships with the theoretical values in Appendix A.4.
Finally, we (1) assess if  $\tau'$  is a novelty by computing the likelihood of  $\mathbf{v}_{\tau'}$  in a pre-fitted GMM on the embeddings  $\mathbf{v}_{\tau}$ 's of the training tasks in  $\mathcal{D}^{\mathrm{tr}}$ , and (2) perform classification on each sample  $\mathbf{x}_i'$  in the query set  $\mathcal{D}_{\tau'}^{\mathrm{q}}$  using the adapted prototypes by
$$
p \left(y _ {i} ^ {\prime} = j ^ {\prime} \mid \mathbf {x} _ {i} ^ {\prime}\right) = \frac {\exp \left(- D \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {i} ^ {\prime}\right) , \bar {\boldsymbol {\mu}} _ {j ^ {\prime}} ^ {\mathrm {c}}\right)\right)}{\sum_ {j = 1} ^ {N} \exp \left(- D \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {i} ^ {\prime}\right) , \bar {\boldsymbol {\mu}} _ {j} ^ {\mathrm {c}}\right)\right)} \tag {8}
$$
which is the posterior probability. The derivation of Eq. (8) is in Appendix A.5.
In this section, we evaluate HTGM's effectiveness on few-shot classification and novel task detection on benchmark datasets, and compare it with SOTA methods.
Datasets. The first dataset is the *Plain-Multi* benchmark [52]. It includes four fine-grained image classification datasets, i.e., CUB-200-2011 (Bird), Describable Textures Dataset (Texture), FGVC of Aircraft (Aircraft), and FGVCx-Fungi (Fungi). In each episode, a task samples classes from one of the four datasets, so that different tasks are from a mixture of the four domains. The second dataset is the *Art-Multi* benchmark [53], whose distribution is more complex than Plain-Multi. Similar to [13], each image in Plain-Multi was applied with two filters, i.e., blur filter and pencil filter, respectively, to simulate a changing distribution of few-shot tasks. Afterward, together with the original four datasets, a total of 12 datasets comprise the Art-Multi, and each task is sampled from one of them. Both benchmarks were divided into the meta-training, meta-validation, and meta-test sets by following their corresponding papers. Moreover, we used the Mini-ImageNet dataset [47] to evaluate the case of uni-component distribution of tasks, which is discussed in Appendix D.6.
Baselines. We compared HTGM with the relevant SOTA methods on meta-learning, including (1) optimization-based methods: MAML [8] and Meta-SGD [26] learn globally shared initialization
In this section,  $\pmb{\mu}_j^{\mathsf{c}}$  ( $1 \leq j \leq N$ ) is the  $j$ -th class mean of support set, its superscript  $\mathsf{s}$  is omitted for clarity.
Setting	Model	Bird	Texture	Aircraft	Fungi	Avg.
5-way, 1-shot	TAML	55.77±1.43	31.78±1.30	48.56±1.37	41.00±1.50	44.28
MAML	53.94±1.45	31.66±1.31	51.37±1.38	42.12±1.36	44.77
Meta-SGD	55.58±1.43	32.38±1.32	52.99±1.36	41.74±1.34	45.67
MUMOMAML	56.82±1.49	33.81±1.36	53.14±1.39	42.22±1.40	46.50
HSML	60.98±1.50	35.01±1.36	57.38±1.40	44.02±1.39	49.35
ARML	62.33±1.47	35.65±1.40	58.56±1.41	44.82±1.38	50.34
ProtoNet	61.54±1.27	38.84±1.42	73.42±1.23	46.52±1.42	55.08
MetaOptNet	62.80±1.29	44.30±1.45	68.64±1.29	47.04±1.38	55.70
ProtoNet-Aug	65.04±1.29	44.68±1.43	70.44±1.32	49.30±1.40	57.37
NCA	62.58±1.25	40.98±1.44	68.70±1.26	46.36±1.34	54.66
FEATS	62.60±1.31	44.12±1.49	68.86±1.28	47.92±1.34	55.88
HTGM (ours)	70.12±1.28	47.76±1.49	75.52±1.24	52.06±1.41	61.37
5-way, 5-shot	TAML	69.50±0.75	45.11±0.69	65.92±0.74	50.99±0.87	57.88
MAML	68.52±0.79	44.56±0.68	66.18±0.71	51.85±0.85	57.78
Meta-SGD	67.87±0.74	45.49±0.68	66.84±0.70	52.51±0.81	58.18
MUMOMAML	70.49±0.76	45.89±0.69	67.31±0.68	53.96±0.82	59.41
HSML	71.68±0.73	48.08±0.69	73.49±0.68	56.32±0.80	62.39
ARML	73.34±0.70	49.67±0.67	74.88±0.64	57.55±0.82	63.86
ProtoNet	78.88±0.72	57.93±0.75	86.42±0.57	62.52±0.79	71.44
MetaOptNet	81.66±0.71	61.97±0.78	84.03±0.56	63.80±0.81	72.87
ProtoNet-Aug	80.62±0.71	58.30±0.77	87.05±0.53	63.62±0.81	72.39
NCA	79.16±0.75	58.69±0.76	85.27±0.53	61.68±0.80	71.20
FEATS	78.37±0.72	57.02±0.73	85.55±0.54	61.56±0.80	70.63
HTGM (ours)	82.27±0.74	60.67±0.78	88.48±0.52	65.70±0.79	74.28
Table 1: Results (accuracy  $\pm  {95}\%$  confidence) of the compared methods on Plain-Multi dataset.
among tasks. MUMOMAML [48] is a task-specific method. TAML [21] handles imbalanced tasks. HSML [52] and ARML [53] learn locally shared initial parameters in clusters of tasks and neighborhoods of a meta-graph of tasks, respectively; and (2) Metric-based methods: ProtoNet [41] is the prototypical network. MetaOptNet [24] uses an SVM classifier with kernel metrics. ProtoNet-Aug [42], FEATS [54] and NCA [19] were built upon ProtoNet by augmenting images (e.g., rotation, jigsaw), adding prototype aggregator (e.g., Transformer), and using contrastive training loss (instead of prototype-based loss), respectively. The detailed setup of these methods is in Appendix C.1.
Implementation. Following [44], the optimization-based baselines used the standard four-block convolutional layers as the base learner, and the metric-based methods used ResNet-12. The output dimension of these networks is 640 (MetaOptNet uses 16000 as in its paper). In our experiments, we observed the optimization-based methods reported out-of-memory errors when using ResNet-12, indicating their limitation in using large backbone networks. To test them on ResNet-12, we followed the ANIL method [36] by pre-training ResNet-12 via ProtoNet, freezing the encoder, and fine-tuning the last fully-connected layer. In this case, HSML and ARML cannot model the mixture task distribution properly as they require joint training of the encoder and other layers. The details are in Appendix D.5. For training, Adam optimizer was used. Each batch contains 4 tasks. Each model was trained with 20000 episodes. The learning rate of the metric-based methods was  $1e^{-3}$ . The learning rates for the inner- and outer-loops of the optimization-based methods were  $1e^{-3}$  and  $1e^{-4}$ . The weight decay was  $1e^{-4}$ . For HTGM, we set  $\sigma = 1.0$ ,  $\bar{\sigma} = 0.1$ ,  $\alpha = 0.5$  (0.9) for 1-shot (5-shot) tasks. The number of mixture components  $r$  varies w.r.t. different datasets, and was grid-searched within [2, 4, 8, 16, 32]. All hyperparameters were set using the meta-validation sets.
4.1 Experimental Results
Few-shot classification. Following [44], we report the mean accuracy and  $95\%$  confidence interval of 1000 random tasks with 5-way 1-shot/5-shot, 5/25-query tests. Following [53], we report the accuracy of each domain and the overall average accuracy for Plain-Multi, and report the accuracy of each image filtering strategy and the overall average accuracy for Art-Multi. Table 1 and 2 summarize the results. From the tables, we have several observations. First, metric-based methods generally outperform optimization-based methods. This is because of the efficiency of metric-based methods, enabling them to fit a larger backbone network, which is consistent with the results in [44]. Built upon the metric-based method, HTGM only introduces a few distribution-related parameters and thus has the flexibility to scale with the encoder size. Second, the baselines designed for dealing
Setting	Model	Original	Blur	Pencil	Avg.
5-way, 1-shot	TAML	42.22±1.39	40.02±1.41	35.11±1.34	39.11
MAML	42.70±1.35	40.53±1.38	36.71±1.37	39.98
Meta-SGD	44.21±1.38	42.36±1.39	37.21±1.39	41.26
MUMOMAML	45.63±1.39	41.59±1.38	39.24±1.36	42.15
HSML	47.92±1.34	44.43±1.34	41.44±1.34	44.60
ARML	45.68±1.34	42.62±1.34	39.78±1.34	42.69
ProtoNet	55.23±1.31	51.70±1.42	49.22±1.44	52.05
MetaOptNet	56.10±1.35	52.33±1.43	49.08±1.45	52.50
ProtoNet-Aug	57.63±1.34	55.00±1.40	49.73±1.53	54.12
NCA	56.12±1.35	50.80±1.49	47.99±1.45	51.64
FEATS	54.33±1.33	50.90±1.48	47.96±1.48	51.07
HTGM (ours)	61.18±1.34	58.80±1.42	53.23±1.48	57.74
5-way, 1-shot	TAML	58.54±0.73	55.23±0.75	49.23±0.75	54.33
MAML	58.30±0.74	55.71±0.74	49.59±0.73	54.50
Meta-SGD	57.82±0.72	55.54±0.73	50.24±0.72	54.53
MUMOMAML	58.60±0.75	56.29±0.72	51.15±0.73	55.35
HSML	60.63±0.73	57.91±0.72	53.93±0.72	57.49
ARML	61.78±0.74	58.73±0.75	55.27±0.73	58.59
ProtoNet	71.34±0.73	67.28±0.75	64.32±0.76	67.65
MetaOptNet	72.33±0.72	68.90±0.78	63.89±0.71	68.37
ProtoNet-Aug	72.87±0.71	70.50±0.72	63.98±0.73	68.78
NCA	72.44±0.72	67.33±0.71	62.98±0.78	67.58
FEATS	71.99±0.71	67.54±0.72	63.09±0.76	67.54
HTGM (ours)	74.67±0.70	71.24±0.73	65.22±0.77	70.37
with mixture distributions of tasks, i.e., HSML and ARML, outperform their counterparts without such design, demonstrating the importance to consider mixture task distribution in practice. Finally, HTGM outperforms the SOTA baselines in most cases by large margins, suggesting its effectiveness in modeling the generative process of task instances.
Novel task detection. We also evaluate HTGM on the task of detecting novel  $N$ -way- $K$ -shot tasks  $(N = 5, K = 1)$  that are drawn out of the training task distributions. To this end, we train each comapred model in the Original domain in Art-Multi dataset, and test the model on tasks drawn from either Original domain (i.e., known tasks), or  $\{\text{Blur, Pencil}\}$  domains (i.e., novel tasks), and evaluate if the model can tell whether a testing task is known or novel.
For comparison, since none of the baselines detects novel tasks, we adapt them as follows. For metric-based methods, since they use a fixed encoder for all training/testing tasks, we averaged the sample embeddings in each task to represent the task. Then a separate GMM model was built upon the training task embeddings, and its likelihood was adapted to score the novelty of testing tasks (some details of the setup are in Appendix C.2).
However, optimization-based models perform gradient descent on the support set of each task, leading to varying encoders per task. As such, sample embeddings of different tasks are not comparable, and we cannot obtain task
embeddings in the same way as before. Among them, only HSML has an augmented task-level encoder for task embedding, allowing us to include it for comparison. For a fair comparison, our HTGM also trains a GMM on its task embeddings for detecting novel tasks. Moreover, two HTGM variants were included for ablation analysis to understand some design choices: (1) HTGM-Gaussian replaces the Gibbs distribution in Eq. (2) with a Gaussian distribution; (2) HTGM w/o GMM removes the task-level GM, i.e., the third term in Eq. (3). The classification results of the ablation variants are in Appendix D.4. Following [6, 46, 56], we report Area Under ROC (AUROC), Average Precision (AP), and Max-F1. Table 3 summarizes the results, from which we observe HTGM outperforms all
Table 2: Results (accuracy  $\pm  {95}\%$  confidence) of the compared methods on Art-Multi dataset.
Model	AUROC	AP	Max-F1
HSML	55.96	37.94	50.17
ProtoNet	65.17	41.51	56.07
MetaOptNet	72.71	63.77	58.33
NCA	66.28	51.45	52.74
ProtoNet-Aug	72.67	57.93	59.07
FEATS	59.35	42.57	49.31
HTGM w/o GMM	70.24	62.45	57.75
HTGM-Gaussian	74.06	66.18	60.62
HTGM	75.66	68.03	60.51
Table 3: Comparison between HTGM, its variants and applicable baselines on novel task detection.
(a) HSML
(b) MetaOptNet
Figure 3: The frequency of tasks w.r.t. the normalized likelihood for (a) HSML (b) MetaOptNet (c) ProtoNet-Aug (d) HTGM. The x-axis ranges vary as only  $95\%$  tasks with top scores were preserved.
(c) ProtoNet-Aug
(d) HTGM
bases over all evaluation metrics, indicating the superior quality of task embeddings learned by our model. The embeddings follow the specified mixture distribution of tasks  $p(\mathbf{v}_{\tau})$  as described in Sec. 3.2, which fits the mixture data well hence allowing to detect novel tasks that are close to the boundary. Since the baselines learn embeddings without explicit constraint, they don't fit the post-hoc GMM well. Moreover, HTGM outperforms HTGM w/o GMM, which is even worse than some other baselines. This further validates the necessity to introduce the regularization of task-level mixture distribution  $p(\mathbf{v}_{\tau})$ . Also, the drops of AUROC and AP of HTGM-Gaussian demonstrate the importance of our unique design of the Gibbs distribution for the task-conditional distribution in Eq. (2). Similar to [46], in Fig. 3, we visualized the normalized likelihood histogram of known and novel tasks for HSML, MetaOptNet (the best baseline), ProtoNet-Aug (near-best baseline), and HTGM. To better interpret the figures, we calculated the ratio between the non-overlapped area of the two distributions and the total-area in Fig. 3 for HSML: 0.1379, MetaOpt: 0.4952, ProtoNet-Aug: 0.4806, and HTGM: 0.5578. As can be seen, the ratio of the non-overlapped area of HTGM is higher than other methods, which indicates the likelihoods (i.e., novelty scores) of HTGM are more distinguishable for known and novel tasks than the baseline methods. We also analyzed the hyperparameters  $\sigma$ ,  $\bar{\sigma}$ , and  $r$  of HTGM in Appendix D.1, D.2, and D.3.