SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Both transformer-structured inference modules use the refined tokens  $\widehat{\nu}^m$  and  $\widehat{\omega}_o$  to obtain a global latent representation and a local latent parameter, respectively. The introduced tokens preserve the specific meta-knowledge for each latent variable during inference. Compared with the  $\theta$ -parameterised inference module exploring the intra-task relationships, the  $\phi$ -parameterised inference module enables the exploitation of the inter-task relationships to reason over each local latent parameter. Thus, the introduced tokens can be refined with relevant information from the heterogeneous context sets. By integrating meta-knowledge and heterogeneous context sets, HNPs can reduce the negative transfer of task-specific knowledge among heterogeneous tasks in each episode. Please refer to Appendix E for algorithms.
Multi-Task Learning. Multi-task learning can operate in various settings [9]. Here we roughly separate the settings of MTL into two branches: (1) Single-input multi-output (SIMO) [24-30], where tasks are defined by different supervision information for the same input. (2) Multi-input multi-output (MIMO) [11, 10, 31-34], where heterogeneous tasks follow different data distributions. This work considers the MIMO setup of multi-task learning with episodic training.
In terms of modeling methods, from a processing perspective, existing MTL methods can be roughly categorized into two groups: (1) Probabilistic MTL methods [11, 19, 35-41], which employ the Bayes framework to characterize probabilistic dependencies among tasks. (2) Deep MTL models [10, 32, 24-26, 42-48], which directly utilize deep neural networks to discover information-sharing mechanisms across tasks. However, deep MTL models rely on large amounts of training data and tend to overfit when encountering the data-insufficiency problem. Meanwhile, previous probabilistic MTL methods consider a small number of tasks that occur at the same time, limiting their applicability in real-world systems.
Meta-Learning. Meta-learning aims to find strategies to quickly adapt to unseen tasks with a few examples [49, 4, 5, 50]. There exist a couple of branches in meta-learning methods, such as metrics-based methods [6, 51-57] and optimization-based methods [5, 58-68]. Our paper focuses on a probabilistic meta-learning method, namely neural processes, that can quantify predictive uncertainty. Models in this family [7, 12, 13, 15, 16, 18, 69-73] can approximate stochastic processes in neural networks. Vanilla NPs [12] usually encounter the expressiveness bottleneck because their functional priors are not rich enough to generate complicated functions [15, 16]. [7] introduces deterministic variables to model predictive distributions for meta-learning scenarios directly. Most NP-based methods only focus on a single task during inference [7, 12, 15, 16, 14], which leaves task-relatedness between heterogeneous tasks in a single episode an open problem.
This paper combines multi-task learning and meta-learning paradigms to tackle the data-insufficiency problem. Our work shares the high-level goal of exploiting task-relatedness in an episode with [19, 74, 75]. Concerning the multi-task scenarios, the main differences are: [19, 74, 75] handles multiple attributes and multi-sensor data under the SIMO setting, while our work performs for the MIMO setting where tasks are heterogeneous and distribution shifts exist. Moreover, [76] theoretically addresses the conclusion that MTL methods are powerful and efficient alternatives to gradient-based meta-learning algorithms. However, our method inherits the advantages of multi-task learning and meta-learning: simultaneously generalizing meta-knowledge from past to new episodes and exploiting task-relatedness across heterogeneous tasks in every single episode. Thus, our method is more suitable for solving the data-insufficiency problem. Intuitive comparisons with related paradigms such as cross-domain few-shot learning [77-82], multimodal meta-learning [83-87, 56] and cross-modality few-shot learning [88-90] are provided in Appendix A.
We evaluate the proposed HNPs and baselines on three benchmark datasets under the episodic multi-task setup. Sec. 4.1 and Sec. 4.2 provide experimental results for regression and classification, respectively. Ablation studies are in Sec. 4.3. More comparisons with recent works on extra datasets are provided in Appendix F. Additional results under the convectional MIMO setup without episodic training can be found in Appendix G & H.
4.1 Episodic Multi-Task Regression
Dataset and Settings. To evaluate the benefit of HNPs over typical NP baselines in uncertainty quantification, we conduct experiments in several 1D regression tasks. The baselines include conditional neural processes (CNPs [13]), vanilla neural processes (NPs [12]), and attentive neural processes (ANPs [15]). As a toy example, we construct multiple tasks with different task distributions: each task's input set is defined on separate intervals without overlap.
Given four different tasks in an episode, their input sets are  $\mathbf{x}_{\tau}^{1:4}$ . Each input set contains a few instances, drawn uniformly at random from separate intervals, such as  $\mathbf{x}_{\tau}^{1} \in [-4, -2)$ ,  $\mathbf{x}_{\tau}^{2} \in [-2,0)$ ,  $\mathbf{x}_{\tau}^{3} \in [0,2)$ , and  $\mathbf{x}_{\tau}^{4} \in [2,4)$ . All tasks in an episode are related by sharing the same ground truth function. Following [12, 18], function-fitting tasks are generated with Gaussian processes (GPs). Here a zero mean Gaussian process  $y^{(0)} \sim \mathcal{GP}(0,k(\cdot,\cdot))$  is used to produce  $\mathbf{y}_{\tau}^{1:4}$  for the inputs from all tasks  $\mathbf{x}_{\tau}^{1:4}$ . A radial basis kernel  $k(x,x') = \sigma^2 \exp(-(x - x')^2) / 2l^2$ , with  $l = 0.4$  and  $\sigma = 1.0$  is used.
Results and Discussions. As shown in Figure 4, CNPs, ANPs, and our HNPs exhibit more reasonable uncertainty than NPs in Figure 4: lower variances are predicted around observed (context) points with higher variances around unobserved points. Furthermore, NPs and ANPs detrimentally impact the smoothness of the predicted curves, whereas HNPs yield smoother predictive curves with reliable uncertainty estimation. These observations suggest that integrating correlation information across related
Figure 4: Performance comparisons on the episodic multi-task 1-D function regression using 5 context points (black dots) for each task. Black curves are ground truth, and blue ones are predicted results. The shadow regions are  $\pm 3$  standard derivations from the mean [18].
tasks and meta-knowledge in HNPs can improve uncertainty quantification in multi-task regression.
To quantify uncertainty we use the average negative log-likelihood (the lower, the better). As shown in Table 1, our HNPs achieve a lower average negative log-likelihood than baselines, demonstrating our method's effectiveness in uncertainty estimation.
Table 1: Average negative log-likelihoods over target points from all tasks.
Methods	CNPs	NPs	ANPs	HNPs
Avg. NLL	0.0935	0.8649	-0.1165	-0.5207
4.2 Episodic Multi-task Classification
Datasets and Settings. We use Office-Home [91] and DomainNet [92] as episodic multi-task classification datasets. Office-Home contains images from four domains: Artistic (A), Clipart (C), Product (P) and Real-world (R). Each domain contains images from 65 categories collected from office and home environments. Note that all domains share the whole target space. The numbers of meta-training classes and meta-test classes are 40 and 25, respectively. There are about 15,500 images in total. DomainNet has six distinct domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch. It includes approximately 0.6 million images distributed over 345 categories. The
Table 2: Comparative results (95% confidence interval) for episodic multi-task classification on Office-Home and DomainNet. Best results are indicated in bold.
Method	Office-Home	DomainNet
4-task 1-shot	5-way 5-shot	4-task 1-shot	20-way 5-shot	6-task 1-shot	5-way 5-shot	6-task 1-shot	20-way 5-shot
ERM [93]	66.04 ±0.61	73.62 ±0.55	39.25 ±0.24	47.14 ±0.18	59.95 ±0.52	68.52 ±0.44	38.62 ±0.22	47.85 ±0.20
VMTL [11]	49.71 ±0.48	65.75 ±0.47	27.50 ±0.14	42.82 ±0.13	42.24 ±0.39	57.37 ±0.43	18.05 ±0.11	31.38 ±0.15
MAML [5]	60.58 ±0.60	75.29 ±0.53	34.29 ±0.19	48.39 ±0.20	53.21 ±0.46	65.24 ±0.47	17.10 ±0.12	20.35 ±0.14
Proto. Net. [6]	57.19 ±0.53	74.97 ±0.46	32.72 ±0.18	49.75 ±0.16	53.71 ±0.48	68.80 ±0.42	31.90 ±0.19	47.59 ±0.18
DGPs [94]	65.89 ±0.53	79.96 ±0.38	31.48 ±0.18	49.46 ±0.18	50.93 ±0.42	63.32 ±0.38	25.46 ±0.15	38.63 ±0.17
CNPs [13]	43.33 ±0.56	55.07 ±0.63	10.57 ±0.10	12.02 ±0.11	37.90 ±0.45	40.53 ±0.44	5.12 ±0.10	5.14 ±0.10
NPs [12]	33.66 ±0.48	53.99 ±0.60	5.25 ±0.16	11.40 ±0.11	20.58 ±0.51	20.53 ±0.53	5.12 ±0.09	5.11 ±0.09
TNP-D [95]	65.49 ±0.53	78.94 ±0.43	41.61 ±0.22	59.19 ±0.21	49.10 ±0.42	67.39 ±0.40	28.83 ±0.17	47.69 ±0.18
HNPs	76.29 ±0.51	80.80 ±0.42	51.82 ±0.23	59.97 ±0.18	62.36 ±0.53	69.38 ±0.42	39.32 ±0.23	48.56 ±0.19
numbers of meta-training classes and meta-test classes are 276 and 69, respectively. Here one domain corresponds to a specific task distribution in the episodic multi-task setting.
When it comes to the episodic multi-task classification, we compare HNPs with the following three branches: (1) Multi-task learning methods: ERM [93] directly expands the training set of the current task with samples of related tasks. VMTL [11] is one of the state-of-the-art under the MIMO setting of multi-task learning. (2) Meta-learning methods: MAML [5], Proto.Net [6] and DGPs [94] address each task separately with no mechanism to leverage task-relatedness in a single episode. (3) Methods from the NP family: CNPs [13] and NPs [12] are established methods in the NP family. TNP-D [95] is recent NP work in sequential decision-making for a single task in each episode.
Results and Discussions. The experimental results for episodic multi-task classification on Office-Home and DomainNet are reported in Table 2. We use the average accuracy across all task distributions as the evaluation metric. It can be seen that HNPs consistently outperform all baseline methods, demonstrating the effectiveness of HNPs in handling each task with limited data under the episodic multi-task classification setup.
NPs and CNPs do not work well under all episodic multi-task classification cases. This can be attributed to their limited expressiveness of the global representation and the weak capability to extract discriminative information from multiple contexts. In contrast, HNPs explicitly abstract discriminative information for each task in the episode with the help of local latent parameters, enhancing the expressiveness of the functional prior.
We also find that HNPs significantly surpass other baselines on 1-shot Office-Home and DomainNet, both under the 4/6-task 5-way and 4/6-task 20-way settings. This further implies that HNPs can circumvent the effect of the problem of data-insufficiency by simultaneously exploiting task-relatedness across heterogeneous tasks and meta-knowledge among episodes.
Influence of Hierarchical Latent Variables. We first investigate the roles of the global latent representation  $\mathbf{z}_{\tau}^{m}$  and the local latent parameters  $\mathbf{w}_{\tau,1:O}^{m}$  by leaving out individual inference modules. These experiments are performed on Office-home under the 4-task 5-way 1-shot setting. We report the detailed performance for tasks sampled from a single task distribution (A/C/P/R) and the average accuracy across all task distributions (Avg.) in Table 3. The variants without specific latent variables are included in the comparison by removing the corresponding inference modules.
As shown in Table 3, both  $\mathbf{z}_{\tau}^{m}$  and  $\mathbf{w}_{\tau,1:O}^{m}$  benefit overall performance. Our method with hierarchical latent variables performs  $9.20\%$  better than the variant without both latent variables,  $3.00\%$  better than the variant without  $\mathbf{z}_{\tau}^{m}$ , and  $5.18\%$  better than the variant without  $\mathbf{w}_{\tau,1:O}^{m}$ . This indicates that latent variables of HNPs complement each other in representing con
Table 3: Effectiveness of global latent representations  $\mathbf{z}_{\tau}^{m}$  and local latent parameters  $\mathbf{w}_{\tau,1;O}^{m}$  in the model.  $\checkmark$  and  $\times$  denote whether the variants of HNPs have the corresponding latent variable or not.
z_mτ	w_mτ,1:O	A	C	P	R	Avg.
X	X	62.64 ±0.72	56.87 ±0.71	75.18 ±0.79	73.68 ±0.77	67.09 ±0.63
X	✓	69.39 ±0.60	63.10 ±0.61	80.66 ±0.67	79.99 ±0.62	73.29 ±0.51
✓	X	67.02 ±0.67	60.70 ±0.69	78.26 ±0.76	78.47 ±0.72	71.11 ±0.59
✓	✓	73.31 ±0.63	64.92 ±0.68	83.38 ±0.66	83.54 ±0.64	76.29 ±0.51
text sets from multiple tasks and meta-knowledge. The variant without  $\mathbf{w}_{\tau,1:O}^{m}$  underperforms the variant without  $\mathbf{z}_{\tau}^{m}$  by  $2.18\%$ , in terms of the average accuracy. This demonstrates that  $\mathbf{z}_{\tau}^{m}$  suffers more from the expressiveness bottleneck than  $\mathbf{w}_{\tau,1:O}^{m}$ , weakening the models' discriminative ability. For classification, local latent parameters are more crucial than a global latent representation in revealing the discriminating knowledge from multiple heterogeneous context sets.
Influence of Transformer-Structured Inference Modules. To further understand our transformer-structured inference modules (Trans. w learnable tokens), we examine the performance against two other options: inference modules that solely utilize a multi-layer perceptron (MLP) and the variants that do not incorporate any learnable tokens (Trans. w/o learnable tokens). We also compare the probabilistic and deterministic versions of such inference modules. The deterministic variants consider the deterministic embedding for the hierarchical latent variables.
As shown in Table 4, our inference modules consistently outperform the variants, regardless of whether the inference network is probabilistic or deterministic. When using the probabilistic one, our inference modules respectively achieve  $1.04\%$  and  $2.99\%$  performance gains over Trans. w/o learnable tokens and MLP under the 4-task 5-way 1-shot setting. This implies the importance of learn
able tokens and task-relatedness in formulating transformer-structured inference modules, which reduces negative transfer among heterogeneous tasks in each meta-test episode. Moreover, the variants with probabilistic inference modules consistently beat deterministic ones in performance, demonstrating the advantages of considering uncertainty during modeling and inference.
Table 4: Performance comparisons between our transformer inference modules (Trans. w learnable tokens) and other alternatives.
Inference networks	1-shot	5-shot
Deterministic	MLP	64.93 ±0.66	72.39 ±0.56
Trans. w/o learnable tokens	70.22 ±0.62	76.15 ±0.54
Trans. w learnable tokens	70.61 ±0.56	76.70 ±0.50
Probabilistic	MLP	73.30 ±0.59	77.94 ±0.48
Trans. w/o learnable tokens	75.25 ±0.55	80.42 ±0.47
Trans. w learnable tokens	76.29 ±0.51	80.80 ±0.42
Effects of Different Ways to Generate Local Latent Parameters. We investigate the effects of different ways to generate each  $\mathbf{w}_{\tau,0}^{m}$  from the shared condition  $\mathbf{z}_{\tau}^{m}$  and  $\mathcal{C}_{\tau}^{1:M}$ . Given a Monte Carlo sample of global latent variables as  $\mathbf{z}_{\tau}^{m(i)}$ , in Table 5, we compare with two alternatives:
1) Concat directly concatenates each context feature and  $\mathbf{z}_{\tau}^{m(i)}$ , and takes the concatenation as inputs of the transformer-structured inference network  $\phi$ . 2) Add sums up each context feature and  $\mathbf{z}_{\tau}^{m(i)}$  and takes the result as the input. 3) Ours incorporates  $\mathbf{z}_{\tau}^{m(i)}$  into the transformer-structured inference module by merging it with the refined learnable tokens in Eq. (10). As shown in Table 5, Ours consistently performs the best. This implies that incorporating the conditional variables into the inference module is more effective than the direct combinations of  $\mathbf{z}_{\tau}^{m(i)}$  and instance features.
Table 5: Performance comparisons of different implementations of generating each local latent parameter  $\mathbf{w}_{\tau, o}^{m}$  from the condition  $\mathbf{z}_{\tau}^{m}$  and  $\mathcal{C}_{\tau}^{1:M}$ .
Methods	A	C	P	R	Avg.
Concat	65.69 ±0.59	58.64 ±0.61	77.54 ±0.68	77.10 ±0.64	69.74 ±0.51
Add	69.92 ±0.69	63.73 ±0.71	78.81 ±0.77	79.03 ±0.78	72.87 ±0.61
Ours	73.31 ±0.63	64.92 ±0.68	83.38 ±0.66	83.54 ±0.64	76.29 ±0.51
Effects of More "Shots" or "Classes". To investigate the effects of more "shots" or "classes" in the episodic multi-task classification setup, we conduct experiments by increasing  $K$  or  $O$  in the defined  $M$ -task  $O$ -way  $K$ -shot setup.
As shown in Table 6, the proposed HNPs have more advantages over the baseline method with the context data points below ten shots. With shots larger than ten, both methods will reach a performance bottleneck.
Table 6: Performance comparisons on Office-Home under the 4-task 5-way K-shot setup.
Methods	1-shot	5-shot	10-shot	20-shot
TNP-D	65.49 ±0.53	78.94 ±0.43	80.81 ±0.32	81.12 ±0.68
HNPs	76.29 ±0.51	80.80 ±0.42	81.28 ±0.38	81.56 ±0.36
Moreover, Table 7 shows that our method consistently outperforms the baseline method as the number of classes increases from 20 to 40 in step 5. However, the performance gap between them narrows slightly with more classes. The main reason could be that the setting with more classes suffers from less data insufficiency.
Table 7: Performance comparisons on DomainNet under the 6-task 0-way 1-shot setup.
Methods	5-way	20-way	25-way	30-way	35-way	40-way
TNP-D	49.10 ±0.42	28.83 ±0.17	25.93 ±0.14	24.08 ±0.12	22.62 ±0.11	21.64 ±0.53
HNPs	62.36 ±0.53	39.32 ±0.23	35.72 ±0.19	32.27 ±0.17	31.27 ±0.14	29.31 ±0.13
Sensitivity to the Number of Monte Carlo Samples. For the hierarchical latent variables in the HNPs, we investigate the model's sensitivity to the number of Monte Carlo samples. Specifically, the sampling number of the global latent representation  $\mathbf{z}_{\tau}^{m}$  and local latent parameters  $\mathbf{w}_{\tau,1:O}^{m}$  varies from 1 to 30. We examine on Office-Home under the 4-task 5-way 1-shot setting. In Figure 5, the runtime per iteration grows rapidly as the number of samples increases. However, there is no clear correlation between the performance and the num
Figure 5: Average accuracy and runtime of HNPs with different numbers of Monte Carlo samples.  $N_{z}$  and  $N_{w}$  are sampling numbers of  $\mathbf{z}_{\tau}^{m}$  and  $\mathbf{w}_{\tau,1:O}^{m}$ , respectively.
ber of Monte Carlo samples. There are two sweet spots in terms of average accuracy, one of which has favorable computation time. Hence, we set  $N_{z}$  and  $N_{w}$  to 5 and 10, respectively.
We also investigate the inference time of NP-based models per iteration on Office-Home under the 4task5way1shot setup. As shown in Table 8, our model needs more inference time than other NP-based meth
Table 8: Inference time of different NP-based methods.
Methods	CNPs	NPs	TNP-D	HNPs
Inference time(s)	0.04	0.05	0.08	0.15
ods for performance gains. The cost mainly comes from inferring the designed hierarchical latent variables; however, we consider this a worthwhile trade-off for the extra performance.
Technical Discussion. This work develops heterogeneous neural processes by introducing hierarchical latent variables and transformer-structured inference modules for episodic multi-task learning. With the help of heterogeneous context information and meta-knowledge, the proposed model can exploit task-relatedness, reason about predictive function distributions, and efficiently distill past knowledge to unseen heterogeneous tasks with limited data.
Limitation & Extension. Although the hierarchical probabilistic framework could mitigate the expressiveness bottleneck, the model needs more inference time than other NP-based methods for performance gains. Besides, the proposed method requires the target space to be the same across all tasks in a single episode. This requirement could limit the method's applicability in realistic scenarios where target spaces may differ across tasks. Our work could be extended to the new case without the shared target spaces, where the model should construct higher-order task-relatedness to improve knowledge sharing among tasks. Our code is provided to facilitate such extensions.
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
3 https://github.com/autumn9999/HNPs.git
[88] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. In Advances in Neural Information Processing Systems, 2019.
[89] Frederik Pahde, Patrick Jahnichen, Tassilo Klein, and Moin Nabi. Cross-modal hallucination for few-shot fine-grained recognition. arXiv preprint arXiv:1806.05147, 2018.
[90] Frederik Pahde, Mihai Puscas, Tassilo Klein, and Moin Nabi. Multimodal prototypical networks for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[91] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[92] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision, 2019.
[93] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[94] Ze Wang, Zichen Miao, Xiantong Zhen, and Qiang Qiu. Learning to learn dense gaussian processes for few-shot learning. In Advances in Neural Information Processing Systems, 2021.
[95] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179, 2022.