Episodic Multi-Task Learning with Heterogeneous Neural Processes
Authors: Jiayi Shen
Xiantong Zhen
Cheems Wang
Marcel Worring
Venue: NeurIPS 2023 spotlight (2023)
Keywords: data-insufficiency problem
episodic training
multi-task learning and neural processes

Abstract: This paper focuses on the data-insufficiency problem in multi-task learning within an episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta-knowledge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take advantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Meanwhile, transformer-structured inference modules are designed to enable efficient inferences toward meta-knowledge and task-relatedness. In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.

Episodic Multi-Task Learning with Heterogeneous Neural Processes
Jiayi Shen', Xiantong 1. Qi (Cheems) Wang', Marcel Worring'
1 University of Amsterdam, Netherlands, {j shen, n.worring/@uva.i. .nl
2 Inception Institute of Artificial Intelligence, Abu Dhabi, UAE, zhenxt@gmail.cm com
Kaiyuan Mathematical Sciences Institute, Changsha, China, hhg123go0gmail com
Abstract
This paper focuses on the data-insufficiency problem multi-task learning within episodic training setup. Specifically, we explore the potential of heterogeneous information across tasks and meta- kootlknnowedge among episodes to effectively tackle each task with limited data. Existing meta-learning methods often fail to take ad- vantage of crucial heterogeneous information in a single episode, while multi-task learning models neglect reusing experience from earlier episodes. To address the problem of insufficient data, we develop Heterogeneous Neural Processes (HNPs) for the episodic multi-task setup. Within the framework of hierarchical Bayes, HNPs effectively capitalize on prior experiences as meta-knowledge and capture task-relatedness among heterogeneous tasks, mitigating data-insufficiency. Mean- while, transformer-struccured inference modules are designed to enable efficient inferences toward meta- keollleeoweede and musk-relatedness In this way, HNPs can learn more powerful functional priors for adapting to novel heterogeneous tasks in each meta-test episode. Experimental results show the superior performance of the proposed HNPs over typical baselines, and ablation studies verify the effectiveness of the designed inference modules.
Introduction
Deep learning models have made remarkable progress with the help the exponential increase in the amount of available training data [1]. However, many practical scenarios only have access to limited labeled data [2]. Such inuuffiiuuffc sharply degrades the model's performance [2, 3]. Both meta-learning and multi-task learning have the potential to alleviate the data-insufficiency issue. Meta-learning can extract knowlede from past episodes and thus enables rapid adaptation to new episodes with a few examples only [4-7]. Meanwhile, multi-task learning exploits the correlation among several tasks and results in more accurate learners for all tasks simultancously [8- 11]. However, the integration of meta-learning and multi-task learning in overcoming the data- insufficiency problem is rarely investigated.
In episodic training [4], existing meta-learning methods [4-7, 12, 13] in every meta- vettatraining or meta-test episode learn a single-task. this paper, we refer this conventional setting as episodic single-task learning. This setting restricts the potential for these models to explore task-relatedness within each episode, leaving the learning multiple heterogeneous tasks in a single episode under- explored. We consider multiple tasks in each episode episodic multi-task learning. The crux of episodic multi-task learning training to meta-test episodes. The differences between episodic single-task learning and episodic multi-task learning are illustrated in Figure 1. To be specific, we restrict the scope the problem
generalize the ability of exploring rask-relatedness from meta-
Currently with United Imaging Healthcare, Co., Ltd., China.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Episodia Single Learning
Episodio Mutti Task Leamiing
aode
coder
A4
mode
Gptocdo
OA
Figure I: Illustration of episodic multi-task learning. Each row corresponds to meta-training or meta-test episode. Different colors represent different label spaces among episodes; the same color with different shades represents different categories in the same task. Compared with episodic single-task learning, episodic multi-task learning simultancously handles several related tasks in single episode.
setup to the case where tasks in each meta-training or meta-test episode are heterogeneous but also relate each other by sharing the same target space.
The neural process (NP) family [12, 13], as typical meta-learning probabilistic models [14), efficiently quantifies predictive uncertainty with limited data, making it in principle well-suited for tackling the problem of data- iuttaiinsuffciency. However, in practice, it is challenging for vanilla NPs [12] with global latent variable to encode beneficial heterogeneous information in each episode. This issue is also known as the expressiveness bottleneck [15, 16), which weakens the model's capacity to handle insufficient data, especially when faced with diverse heterogeneous tasks.
To better resolve the data-insufficiengs problem, we develop Heterogeneous Neural Processes (HNPs) for episodic multi-task learning. As new member of the NP family, HNPs improve the expressiveness of vanilla NPs by introducing hierarchical functional space with global and local latent variables. The remainder of this work is structured follows: We introduce our method in Section (2). Related work is overviewed Section (3). We report experimental results with analysis in Section (4), after which conclude with a technical discussion, existing limitations, and future extensions. In detail, our technical contributions are two-fold:
Built on the hierarchical Bayes framework, our developed HNPs can simultancously gen- eralize meta-knowledge from past episodes to new episodes and exploit sesteeessss across heterogeneous tasks in every single episode. This mechanism makes HNPs more powerful when encoding complex conditions into functional priors.
We design transformer-structuree inference modules to infer the hierarchical latent variables, capture resk-relat,,ness, and learn set tokens as mnea-eenwledde. The designed modules fuse the meta-knowledge and heterogeneous information from context samples in unified manner, boosting the generalization capability of HNPs across tasks and episodes.
Experimental results show that the proposed HNPs together with transformer--tructuree inference modules, can exhibit superior performance regression and classification tasks under the episodic multi-task setup.
2 Methodology
We will now formally define episodic multi-task learning. For a single episode т, consider M heterogeneous but related tasks I.:. \ j Notably, the subscript denotes an episode, while superscripts are used to distinguish tasks in this episode. In the episodic multi- task setup, tasks in a single episode are heterogeneous since they are sampled from different task distributions (I(Im)) M= but are related at the same time as they share the target space ),,
Notations ?
For ease presentation, we abbreviate set ((-)TMMMM as (-):M, where a positive integer. Likewise, (0))) to is abbreviated as (0),0- For convenience, the notation table is provided in Appendix B.

---

To clearly relate to the modeling of vanilla neural processes [12], this paper follows its nomenclature to define each task. Note that in vanilla neural processes context and target are often respectively called supporf and query in conventional meta-learning [4, 5]. Each task I.M contains a context set with limited training data CT { Dr.JJiml Ni and target set 7 {- V NN where Ne and Ny are the numbers of context samples and target samples, respectively. T and "" represent features of context and target samples: while T y E V are their corresponding targets, where i = 1, ..., j = 1, 2, ... Ny; m = 1,2,... ...,, M. For simplicity, we denote the set of target samples and their corresponding ground-
....
Ny by x = {-= No y" {u j=1' For an episode T, episodicc multi-task learning aims to perform simultaneously well on each corresponding target set T," m """, 1, 2.., M, given the collection of context sets C.:.A.
For classification, this paper follows the protocol of meta models [4, 5, 17], such as O-way K-shot setup, clearly suffering from the data-insufficiency problem. Thus, episodic multi-task classification can be cast as M-task O-way K-shot supervised learning problem. An episode has M related classification tasks, and each of them has a context set with K different instances from each of the O classes [5]. It is worth mentioning that the target spaces of meta-training episodes do not overlap with any categories in those of meta-test episodes.
2.1 Modeling and Inference of Heterogeneous Neural Processes
We now present the proposed heterogeneous neural process. The proposed model inherits the advantages of multi-task learning and meta-learning, which can exploit task-relatedness among heterogeneous tasks and extract meta-knowledde from previous episodes. Next, we characterize the generative process, clarify the modeling within the hierarchical Bayes framework, and derive the approximate evidence lower bound (ELBO) in optimization.
Generative Processes. To get our proposed method HNPs, we extend the distribution over single function p(ff-) as used in vanilla NPs to joint distribution of multiple functions p(f.:M) ) for all heterogeneous tasks in single episode T. In detail, the underlying multi-task function distribution p(ff!:M) is inferred from a collection of context sets Cl:M and learnable meta-knowledde w, y IIM,, Note that represents the shared meta-knowledde meta-knowledge corresponding to the task distribution p(I").
for all tasks, and nt denotes the task-specific
). Hence, we can formulate the predictive
distribution for every single episode as follows:
P(T. I:M (c) 5wo, ;wo," y = ply! 1:M [x. . pl:M (p(!!:M (c) c.: ;w v L:M yar!-M
(1)
where p( P(P.:MICI:MM w. v*) denotes the data-dependent functional prior for multiple tasks of the episode т. The functional prior encodes context sets from all heterogeneous tasks and quantifies uncertainty in the functional space. Nevertheless, is less optimal to characterize multi-task function generative processes with vanilla NPs, since the single latent variable limits the capacity of the latent space to specify the complicated functional priors. This expressiveness bottleneck vanilla NPs is particularly severe for our episodic multi-task learning since each episode has diverse heterogeneous tasks with insufficient data.
Modeling within the Hierarchical Bayes Framework. To mitigate the expressiveness bottleneck of vanilla NPs, we model HNPs by parameterizing each task-specific function within hierarchical Bayes framework. As il- lustrated in Figure 2. HNPs integrate a global latent representation z." and a set of local latent parameters w w to model each task-specific function f.m. Specifically, the latent variables are introduced at different levels: z, encodes task-specift context information from C. and T in the representation level. ",,1:0 encode prediction-aware information for a task-specific decoder from CI-M and w in the parameter level, where is the dimension of the decoder. For example, the dimension is the size of the target space when performing classification tasks
C
W1
W
WO
Tt A.
c
w
W 5,0
Th 7.O
y
CM
CT
m
Graphical model of the proposed HÑPs in a single episode. Filled shapes indi- cate observations. Probabilistic and deterministic variables are indicated by unfilled circles and dia- monds, respectively.
Figure 2:
3

---

Notably, each local latent parameter is conditioned on the global latent representation, which controls access all context sets the episode for the corresponding task. Our method differs from previous hierarchical architectures [16. 18 18-20] in the NP family since the local latent parameters of our HNPs are prediction-awar and explicitly constitute a decoder for the subsequent inference processes.
In practice, we assume that distributions of each task-specific function are conditionally independent. Thus, with the introduced hierarchical latent variables for each task in the episode, we can factorize the prior distribution over multiple functions in Eq. (1) into:
M
1:M C, "7,",,, :w,v J:M = II plf
plzz" v pp(w ( 7..00, 1 CllM, w),
m==
where p(zz") CT,; v") and p( wT "..0o"""""" CL:M;,) :w) are prior distributions of the global latent represen- tation and the local latent parameters to induce the task-specific function distribution.
By integrating Eq. (2) into Eq. (I), we rewrite the modeling of HNPs the following form:
P(T M:M c. ; v J:M)
II /
/ Ply" wh..))
(3)
p(w "z,""
CC ;w)dw" Jo(z." |(, ; :""
where p(y"" |x"" 17)) is the function distribution for the task I in HNPs. This distribution is obtained by the matrix multiplication of x and all local latent parameters w WT,1.O
Compared with most NP models [12, 16, 18, 19] employing only latent representations, HNPs infer both latent representations and parameters in the hierarchical architecture from multiple heterogeneous context sets and learnable meta-knowledge. Our model specifies richer and more intricate functional space by leveraging the hierarchical uncertainty inherent in the context sets and meta-knowledge. This theoretically yields more powerful functional priors induce multi-task function distributions.
Moreover, we claim that the developed model constitutes an exchangeable stochastic process and demonstrate this via Kolmogorov Extension Theorem [21]. Please refer to Appendix B for the proof.
Approximate ELBO. Since both exact functional posteriors and priors intractable, we apply variational inference to the proposed HNPs in Eq. (3). This results in the approximate ELBO:
M
THNNoo J.M 0)
F (w (w mo 17... (w) wog ply" |x" w [.:o))
- -DKilqo(w",, T T w) (po o0 c. [w)]] - Dsr, 19o (2," (T""","" (| Ilp
(4)
where go (o(2 1, " and go 1. T.I.M ww) variational posteriors of their corresponding
latent variables. 0 and & parameters of inference modules for z and w w. respectively.
Following the protocol of vanilla NPs [12], the priors use the same inference modules as variational posteriors for tractable optimization. In this way, the KL--ivergence terms in Eq. (4) encourage all latent variables inferred from the context sets to stay close to those inferred from the target sets, enabling effective function generation with few examples. Details the derivation of the approximate ELBO and its tractable optimization are attached in Appendix C.
2.2 Transformer-Strutture Inference Module
In order to infer the prior and variational posterior distributions in Eq. (4), is essential to develop well- designed approximate inference modules. This is non-trivial and closely related to the performance of HNPs. Here we adopt transformer structure the inference module to better exploit task- relatedness from the meta-knowledge and the context sets in the episode. More specifically, the previously mentioned meta-knowledge induce the distributions of hierarchical latent variables in the proposed model.
w1.0 and I:M are instantiated learnable tokens to
Without loss of generality, in the next, we provide an example of transformer-structured inference modules for prior distributions in classification scenarios. Details of the inference modules in regression scenarios can be found in Appendix D. In Figure 3, diagram of the transformer-structtue

---

w
Transformer Inference Module { w1ss}
Clss
Class
Class
Class
Cass
Sampling
Latent Sparse for
Transformer Inference Module [0, [:)
Teak
Figure 3: diagram transtormer--ttructurrd inference modules of HNPS for the first meta- training episode in Figure under the 3-task 5-way 1 1-shot setting, For clarity, we display the inference process of the local latent parameters specific the third task the episode.
inference modules is displayed under the 3-task 5-way 1-shot setting. In this case, the number of context samples the same the size of the target space, and thus have C. { 1 -= where O is set as 5. In episodic training, labels in context sets are always available during inference.
Transformer--truutuuee Inference Module {6, Dー> for z,"" In the proposed HNPs, each global latent representation encodes task-specific information relevant to the considered task in the episode as po(" [C;;; u²)). The learnable token n preserves the enttllnowwedge from previous episodes for specific tasks, which are sampled from the corresponding task distribution p(Z"). The role of u to help the model adapt efficiently to such specific tasks meta-test episodes.
In detail, we set the dimension of the learnable token nm to the same as that of the features T 7,,100* Then the transformer-structure
inference module 0 fuses them in a unified manner by taking [x 11 =** as the input. The module outputs the mean and variance of the corresponding prior distribution. The inference steps for the global latent representation z are:
T
MSA(LN ([SA(L(( (F""):: + ""100:
(5)
11 D
MLP 7,10: (T"l)) 7,100 D
(6)
pe (0 CT vTT) = N( Nlmm T2m))
where Mam MLP(D""), Jzm Sof (MLP(?"")), The transformer-structuredd inference module includes multi-headed self-attention (MSA) and three multi-layer perceptrons (MLP). The layer normalization (LN) is "pre-norm" as done in [22]. Softplus is the activation function to output the appropriate value as the variance of the prior distribution [23].
Seansormer-Structtuee Inference Module {Ø, w1:0) for w ", Likewise, each learnable token wo corresponds a local latent parameter w With the learnable tokens w1:0, we reformulate the prior distribution of local latent parameters as Pa (w 127h (1:M:wr:o) this way, we learn the shared knowledge, inductive biases across all tasks, and their distribution a parameter level, which in practical settings can capture epistemic uncertainty.
To be specific, the prior distribution can be factorized as II (w "","" " C. Cl:M. w.), where all local latent parameters are assumed to be conditionally independent. For each local latent parameter w.... the rransformer-strucuuree inference module takes [x!:MM wo] as input and outputs the corresponding prior distribution, where 11:M are deep features from the same class o in the episode and w, the corresponding learable token. Here the inference steps for with
follows:
I:M [ T.o [o] MSA(LN([Z)))) ;wol)) 1MIM [wo]
(8)
5

---

[ [.o] = MLP(LNN M:M, ; + ( ;0o,
(9)
Po Po(wT, 1,, Cl:M, wo) = -N(w".. T.oi Mww. Twm.
(10)
where Mwr. ML. zz (m)()) Own. = Softplus (MLP((.. 27nh ()) z 2m is Monte Carlo sample
from the variational posterior of the corresponding global latent representation during meta-training.
Both transformer-struccureed inference modules use the refined tokens 7 and 3, to obtain a global latent representation and a local latent parameter, respectively. The introduced tokens preserve the specific mheaa-knowlcde for each latent variable during inference. Compared with the 0-parameterised inference module exploring the intra-task relationships, the d-parameterised inference module enables the exploitation of the inter-task relationships to reason over each local latent parameter. Thus, the introduced tokens can be refined with relevant information from the heterogeneous context sets. By integrating meta-knowledd and heterogeneous context sets, HNPs can reduce the negative transfer of task-specific knowledge among heterogeneous tasks in each episode. Please refer to Appendix for algorithms.
3 Related Work
Multi-Task Learning. Multi-task learning can operate in various settings [9]. Here we roughly separate the settings of MTL into two branches: (1) Single-input multi-output (SIMO) [24-30], where tasks are defined by different supervision information for the same input. (2) Multi-input multi-output (MIMO) [11, 10, 31-34]. where heterogeneous tasks follow different data distributions. This work considers the MIMO setup of multi-task learning with episodic training.
In terms of modeling methods, from a processing perspective, existing MTL methods can be roughly categorized into two groups: (1) Probabilistic MTL methods [11, 19, 35-41), which employ the Bayes framework to characterize probabilistic dependencies among tasks. (2) Deep MTL models [10, 32, 24-26, 42-48], which directly utilize deep neural networks to discover information-sharinn mechanisms across tasks. However, deep MTL models rely on large amounts of training data and tend to overfit when encountering the data-insufficiency problem. Meanwhile, previous probabilistic MTL methods consider small number of tasks that occur at the same time, limiting their applicability in real-world systems.
Meta-Learning. Meta-learning aims to find strategies to quickly adapt to unseen tasks with few examples [49, 4, 5, 50). There exist a couple of branches in meta-learning methods, such as metrics-based methods [6. 51-57] and optimization-bart
methods [5, 58-68]. Our paper focuses on probabilistic meta-learning method, namely neural processes, that can quantify predictive uncertainty. Models this family [7. 12, 13, 15, 16, 18, 69-73] can approximate stochastic processes in neural networks. Vanilla NPs [12] usually encounter the expressiveness bottleneck because their functional priors are not rich enough to generate complicated functions [15, 16). [7] introduces deterministic variables to model predictive distributions for meta-learnin scenarios directly. Most
NP-based methods only focus on a single task during inference [7, 12, 15, 16, 14], which leaves task-relatedness between heterogeneous tasks in a single episode an open problem.
This paper combines multi-task learning and meta-learning paradigms to tackle the data-insufficiency problem. Our work shares the high-level goal of exploiting task-relatedness in an episode with [19, 74, 75]. Concerning the multi-task scenarios, the main differences are: [19, 74, 75] handles multiple attributes and multi-sensor data under the SIMO setting, while our work performs for the MIMO setting where tasks are heterogeneous and distribution shifts exist. Moreover, [76] theoretically addresses the conclusion that MTL methods are powerful and efficient alternatives to gradient-based meta-learning algorithms. However, our method inherits the advantages of multi-task learning and meta-learning: simultaneously generalizing meta- neta--nooledde from past to new episodes and exploiting task-relatedness across heterogeneous tasks in every single episode. Thus, our method is more suitable for solving the data-insufficiency problem. Intuitive comparisons with related paradigms such as cross-domain few-shot learning [77-82], multimodal meta-learning [83-87, 56] and cross-modality few- shot learning [88-90] are provided in Appendix A.

---

Experiments
We evaluate the proposed HNPs and baselines on three benchmark datasets under the episodic multi-task setup. Sec. 4.1 and Sec. 4.2 provide experimental results for regression and classification, respectively. Ablation studies are in Sec. 4.3. More comparisons with recent works on extra datasets are provided in Appendix F. Additional results under the convectional MIMO setup without episodic training can be found in Appendix &
4.1 Episodic Multi-Task Regression
Dataset and Settings. To evaluate the benefit of HNPs over typical NP basclines in uncertainty quan- tification, we conduct experiments in several ID regression tasks. The baselines include conditional neural processes (CNPs [13]), vanilla neural processes (NPs [12[). and attentive neural processes (ANPs [15]. As toy example, we construct multiple tasks with different task distributions: each task's 's imput set is defined on separate intervals without overlap.
Given four different tasks in episode, their input sets are x Each input set contains random few instances, drawn uniformly
Task1
Task2
Task3
from separate intervals, such as x, E -2), 4).
x (-2,0), x² [0, 2). and x
All tasks in an episode are related by shar- ing the same ground truth function. Follow- ing [12, 18], function-fin fittingiinnnnnn tasks are generated with Gaussian processes (GPs). Here zero mean Gaussian process y
CNP
((:),
is used to produce y= for the inputs from all tasks x1:4 A radial basis kernel k(2,x')
o² exp(-(2> 2212)/ with = 0.4 and 1.0 used.
ANPs
Results and Discussions. As shown in Fig- CNPs, ANPs, and our HNPs exhibit more reasonable uncertainty than NPs in Figure 4: lower variances are predicted around observed (context) points with higher variances around unobserved points. Furthermore, NPs and ANPs detrimentally impact the smoothness of the pre- dicted curves, whereas HNPs yield smoother predictive curves with reliable uncertainty esti- mation. These observations suggest that inte- grating correlation information across related tasks and mnea-knoolccder in HNPs can improve
HPP
Figure 4: Performance comparisons the episodic multi-task 1-D function regression us- ing context points (black dots) for each task. Black curves are ground truth, and blue ones are predicted results. The shadow regions are 43 stan- dard derivations from the mean [18].
uncertainty quantification in multi-tas multi-task regression.
To quantify uncertainty we use the average neg- ative log-likelihood (the lower, the better). As shown in Table 1, our HNPs achieve a lower average negative log-likelihood than baselines, demonstrating our method's 's effectiveness in un- certainty estimation.
Table 1: Average negative log-likelihoods over target points from all tasks.
Methods
CNPs
NPs
ANPs
INNPs
Avg. NLL 0.0935
0.8649
-0.1
-0.5207
4.2 Episodic Multi-task Classification
Datasets and Settings. We use Office-Home [91] and DomainNet [92] as episodic multi-task classification datasets. Office-Home contains images from four domains: Artistic (A), Clipart (C), Product (P) and Real-world (R). Each domain contains images from 65 categories collected from office and home environments. Note that all domains share the whole target space. The numbers of meta-training classes and meta-test classes are 40 and 25, respectively. There are about 15, 500 images total. DomainNet has six distinct domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch. It includes approximately 0.6 million images distributed over 345 categories. The
7

---

Table 2: Comparative results (95% confidence interval) for episodic multi-task classification on Office Home and DomainNet. Best results are indicated in bold.
Office Hifice-Home
Domainlet
4-task
5-way
4-taak
20-way
6-taak
5-way
6-taax
20-way
Method
1-shot
G-shot
1-shot
5-shot
1-shot
5-ahot
1-shot
5-shot
ERM [93]
66.04 1a6
73.62 20ss
39.25 1024
47.14 10.1
59.95
20.52
68.52 20.4
38.62 s0m
47.85 10.20
[1T][11]
49.71
65.75 an4
27.50
42.82
42.24
57.37
2043
18.05
2011
31,38 1015
MAML [5]
60.58
75.29 203
34.29 1011
48.39
53.21
65.24
20.47
17.10 20.12
20.35 20.14
Proto. Net. [6]
57.19 aust
74.97 2046
32.72
1018
49.75 1015
53,71 1048
68.80
2044
31.90 20.19
47.59 10
DGPs (94]
65.89 #053
79.96 +0.3
31.48
1018
49.46 #0.18
50.93
#042
63.32
#0.38
25.46 #0.15
38.63 1017
CNPs [13]
43.33 1036
55.07 2000
10.57 10
12.02 2011
37.90
1044
40.53
10.4
5.12 20:10
5.14 2010
NPs [12]
33.66 #044
53.99 #0.0
5.25 #0.16
11.40 +0.11
20.58 #0.51
20.53
=0.53
5.12 =0.00
5.11 #009
TNP-D [95]
65.49 #053
78.94
+:44
41.61
2022
59.19
49.10
67.39
20.40
28.83 10.17
47.69 1018
HNPs
76.29 #0.1
80,80 #0.42
51.82
+:025
59.97
=0.15
62.36 #0.53
69.38 #0.42
39.32
+0.23
48.56 #0.19
numbers of meta-training classes and meta-test classes are 276 and 69, respectively. Here one domain corresponds to a specific task distribution in the episodic multi-task setting.
When it comes to the episodic multi-task classification,
compare HNPs with the following three branches: (1) Multi-task learning methods: ERM [93] directly expands the training set of the current task with samples of related tasks. VMTL [11] is of the state-of-the-an under the MIMO setting of multi-tast learning. (2) Meta-learning methods: MAML [5], Proto.Net Net [6] and DGPs [94] address
each task separately with mechanism leverage task-relatedness in single episode. (3) Methods from the NP family: CNPs [13] and NPs [12] are established methods in the NP family. TNP-D [95] recent NP work in sequential decision-making for a single task in each episode.
Results and Discussions. The experimental results for episodic multi-task classification on Office-Home and DomainNet are reported in Table 2. We use the average accuracy across all task distributions as the evaluation metric. It can be seen that HNPs consistently outperform all baseline methods, demonstrating the effectiveness of HNPs in handling each task with limited data under the episodic multi-task classification setup.
NPs and CNPs do not work well under all episodic multi-task classification cases. This can be attributed to their limited expressiveness of the global representation and the weak capability to extract discriminative information from multiple contexts. In contrast, HNPs explicitly abstract discriminative information for each task the episode with the help of local latent parameters, enhancing the expressiveness of the functional prior.
We also find that HNPs significantly surpass other baselines on 1-shot Office- Home and DomainNet, both under the 4/6- t/sktask 5-way and 4/6-task 20-way settings. This further implies that HNPs can circumvent the effect of the problem of iata-insuuficcencyy by simultancously exploiting task-relatedness across heterogeneous tasks and meta-knowledge among episodes.
4.3 Ablation Studies
Influence of Hierarchical Latent Variables. We first investigate the roles of the global latent representation z,"h and the local latent parameters w by leaving out individual inference modules. These experiments are performed on Office- office-home under the 4-task 5-way 1-shot setting. We report the detailed performance for tasks sampled from single task distribution (A/C/P/R) and the average accuracy across all task distributions (Avg.) in Table 3. The variants without specific latent variables are included in the comparison by removing the corresponding inference modules.
As shown in Table 3, both z and w W benefit overall performance. Our method with hierarchical latent variables performs 9.20% better than the variant without both latent vari- ables, 3.00% better than the variant without z and 5. 18% better than the variant without w 7.1:0- This indicates that latent variables of HNPs comple- ment each other in representing con-
Table 3: Effectiveness of global latent representations z. and local latent parameters wi in the model. / and X denote whether the variants of HNPs have the corre- sponding latent variable not.
2Thh w A
R
Avg.
62.64 2077
56.87 =07)
75.18 1029
73.68 10.77
67.09 20.3
69.39
2050
80.66 #367
79.99
40.2
73.29 #051
67.02 2057
60.70
2009
78.26 +376
78.47 20.22
71.11 20.50
73.31 18as
64.92 .688
83.38 18ss
83.54 18M
76,29 jast
8

---

text sets from multiple tasks and meta- meta--nowledge. The variant without w underperforms the variant without z," by 2.18, in terms of the average accuracy. This demonstrates that z suffers more from the expressiveness bottleneck than "*.10- weakening the models' discriminative ability. For classification, local latent parameters are more crucial than global latent representation in revealing the discriminating knowledge from multiple heterogeneous context sets.
Influence of Transformer--Sructuree Inference Modules. To further understand our transformer- structured inference modules (Trans. learnable tokens), we examine the performance against two other options: inference modules that solely utilize a multi-layer perceptron (MLP) and the variants that do not incorporate any learnable tokens (Trans. w/o learnable tokens). We also compare the probabilistic and deterministic versions of such inference modules. The deterministic variants consider the deterministic embedding for the hierarchical latent variables.
As shown in Table 4, our infer- ence modules consistently outperform the variants, regardless of whether the inference network is probabilis- tic or deterministic. When using the probabilistic one, our inference mod- ules respectively achieve 1.04% and 2.99% performance gains over Trans. w/o learnable tokens and MLP under the 4-task 5-way 1-shot setting. This implies the importance of learn-
Table 4: Performance comparisons between our trans- former inference modules (Trans. w learnable tokens) and other alternatives.
Inference networks
1-shot
5-shot
MLP
64.93 #0.66
72.39 40.56
Deterministic
Trans, w/o learnable tokens Trans, w learnable tokens
70.22 #0.62
76. +0.54
70.61 #0.56
76.70 #0.50
MLP
73.30 =0,99
77.94 #0.48
Probahilistic
Trans, w/o learnable tokens
75.25 =0.55
80.42 +0.47
Trans, w learnable tokens
76.29 =0.51
80.80 10.42
able tokens and task-relatedness in formulating transformer--truccures inference modules, which reduces negative transfer among heterogeneous tasks in each meta-test episode. Moreover, the variants with probabilistic inference modules consistently beat deterministic ones in performance, demonstrating the advantages of considering uncertainty during modeling and inference.
Effects of Different Ways to Gener- ate Local Latent Parameters. We investigate the effects of different ways to generate each w from the shared condition zi"h and Cl:M, Given a Monte Carlo sample of global la- tent variables as (,, in Table 5, we compare with two alternatives:
Table 5: Performance comparisons of different implemen- tations of generating each local latent parameter w from the condition 2. and C.-M
Methods
C
R
Avg.
Concat
65.69 25.69 20.50
58.64 1081
77.54 10.8
77. 10 :0s4
69.74
10.51
10.0
63.73
1071
78.81
107
79.03
1078
72.87
1061
Oura 73.31 19:3
64.92
10.8
83,38 20,66
83,54
76.29 14.51
1) Concat directly concatenates each context feature and (,,(()), and takes the concatenation as inference network Φb 2) Add sums up each context feature and (i)( and takes the result as the input. 3) Ours incorporates 2.7((() into the transformer-structtur inference module by merging it with the refined Iearnable tokens in Eq. (10). As shown in Table 5,
inputs of the transformer-strucct
Ours consistently performs the best. This implies that incorporating the conditional variables into the inference module is more effective than the direct combinations of mi) and instance features.
To investigate the effects of more "shots" or "classes" in the episodic multi-task classification setup, we conduct experiments by increasing K or O in the defined task O-way K-shot setup.
Effects of More "Shots" or "Classes".
As shown in Table 6, the proposed HNPs have more advantages over the baseline method with the context data points below ten shots. With shots larger than ten, both methods will reach a performance bottleneck.
Table 6: Performance comparisons on Office-Home un- der the 4-task 5- 5-way K-shot setup.
Methods
1-shot
5-shot
10-shot
20-shot
TNP-D
65.49 20.53
78.94 10.43
80.81 10.3
81.12 20.68
HNPs
76.29 #0.51
80.80 40.4
81.28 10.38
81.56 20.36
Moreover, Table 7 shows that our method consistently outperforms the baseline method as the number of classes increases from 20 to 40 in step 5. However, the performance gap between them narrows slightly with more classes. The main reason could be that the setting with more classes suffers from less data insufficiency.

---

Table 7: Performance comparisons DomainNet under the 6-task O-way
1-shot setup.
Methods
5-way
20-way
25-way
30-way
35-way
40-way
TNP-D
49.10 1042
28.83 10.17
25.93 10.14
24.08 10.12
22.62
10.11
21.64 10.53
HNPs 62.36 10.53
39.32
10.23
35.72 10.19
32.27 10.17
31.27
10.14
29.31
10.13
Average socurasy((4)
Runtimet
Sensitivity to the Number Monte Carlo Samples. For the hierarchi- cal latent variables the HNPs, we investigate the model's sensitivity to the number of Monte Carlo samples. Specifically, the sampling number of the global latent representation z and local latent parameters 7,1:0 varies from to 30. We examine Office-Home under the 4-task 5-way 1-shot setting. In Figure 5, the runtime per iteration grows rapidly as the number of samples increases. However, there no clear correlation between the performance and the num- ber of Monte Carlo samples. There are two sweet spots in terms of average accuracy, of which has favorable computation time. Hence, we set N₂ and N to and 10, respectively.
26.72
76.08
36
20
53
N, 15
Ng
15
0.30
75.42
0.2
75.26
75.00
021
26:44
Figure Average accuracy and runtime of HNPs with different numbers of Monte Carlo samples. N₂ and Nu. are sampling numbers of z," and w 7,1:0, respectively.
We also investigate the inference time of NP-based models per iter- ation Office-Home under the 4task5wayl Isot setup. As shown in Table 8, our model needs more infer- ence time than other NP-based meth- ods for performance gains. The cost mainly comes from inferring the designed hierarchical latent variables; however, we consider this worthwhile trade-off for the extra performance.
Table 8: Inference time of different NP-based methods.
Methods
CNPs
NPs
TNP-D
HNPs
Inference time(s)
0.04
0.05
0.08
0.15
Conclusion
Technical Discussion. This work develops heterogeneous neural processes by introducing hierar- chical latent variables and transformer-structured inference modules for episodic multi-task learning. With the help of heterogeneous context information and meta- knowledge, the proposed model can exploit task-relatedness, reason about predictive function distributions, and efficiently distill past knowledge unseen heterogeneous tasks with limited data.
Limitation & Extension. Although the hierarchical probabilistic framework could mitigate the expressiveness bottleneck, the model needs more inference time than other NP-based methods for performance gains. Besides, the proposed method requires the target space to be the same across all tasks single episode. This requirement could limit the method's applicability in realistic scenarios where target spaces may differ across tasks. Our work could be extended to the new case without the shared target spaces, where the model should construct higher-order fask-relatedness improve knowledge sharing among tasks. Our code is provided to facilitate such extensions.
Acknowledgment
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
10

---

References
[1] lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. //www. wwwppeerninggboo..... org.
MIT Press, 2016. http:
[2] Shichao Xu, Lixu Wang, Yixuan Wang, and Qi Zhu. Weak adaptation learning: Addressing cross-domain data insufficiency with weak annotator. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[3] Justin M Johnson and Taghi M Khoshgoftaar. Survey deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
Oriol Vinyals. Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Procexsing Systems, 2016.
[5] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fäst adaptation of deep networks. International Conference on Machine Learning, 2017.
[6] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for fow-shot learning. In Advances in Neural Information Processing Systems, 2017.
[7] James Requeima, Jonathan Gordon, John Bronskill, Schastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances Neural Information Processing Systems, 2019.
[8] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75,11 1997.
[9] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
[10] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multilinear relationship networks. In Advances in neural information processing systems, 2017.
[11] Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel- softmax priors. In Advances in Neural Information Processing Systems, 2021.
[12] Marta Gamelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 187.0:880. 2018.
[13] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, 2018.
[14] Wessel Bruinsma, Stratis Markou, James Requeima, Andrew Y. K. Foong, Tom Andersson, Anna Vaughan, Anthony Buonomo, Scott Hosking, and Richard E Tumner Autoregressive conditional neural processes. In International Conference on Learning Representations, 2023.
[15] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yoe Whye Tch. Attentive neural processes. International Conference on Learning Representations, 2019.
[16] Qi Wang and Herke van Hoof. Learning expressive meta-represenntttions with mixture of expert neural processes. In Advances in Neural Information Processing Systems, 2022.
[17] Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations, 2017.
[18] Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. International Conference on Machine Learning. 2020.
[19] Donggyun Kim, Seongwoong Cho. Wonkwang Lee, and Seunghoon Hong. Multi-task processes. arXiv preprint arXiv:21 10.1453,, 2021.
[20] Zongyu Guo, Cuiling Lan. Zhizheng Zhang, Yan Lu, and Zhibo Chen. Versatile neural processes for learning implicit neural representations. International Conference on Learning Representations, 2023.
[21] Achim Klenke. Probability theory: comprehensive course. Springer Science & Business Media, 2013.
[22] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-am aad--andd-aaaaa language transformer without convolution or region supervision. In International Conference Machine Learning, 2021.
11

---

[23] Casper Kaae Sønderby, Tapani Raiko, Lars Maalde. Spren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems, 2016.
[24] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[25] Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In /EEE Conference on Computer Vision and Pattern Recognition, 2019.
[26] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[27] Liyang Liu, Li,, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2020.
[28] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, 2018
[29] Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[30] Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk, and Mathieu Salzmann. Mult: an end-to-end multitask learning transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[31] Yi Zhang, Yu Zhang, and Wei Wang. Multi-task learning via generalized tensor trace norm. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
[32] Jiayi Shen, Zehao Xiao, Xiantong Zhen. Cees GM Snoek, and Marcel Worring. Association graph learning for multi-task classification with category shifts. arXiv preprint arXiv:2210.04633 2022.
[33] Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. Scheduled multi-task lcarning for neural chat translation. arXiv preprint arXiv:2205.03777 2022.
[34] Yi Zhang, Yu Zhang, and Wei Wang. Learning linear and nonlinear low-rank structure in multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2022.
[35] Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research, 2003.
[36] Kai Yu, Volker Tresp, and Anton Schwaighofer. Learning gaussian processes from multiple tasks. In International Conference on Machine Learning. 2005.
[37] Michalis K Titsias and Miguel Lázaro-Gredilla Spike and slab variational inference for multi-task and multiple kernel learning. In Advances in neural information processing systems, 2011.
[38] Neil D Lawrence and John C Platt. Learning to learn with the informative vector machine, In International Conference on Machine Learning, 2004.
[39] Fariba Yousefi, Michael Thomas Smith, and Mauricio A Álvarez. Multi-task learning for aggregated data using gaussian processes. arXiv preprint arXiv: (wt.i.906999919 2019.
[40] Diane Oyen and Terran Lane. Leveraging domain knowledge in multitask bayesian network structure learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2012.
[41] Weizhu Qian, Bowei Chen, Yichao Zhang, Guanghui Wen, and Franck Gechter. Multi-task variational information bottleneck. arXiv preprint arXiv:2007.000 20033999..00....... 2020.
[42] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Learning task relatedness in multi-task learning for images in context. In International Conference on Multimedia Retrieval, 2019.
[43] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. arXiv preprint 1911. 2019.
[44] Gjorgji Strezoski. Nanne van Noord, and Marcel Worring. Many task learning with task routing. In IEEE International Conference Computer Vision, 2019.
12

---

[45] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman. and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.0677 2020.
[46] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conffict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, 2021.
[47] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems, 2021.
[48] Pengsheng Guo, Chen- Yu Lee. and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, 2020.
[49] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview, In Learning to learn, pages 3- 17. Springer. 1998.
[50] Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[51] Kelsey R Allen, Evan Shelhamer, Hanul Shin, and Joshua B Tenenhaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, 2019.
[52] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: improved few-shot learning. In Advances in neural information processing systems, 2018.
Task dependent adaptive metric for
[53] Sung Whan Yoon, Jun Seo, and Jackyun Moon. Tapnet: Neural network augmented with task ada-tiee projection for few-shot learning. In International Conference on Machine Learning, 2019.
[54] Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
[55] Tianshi Cao, Mare Law. and Sanja Fidler. theoretical analysis of the number shots few-shot learning. arXiv preprint 1rXiv:1909.111 11722, 2019.
[56] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.0 03096, 2019.
[57] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[58] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-lcarning algorithms. arXiv preprint arXiv: 1803.0299, 2018.
[59] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta Deaannn with latent embedding optimization. In International Conference on Learning Representations, 2019.
[60] Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: 106.00/776609999, 2017.
[61] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.0000 09835, 2017.
[62] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Representations, 2019.
[63] Harrison Edwards and Amos Storkey. Towards neural statistician. arXiv preprint arXiv.16060021855. 2016.
[64] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018
[65] Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforccment learning with latent variable gaussian processes. arXiv preprint arXiv: 1803.07551, 2018.
[66] Sungyong Baik, Junghoon Oh, Seokil Hong, and Kyoung Mu Lee. Learning to forget for meta-learning via lask-and- -nd--lyyer-wie attenuation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
13

---

[67] Myungsub Choi. Janghoon Choi, Sungyong Baik, Tae Hyun Kim, and Kyoung Mu Lee. Test-time adaptation for video frame interpolation via meta-learning IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[68] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint urXiv:190900917, 2019.
[69] Ying Wei, Peilin Zhao, and Junzhou Huang. Meta-learning hyperparameter performance prediction with neural processes. In International Conference on Machine Learning, 2021.
[70] Stratis Markou, James Requeima, Wessel P Bruinsma, Anna Vaughan, and Richard E Turner. Practical conditional neural processes via tractable dependent predictions. arXiv preprint arXiv:2203.0877 2022.
[71] Zesheng Ye and Lina Yao. Contrastive conditional neural processes. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[72] Mingyu Kim, Kycongryeol Go, and Se- Young Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. arXiv preprint (rXXi:2204.05449) 2022.
[73] Qi Wang. Marco Federici, and Herke van Hoof. Bridge the inference gaps of neural processes via expectation maximization. In International Conference on Learning Representations, 2023.
[74] Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. Efficient and effective multi-task grouping via meta learning on task combinations. Advances Neural Information Processing Systems, 2022.
[75] Richa Upadhyay, Prakash Chandra Chhipa, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Multi- task meta learning: learn how to adapt to unseen tasks. In 2023 International Joint Conference on Neural Networks (JCCN). IEEE, 2023.
[76] Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-lcaming: Towards efficient training and effective adaptation. In International Conference Machine Learning, 2021.
[77] Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Bin Huang, and Ming-Hsuar Yang. Cross-domain sho-shot classifica- tion via learned feature-wise transformation. In International Conference on Learning Representations, 2020.
[78] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. arXiv preprint arXiv:1904.04233 0422322, 2019.
[79] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In European Conference on Computer Vision, 2020.
[80] Yingjun Du, Xiantong Zhen, Ling Shao, and Cees G M Snock. MetaNomm: Leamnngg to normalize few-shot batches across domains. In International Conference on Learning Representations, 2021.
[81] Debasmit Das, Sungrack Yun, and Fatih Porikli. Confess: A framework for single source cross-domain few-shot learning. In International Conference on Learning Representations, 2022.
[82] Li, Xialei Liu, and Hakan Bilen. mooss-domain few-shot learning with task specific adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7161-7170, 2022.
[83] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- meta-lcarnin via task-aware modulation. In Advances in neural information processing systems, 2019.
[84] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph Lim. Toward multimodal model-agnostic meta- lcarning. In arXiv preprint arXiv: 17,, 2018
[85] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hicrarchically structured meta- meta-learning In International Conference on Machine Learning, 2019.
[86] Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung. Revisit multimodal meta-learning through the lens of multi-task lcarning. In Advances in Neural Information Processing Systems, 2021.
[87] Jiayi Chen and Aidong Zhang. Hetmaml: Task hesk-heteroggneous model-agnostic meta-learning for few-shot learning across modalities. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021.
14

---

[88] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O Pinheiro. Adaptive cross-modal few-shot learning. In Advances in Neural Information Processing Systems, 2019.
[89] Frederik Pahde, Patrick Jähnichen, Tassilo Klein, and Moin Nabi. Cross-modal hallucination for few-shot line-grained recognition. arXiv preprint arXiv:1806.05144 001414000..117 2018.
[90] Frederik Pahde. Mihai Puscas, Tassilo Klein, and Moin Nabi. Multimodal prototypical networks for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications Computer Vision, 2021.
[91] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[92] Xingchao Peng. Qinxun Bai, Xide Xia. Zijun Huang, Kate Saenko, Bo Wang. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision, 2019.
[93] Ishaan Gulrajani David Lopez-Paz. search lost domain generalization. arXiv preprint uxXiv.200.004334 2020.
[94] Wang. Zichen Miao, Xiantong Zhen, and Qiang Qiu. Learning to learn dense gaussian processes for few-shot learning. In Advances in Neural Information Processing Systems, 2021.
[95] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179, 2022.
15


Reviews:
[{"review_id": "mHRm4DTr3m", "reviewer": "Reviewer_n4KJ", "summary": "The paper introduces a meta-learning paradigm to perform few-shot multi-task learning. It introduces a local latent representation $\\omega$, which effectively combines heterogeneous information from various tasks. Additionally, the paper suggests using transformer-structured inference modules to infer both hierarchical latents for task-relatedness and learnable tokens as meta-knowledge. The experimental results demonstrate that the proposed method achieves good performance in the few-shot multi-domain image classification task.", "questions": "- more thorough comparison and experiment discussion as stated in the weakness section. \n- can the proposed method work when target space across tasks differs from each other?\n- What exactly does meta-knowledge $\\omega = \\omega_{1:O}$ learn? \n", "limitations": "One limitation that can be highlighted is that the proposed method requires the target space to be strictly the same across all tasks during meta-training. This requirement could potentially limit the applicability of the method in realistic scenarios where taxonomies may differ across tasks. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- the problem formulation and the proposed method is novel to the best of my knowledge.\n- the proposed meta knowledge and task-specific latent representation are well motivated.\n- the method is proved to be effective in the few shot multi-domain image classification task.", "weaknesses": "- Minor improvement in 5-shot setting: while the proposed method shows promising improvement on 1-shot setting across the two datasets. The performance gap diminishes quickly when scale to 5-shot setting (less than 2% improvement v.s. TNP-D). It is questionable whether the method can maintain the superiority when number of unseen samples is further increased.\n- Comparison Fairness: In the classification experiment, the paper evaluates the effectiveness of meta-learning methods (e.g., MAML) by treating each task separately. A more fair baseline implementation would merge all the tasks into a single task and provide the same amount of meta-training/meta-testing data. For instance, in a 4-task 20-way 1-shot experiment setup, baseline meta-learning methods should have a 20-way 4-shot setup.\n- Evaluation Metrics: The claim regarding the regression task and the smoother predictive curves with reliable uncertainty estimation from HNPs lacks quantitative support and comes across as vague. Furthermore, the experimental setup from this single example raises doubts about whether it was cherry-picked.\n- Clarification: The caption of Figure 1 stating, \"different colors denote different categories\" is not clear. Instead, different shades should be associated with class categories, while different colors (yellow, blue, red, etc.) should correspond to various training examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer n4KJ for the positive review and thorough feedback. We are glad that the review found that our method is novel, the proposed meta-knowledge and latent representation are well motivated.*\n\n---\n\n\n**1. Performance with more “shot”.**\n\nThanks for this question. We’ve performed additional experiments on the Office-home dataset by increasing the number of context samples. The results and discussions are:\n| Methods |    1-shot   |    5-shot   |   10-shot   |   20-shot   |\n|--------:|:-----------:|:-----------:|:-----------:|:-----------:|\n|    TNPs | 65.49 ±0.53 | 78.94 ±0.43 | 80.81 ±0.32 | 81.12 ±0.68 |\n|    Ours | 76.29 ±0.51 | 80.80 ±0.42 | 81.28 ±0.38 | 81.56 ±0.36 |\n\nWe observe that the proposed method has more advantage over TNPs with the context data points below ten shots. It seems that with shots greater than ten, both methods will reach a performance bottleneck in fast adaptation.\n\n\n**2. New evaluation in classification.**\n\nThanks for this suggestion. We conducted experiments with the requested 20-way 4-shot setup, which needs to mix samples from all tasks in one episode.\n\n|       Methods    |     4task20way1shot    |       20way4shot     |\n|-----------------:|:----------------------:|:--------------------:|\n|          MAML    |      34.29   ±0.19     |     37.23   ±0.25    |\n|     Proto.Net    |      32.72   ±0.18     |     37.12   ±0.22    |\n|          Ours    |      51.82   ±0.23     |          -           |\n\nAs shown in the table, we find that MAML and Proto.Net perform better under the 20-way 4-shot but cannot outperform our method. The main reason is that our method can better handle distribution shifts among tasks by exploring task-relatedness rather than simply mixing them together.\n\n\n**3. Uncertainty quantification in the regression.**\n\nWe’ve reported the negative loglikelihood (NLL) results in uncertainty quantification of regression tasks:\n\n| Methods  | CNPs   | NPs    | ANPs    | Ours    |\n|----------|--------|--------|---------|---------|\n| Avg. NLL | 0.0935 | 0.8649 | -0.1165 | -0.5207 |\n\nAs shown in the table, the proposed method obtains a lower average NLL, quantitatively outperforming other baselines in the toy-example experiment. \n\n\n**4. Refining Figure 1.**\n\nThank you for this suggestion. Different colors (yellow, blue, red, etc.) represent different label spaces among episodes; the same color with different shades (adding different percentages of black) represents different categories in the same task. We’ll polish Figure 1 in the final version.\n\n\n**5. Other questions.**\n\n* When the target space differs from each other, can the proposed method work?\n\n    Thanks for your insightful question.  This constraint of the same target space is indeed the limitation of our work, and HNPs could work under certain conditions. Our model is to explore task-relatedness across tasks in a single episode and considers this as a new kind of meta-knowledge. The shared target space is a simple way to ensure positive transfer among related tasks.\n\n    However, the shared target space could not be the only condition to yield task-relatedness. If tasks have higher-level semantic relatedness, our work could be extended to the new case without the shared target spaces. But this requires more modifications to encode the target space, so we’ll add this discussion to the limitation section in Line 374 in the final version.\n\n* What does meta-knowledge $\\omega=\\omega_{1:O}$ learn?\n\n    Thank you for this question. The $\\omega=\\omega_{1:O}$ learns the shared knowledge and inductive biases across all tasks, and its distribution at a parameter level can capture epistemic uncertainty in practice. \n\n---\n\n*We will add these collected results to the final version of the paper. We hope that this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need any further clarifications. Thank you.*\n"}, {"review_id": "PfdtfYk3ZQ", "reviewer": "Reviewer_YPdL", "summary": "The paper proposes an NP-based approach to episodic multi-task learning called HNPs that uses transformer-based hierarchically arranged inference modules to model class-specific distributions. As a result, the predictions are obtained by a dot product of the input features and the sampled class-specific latent variables, thus making the latent variables prediction-aware.", "questions": "- Further on the use of priors, NPs are known to suffer from posterior collapse, and in classification settings, it could be the case that the main classification loss (cross-entropy, etc.) dominates over the posterior approximation loss (KL div) to the extent where the model could perform equally well without the latter loss. A comparison with and without the KL losses could be useful here.\n- A standard framework for NP uses a concatenation of inputs x with outputs y to be fed into the NP layers. However, the HNP does not seem to be using such concatenated features. Can the authors clarify if they have any observations with and without such concatenated features for generative modeling?\n- Can the authors clarify further how they make use of the context set at train time vs test time?", "limitations": "Overall, this paper presents a valuable contribution to the field of episodic multi-task learning. However, the current empirical results may limit its impact and applicability in real-world scenarios. Therefore, the authors may want to elaborate further on their empirical evaluations,  include more recent methods for comparison, discuss their saturating performance with more number of classes, or alternatively, provide a clear explanation for this shortcoming.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on the setting of episodic multi-task learning and proposes Heterogenous Neural Processes (HNPs) -- a new member to the NP family for classification and regression tasks.\n- The authors provide a well-motivated and well-explained method that emphasizes the potential of HNPs. A good highlight of their method remains a two-layer cross-task and intra-task arrangement of latent variables where the latter variables are prediction-aware. \n- The authors adapt transformer encoder layers as inference modules for HNPs which is a good alternative to traditional MLP-based NP layers that lack ample inductive biases, particularly for classification tasks.\n- The experiments and ablations look quite diverse ranging from episodic multi-task regression to classification settings.", "weaknesses": "- While the NP-based methods in Table 1 sound fairly recent, the methods used from multi-task learning and meta-learning branches remain fundamental. Can the authors look into comparing their method with more up-to-date methods from these two branches?\n- The main table of results compares HNPs with a range of other NPs with and without transformer-based inference modules. Besides Figure 5, it would also be valuable for the authors to provide a comparison of the computational and/or runtime cost at inference for these models.\n- It would be valuable to clarify further the working of the inference modules under a more generic classification setting. E.g., does the episodic multi-task learning setup guarantee that each training batch consists of samples from all possible classes? In particular, if a minibatch has samples from a few classes missing, how does the HNP handle this given that the prediction-aware transformer encoder layer (possibly) expects samples from classes to be arranged as a sequence?\n- Can the authors comment on the saturating gap between the performance of HNPs and TNP-D in settings with a larger number of classes? It would be interesting to see an analysis of the performance comparison between the two on settings with more number of classes.\n- While the authors show that the probabilistic version of the HNP outperforms its deterministic counterpart (Table 3), it would be valuable to mention how well the probabilistic HNP makes use of the prior. In particular, how the two KL values in eq. (4) vary with training, do they approach to nearly zero in the latter training stages? And if so, is the model carefully considering the prior?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer YPdL for the positive review and constructive feedback. We are glad that the review found that our method is well-motivated and well-explained.* \n\n---\n\n**1. More experimental comparison and discussions.**\n\nWe’ve reimplemented recent multi-task and meta-learning methods, such as, MTL-bridge [a] and MLTI [b] on Office-home under the 4way5way1shot and 4way5way5shot setups as follows:\n\n| Methods      |  1-shot      | 5-shot      |\n|--------------:|:--------------:|:-------------:|\n| MTL-bridge [a]| 64.31 ± 0.55 | 75.10 ±0.51 |             \n| MLTI [b]      | 70.69 ± 0.73 | 79.59 ±0.58 |\n| Ours         | 76.29 ± 0.51 | 80.80 ±0.42 |\n\nFrom this table, we find that our method significantly outperforms other baselines with severely insufficient data, such as 1-shot. This is consistent with the conclusion obtained in the main paper. \n\n**2. Runtime cost in the inference of models.**\n\nWe’ll include comparisons on runtime cost per iteration of models as follows:\n\n| Methods        | CNPs | NPs  | TNPs | Ours |\n|----------------:|:------:|:------:|:------:|:-------:|\n| Inference time(s) | 0.04 | 0.05 | 0.08 | 0.15  |\n\nFrom this table, we can see our model needs a bit more inference time than other NP-based methods for performance gains. The cost mainly comes from inferring the designed hierarchical latent variables; however, we consider this a worthwhile trade-off for the extra performance. We will add this limitation to our conclusion section.\n\n**3. Each training batch consists of all possible classes?**\n\nThanks for this question.\n\nYes, we indeed include all possible classes in a single episode during the training and testing times. Under the episodic setup, e.g., O-way K-shot, the inference module works on the task level rather than the sample level. Thus, the module is fed with data from a whole task, where the context set contains samples from all O classes.\n\nMoreover, when samples from a few classes are missing, we can also provide an alternative solution to this scenario. As shown in the formulation of our model, latent variables are inferred from the corresponding samples and learnable tokens. Without the samples, the transformer encode layer only takes the learnable tokens as input and outputs the distribution of the corresponding latent variables.\n\n**4. Discussion on the gap between HNPs and TNP-D’s performance with a larger number of classes.**\n\nWe’ve performed the experiment on the DomainNet dataset and reported the result as follows:\n\n|     Methods    |         5way       |        20 way      |        25way       |        30way       |        35way       |        40way       |\n|---------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|\n|        TNPs    |     49.10 ±0.42    |     28.83 ±0.17    |     25.93 ±0.14    |     24.08 ±0.12    |     22.62 ±0.11    |     21.64 ±0.10    |\n|        HNPs |     62.36 ±0.53    |     39.32 ±0.23    |     35.72 ±0.19    |     32.27 ±0.17    |     31.27 ±0.14    |     29.31 ±0.13    |\n\nFrom this table, we find that our method consistently outperforms TNPs as the number of classes increases from 20 to 40 in step 5. The performance gap between them narrows slightly. The main reason could be that the setting with more classes suffers from less data insufficiency. \n\n**5. Role of probabilistic HNPs, KL values in meta training, and without KL scenarios.**\n\n- The probabilistic HNPs encode the context as the heterogeneous prior and reveal the uncertainty resulting from data insufficiency and the extent of observations in tasks. Additionally, minimizing KL terms encourages priors inferred from context sets to stay close to posteriors inferred from target sets, guiding more efficient conditional generation.\n \n- In training processes, we observed the KL divergence value does not decrease to 0 after convergence, e.g., KL values in the scale of e-1 on Office-home. This is also part of traits in NPs family, suggesting the approximate posterior and the approximate prior encode different conditional information during the generation of latent variables.\n\n- Note that our primary contribution is a new variant of the stochastic NP model. Without the constraint of the KL divergence, this no longer constitutes the stochastic NP model, and we cannot find the appropriate stochastic NP baseline to compare. Instead, we consider the CNP as the appropriate baseline without the KL divergence in the discussion and results are already reported in Table 1 of the main paper.\n\n**6. Concatenation of $x$ and $y$ and observations.**\n\nThanks for this insightful question. For various benchmarks, there are different operations w.r.t. the couple of $x$ and $y$ in our model. In regression scenarios, we concatenate the x and y as the input of the inference layers, as done in vanilla NPs. In classification scenarios, we exploit the label information to infer the local latent variables by incorporating the prediction-aware token and context samples from the corresponding classes. In practice, this operation helps us to generate more discriminative local latent parameters. \n\n**7. Use of the context set at train vs. test time.**\n\nThanks for this question. During the meta-test, we make predictions with priors inferred from the context set. This is the same as that in meta-training. Also, note that target sets are unseen to infer the corresponding posteriors in the meta-test. \n\n**References:**\n\n[[a] Wang, Haoxiang, Han Zhao, and Bo Li. \"Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation.\" ICML2021.](https://arxiv.org/abs/2106.09017)\n\n[[b] Yao, Huaxiu, Linjun Zhang, and Chelsea Finn. \"Meta-learning with fewer tasks through task interpolation.\" ICLR2022.](https://arxiv.org/abs/2106.02695)\n\n---\n\n*We hope this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need further clarification. Thank you.*\n"}, {"review_id": "hLOtZBL8As", "reviewer": "Reviewer_21Dc", "summary": "Learning multiple tasks with limited data poses a significant challenge. Addressing this issue, the paper proposes a novel approach that leverages heterogeneous information across tasks by combining the benefits of multi-task learning and meta-learning. This is achieved through the introduction of hierarchical neural processes (HNPs) within a hierarchical Bayes framework. To approximate the intractable integral in the posterior, variational inference is employed, facilitated by a transformer inference module. The performance of the proposed HNPs is thoroughly evaluated on both regression and classification tasks. The results demonstrate improved performance compared to typical baselines, highlighting the efficacy of the approach. The integration of multi-task learning and meta-learning through the HNPs framework showcases its potential for addressing the challenges associated with learning from limited data across multiple tasks.", "questions": "* Why have “Multi-Task Processes” not been included in baselines? As the authors have\nmentioned, it is highly relevant to this work.\n*  Authors have claimed how the local and global representations capture task-specific functions but have not properly supported this claim experimentally other than showing performance gain by including the representations.\n* Line 260 states that \"... reliable uncertainty estimation ...\", can the authors be more specific about \"reliable\"?\n* The datasets used in classification tasks contain thousands of images, the reviewer was wondering how HNPs exactly handle the data-insufficiency issue. \n* How about the time complexity of HNPs? Can the authors report the running time for example?\n", "limitations": "The authors have briefly discussed the limitation in Section 5. As mentioned in the Weakness session, the requirement of shared target space can be one potential limitation of the proposed method. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The concept of leveraging meta-learning and multi-task learning to enhance neural processes is both intriguing and well-grounded. The development of hierarchical neural processes (HNPs) is supported by established theoretical frameworks such as existing neural processes, hierarchical bases, and the natural alignment with variational inference.\n* The incorporation of local and global latent representations to capture and share task-specific information is a logical choice, aligning well with the Bayesian interpretation of meta-learning. To facilitate approximate inference, a transformer-based inference module is employed, allowing for the fusion of meta-knowledge and information from various tasks.\n* The experimental results showcase the adaptability of HNPs in both regression and classification tasks. Additionally, the conducted ablation studies shed light on the specific components of the models that influence the overall model performance, providing valuable insights into the model's behavior.", "weaknesses": "* While the utilization of neural processes in the proposed framework is intriguing, it is worth noting that the concept of combining multi-task learning and meta-learning is not novel. To provide a comprehensive understanding, it would be valuable for the authors to explicitly highlight the differentiating factors of their proposed work compared to existing approaches, as referenced in [1]-[3]. Furthermore, including these existing methods in the experimental comparisons would enhance the thoroughness and relevance of the evaluation. This would enable a more comprehensive analysis and facilitate a clearer understanding of the unique contributions and advantages of the proposed framework.\n* The proposed framework imposes a constraint that the different tasks must share the same target space. This requirement significantly restricts the applicability of the method and narrows its scope. From the reviewers point of view, the framework aligns more closely with the domains of domain adaptation or multimodal learning rather than traditional multi-task learning. Acknowledging and discussing this limitation would provide a clearer understanding of the proposed method's potential use cases and its relationship with related research areas.\n* References:\n  * [1] “Multi-Task Meta Learning: learn how to adapt to unseen tasks”\n  * [2] “Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation”\n  * [3] “Efficient and Effective Multi-task Grouping via Meta Learning on Task Combinations”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate Reviewer 21Dc the positive review and helpful feedback. We are glad that the review found that leveraging meta-learning and multi-task learning is intriguing and well-grounded.* \n\n---\n\n\n**1. Related works [1-3] and MTPs.**\n\nThanks for this suggestion. \n\nWe highlight the differences between these related works and our work as follows: \n\n- [1] and MTPs are proposed for the single-input multi-output setting of multi-task learning in every single episode, where several tasks share the same input. However, in our work, episodic multi-task learning is designed based on the multi-input multi-output setting, where each episode suffers from distribution shifts between heterogeneous tasks, and each task has its individual input.\n\n- [2] theoretically addresses the conclusion that MTL methods are powerful and efficient alternatives to GBML for meta-learning applications. However, our method inherits the advantages of multi-task learning and meta-learning: simultaneously generalizing meta-knowledge from past to new episodes and exploiting task-relatedness across heterogeneous tasks in every single episode. Thus, our method is more suitable for solving the data-insufficiency problem.\n\n- [3] casts multi-task grouping as a meta-learning problem and aims to generalize the grouping ability from meta-train with a small number of combinations to meta-test with all the rest combinations. In [3], meta-train and meta-test share a fixed set of tasks. However, our work introduces episodic training to multi-task learning, where meta-test tasks differ from meta-train tasks. Thus, our method is more suitable to fast adapt the ability of learning task-relatedness to unseen tasks. \n\nWe also provide experimental results of [2] on Office-home under the 4task5wayXshot setup.  Our method significantly outperforms [2] with severely insufficient data, such as 1-shot. \n\n| Methods      |  1-shot      | 5-shot      |\n|--------------|--------------|-------------|\n| MTL-bridge [2]  | 64.31 ± 0.55 | 75.10 ±0.51 |             \n| Ours         | 76.29 ± 0.51 | 80.80 ±0.42 |\n\nDue to differences in setups between [1,3] MTP and our method, it is intractable to find standard implementations under our episodic multi-task setup. However, these interesting papers support our work that it is reasonable to meta-learn the ability of exploring task-relatedness by combining meta-learning and multi-task learning. We’ll cite these related papers and add discussions in the Line198 Related Work.\n\n**2. The same target space shared across different tasks.**\n\nThis constraint is indeed the limitation of our work, and HNPs work under certain conditions. The shared target space is a simple way to ensure positive transfer among related tasks. However, the shared target space could not be the only condition to yield task-relatedness. If tasks have higher-level semantic relatedness, our work could be extended to the new case without the shared target spaces. But this requires more modifications to encode the target space.  We’ll add this discussion to the limitation section in Line 374.\n\n**3. More discussions on the local and global representations.** \n\n- In a theoretical sense, a hierarchical Bayesian framework with global and local latent variables yields a more rich and complex latent space to mitigate the expressiveness bottleneck, thus better parameterizing task-specific functions in stochastic processes.\n\n- In an empirical sense, global and local latent variables capture epistemic uncertainty in representation and parameter levels, respectively, and show improved performance in Table 2 and Figure 5 of the main paper. \n\n**4. Reliable uncertainty estimation in the regression of Line 260.**\n\nTo show the reliability of our model, we’ve reported the negative loglikelihood (NLL) results in uncertainty quantification of regression tasks. The lower, the better.\n\n| Methods  | CNPs   | NPs    | ANPs    | Ours    |\n|----------|--------|--------|---------|---------|\n| Avg. NLL | 0.0935 | 0.8649 | -0.1165 | -0.5207 |\n\nThe table shows that the proposed method obtains a lower average NLL, quantitatively outperforming other baselines in the toy-example experiment. We’ll add this discussion to the main paper.\n\n**5. Handling the data insufficiency issue.**\n\nFor episodic multi-task classification, we feed the data in a meta-learning manner where each task suffers from the data-insufficiency problem. In detail, we define O-way K-shot as the context set of a single task, which first randomly samples O categories from the given label space and randomly samples K images for each individual category. \n\nTo address this data insufficiency problem of the task, we propose to fully utilize the positive transfer from its related tasks in the same episode and reuse experiences from earlier episodes. Experimental results show that our model combining the advantages of both multi-task learning and meta-learning alleviates the data-insufficiency problem.\n\n\n**6. Runtime cost in the inference of models.**\n\nThanks for this suggestion. The time complexity positively correlates with the number of Monte Carlo samples of latent variables. As shown in the right part of Figure 5, the runtime per iteration grows rapidly as the number of samples increases. For interests, we also compare the runtime cost of other NP-based methods with that of our model as follows:\n\n| Methods        | CNPs | NPs  | TNPs | Ours  |\n|----------------|------|------|------|-------|\n| inference time(s) | 0.04 | 0.05 | 0.08 | 0.15  |\n\nThis table shows our model needs a bit more inference time than other NP-based methods for performance gains. The cost mainly comes from inferring the designed hierarchical latent variables, however, we consider this a worthwhile trade-off for the extra performance. We will add this limitation to our conclusion section.\n\n---\n\n*We hope this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need further clarification. Thank you.*\n"}, {"review_id": "IKTQr3ybNF", "reviewer": "Reviewer_LXqe", "summary": "The paper tackles the problem of learning tasks with insufficient data within the context of multi-task meta-learning. Specifically, the method proposed, namely Heterogenous Neural Processes (HNPs), is designed to leverage information amongst tasks in a single episode to improve performance while retaining learning from previous episodes during episodic training. The model leverages the function distributions task-wise prior to modeling each task in the episode while using a global prior and learned weights to meta-learn from previous episodes. A transformer architecture is adopted for inference in HNPs. Experiments on regression and classification tasks demonstrate the empirical improvements of HNPs over NP baselines and other methods.", "questions": "See the section above.", "limitations": "There is a clear discussion of the potential limitations of HNPs, although there is no explicit discussion of potential negative impacts; which I strongly encourage the authors to include in the submission.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- HNPs are interesting and the architectural choices in the model are well-motivated and supported with ablation studies.\n- Experiments clearly establish the efficacy of HNPs in regression and classification tasks.\n- Paper is well-written and uses proper notation which helps with understanding the submission better.", "weaknesses": "- Multi-task meta-learning is an interesting problem paradigm that may not possess the same applicability as meta-learning and multi-task learning do in applied settings individually. What are some examples of real-life environments where multiple tasks are needed to process during a single episode beyond the benchmarks experimented on?\n- Classification performance is compared to older baselines, especially in the context of meta-learning methods. More recent baselines should be included for completeness of comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer LXqe for the positive review and helpful comments. We are glad that the review found that HNPs are interesting, the architectures are well-motivated, and the paper is well-written.*\n\n---\n\n**1. Real-world examples or benchmarks in multi-task meta-learning.**\n\nEpisodic multi-task learning has several potential applications in the real world, such as autonomous driving and robotic manipulations. In detail, the autonomous driving system needs to deal with different and related sensor data in an environment. However, the driving environment constantly changes along with the weather, time, country, etc. Thus, fast adapting the current multi-tasker to new environments is challenging in this industry track and our method can be a plausible solution for this challenge.  \n\n**2. More experimental comparisons.**\n\nThanks for your suggestion. We’ve included a recent multi-task and meta-learning method, such as MTL-bridge [a] and MLTI [b], and reported results on the Office-home dataset under the 4way5way1shot and 4way5way5shot setups as follows:\n\n| Methods      |  1-shot      | 5-shot      |\n|--------------|--------------|-------------|\n| MTL-bridge [a]| 64.31 ± 0.55 | 75.10 ±0.51 |             \n| MLTI [b]      | 70.69 ± 0.73 | 79.59 ±0.58 |\n| Ours         | 76.29 ± 0.51 | 80.80 ±0.42 |\n\nFrom this table, we find that our method significantly outperforms other baselines with severely insufficient data, such as 1-shot. This is consistent with the conclusion obtained in the main paper. We will add these results and discussion in Table 1 of the final version.\n\n**References:**\n\n[[a] Wang, Haoxiang, Han Zhao, and Bo Li. \"Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation.\" ICML2021.](https://arxiv.org/abs/2106.09017)\n\n[[b] Yao, Huaxiu, Linjun Zhang, and Chelsea Finn. \"Meta-learning with fewer tasks through task interpolation.\" ICLR2022.](https://arxiv.org/abs/2106.02695)\n\n---\n\n*We will add these collected results to the final version of the paper. We hope this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need further clarification. Thank you!*\n"}, {"review_id": "bjD6mWoTn2", "reviewer": "Reviewer_xREG", "summary": "This paper proposes HNP: Heterogeneous Neural Processes, an approach that combines multitask learning with metalearning to address the problem of insufficiency of data in the multitask learning setting. The paper introduces episodic multitask learning as a way to exploit task relatedness from meta-training to meta-test episodes. HNP combines a global latent representation and a set of local parameters to model each task-specific function, with each local latent-parameter conditioned on the global representation. The inference module for prior and posterior variational distributions is implemented through a transformer-structure where meta-knowledge is instantiated as learnable tokens. Experimental results are provided for both regression and classification tasks, in comparison to other NP-based, multitask learning and metalearning methods. \n\n---Rebuttal---\nI read the authors rebuttal and considered it, along with other reviews. I left my score unchanged since I think this already reflected my overall evaluation of the paper.", "questions": "- Please refer to the \"weaknesses\" section.", "limitations": "Limitations have been addresses. There are no potentially negative societal impact of this work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Regarding originality, the paper makes what seems to be a relevant contribution to the field in the sense of proposing a method that combines the benefits of metalearning with multitask learning for contexts where tasks are related, relying on a neural process-based approach. \n\n- Regarding significance, the paper shows important improvements mainly in classification tasks, compared to baselines in NP-based models, multitask-based models and metalearning-based approaches. \n\n- The quality and clarity of the paper are remarkable. The paper is very well-written and easy to follow, even for people not specifically into NP-based approaches.", "weaknesses": "- The results on regression tasks (Figure 4) seem to show that approaches such as ANPs and even NPs only can be quite competitive with respect to the proposed HNPs for at least a few of the tasks. Results in classification tasks, however, seem to denote much higher advantages of HNPs, especially in 1-shot learning, and considering the much larger number of tasks (or classes, in this case). This leads me to guess that the number of tasks has an influence on the amount, and quality, of sharing knowledge for HNPs. Does this imply that the method relies on the number of tasks to be large? What is the influence of the number of tasks, and the degree of relatedness of these tasks, in the way in which the local parameters and the global latent representation interact with each other, if any?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer xREG for the positive review and constructive feedback. We are glad that the review found that the quality and clarity of our paper are remarkable; the paper is very well-written and easy to follow.*\n\n---\n\n**1. Influence of the number of tasks.**\n\nThanks for the insightful questions. We’ve performed experiments and investigated the performance by gradually increasing tasks during inference. \n\n| Number   of tasks  |        1        |       2      |        3       |        4       |\n|--------------------:|:---------------:|:------------:|:--------------:|:--------------:|\n| Average   accuracy | 64.33     ±0.85 | 70.75 ±0.67 | 74.95    ±0.57 | 76.29    ±0.51 |\n\nFrom this table, we observe the average accuracy increases with more tasks. The main reason is that more tasks can provide richer transferrable information. Our model benefits from the positive transfer among tasks, and thus can obtain higher performance gain from more tasks.\n\n\n**2. Influence of the degree of task-relatedness.**\n\nThe degree of task-relatedness is particularly essential to the learning of hierarchical latent variables. Global latent representation is task-specific and controls the access of each local latent parameter to all context sets in the episode for the corresponding task. With a higher degree of task-relatedness, such as fewer concept shifts among tasks, each local latent parameter can access richer prediction-aware information from related tasks, yielding a more discriminative classifier in practice.\n\n---\n\n*We hope this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need further clarification. Thank you.*\n"}, {"review_id": "PGi2Kmd5UV", "reviewer": "Reviewer_DuFB", "summary": "The article is innovative in addressing the problem of data insufficiency by introducing the mutli-task learning method into the meta-learning paradigm. In this process, the author identify the issue of task-relatedness between heterogeneous tasks in a single episode and propose a solution called Heterogeneous Neural Processes (HNPs) for the episodic multi-task set-up.\n\nThe strengths of the article include:\n1) The author clearly explains the differences and connections between the proposed method and existing works, particularly the contribution of handling heterogeneous tasks and distribution shifts.\n2) The article provides impressive theoretical proofs and derivation processes of the algorithm in the appendix.\n3) The code of the method is provided in the appendix, ensuring reproducibility. \n4) The ablation study is comprehensive in exploring the impact of different factors on the method's performance, which is also impressive.\n\nThe weaknesses of the article:\nThe number of datasets used in the experiments is limited. While, the article provides more experiments in the appendix to supplement this issue. It does not significantly affect the credibility of the article's results. It would be helpful to provide more experimental results for the classification and regression tasks if possible.", "questions": "I wonder if any more experiments are conducted on other datasets, including classification and regression tasks. How is the performance of the proposed method? Especially considering the limitations that the author mentions in the Conclusion. ", "limitations": "The authors adequately addressed the limitations in the Conclusion part. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "As already written in summary, this paper is impressive considering its innovation:\n\n1) The author clearly explains the differences and connections between the proposed method and existing works, particularly the contribution of handling heterogeneous tasks and distribution shifts.\n2) The article provides impressive theoretical proofs and derivation processes of the algorithm in the appendix.\n3) The code of the method is provided in the appendix, ensuring reproducibility. \n4) The ablation study is comprehensive in exploring the impact of different factors on the method's performance, which is also impressive.", "weaknesses": "The number of datasets used in the experiments is limited. While, the article provides more experiments in the appendix to supplement this issue. It does not significantly affect the credibility of the article's results. It would be helpful to provide more experimental results for the classification and regression tasks if possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate Reviewer DuFB for the positive review and helpful feedback. We are glad the review found that our paper is innovative and impressive.*\n\n---\n\n**1. More experimental results on new benchmarks or datasets.**\n\nThanks for this suggestion. We’ve performed new experiments on the Office31 dataset [c, d] under the \"3task5way1shot\" setup. The dataset contains 31 object categories in three domains: Amazon, DSLR, and Webcam. \n\n|            Methods |      ERM      |   Proto.Net   |      CNPs     |      NPs      |     TNP_D     |      Ours     |\n|-------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| Average   Accuracy | 63.53   ±0.71 | 64.54   ±0.64 | 49.02   ±0.74 | 40.52   ±0.75 | 69.69   ±0.87 | 71.89   ±0.52 |\n\nFrom this table, we find that our method outperforms baseline methods, demonstrating our model's effectiveness in addressing the data insufficiency under the episodic setup. We will add this table and related discussions in the final version of the paper.\n\n**2. Experiments with considering the limitation in the conclusion section.**\n\nExperiments in Appendix F avoid the negative influence of relabeling scheme by setting the conventional multi-task learning without the episodic training scheme. The related results are shown in Appendix F, where we apply the Office-Caltech and ImageCLEF datasets for classification and the Rotated MNIST dataset for regression. Empirical observations show that HNPs consistently outperform baselines, suggesting the effectiveness of the proposed model.\n\n**References:**\n\n[[c] K. Saenko, B. Kulis, M. Fritz and T. Darrell. \"Adapting Visual Category Models to New Domains.\" ECCV2010.]( https://link.springer.com/chapter/10.1007/978-3-642-15561-1_16)\n\n[[d] B. Kulis, K. Saenko, and T. Darrell. \"What You Saw is Not What You Get: Domain Adaptation Using Asymmetric Kernel Transforms.\" CVPR2011.]( https://ieeexplore.ieee.org/document/5995702)\n\n---\n\n*We will add these collected results to the final version of the paper. We hope this rebuttal solidifies your positive outlook on our work, and we are happy to discuss it if you need further clarification. Thank you!*\n"}]

Author rebuttal:
*We thank all reviewers for their insightful comments and supportive suggestions. These feedbacks are constructive in improving this paper’s quality. Here, we provide a global response to summarize the primary merits of this paper and concerns to address as follows.*

---

## **I. Merits of this Work**

We summarize the following supportive points from reviewers: 

* The proposed method is novel, well-motivated, original, and innovative by Reviewers **n4KJ/ YPdL/ LXqe/ xREG/ DuFB**.

* The paper is well-written with clear contributions by Reviewers **LXqe/ xREG/ DuFB**.

* Experimental results show HNPs’ effectiveness with diverse tasks and ablation studies by Reviewers **n4KJ/ YPdL/ 21Dc/ LXqe/ xREG/ DuFB**.

* The theoretical proofs and derivations of algorithms are impressive by Reviewer **DuFB**.

---

## **II. Primary Concerns**

**1. Additional experimental comparisons and discussions by Reviewers n4KJ/ YPdL/ 21Dc/ LXqe.**

(1) We’ve performed experiments on Office-home under the 4task5wayXshot setup. 

| Methods |    1-shot   |    5-shot   |   10-shot   |   20-shot   |
|:--------:|:-----------:|:-----------:|:-----------:|:-----------:|
|    TNPs | 65.49 ±0.53 | 78.94 ±0.43 | 80.81 ±0.32 | 81.12 ±0.68 |
|    Ours | 76.29 ±0.51 | 80.80 ±0.42 | 81.28 ±0.38 | 81.56 ±0.36 |

Please refer to Response to Reviewer **n4KJ** for more details and discussions.

(2) We’ve performed experiments on DomainNet under the 6taskXway1shot setup. 

|     Methods    |         5way       |        20 way      |        25way       |        30way       |        35way       |        40way       |
|:---------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|
|        TNPs    |     49.10 ±0.42    |     28.83 ±0.17    |     25.93 ±0.14    |     24.08 ±0.12    |     22.62 ±0.11    |     21.64 ±0.10    |
|        Ours    |     62.36 ±0.53    |     39.32 ±0.23    |     35.72 ±0.19    |     32.27 ±0.17    |     31.27 ±0.14    |     29.31 ±0.13    |

Please refer to Response to Reviewer **YPdL** for more details and discussions.

(3) We've compared meta-learning baselines and our method on DomainNet under the 20way4shot setup.

|       Methods    |     4task20way1shot    |       20way4shot     |
|-----------------:|:----------------------:|:--------------------:|
|          MAML    |      34.29   ±0.19     |     37.23   ±0.25    |
|     Proto.Net    |      32.72   ±0.18     |     37.12   ±0.22    |
|          Ours    |      51.82   ±0.23     |          -           |

Please refer to Response to Reviewer **n4KJ** for implementation and discussions. 

(4) We’ve implemented two recent works [a] and [b] and performed experiments on Office-home under the 4task5way1shot and 4task5way5shot setups. 

| Methods      |  1-shot      | 5-shot      |
|--------------|--------------|-------------|
| MTL-bridge [a]   | 64.31 ± 0.55 | 75.10 ±0.51 |             
| MLTI [b]         | 70.69 ± 0.73 | 79.59 ±0.58 |
| Ours         | 76.29 ± 0.51 | 80.80 ±0.42 |

Please refer to Response to Reviewer **YPdL/ 21Dc/ LXqe** for more details and discussions. 

(5) We've provided the runtime cost in the inference of NP-based models per iteration.

| Methods        | CNPs | NPs  | TNPs | Ours  |
|----------------|------|------|------|-------|
| inference time(s) | 0.04 | 0.05 | 0.08 | 0.15  |

Please refer to Response to Reviewer **YPdL and 21Dc** for more details and discussions. 

(6) We’ve reported the negative loglikelihood (NLL) results in uncertainty quantification of regression tasks (the lower, the better).

| Methods  | CNPs   | NPs    | ANPs    | Ours |
|----------|--------|--------|---------|---------|
| Avg. NLL | 0.0935 | 0.8649 | -0.1165 | -0.5207 |

Please refer to Response to Reviewer **n4KJ and 21Dc** for more details and discussions. 

&nbsp;

**2. More benchmark datasets by Reviewer DuFB.**

We've validated the performance of methods on the Office31 [c, d] dataset. 

|            Methods |      ERM      |   Proto.Net   |      CNPs     |      NPs      |     TNP_D     |      Ours |
|-------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| Average   Accuracy | 63.53   ±0.71 | 64.54   ±0.64 | 49.02   ±0.74 | 40.52   ±0.75 | 69.69   ±0.87 | 71.89   ±0.52 |

Please refer to Response to Reviewer **DuFB** for details and discussions. 

&nbsp;

**3. Updates on the limitation of the shared target space by Reviewers n4KJ and 21DC.**

It is indeed challenging to ensure all tasks share the same target space. In this case, more modifications are required to encode the target space when using our method. We'll add this discussion to the limitation section in Line 374.

&nbsp;

**4. Other questions and future updates in the manuscript.**

For detailed concerns and questions, we respectively reply to individual reviewers. The newly collected experimental results, together with discussions, will be included in the final version.

&nbsp;

**References:**

[[a] Wang, Haoxiang, Han Zhao, and Bo Li. "Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation." ICML2021.](https://arxiv.org/abs/2106.09017)

[[b] Yao, Huaxiu, Linjun Zhang, and Chelsea Finn. "Meta-learning with fewer tasks through task interpolation." ICLR2022.](https://arxiv.org/abs/2106.02695)

[[c] K. Saenko, B. Kulis, M. Fritz and T. Darrell. "Adapting Visual Category Models to New Domains." ECCV2010.]( https://link.springer.com/chapter/10.1007/978-3-642-15561-1_16)

[[d] B. Kulis, K. Saenko, and T. Darrell. "What You Saw is Not What You Get: Domain Adaptation Using Asymmetric Kernel Transforms." CVPR2011.]( https://ieeexplore.ieee.org/document/5995702)

---

*Thank all reviewers and area chairs again for collaborating with us to improve our manuscript.*


Meta-review: This paper proposes a meta-learning method for multi-task learning. The proposed method, called heterogeneous neural processes, can capture relationships among heterogeneous tasks and use knowledge for tasks with a limited amount of data. The experimental results demonstrate the effectiveness of the proposed method in both classification and regression problems. This paper is well-written. The problem formulation and the proposed method are novel and interesting. The architectures of the proposed model are well-motivated, and their advantages are supported by their ablation study. The additional experiments in the rebuttals strength the paper.
Decision: Accept (spotlight)