Multi-Task Neural Processes
Authors: Jiayi Shen
Xiantong Zhen
Marcel Worring
Ling Shao
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-task learning
Neural processes
Variational Inference

Abstract: Neural processes have recently emerged as a class of powerful neural latent variable models that combine the strengths of neural networks and stochastic processes. As they can encode contextual data in the network's function space, they offer a new way to model task relatedness in multi-task learning. To study its potential, we develop multi-task neural processes, a new variant of neural processes for multi-task learning. In particular, we propose to explore transferable knowledge from related tasks in the function space to provide inductive bias for improving each individual task. To do so, we derive the function priors in a hierarchical Bayesian inference framework, which enables each task to incorporate the shared knowledge provided by related tasks into its context of the prediction function. Our multi-task neural processes methodologically expand the scope of vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning. The proposed multi-task neural processes are capable of learning multiple tasks with limited labeled data and in the presence of domain shift. We perform extensive experimental evaluations on several benchmarks for the multi-task regression and classification tasks. The results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning and superior performance in multi-task classification and brain image segmentation.

Under review as a conference paper at ICLR 2022
MULTI-TASK NEURAL PROCESSES
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural processes have recently emerged as a class of powerful neural latent vari- able models that combine the strengths of neural networks and stochastic pro- cesses. As they can encode contextual data in the network's function space, they offer a new way model task relatedness in multi-task learning. To study its potential, we develop multi-task neural processes, a new variant of neural pro- cesses for multi-task learning. In particular, we propose explore transferable knowledge from related tasks in the function space to provide inductive bias for improving each individual task. To do so, we derive the function priors in hi- erarchical Bayesian inference framework, which enables each task to incorporate the shared knowledge provided by related tasks into its context of the predic- tion function. Our multi-task neural processes methodologically expand the scope vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning. The proposed multi-task neural pro- cesses are capable of learning multiple tasks with limited labeled data and in the presence of domain shift. We perform extensive experimental evaluations sev- eral benchmarks for the multi-task regression and classification tasks. The results demonstrate the effectiveness of multi-task neural processes in transferring use- ful knowledge among tasks for multi-task learning and superior performance in multi-task classification and brain image segmentation'.
INTRODUCTION
As deep neural networks are black-box function approximations, is difficult to introduce prior domain expert knowledge into a prediction function (Jakkala, 2021). In contrast, Gaussian processes (Rasmussen, 2003) explicitly define distributions over functions and perform inference these functions given some training examples. This enables reliable and flexible decision- making. However, Gaussian processes can suffer from high computational complexity due to the manipulation of kernel matrices. Therefore, there has been continuous interest in bringing together neural networks and Gaussian processes (Damianou & Lawrence, 2013; Wilson et al., 2016; Garnelo al., 2018a; Jakkala, 2021) into so-called neural processes.
Neural processes (Garnelo et al., 2018b) combine the computational efficiency of neural networks with the uncertainty quantification of stochastic processes. They are class of neural latent variables model, which deploy deep neural network to encode context observations into a latent stochastic variable to model prediction functions. Neural processes provide an elegant formalism to efficiently and effectively incorporate multiple datasets into learning distributions over functions. This for- malism is also promising in multi-task learning to improve individual tasks by transferring useful contextual knowledge among related tasks. Their capability of estimating uncertainty over predic- tions also makes them well-suited for multi-task learning with limited data, where each task has only few training samples. However, neural processes rely the implicit assumption that the context and target sets are from the same distribution and can be aggregated by a simple average pooling operation (Kim al., 2019; Volpp et al.. 2020). This makes non-trivial to directly apply neural processes to modeling multiple heterogeneous tasks from different domains, where the context data of different tasks are from distinctive distributions (Long al., 2017).
Our code is available in http://aaannnnaa..oooommccccccca

---

Under review as a conference paper at ICLR 2022
In this paper, develop multi-task neural pro- cesses (MTNPs), methodological extension of neural processes for multi-task learning, which fills the theoretical gap of neural processes for multi-task learning. Partieularly, we propose to explore task relatedness in the function space by specifying the function priors in hierarchi- cal Bayesian inference framework. The shared knowledge from related tasks incorporated into the context of each individual task, which serves as the inductive bias for making pre- dictions in this task. The hierarchical archi- tecture allows us to design expressive data- dependent priors. This enables the model to capture the complex task relationships in multi- task learning- By leveraging hierarchical mod- eling, multi-task neural processes are capable of exploring shared knowledge among related tasks in principled way by specifying the function prior.
X
Y
M
D1
Z
+
α1
Y'
X
Yi
X
(a) Neural processes
(b) Multi-task neural processes
Graphical illustration of neural pro- cesses and multi-task neural processes. Shaded nodes indicate observed variables, and white nodes indicate the introduced latent variables.
Figure 1:
We validate the effectiveness of the proposed multi-task neural processes by extensive experiments in both multi-task classification and regression. The results demonstrate that multi-task neural pro- cesses can effectively capture task relatedness in the function space and consistently improve the performance of each individual task, especially in the limited data regime.
PRELIMINARIES: NEURAL PROCESSES
In this section, we briefly review vanilla neural processes (Garnelo al., 2018b) based on which derive our multi-task neural processes.
Let data be given composed of training samples and corresponding labels. In order to better reflect the desired model behaviour at test time (Garnelo et al., 2018b), 8b). the training data is split into the context set = (X, Y) and target set D = (X* Y*), where
{x1,-' xx, is a subset of the training data, Y {y1, y the corresponding set of labels and the size of the context
set. To ensure both sets have the same data distribution, the context set is split from the target set.
Now, given the context set (X, Y). we would like to estimate function that make predict the labels Y for target samples X
In general, we define stochastic process by random function : + V. Given the context set, we define the joint distribution over the function values (f(x)), f(x,)] which in Gaussian processes multivariate Gaussian distribution parameterized by kernel function. The rationale of neural processes to extract knowledge from the context set to specify the prior over the prediction function. Instead of kernel function, neural processes adopt deep neural network define the prior distribution. Specifically, the model introduces a latent variable z to account for uncertainty in the predictions of Y". The observed context conditioned on the context set, i.c. it follows the prior distribution p(z[X, Y). The latent variable high-de digeesinnns random vector parameterising the stochastic process by f(X*) 9(X*, z). The function ((...) is an extra fixed and learnable decoder function, which is also implemented by a neural network. Thus, the neural process model can formulated as follows:
encoded into the latent variable which is
p ,Y X,Y
Y X 2)p(z|XX Y)dz.
(1)
The graphical model for neural processes shown in Figure (a).
The neural process model is optimized using amortized variational inference. Let q(z)X*, Y*) be variational posterior of the latent variable z. The evidence bound (ELBO) for neural processes given follows:
log Y X ('ppp("'''""" X, 2 E (o)le' z)] [zz Y*)]|p(z|X, Y)].
In neural processes, the function space defined by deep neural networks allows the model to extract deep features while retaining probabilistie interpretation (Jakkala, 2021). In multi-task learn-
2

---

Under review as a conference paper at ICLR 2022
ing, usually different tasks can be from different domains and have their specific data distributions (Lawrence & Platt, 2004; Long et al., 2017). Due to the complex data structure of multi-task learn- ing, it is not straightforward to explore task relatedness in such function spaces. In this paper, aim to extend the methodology of the neural process to the scenarios of multi-task learning to learn the shared knowledge among tasks for improving individual tasks.
3 MULTI-TASK NEURAL PROCESSES
The common setup of multi-task learning is that there are multiple related tasks for which would like to improve the learning of each individual one by sharing information across the different tasks (Williams al., 2007). Multi-task learning has been studied under different settings (Re- queima et al., 2019; Williams et al., 2007; Lawrence & Platt, 2004; Yu et al., 2005; Long et al., 2017). In this paper, we tackle the multi-input multi-output setting, where each task has a different distribution while different tasks share the same target space (Lawrence & Platt, 2004; Yu et al., 2005; Long al., 2017; Zhang al., 2020). It aims improve the overall performance of all mul- tiple tasks simultaneously different from the sequential multi-task leaning (Requeima et al., 2019; Garnelo et al., 20) This a challenging scenario due to the domain shift between tasks, which makes it sub-optimal to directly apply neural processes by incorporating the data from other related tasks into the context of each individual task.
3.1 HIERARCHICAL CONTEXT MODELING
Multi-task learning considers the estimation of random functions fi, 1 1,.... L for each of the L related tasks. Each task has its own training data, which is split into a context set Dj = (X,, Y,) and a target set Di = (X), ; Yf). Xi E IR" xd and X; E RR" xd are inputs while Y₁ E R xC and Y, E R are outputs. d and C' are the sizes of the input space and output space, respectively, nf and ny are the sizes of respectively the target and context task-specific latent variables fĄ = fiX,)) E XC We use {Di to denote all context sets in the dataset, which for brevity we represent as {Di) and likewise will do for other sets.
for the I-th task. Thus, obtain the
Formulated this way the goal of multi-task learning is to predict { (Yi} for given (X;) simultane- ously with the assistance information {Di from all tasks. To this end, we construct joint prediction distribution with respect to the latent random functions [fi] as:
L p (((X))(())) (Di)) = - PIXXX (Di)) = II
Here, enable shared knowledge be transferred among tasks, we introduce global variable M which works as a container to collect the useful information from (Dı} of all tasks. In contrast to neural processes for single tasks, the global variable M provides the contextual information from all tasks for each individual task. The concrete formation of M depends on the learning scenarios. For regression tasks, M TR. Led and each row corresponds one task, which is the average of all feature vectors from task. For classification tasks, M E IR L-Cadd where each vector M, is the average of all features of each category from the corresponding task.
Similar to Gaussian processes, we assume that the function value is = f(;;) + where e (000, d²) is the observation noise. For regression tasks, we can define the predictive like- lihood on the target set as p p(Y; fi) = (((Y)fff( o?) For classification tasks, we use log x fi) = E yo.log(f((x).) as the log- likeliiol function, where xi,; is the i-th target sample from the 1-th task.
In order to combine Gaussian processes and neural networks in the context of multi-task learning, we define p(fi M) in (3) as a deep neural network in place of a Gaussian distribution parameterized by kernel function. To be more specific, we assume that fi is parameterized by random variable db by defining fiX = X We specify a data dependent prior by conditioning z/1 on the global variable M. In this way, incorporate the transferable knowledge into the learning of the prediction function of the current tasks. Thus, the predictive distribution for the I-th task over its target set can be formulated as follows:
3

---

Under review as a conference paper at ICLR 2022
PXIIX, (DI)) - f /XYIIX, (o) M))b..
(4)
In effect, ub denotes the parameters of classifier for classification tasks or regressors for regression tasks. Particularly, the introduced latent variable th Ixxdd denotes the task specific classifier, where d is the dimension of the input feature and C' is the number of classes in the dataset. As done in (Requeima et al., 2019), we generate each column of abt independently from the context samples of the corresponding class. In our case, each column of ahr encodes the context information of its class from all tasks po (1br MM IT= (t),,
Directly aggregating M as done in neural processes for single tasks is not applicable for multi-task learning due to the distribution shift between tasks. The data from related tasks should be processed and adapted to the current task as the contextual information. To this end, we introduce a higher- level latent variable on extract the shared knowledge from M, which is conditioned Dj of each task. This results in a hierarchical Bayesian modeling of functions in the neural process:
the data
pu<bb/M) = / po, (bbilou,
where ar is the latent variable to control the access shared knowledge for each task, which is used to explore the relevant knowledge to the task I. pe, |y, M) and pe, (t) are prior distributions of the latent variable 1 and au, respectively, which parameterized with neural networks. To be specific, we define po, on, M) (Nbb ((m)). E(m))). Here m, contains the relevant knowledge to the task 7, which is adapted from the global variable M by deterministic function m; = h((a,, M), where ft-)) is learnable function implemented with a neural network.
By substituting (5) into (4), obtain the model of multi-task neural processes as follows:
L p((Y))((X1) ( = III /XY:IX; thippo (3/ M)po, (D..)ddddd [=1
(6)
The designed hierarchical context modeling provides a principled way explore task relatedness in the function space, which allows task-specific function variables to leverage the shared knowledge from related tasks. We provide theoretical proof in Appendix to show that the proposed multi-tass neural processes are valid stochastic processes, which completes the theory of multi-task neural processes. The graphical model of the multi-task neural processes is shown Figure (b).
3.2 VARIATIONAL HIERARCHICAL INFERENCE
The previous section developed the model of multi-tasks neural processes with hierarchical context model. We now describe how to optimize the model and obtain the predictions by leveraging a varia- tional Bayesian inference framework. To that end, based on conditional independence assumption, we introduce the variational joint posterior distribution factorized as follows:
L
90 (( fau))( = II (zb, anl
II 1=1
g.s (abr Di)q. (aul
(7)
where 4o DF) and yos (o)lD7) are variational posteriors of the latent variables th and a for the task I, respectively. Both variational posteriors are parameterized as Gaussian distributions. 61 and 42 amortized inference networks to generate variational posteriors and shared by all tasks. We make use of the amortized variational inference technique (Kingma & Welling, 2013) to learn such distributions over latent variables.
Learning By incorporating the variational posteriors into (7), we derive the ELBO for the multi- task neural processes as follows:
log p( (XX))
Σ
Eg (o))))))) (0 vh)
- BKLL (wrl (sbr ab M)] - (au (ar
4

---

Under review as a conference paper at ICLR 2022
The detailed derivation is provided in Appendix A. By adopting the Monte Carlo sampling, obtain the empirical objective for the proposed multi-task neural processes:
(0,)
+ XyDKLL (zhy) DJ) lpp, (vrila)) ( M) + A
qp 4p, (ayl Ny and Na are the number Monte Carlo samples for the variational posteriors of so and a, respectively. 'y and X the hyperparameters to help stably train the Ki-diiveeeence terms, which are set following the annealing scheme of (Bow- man et al., 2015). In practice, we apply the local reparameterization trick (Kingma al., 2015) to reduce the variance of stochastic gradients.
where (5(2) ~ 9p. (b):) and aj
Prediction Having the learned model, we can make prediction on the target set. Given a test sample x₃ from the I-th task, can produce the predictive distribution which involves the prior distributions po, (*h) or, M) and pa, (arl Dt). The predictive distribution with the integration for the introduced latent variables is formulated as:
p (Di)) = f p(ys)x,, do) (b/)) M)po, (ax)
(10)
Here we also need to apply the Monte Carlo estimation over (10) and obtain the predictions as follows:
Na
N ply:|xi, p(y))x,,+()))
p(yl(x)) 2
(11)
N₂
NJ
(i)) In particularly, the prior distribution of the latent variable tir is generated from the output of the learned function l((a)) (t) .M.
where w/ pe, (tb (sbila)) M) and ax
4 RELATED WORK
Neural network based models have achieved impressive results various applications (LeCun et al., 2015). However, due to the large number of parameters these neural models need large-scale data with annotation. It is challenging to train deep neural model that generalizes well with number- limited labeled data. To reduce such labeling consumption, recent works (Long et al., 2017: Liu et al., 2016) follow a multi-task learning strategy fully leverage information from relevant tasks as inductive bias (Caruana, 1997) to improve each task's performance. The main challenge multi- task learning is that each task is provided with limited amount of labeled data, which is insufficient to build reliable classifiers without overfitting (Long et al., 2017).
Multi-task learning (MTL) aims learn several tasks simultaneously and improve their overall performance. The crux of MTL is how to explore task relatedness from different tasks, which could be particularly significant when limited data for each task available (Long al., 2017: Zhang et al., 2020). Recently, the task relatedness is learned in many different aspects of the model, e.g., loss functions (Liu al., 2020; Qian et al., 2020; Kendall al., 2018), parameter space (Long al., 2017: Bakker & Heskes, 2003), or representation space (Misra et al., 2016). In this paper, we focus on the data-insufficient problem for multi-task learning. Different from other branches of MTL models (Huang et al., 2021; Fu et al., 2021; Phillips et al., 2021) which leverage several channels of supervision information simultaneously included by the same input, our data setting allows each task to have its own individual data. This is much more challenging due to the distribution shift between inputs of different tasks.
In previous works (Lawrence & Platt, 2004; Yu et al., 2005; Yousefi al., 2019), multi-task learning benefits from Gaussian Processes by generally incorporating the shared information in the Gaussian Processes prior to synergize several random functions of different tasks simultaneously. The multi- task Gaussian processes (Yu et al., 2005) proposes a hicrarchical prior which enables task specific random functions to share some common structure in the hyper prior, As alternative to using
5

---

Under review as a conference paper at ICLR 2022
4 context samples
context samples
context samples
NPs
MTNP
Task1
Task2
Task3
Task
Task1
Task2
Task3
Task
Task1
Task2
Task3
Task 4
Figure 2: 1-D multi-task function regression. The plots show sample of curves conditioned an increasing number of context samples for each task (4, 8 and 12 each column respectively), The ground truth is shown in black and the context samples as black dots. The predictions of our MTNPs more resemble the ground truth than that of NPs, especially in the boundary of different tasks.
Gaussian processes for multi-task learning, nowadays deep neural networks have become popular as parameterized functions, which construct information sharing architecture between tasks. Some methods (Misra et al., 2016; Liu et al., 2019) learn expressive combinations of features from different tasks by deep neural networks. Some approaches (Gao et al., 2020; Sun et al., 2019) flexibly adjust the deep architecture for each individual tasks. Yu et al. (2020) mitigates gradient interference by performing the proposed gradient surgery. However, these deep multi-task learning methods relies on large amounts of training data and therefore tends to overfit with limited data.
Recently, neural processes (Garnelo al., 2018b) and the related works (Garnelo et al., 2018a; Kim et al., 2019; Wang & Van Hoof, 2020; Requeima et al., 2019; Gordon et al., 2019) combine Gaussian Processes and neural networks, which are not only computationally efficient but also re- tain a probabilistic interpretation of the model. (Garnelo et al., 2018b; Kim et al., 2019; Wang & Van Hoof, 2020) introduce the latent representation variables to model the function randomness. (Requeima al., 2019; Gordon et al., 2019) introduce a latent parameter variable to directly model the predictive distribution, which acts as the parameters of the neural network.
Hierarchical modeling in the Bayesian framework has been successful to design the form of the prior (Daumé III. 2009; Zhao et al., 2017; Klushyn et al., 2019: Wang & Van Hoof, 2020) and posterior distributions (Ranganath al., 2016; Krueger et al., 2017; Zhen et al., 2020) based on many observations. allows the latent variable to follow a complicated distribution and forms a highly flexible approximation (Krueger et al., 2017).
EXPERIMENTS AND RESULTS
5.1 1-D MULTIITASE FUNCTION REGRESSION
Our method fully utilizes the transferable knowledge provided by related tasks to improve each task's performance. To show this, test the proposed multi-task neural processes the 1-D multi-task function regression.
Setup. We define several tasks with different data distributions: inputs of cach task are sampled from the separated intervals without overlapping, such [-2m, π). [- 0), [0, x), and [m, 2m). Each r-value is drawn uniformly at random in its belonging intervals. We assume that all tasks share the same ground truth function ensure that there is transferable knowledge between them. The ground truth function is characterized as a sum of sine or cosine functions (Wang & Van Hoof, 2020), such as y = sin(xr) (in(2r)+si)) +inn (022))0 cos(0.5.z) +e, where is the noise drawn from (7 0.0003"). At each training step. the hyperparameters of the ground truth function are fixed as set in (Kim et al., 2019; Wang & Van Hoof, 2020).

---

Under review as a conference paper at ICLR 2022
Results. As shown in Figure 2, we visualize prediction results of different tasks for the comparison of neural processes and multi-task neural processes. The predictions of the proposed multi-task neural processes (the second row) more resemble the ground truth functions than that of neural processes (the first row), especially in the boundary of different tasks. Moreover, we find that when training with fewer context samples, the improvement is more significant. This demonstrates that our method enhances the generalization of each task by fully utilizing the transferable knowledge from related tasks.
5.2 MULTI-TASK CLASSIFICATION
Datasets. We evaluate the performance of our multi-task neural processes on real-world multi- task classification, where different tasks are defined as image classífication problems in different domains. The tasks are related to each other since they share the same label space. Of fice ce-oeee (Venkateswara et al., 2017) contains images from four domains/tasks: Artistic (A), Clipart (C), Product (P) and Real-world (R). Each task contains images from 65 categories collected under office and home settings. There are about 15, 500 images in total. Office Caltech (Gong et al., 2012) contains the ten categories shared between Office-31 (Saenko et al., 2010) and Caltech- 256 (Griffin et al., 2007). One task uses data from Caltech-256 (C), and the other three tasks data from Office-31, whose images were collected from three distinct domains/tasks, namely Amazon (A), Webcam (W) and DSLR (D). There are 151 samples per category per task, and 2, 533 images in total. ImageCLEF (Long et al., 2017), the benchmark for the ImageCLEF domain adaptation challenge, contains 12 common categories shared by four public datasets/tasks: Caltech-25t (C), ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Bing (B). There are 2, 400 images in total.
Setup. We adopt the standard evaluation protocols (Zhang & Yang, 2021) for multi-task classifi- cation datasets. We randomly select 5%, 10% and 20% labeled data for training, which correspond to about 3, 6 and 12 samples per category per task, respectively. In this case, each task has a limited amount of training data, which is insufficient for building the reliable classifier without overfitting. For all three benchmarks, we extract the input features by the pretrained VGGnet as (Long et al., 2017). The architectures of inference networks used in our model are provided in Appendix C. All the results with error bars are obtained based on 95% confidence interval from five runs.
Compared methods. To show the effectiveness of the proposed multi-tasks neural processes, we conduct a thorough comparison implementing multiple different baseline models. Single task learn- ing (STL) is implemented by task-specific feature extractors and classifiers without knowledge shar- ing among tasks. Basic multi-task learning (BMTL) shares feature extractors and adds task specific classifiers. We also define variational extensions of the single task learning (VSTL) and basic multi- task learning (VBMTL), which cast models as variational Bayesian problems and treat classifiers as latent variables (Shen et al., 2021). We implement neural processes (NPs) and its variant, neural processes with all task context for comparison. NPs with all task context is a straightforward exten- sion of NPs for MTL, which treats context sets from all tasks equally without hierarchical context modeling. For a fair comparison, all the above-mentioned methods share the same architecture of the feature extractor.
Results. We provide comprehensive compar- isons on ce-Home in Table 1, which is more challenging multi-task classification dataset with 65 categories. The results show that our MTNPs outperform other counterpart methods. NPs with all task context performs better than NPs by a large margin when 5% la- beled data is available, showing the benefit of exploring shared knowledge from related tasks with limited data. Noticeably compared to NPs with all task context, the proposed multi-task neural processes benefit from hicrarchical con- text modeling and show even better performance.
Table 1: Performance comparison (average accu- racy) on Of f ce-Home training with 5%, 10%, 20% labeled data. The higher the better.
Methods
5%
10%
20%
STL VSTL BMTL.
49.2
58.3
64.910.
51.1
60.2+0.2
65.8+02
50.4
59.5
65.6+a1
VHMTI.
51.3:10.
60.94..1
67.0+a3
NPs
54.7
59.4-10.2
69.3+02
NPs with all task context
59.1
62.4+t0.1
+
MTNPs
60.0::0.1
63.3
69.9:0.3
7

---

Under review as a conference paper at ICLR 2022
Table Performance comparison of different methods on the Office Home dataset.
5%
10%
Methods
R
Avg.
P
R
Avg.
STL
m6.7
30.8.0s
67.5100
61.7103
49.2
50.4
40.8100
74.4104
67.5
$8.3..0
Bakker & Heskes (2003)
40.0_m1
33.6
69
63.6103
52.8
52.5
42.3
78.7
69.5
60.0.nn
Lang et al. (2017)
47.8.04
37.9102
73.6103
70.4102
56.4
57.21.9
43.3100
78.7
74.4
63.4_.02
Kendall al. (2018)
40.2-ma
33.62n4
69.5aaa
51.8 63.7
49.1
38.7
73.4
67.4
57.2.mn
Guo et al. (2020)
25.8=14
26.7=ns
55.8+07
46.0+06
38. 380.5
38.3+38
41.5+a
67. 64.6444
58.8
51.5=00
Qian et al. (2020)
37,9=03
31.=02
67.7+03
62.4+002
49.9.002
47.1=94
37.2+001
70.5+02
66.3+a03
55.3=01
MTNPs
55,0ma2
40.8=03
74.2+02
69.9
60.0
59.5+a3
44.4+05
37.3+03
72.0a03
63,3-a1
Table 3: Performance comparison of different methods on the Off Owf.cee---ee............ Ca 1 tech dataset.
5%
10%
Methods
A
W
D
Avg.
A
W
D
C
Avg.
STL
87.4=04
87,9=03
96.4+05
82. 8+02
88.6+0.3
92.8 92.8
97.7=03
87,8+02
84,3
90.7=02
Bakker & Heskes (2003)
93.2-00
94.0203
94.7+03
85,4s04
91.8
94.9234
97.6*95
96,6295
90.9s04
95.0=00
Long et al. (2017)
92.7=02
94.3=02
97.1+02
89.2#06
93.4402
95.0=a3
98.1+04
25.0=05
91.3=a2
94,8=03
Kendall al. (2018)
93.6:04
92.5:01
95.0zus
83.9
91.2s63
94.9205
96.2494
93.6203
90.4302
93.8=00
Guo et al. (2020)
73.9
76.0ux6
78.3113
70.3100
74.6164
80.411
89.4138
73.4
78.5
80.4
Qian al. (2020)
92,6:03
90.9:02
95.7204
85.2
91.1463
94.2::0
97.0204
95.0203
90.2s03
=4,1
MTNPs
94.6.m1
95.8
97.9100
90.2..01
94.61
97.7.19
97.1
91.6.00
95.4
Table 4: Performance comparison of different methods on the ImageCLEF dataset.
5%
10%
Methods
Ave
Avg.
STL
85,4
71.4204
57.7=02
36,0#02
62.6
88. 93.3
77.8+03
64.3e02
47.6
69.7=03
Bakker & Heskes (2003)
90.9_.+
85.4.s
68.1
51.410s
73.9
91.01as
87.1
73.4.00
54,5102
76.5_m
Long et (2017)
=0.
76.52ns
72.8103
54.9104
73.7
93.3224
83.2
70.4
56.3:04
75.8-00
Kendall al. (2018)
93.2.ms
86.1
68.6143
50.4164
74.6102
91.9193
88.9108
74.3.00
52,4.02
76.9_ns
Gao (2020)
$0.1
55.5g11
46.7111
24.4
51.7
86. 16.
68.9111
56.0.1 56.0.115
39.3
62.6-08
Qian ct al. (2020)
91.6=03
85.8=04
68 68.4
50.2=00
74.0 74.0=00
90.7=04
88. 88.1
75.6=04
54,6=03
77.3=003
MINPS
90.5=a3
84.9=01
70.2+02
58.9
76.1:9.1
93.5=a0
88.5+00
74.6+04
61.7503
79.6=01
Table 5: Performance of multi-task regression (normalized mean squared errors) for rotation angle estimation. The lower the better.
Mathods
9
Avg.
1338: 303
215
2216
337
150
.229-109
1866
119
VSTL
J44: m:
_161=.m
m3
.197
33822.m
208:::16
1189-102
12662.04
133
1144-1e
BMTI
114g m
.124g res
1155
.124g 6s
m15
114g
.14:2 (1)
1125 mm
115: re
118:.m
VRMTL
121g mm
.124:res
.125g 065
121
121
_12
.134:/
g
_121_10
_121 am
Yu (2005)
1.45
.126
1163
.234
.145
.1222_res
.153
Liu al. (2019)
196
.166..1
.162
.124
112aam
.140s.225
.249:004
.195
081
.119-10
.152
Gan et al. (2020)
=188= 202
.078-
_103::00
0063:
.18:
099::00
.156-1104
.090s.0e
082=0094
.138=1e9
.092
NPs
193: m
1058:100
.005+aa
0667
101
120
1158-2104
107
083
.126=1e9
112s:0
NPs with all task context
:88 302
0044:
.114..00
0663 065
103
11161.00
.167=.400
096 1.00
007
_111=re0
.00
MINPs
183
.060:xe0
.098.acc
.067s
109
109::.02
.160=1004
5092
0077
.113=104
.106:00
More comparison results the Office-Home: Office-Caltech. ImageCLEF datasets shown in Tables 2, 3 and 4, respectively. The average accuracy of all tasks is used for overall performance measurement. The best results are marked in bold. Our MTNPs achieve competitive and even better performance on such multi-task classification datasets with different train-test splits. Compared with Bayesian baselines, including VSTL, VBMTL, NPs and (Bakker & Heskes, 2003), our MTNPs directly infer the parameters of prediction functions rather than the input representation, which is able to model a broader range of functional distribution. Moreover, in function space the hierarchical context modeling can better explore the task relatedness, which enable the models to capture the relevant knowledge even in presence of distribution shift among tasks. Experimental results on all three benchmarks with 20% labeled data are provided in Appendix D.
5.3 MULTI-TASK REGRESSION
Setup. In order to show the effectiveness of MTNPs for multi-task regression, we conduct ex- periments on the Rotated MNIST dataset (LeCun et al., 1998). We adopt this dataset to study multi-task regression, where each task is angle estimation problem for each digit and different tasks corresponding to different digits are related because they share the same rotation angle space. Each image is rotated by 0 through 90 in intervals of 10 where the rotation angle is the regression target. We randomly choose 0.1% samples per task per angle as the training set.
Results. Since we would like improve the overall performance of all regression tasks, we the average of normalized mean squared errors of all tasks as the measurement. As shown in Table 5, our MTNPs outperform other counterpart methods by yielding overall lower mean error.
8

---

Under review as a conference paper at ICLR 2022
0.9677
0.8173
0.8954
09220
0,8888
0.0861
G.7471
0.8894
0.9817
0-9504
0,9079
0.5306
Figure 3: The segmentation results by the proposed multi-task neural processes (bottom row) and the U-Net (upper row). Green outline corresponds to the ground truth and red to the segmentation output. The numbers are the DSC scores compared against ground truth. Our multi-task neural processes can predict contours closer to the ground truth ones with higher DSC scores than U-Net, indicating the advantages of exploring spatial context information for brain image segmentation.
5.4 BRAIN IMAGE SEGMENTATION
In this section, we demonstrate that multi-task neural processes are also able to explore spatial con- text information to improve image segmentation. To this end, we adopt brain image dataset (Buda et al., 2019) with lower-grade gliomas collected from 110 patients. The number of images varies among patients from 20 to 88. The goal to segment the tumor in each brain image.
Setup. To apply our multi-task neural processes, we reformulate the segmentation task pixel- wise regression problem, where each pixel corresponds to a regression task to predict the probability of this pixel belonging to the tumor. In doing so, the spatial correlation and dependency among pixels are effectively modeled by capturing the task relatedness. To be specific, we consider the prediction of each pixel to be regression task. For the task I, we define 12 local region centered at the spatial position I, which provides the local context information. In this case, the region centered the pixel provides the local context information. Each task incorporates the shared knowledge provided by related tasks into its context of the prediction function. This offers an effective way to model the long-range interdependence pixels one image. For implementation, we use the U-Net architecture (Ronneberger et al., 2015) as the backbone and add our model as the final layer.
Results. We compare our method and U-Net on the brain segmentation dataset. The results show that the proposed multi-task neural processes surpass the baseline U-Net by 0.5% in terms of dice similarity coefficients (DSC) for the overall validation set. Figure 3 shows segmentation results of the proposed multi-task neural processes (bottom row) and the U-Net (upper row), where the green outline corresponds to the ground truth and the red to the segmentation output. Our multi-task neural processes predict contours closer the ground truth. This demonstrates the advantages of exploring context information by multi-task neural processes for segmentation.
6 CONCLUSION
In this paper, we develop multi-task neural processes, a new variant of neural processes for multi- task learning. We propose explore the task relatedness in the function space by specifying the function priors in a hierarchical Bayesian inference framework. The shared knowledge from related tasks is incorporated into the context of each individual task, which serves as the inductive bias for making predictions of this task. The hierarchical architecture allows us to design expressive data- dependent prior, enabling the model to explore the complex task relationships in multi-task learning- By leveraging the hierarchical modeling, multi-tasks neural processes are capable of capturing the shared knowledge from other tasks in a principled way by specifying the function prior. We evaluate multi-task neural processes multi-task regression and classification datasets. Results demonstrate the effectiveness multi-task neural processes in transferring useful knowledge among tasks for multi-task learning.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Since our method a general contribution to the multi-task learning field, it has the potential to bene- fit any societal problem where multiple tasks are performed simultaneously, e.g., medical diagnosis. Accordingly, the method would also potentially face some negative social impacts accompanying with these applications. e.g., lack of fairness with the model trained by incomplete data due the privacy of patients in medical imaging tasks.
REPRODUCIBILITY STATEMENT
To make our method reproducible, we provide the details of the method in Section 3. We also give the derivations of the evidence lower bound proposed Section 3 Appendix A. The datasets used in this paper can be obtained by referencing Section 5. There are also the experimental settings and implementation details for each dataset in Section We also provide an anonymous link for the code of our method, which is in the abstract of this paper.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
BJ Bakker and TM Heskes. Task clustering and gating for bayesian multitask learning. 2003.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv:/511.06549. 2015.
Mateusz Buda, Ashirbani Saha, and Maciej A Mazurowski. Association of genomic subtypes of
lower-grade gliomas with shape features automatically extracted by deep learning algorithm. Computers in biology and medicine, 109:218-222 225, 2019.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMIR, 2013.
Hal Daumé III. Bayesian multitask learning with latent hierarchies. arXiv preprint arXiv:0907.0783, 2009.
Gang Fu, Qing Zhang, Lei Zhu, Ping Li, and Chunxia Xiao. A multi-task network for joint specular highlight detection and removal. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7752-7761, 2021.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11525 11535, 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1713-171. PMLR, 2018a.
Marta Garelo,, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.0122. 2018b.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp- 2066-2073. IEEE, 2012.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Jonathan Gordon, Wessel Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv: /9/0. 13556, 2019.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pp. 3854-3863, PMLR, 2020.
Zhizhong Huang, Junping Zhang, and Hongming Shan. When age-invariant face recognition meets face age synthesis: A multi-task learning framework. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7282-7291, 2021.
Kalvik Jakkala. Deep gaussian processes: survey. arXiv preprint arXiv:2106. 10X.::2106./233, 2021.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Hyunjik Kim, Andriy Mnih Jonathan Schwarz, Marta Garnel,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv: arXiv: /901:0560000 2019.

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6988, 2014.
arXiv preprint
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 2013.
arXiv preprint
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. Advances neural information processing systems, 28:2575-2588, 2015.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes. arXiv preprint urXiv:1905. 1900.049994 2019.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 17X.00477 2017.
Neil Lawrence and John C Platt. Learning to learn with the informative vector machine. In Proceedings of the twenty-first international conference on Machine learning, pp. 65, 2004.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
Liyang Liu, Li, Zhanghui Kuang, Jing- Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2020.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Deep multi-task learning with shared memory.
arXiv preprint arXiv:1609.07222, 2016.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1871-1880, 2019.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multilinear relationship networks. In Advances in neural information processing systems, pp. 1594-1603, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings the IEEE conference on computer vision and pattern recog- nition, pp. 3994-4003, 2016.
John Phillips, Julieta Martinez, Ioan Andrei Bårsan, Sergio Casas, Abbas Sadat, and Raquel Urtasun. Deep multi-task learning for joint localization, perception, and prediction. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4679-4689, 2021.
Weizhu Qian, Bowei Chen, Yichao Zhang. Guanghui Wen, and Franck Gechter. Multi-task varia- tional information bottleneck. arXiv preprint arXiv:2007.00333) 2020.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333. PMLR, 2016.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 12:7959-79707 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241 Springer, 2015.
12

---

Under review as a conference paper at ICLR 2022
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning, 2021.
URL httpst:///ppeeeeieeeeeeee wttprriewweee/frrruee -i-.ummii---keeeemoo
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what share for efficient deep multi-task learning, arXiv preprint urXiv:/911.12: 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In (IEEE) Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Michael Volpp,
Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations, 2020.
Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hi- erarchical latent variables. In International Conference on Machine Learning, pp. 10018-10028. PMLR, 2020.
Chris Williams, Edwin Bonilla, and Kian M Chai. Multi-task gaussian process prediction. Ad- vances in neural information processing systems, pp. 153-160, 2007.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Fariba Yousefi, Michael Thomas Smith, and Mauricio A Álvarez. Multi-task learning for aggregated data using gaussian processes. arXiv preprint arXiv:1906.0940 094199000941,,... 2019.
Kai Yu, Volker Tresp, and Anton Schwaighofer. Learning gaussian processes from multiple tasks. In Proceedings of the 22nd international conference on Machine learning, pp. 1012-1019, 2005.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.
Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06755 2020.
Yi Zhang, Yu Zhang, and Wei Wang. Deep multi-task learning via generalized tensor trace arXiv preprint arXiv:2002.04799 047999 2020.
Yu Zhang and Qiang Yang. A survey multi-tase task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. arXiv preprint arXiv: arXi::1702.08396 2017.
Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees GM Snoek, and Ling Shao. Learning learn variational semantic memory. arXiv preprint 2rXiv:2010.10341, 2020.
13

---

Under review as a conference paper at ICLR 2022
A DERIVATION OF THE ELBO FOR MULTI-TASK NEURAL PROCESSES
We provide derivation of ELBO of the proposed multi-task processed with hierarchical context modeling. The likelihood of multi-task learning is as follows:
((0)))(77)))MM)
II (x) (Di))
(12)
P(Y7XX II S
Based on conditional independence assumption, we introduce the variational joint posterior distri- bution factorized as (7). By incorporating the variational posteriors the log likelihood, we can obtain the ELBO as follows:
log p( (XT))IX)))
log
((XiIX)) vi) (vhi)a, M)po, (oo Di)douddbr
Σ log / PXXXXX p shipo, (bbiaa M) O51 (1b1 (D])
2r0 (1, (ax) daa qos (au Di)
debi (ax (axi
go, (2b) D
1=1
Σ (au (n7) f P(XF)X) vi)po, (Vilaa, M) 9p ( (Di) Oi (1b) Di) depi)
(o
Eaz (on DDF (w/17)) ( bb)) DRL la (wbr) (151)an, M)]]
1=1
DKLL Daule:/aa:1D)))pppaaaDD....
(13)
B PROOF OF EXCHANGABILITY AND CONSISTENCY
We further provide theoretical proof to show that the proposed multi-task neural processes are valid stochastic processes, which completes the theory of multi-task neural processes. the statement in Garnelo et al. (2018b): the conditions, including (finite) exchangeability and consistency, are suf- ficient to define stochastic process. our multi-input multi-output setting, we observe L tasks, [X7] where X = 1 and {Y,} where Y; = {.r,,) 27,, denotes the i-th target samples from task / and vf, is its corresponding target or label. Here are the two propositions to state the ex- changeability and consistency of the proposed multi-task neural processes. We model the functional posterior distribution of the stochastic process by approximating the joint predictive distribution each target set (( (Di)), which is conditioned on all context samples {D)).
Proposition 1 (Exchangability) For finite n Σt. ni, T {m)te a permutation of (1, n where Ti is a permutation of the corresponding order set {1, (1,..., ni then:
p(
(D:3) =x(XX))(X)))
(14)
where (((11)) = m (m (Xi), FL (1 (Xi)) "*
(02t0))-.. .... Fn(m) and ((x;1) =
14

---

Under review as a conference paper at ICLR 2022
Proof.
p( ((T((T)))<((X1)) (Di))
- -/1 pl F
= =/1
L
plu 1) (1) v1)
IIvosso p(pbbb Mi) (II p(au) Di) d dep..dden.. (15)
1=1
L Th
/
PlUl 41) [II p( on M)) (II p(aul doa
p( ((
Proposition 2 (Consistency) Given m = = m1=1, if m S n for each task 1 S my S ni.
then:
/ + (DIT)..DDD))LLCC D = = ((X) (Di)),
(16)
where (TYT))m ((Y (1mm)mmm: ..... (Yi)ımm.) = (vi,..., Vom) and ((X7))m
((Xi) 1:m (Xi)amm)) (zf...., "e.........) mm)
Proof.
/ (XXx))(x),,
=
I p v122)
(Ire M)) (íI p( (epii)ddprraa dep (YY))))
p(ul lact vb...))
II
plu lat (b..t)(((i)) mm+
i=m+1
L p(aa debi.ddaa..
I p( on, M))
=/ʃ
P[ M)) (Io()))D)) deb deh1.
II p(u (/..)
p(((1 ((Y7))) (((x7) M)
(17)
C MORE EXPERIMENTAL DETAILS
Details of iteration numbers and batch sizes for different benchmarks are provided in Table 6. In each batch, the number of training samples from each task and category is identical. We train all models and parameters by the Adam optimizer Kingma & Ba (2014) using an NVIDIA Tesla V100 GPU. The learning rate is initially set le 4 and decreases with factor of 0.5 every 3, 000 iterations. The network architectures the proposed multi-task neural processes for multi-task classification are provided follows.
15

---

Under review as a conference paper at ICLR 2022
Table 6: The iteration numbers and batch sizes on different datasets, where C' and denotes the number of classes and tasks in the specific dataset, respectively.
Dataset
Iteration
Batch size
Office-Hane
15, 000
8
Office-Caltech
15, 000
8+C
ImageCLEF
15, 000
8+C+
Table 7: The architecture of inference networks ş1(() for latent variable ab of multi-task neural processes.
Output size
Layers
4096
Input feature
4096
Dropout (p=0.7)
4096
Fully connected, ELU
4096
Fully connected, ELU
4096
Reparameterization to pha 0
Table 8: The architecture inference networks (0(()) for the latent variable a of multi-task neural
processes.
Output size
Layers
1096
Input feature
4096
Dropout (p=0.7)
2048
Fully connected, ELU
2048
Fully connected, ELU
2048
Reparameterization to Na o²
Table 9: The architecture the neural network f(-) of multi-task neural processes.
Output size
Layers
2048
Input feature
1024
Fully connected, ELU
512
Fully connected, ELU
L
Fully connected
T
Normalization
4096
Multiply with the global variable
The architecture of the inference network ip1 is provided in Table 7. The architecture of the inference network &22 is provided Table 8. We note that the inference network 01 and 0₂ share the same architectures with (P: and (22, respectively. The architecture of the neural network h is provided in "abbl 9. The network is needed because provides a data-driven way for the model to incorporate the task-speeific latent variable ay and the global variable M, which are usually defined in different feature spaces, During inference, we apply the reparameterization trick to generate the samples for the latent variables (Kingma & Welling, 2013).
D MORE EXPERIMENTAL. RESULTS
D.1 MULTI-TASK REGRESSION AND CLASSIFICATION WITH THE 20% SPLIT
Further, provide experiments results the three multi-task classification datasets with 20% training samples in Table 10, and 12. The proposed multi-task neural processes consistently achieve the best performance all three benchmarks.
16

---

Under review as a conference paper at ICLR 2022
Table 10: Performance comparison of different methods the Office-Home dataset with 20% training samples.
Methods
A
C
P
R
Avg.
Single task learning
54.6104
50.6104
81.3=002
73. 10.3
64,940.1
Bakker & Heskes (2003)
61.3+002
56.5+02
81.7=0.3
75,4+0.2
68,7+002
Long et al. (2017)
65.1
46.7=0.2
79.9=0.3
76.6:40.
67.1
Kendall et al. (2018)
59.5+03
53.8+0.3
80.1=0.1
73.6=00.4
66.8+0.2
Qian et al. (2020)
58.3102
53.510.3
79.8
73.1
66.2.10.1
Multi-task neural processes
64.2.10.1
55.710.3
82.6:10.
77.2.10.3
69.9
Table 11:
Performance comparison of different methods on the Office-Caltech dataset with 20% training samples.
Methods
A
W
D
C
Avg.
Single task learning
94.9102
92.8104
95.21006
86.7106
92.4.10.
Bakker & Heskes (2003)
95.2+02
94.4+04
99.5=0.3
91.3++0.1
+0.1
Long et al. (2017)
95.5+003
94.9=0.1
99.2=0.3
91.0+04
10.1
Kendall et al. (2018)
95.41107
93.2+0.4
99.2.10.4
91.2110.
94.7:0.3
Qian et al. (2020)
95.7+00
94.1
99.2.10.5
91.14004
95.010.
Multi-task neural processes
94.940.
96.61n2
99.240.4
92.34002
95.740..
Table 12:
Performance comparison of different methods on the ImageCLEF dataset with 20%
training samples.
Methods
I
P
B
Avg.
Single task learning
92.9+0.6
84.6+0.3
72.5
54.610.
76.2+0.3
Bakker & Heskes (2003)
94.4+00
90.6+04
74.2
57.940.
7
Long et al. (2017)
94.4::04
89.2+0.5
75.8
59.410.
79.7 79.7+0.3
Kendall et al. (2018)
93.3404
91.0+u02
75.6=10.2
56.9404
79.2+0.3
Qian et al. (2020)
93.1
1
74.4=00.
55.8+006
78.9
Multi-task neural processes
92.1100
91.5
79.2.104
60.6110.3
80.8
D.2 MULTI-TASK REGRESSION WITH LESS DATA
To show the advantages our model, we compare them on the setting less data with the 0.05% split. In this case, there are only 20 samples task during training. As shown in the Table 13, the improvement of our method becomes larger.
Table 13: Performance (Average NMSE on Rotated MNIST(0.05% split).
Aug.
0.214 souee
MTNP
0.185
0.152 0.1699
0.17333
0.138
0.106
0.152anmm
sove?
0.165
D.3 STABILITY OF OUR METHOO
The computational advantage our method can be illustrated by the training loss as function of iteration Home. As shown in Fig. 4, our MTNPs converges stable than NPs under 5%. 10% and 20% train-test splits. Moreover, we have added a new experiment to investigate the training stability by introducing the noise to the input data.
17

---

Under review as a conference paper at ICLR 2022
MTNPs, 5%
MTNPS, 20%
NPs. 5%
10%
NPs. 20%
Treining loss
250
750
1000
1250
1900
Iteration
Figure 4: Ilusstationn of training loss with iterations on Office Home. MTNPs converges more stable than NPs under 5%, 10% and 20% train-test splits.
We apply the the fast gradient sign method (Goodfellow al., 2014) generate the noise. The results are given in Table 14, where denotes the noise level. We observe that our MTNPs show better stability than NPs different noise levels.
Table 14: The performance under different noise level the Of f dataset with 5% split.
7
0.0
0.2
0.4
0.6
0.8
1.0
NPs MTNPs
92.6
89.5
86.6
82.7
77.7
75.1
94.5
91.2
87.9
84.3
82.5
81.0
D.4 SENSITIVITY OF THE HYPERPARAMETER Ny AND N
In practice, we set y and Na to be 10 and which offer a good balance between performance and efficiency. We determine them by grid search as shown in Table 15 and 16.
Table 15: Sensitivity of N2 (and N₂ 5) -ffice-Home with the 5% split.
N Avg.
5
10
20
30
59.8.10.1
59.9.1..1
60.0
59.9.10.1
60.0
18

---

Under review as a conference paper at ICLR 2022
Table 16: Sensitivity of Na (and Ny = 10) on Office Home with the 5% split.
Na
10
30
Avg.
59.8
60.0.10.1
59.6.10.1
59.910.1
19.8
19


Reviews:
[{"review_id": "yotQ06EF92v", "reviewer": "Reviewer_Ef49", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nEven though rather limited, the proposed method shows better performance over compared methods for most of the learning tasks in their experiment. \n\nWeakness: \nThe paper is poorly written, many places lack of clarity. (1) The description of the method is very confusing. In the main text, on page 4, it seems to suggest the \\mu of and \\Sigma of \\psi are functions of m_l. However, in the network specifications in appendix, it seems \\mu of and \\Sigma of \\psi has nothing to with m_l, because the input size of the \\psi_1 network is 4096 but output size of h network is L. In addition, it is not clear why the h network is even needed. I did not see it is involved in either Eq. (12) or (13) to make predictions. (2) In section 3.1 it is mentioned that concrete formation of global variable M depends on the learning scenarios. However, it is not made clear anywhere in the paper how M is constructed from the context data. (3) It is unclear the multi-task setup in the brain image segmentation experiment. (4) How many Monte Carlo samples, i.e., N_f and N_a are drawn for the variational posteriors \\psi and \\alpha? And how they were determined?\n\nAre networks \\psi_1 and \\psi_2 common networks shared among all tasks? Or are they task specific? If both are task specific, I do not see mathematically it would make much difference to train one single network comparing to the two that the authors chose to do. In other words, I do not see a clear value of proposed hierarchical design mathematically. More discussion on this would be very helpful. \n\nMore empirical studies to support their claims are desired from both aspects of the methods to compare and learning tasks to test on. Most of the recently published multi-task learning methods were neither discussed or compared, for example Quo et. al., 2020, ICML, Sun et. al., 2020, NeurIPS, and Yu et. al., 2020 NeurIPS. The learning tasks used are limited to image recognition related tasks. The results could be more persuasive if tasks from diverse domains are included.   \n", "summary_of_the_paper": "This paper presents a multi-task neural processes approach in which function priors are derived in a hierarchical Bayesian inference framework to incorporate the shared knowledge into its context of the prediction function. Authors introduced a higher-level latent variable derived from the context data of related tasks to control the sharing of common knowledge between the tasks. Previous works used context data from its own task to generate the task-specific latent variables, whereas this work uses data from other tasks as well. The shared knowledge from all the tasks added to context of each individual task acts as inductive bias for predictions. Their experiment results show better performance of the proposed method than compared ones. ", "summary_of_the_review": "The paper is poorly written. The motivation of the proposed formulation is not clear. The empirical study is not adequate. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "qftnt-IJ2gt", "reviewer": "Reviewer_fDzq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nThe general idea of the model seems to be a valid one for multitask learning, and to my knowledge, it seems original.\n\nFor the most part, the experimental section shows the strength of this model and the performance gains it brings against some other models in the literature. I particularly liked the last example on Brain Image segmentation.\n\n**Weaknesses:**\n\nI was a bit confused with the model construction. What exactly is the global variable $\\mathbf{M}$? Is it simply the collection of the context datasets for all the tasks? Or is it a latent variable? If it is a latent variable, why do you opt for doing MAP estimation instead of placing a prior over it? I think this needs to be clarified further in the manuscript.\n\nThere are also some errors in the model construction section. The Gaussian likelihood is used to derive the generic multitask model. While this is a reasonable choice for regression tasks, it is a wrong assumption for classification tasks which are also considered in this paper. The inference section seems to have fixed this problem though. I suggest that section 3 is revisited to correct those discrepancies.\n\nI would have liked to see a synthetic example exploring the properties of this model. The experiments jump straight to comparison on real-world datasets, but it would have been better to first start with a synthetic dataset to validate the modelling choices in Section (3).\n\nTable (5) is missing confidence intervals. It also looks like the performance difference might not be significantly different from NPs for this example. I think this should be discussed.\n\nQuestion: do the nested expectations in eq (11) cause any problems with the variance? How stable is the training of this model in comparison to standard neural processes?\n", "summary_of_the_paper": "The paper proposes a multi-task learning model with neural processes. The model is based on a hierarchical construction whereby each task is conditioned on global and local information. The paper derives a “hierarchical” ELBO for this model that is evaluated with MC sampling. Experiments are presented to validate the proposed model.", "summary_of_the_review": "In my opinion, this work is original and has the potential to be an impactful contribution. However, in its current state, it is not ready yet. My main issues are the ambiguity in the model construction exposition and the lack of experiments that explore the model’s behaviour and properties.\n\n**********\nEdit: My score was updated to 6 after the rebuttal.\n**********", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g_WcXXhLDX7", "reviewer": "Reviewer_H8nA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Many Bayesian models have been proposed for multi-task learning. Why does the proposed multi-task neural process achieve superior performance? It is better to compare different models. \n\nThe Gaussian likelihood is not suitable for classification tasks. Authors should consider other likelihood function.\n\nIn experiments, important baselines are missing. Authors should compare with MTL models such as cross-stitch network and MTAN, which can be modified to solve the multi-input multi-output setting. Moreover, multi-task Gaussian process should be compared.", "summary_of_the_paper": "This paper proposes a multi-task neural process as a multi-task Bayesian model. ", "summary_of_the_review": "This paper proposes a multi-task Bayesian model. Compared with existing multi-task Bayesian models, I don't know why the proposed model is better. The proposed model is not suitable for classification tasks. Important baselines are missing in experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MihxQW2hbF", "reviewer": "Reviewer_3xsW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strong point is the extensive experiments. The authors have done extensive experiments to evaluate the proposed idea on various tasks. The results are very promising and significant. \n\nThe weakness is possibly the weak theatrical contribution. The hierarchical model design and inference are straightforward. The authors did not compare with other possible solutions to show why this design is the best. The design of the original NP is guaranteed to be a valid stochastic process. It is pity that this discussion is missed for multi-task NP. Is multi-task NP still a valid stochastic process? is any marginal NP on a single task in multi-task NP still a valid stochastic process? ", "summary_of_the_paper": "This paper proposes a new multi-task neural process based on the classical neural process. The idea is to introduce additional global variables to share knowledge from different tasks. That is to say, a hierarchical bayesian model is constructed to link single-task neural processes together. The new model is able to handle multi-task applications contract to the classical neural process", "summary_of_the_review": "The paper is clearly organized and written. It is easy to follow. The results are very impressive. Although some important discussions are missed, this contribution of this work is good. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends neural processes (NPs) to the multi-task setting (MTNPs). The approach uses a hierarchical Bayesian construction, where the latent variables of an NP are conditioned on a set of global task-specific context variables. This allows the NP to share knowledge across related tasks.

There were a few issues raised in the reviews. Consistently, the reviewers noted that the writing could be improved. There were variables, like the context variables M, that lacked explanation. There was also confusion between the use of a Gaussian likelihood for classification vs regression. These were resolved with the author’s response and updated draft.

There were also requests for additional experiments and baselines: 1) a synthetic task, to which the authors included a 1D regression task. 2) More baselines against other multi-task methods, to which the authors included a comparison to Guo et al., 2020, MTAN, and multi-task Gaussian Processes.

Finally, there were questions around whether MTNPs are valid stochastic processes. This has been proven theoretically by the authors, albeit in the appendix.

Currently, this paper remains borderline. The main remaining criticisms are a) A desire for more experiments and analysis to highlight the particular strengths of the approach. b) That the approach is a straightforward extension of NPs, and may not be sufficiently novel. c) That the authors include more baselines from the recent multi-task literature (Yu et al. and Sun et al.). In the end it was determined that the paper does not quite meet the bar for acceptance. I think in future submissions, it would be worthwhile to further highlight MTNP’s performance in the low-data regime, where it particularly seems to do well, and to complete the full set of comparisons (e.g., Sun et al. and Yu et al.) that were requested by the reviewers.
Decision: Reject