Year: 2021

Abstract:
Neural processes have recently emerged as a class of powerful neural latent variable models that combine the strengths of neural networks and stochastic processes. As they can encode contextual data in the network’s function space, they offer a new way to model task relatedness in multi-task learning. To study its potential, we develop multi-task neural processes, a new variant of neural processes for multi-task learning. In particular, we propose to explore transferable knowledge from related tasks in the function space to provide inductive bias for improving each individual task. To do so, we derive the function priors in a hierarchical Bayesian inference framework, which enables each task to incorporate the shared knowledge provided by related tasks into its context of the prediction function. Our multi-task neural processes methodologically expand the scope of vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning. The proposed multi-task neural processes are capable of learning multiple tasks with limited labeled data and in the presence of domain shift. We perform extensive experimental evaluations on several benchmarks for the multi-task regression and classification tasks. The results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning and superior performance in multi-task classification and brain image segmentation1.

text: Neural processes have recently emerged as a class of powerful neural latent variable models that combine the strengths of neural networks and stochastic processes. As they can encode contextual data in the network’s function space, they offer a new way to model task relatedness in multi-task learning. To study its potential, we develop multi-task neural processes, a new variant of neural processes for multi-task learning. In particular, we propose to explore transferable knowledge from related tasks in the function space to provide inductive bias for improving each individual task. To do so, we derive the function priors in a hierarchical Bayesian inference framework, which enables each task to incorporate the shared knowledge provided by related tasks into its context of the prediction function. Our multi-task neural processes methodologically expand the scope of vanilla neural processes and provide a new way of exploring task relatedness in function spaces for multi-task learning. The proposed multi-task neural processes are capable of learning multiple tasks with limited labeled data and in the presence of domain shift. We perform extensive experimental evaluations on several benchmarks for the multi-task regression and classification tasks. The results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning and superior performance in multi-task classification and brain image segmentation1.

heading: 1 INTRODUCTION
text: As deep neural networks are black-box function approximations, it is difficult to introduce prior domain or expert knowledge into a prediction function (Jakkala, 2021). In contrast, Gaussian processes (Rasmussen, 2003) explicitly define distributions over functions and perform inference over these functions given some training examples. This enables reliable and flexible decisionmaking. However, Gaussian processes can suffer from high computational complexity due to the manipulation of kernel matrices. Therefore, there has been continuous interest in bringing together neural networks and Gaussian processes (Damianou & Lawrence, 2013; Wilson et al., 2016; Garnelo et al., 2018a; Jakkala, 2021) into so-called neural processes.
Neural processes (Garnelo et al., 2018b) combine the computational efficiency of neural networks with the uncertainty quantification of stochastic processes. They are a class of neural latent variables model, which deploy a deep neural network to encode context observations into a latent stochastic variable to model prediction functions. Neural processes provide an elegant formalism to efficiently and effectively incorporate multiple datasets into learning distributions over functions. This formalism is also promising in multi-task learning to improve individual tasks by transferring useful contextual knowledge among related tasks. Their capability of estimating uncertainty over predictions also makes them well-suited for multi-task learning with limited data, where each task has only a few training samples. However, neural processes rely on the implicit assumption that the context and target sets are from the same distribution and can be aggregated by a simple average pooling operation (Kim et al., 2019; Volpp et al., 2020). This makes it non-trivial to directly apply neural processes to modeling multiple heterogeneous tasks from different domains, where the context data of different tasks are from distinctive distributions (Long et al., 2017).
1 Our code is available in https://anonymous.4open.science/r/Multi-Task-Neural-Processes/.
In this paper, we develop multi-task neural processes (MTNPs), a methodological extension of neural processes for multi-task learning, which fills the theoretical gap of neural processes for multi-task learning. Particularly, we propose to explore task relatedness in the function space by specifying the function priors in a hierarchical Bayesian inference framework. The shared knowledge from related tasks is incorporated into the context of each individual task, which serves as the inductive bias for making predictions in this task. The hierarchical architecture allows us to design expressive datadependent priors. This enables the model to capture the complex task relationships in multitask learning. By leveraging hierarchical modeling, multi-task neural processes are capable of exploring shared knowledge among related
tasks in a principled way by specifying the function prior.
We validate the effectiveness of the proposed multi-task neural processes by extensive experiments in both multi-task classification and regression. The results demonstrate that multi-task neural processes can effectively capture task relatedness in the function space and consistently improve the performance of each individual task, especially in the limited data regime.

heading: 2 PRELIMINARIES: NEURAL PROCESSES
text: In this section, we briefly review vanilla neural processes (Garnelo et al., 2018b) based on which we derive our multi-task neural processes.
Let a data set be given composed of training samples and corresponding labels. In order to better reflect the desired model behaviour at test time (Garnelo et al., 2018b), the training data is split into the context set D = (X,Y) and a target set D∗ = (X∗,Y∗), where X = {x1, · · · ,xn} is a subset of the training data, Y = {y1, · · · ,yn} the corresponding set of labels and n the size of the context set. To ensure both sets have the same data distribution, the context set is split from the target set. Now, given the context set (X,Y), we would like to estimate a function that can make predict the labels Y∗ for target samples X∗.
In general, we define a stochastic process by a random function f : X → Y . Given the context set, we define the joint distribution over the function values {f(x1), · · · , f(xn)}, which in Gaussian processes is a multivariate Gaussian distribution parameterized by a kernel function. The rationale of neural processes is to extract knowledge from the context set to specify the prior over the prediction function. Instead of a kernel function, neural processes adopt a deep neural network to define the prior distribution. Specifically, the model introduces a latent variable z to account for uncertainty in the predictions of Y∗. The observed context set is encoded into the latent variable which is conditioned on the context set, i.e. it follows the prior distribution p(z|X,Y). The latent variable z is a high-dimensional random vector parameterising the stochastic process by f(X∗) = g(X∗, z). The function g(·, ·) is an extra fixed and learnable decoder function, which is also implemented by a neural network. Thus, the neural process model can be formulated as follows:
p(Y∗|X∗,X,Y) = ∫ p(Y∗|X∗, z)p(z|X,Y)dz. (1)
The graphical model for neural processes is shown in Figure 1 (a).
The neural process model is optimized using amortized variational inference. Let q(z|X∗,Y∗) be a variational posterior of the latent variable z. The evidence lower-bound (ELBO) for neural processes is given as follows:
log p(Y∗|X∗,X,Y) ≥ Eq(z|X∗,Y∗)[p(Y∗|X∗, z)]− DKL[q(z|X∗,Y∗)||p(z|X,Y)]. (2)
In neural processes, the function space defined by deep neural networks allows the model to extract deep features while retaining a probabilistic interpretation (Jakkala, 2021). In multi-task learn-
ing, usually different tasks can be from different domains and have their specific data distributions (Lawrence & Platt, 2004; Long et al., 2017). Due to the complex data structure of multi-task learning, it is not straightforward to explore task relatedness in such function spaces. In this paper, we aim to extend the methodology of the neural process to the scenarios of multi-task learning to learn the shared knowledge among tasks for improving individual tasks.

heading: 3 MULTI-TASK NEURAL PROCESSES
text: The common setup of multi-task learning is that there are multiple related tasks for which we would like to improve the learning of each individual one by sharing information across the different tasks (Williams et al., 2007). Multi-task learning has been studied under different settings (Requeima et al., 2019; Williams et al., 2007; Lawrence & Platt, 2004; Yu et al., 2005; Long et al., 2017). In this paper, we tackle the multi-input multi-output setting, where each task has a different distribution while different tasks share the same target space (Lawrence & Platt, 2004; Yu et al., 2005; Long et al., 2017; Zhang et al., 2020). It aims to improve the overall performance of all multiple tasks simultaneously different from the sequential multi-task leaning (Requeima et al., 2019; Garnelo et al., 2018a). This is a challenging scenario due to the domain shift between tasks, which makes it sub-optimal to directly apply neural processes by incorporating the data from other related tasks into the context of each individual task.

heading: 3.1 HIERARCHICAL CONTEXT MODELING
text: Multi-task learning considers the estimation of random functions fl, l = 1, ..., L for each of the L related tasks. Each task l has its own training data, which is split into a context set Dl = (Xl,Yl) and a target set D∗l = (X ∗ l ,Y ∗ l ). X ∗ l ∈ Rn ∗ l×d and Xl ∈ Rnl×d are inputs while Y∗l ∈ Rn ∗ l×C and Yl ∈ Rnl×C are outputs. d and C are the sizes of the input space and output space, respectively. n∗l and nl are the sizes of respectively the target and context set for the l-th task. Thus, we obtain the task-specific latent variables fl = fl(Xl) ∈ Rnl×C . We use {Dl}Ll=1 to denote all context sets in the dataset, which for brevity we represent as {Dl} and likewise will do for other sets. Formulated this way the goal of multi-task learning is to predict {Y∗l } for given {X∗l } simultaneously with the assistance information {Dl} from all tasks. To this end, we construct a joint prediction distribution with respect to the latent random functions {fl} as:
p({Y∗l }|{X∗l }, {Dl}) = L∏ l=1 p(Y∗l |X∗l , {Dl}) = L∏ l=1 ∫ p(Y∗l |X∗l , fl)p(fl|M)dfl. (3)
Here, to enable shared knowledge to be transferred among tasks, we introduce a global variable M which works as a container to collect the useful information from {Dl} of all tasks. In contrast to neural processes for single tasks, the global variable M provides the contextual information from all tasks for each individual task. The concrete formation of M depends on the learning scenarios. For regression tasks, M ∈ RL∗d and each row corresponds to one task, which is the average of all feature vectors from a task. For classification tasks, M ∈ RL∗C∗d where each vector Ml,c is the average of all features of each category from the corresponding task.
Similar to Gaussian processes, we assume that the function value is Y∗l = fl(X ∗ l ) + , where
∼ N (0, σ2) is the observation noise. For regression tasks, we can define the predictive likelihood on the target set as p(Y∗l |X∗l , fl) = N (Y∗l |fl(X∗l ), σ2). For classification tasks, we use log p(Y∗l |X∗l , fl) = ∑n∗l i=1 y ∗ l,ilog(fl(x ∗ l,i)) as the log-likelihood function, where x ∗ l,i is the i-th target sample from the l-th task.
In order to combine Gaussian processes and neural networks in the context of multi-task learning, we define p(fl|M) in (3) as a deep neural network in place of a Gaussian distribution parameterized by a kernel function. To be more specific, we assume that fl is parameterized by a random variable ψl by defining fl(X) = Xψ>l . We specify a data dependent prior by conditioning ψl on the global variable M. In this way, we incorporate the transferable knowledge into the learning of the prediction function of the current tasks. Thus, the predictive distribution for the l-th task over its target set can be formulated as follows:
p(Y∗l |X∗l , {Dl}) = ∫ p(Y∗l |X∗l ,ψl)pθ(ψl|M)dψl. (4)
In effect, ψl denotes the parameters of classifier for classification tasks or regressors for regression tasks. Particularly, the introduced latent variable ψl ∈ RC×d denotes the task specific classifier, where d is the dimension of the input feature and C is the number of classes in the dataset. As done in (Requeima et al., 2019), we generate each column of ψl independently from the context samples of the corresponding class. In our case, each column of ψl encodes the context information of its class from all tasks pθ(ψl|M) = ∏C c=1 pθ(ψl,c|Mc).
Directly aggregating M as done in neural processes for single tasks is not applicable for multi-task learning due to the distribution shift between tasks. The data from related tasks should be processed and adapted to the current task as the contextual information. To this end, we introduce a higherlevel latent variable αl to extract the shared knowledge from M, which is conditioned on the data Dl of each task. This results in a hierarchical Bayesian modeling of functions in the neural process:
pθ(ψl|M) = ∫ pθ1(ψl|αl,M)pθ2(αl|Dl)dαl, (5)
whereαl is the latent variable to control the access to shared knowledge for each task, which is used to explore the relevant knowledge to the task l. pθ1(ψl|αl,M) and pθ2(αl|Dl) are prior distributions of the latent variable ψl and αl, respectively, which are parameterized with neural networks. To be specific, we define pθ1(ψl|αl,M) = N (ψl|µ(ml),Σ(ml)). Here ml contains the relevant knowledge to the task l, which is adapted from the global variable M by a deterministic function ml = h(αl,M), where h(·) is a learnable function implemented with a neural network. By substituting (5) into (4), we obtain the model of multi-task neural processes as follows:
p({Y∗l }|{X∗l }, {Dl}) = L∏ l=1 ∫ ∫ p(Y∗l |X∗l ,ψl)pθ1(ψl|αl,M)pθ2(αl|Dl)dψldαl. (6)
The designed hierarchical context modeling provides a principled way to explore task relatedness in the function space, which allows task-specific function variables to leverage the shared knowledge from related tasks. We provide theoretical proof in Appendix to show that the proposed multi-task neural processes are a valid stochastic processes, which completes the theory of multi-task neural processes. The graphical model of the multi-task neural processes is shown in Figure 1 (b).

heading: 3.2 VARIATIONAL HIERARCHICAL INFERENCE
text: The previous section developed the model of multi-tasks neural processes with a hierarchical context model. We now describe how to optimize the model and obtain the predictions by leveraging a variational Bayesian inference framework. To that end, based on a conditional independence assumption, we introduce the variational joint posterior distribution factorized as follows:
qϕ({ψl}, {αl}|{D∗l }) = L∏ l=1 qϕ(ψl,αl|D∗l ) = L∏ l=1 qϕ1(ψl|D∗l )qϕ2(αl|D∗l ), (7)
where qϕ1(ψl|D∗l ) and qϕ2(αl|D∗l ) are variational posteriors of the latent variables ψl and αl for the task l, respectively. Both variational posteriors are parameterized as Gaussian distributions. ϕ1 and ϕ2 are amortized inference networks to generate variational posteriors and shared by all tasks. We make use of the amortized variational inference technique (Kingma & Welling, 2013) to learn such distributions over latent variables.
Learning By incorporating the variational posteriors into (7), we derive the ELBO for the multitask neural processes as follows:
log p({Y∗l }|{X∗l },{Dl}) ≥ L∑ l=1 { Eqϕ2 (αl|D∗l ) { Eqϕ1 (ψl|D∗l )[log p(Y ∗ l |X∗l ,ψl)]
− DKL[qϕ1(ψl|D∗l )||pθ1(ψl|αl,M)] } − DKL[qϕ2(αl|D∗l )||pθ2(αl|Dl)] } .
(8)
The detailed derivation is provided in Appendix A. By adopting the Monte Carlo sampling, we obtain the empirical objective for the proposed multi-task neural processes:
L̂MTNPs(θ, ϕ) = L∑ l=1 { 1 Na Na∑ i=1 { 1 Nf Nf∑ j=1 [− log p(Y∗l |X∗l ,ψ (j) l )]
+ λfDKL[qϕ1(ψl|D∗l )||pθ1(ψl|α (i) l ,M) } + λaDKL[qϕ2(αl|D∗l )||pθ2(αl|Dl)] } ,
(9) where ψ(j)l ∼ qϕ1(ψl|D∗l ) and α (i) l ∼ qϕ2(αl|D∗l ). Nf and Na are the number of Monte Carlo samples for the variational posteriors of ψ and α, respectively. λf and λa are the hyperparameters to help stably train the KL-divergence terms, which are set following the annealing scheme of (Bowman et al., 2015). In practice, we apply the local reparameterization trick (Kingma et al., 2015) to reduce the variance of stochastic gradients.
Prediction Having the learned model, we can make prediction on the target set. Given a test sample xl from the l-th task, we can produce the predictive distribution which involves the prior distributions pθ1(ψl|αl,M) and pθ2(αl|Dl). The predictive distribution with the integration for the introduced latent variables is formulated as:
p(yl|xl, {Dl}) = ∫ ∫ p(yl|xl,ψl)pθ1(ψl|αl,M)pθ2(αl|Dl)dαldψl (10)
Here we also need to apply the Monte Carlo estimation over (10) and obtain the predictions as follows:
p(yl|xl) ≈ 1
Na Na∑ i=1 1 Nf Nf∑ j=1 p(yl|xl,ψ(j)l )], (11)
where ψ(j)l ∼ pθ1(ψl|α (i) l ,M) and α (i) l ∼ pθ2(αl|Dl). In particularly, the prior distribution of the latent variable ψl is generated from the output of the learned function h(α (i) l ,M).

heading: 4 RELATED WORK
text: Neural network based models have achieved impressive results on various applications (LeCun et al., 2015). However, due to the large number of parameters these neural models need large-scale data with annotation. It is challenging to train a deep neural model that generalizes well with numberlimited labeled data. To reduce such labeling consumption, recent works (Long et al., 2017; Liu et al., 2016) follow a multi-task learning strategy to fully leverage information from relevant tasks as inductive bias (Caruana, 1997) to improve each task’s performance. The main challenge of multitask learning is that each task is provided with a limited amount of labeled data, which is insufficient to build reliable classifiers without overfitting (Long et al., 2017).
Multi-task learning (MTL) aims to learn several tasks simultaneously and improve their overall performance. The crux of MTL is how to explore task relatedness from different tasks, which could be particularly significant when limited data for each task is available (Long et al., 2017; Zhang et al., 2020). Recently, the task relatedness is learned in many different aspects of the model, e.g., loss functions (Liu et al., 2020; Qian et al., 2020; Kendall et al., 2018), parameter space (Long et al., 2017; Bakker & Heskes, 2003), or representation space (Misra et al., 2016). In this paper, we focus on the data-insufficient problem for multi-task learning. Different from other branches of MTL models (Huang et al., 2021; Fu et al., 2021; Phillips et al., 2021) which leverage several channels of supervision information simultaneously included by the same input, our data setting allows each task to have its own individual data. This is much more challenging due to the distribution shift between inputs of different tasks.
In previous works (Lawrence & Platt, 2004; Yu et al., 2005; Yousefi et al., 2019), multi-task learning benefits from Gaussian Processes by generally incorporating the shared information in the Gaussian Processes prior to synergize several random functions of different tasks simultaneously. The multitask Gaussian processes (Yu et al., 2005) proposes a hierarchical prior which enables task specific random functions to share some common structure in the hyper prior. As an alternative to using
Gaussian processes for multi-task learning, nowadays deep neural networks have become popular as parameterized functions, which construct an information sharing architecture between tasks. Some methods (Misra et al., 2016; Liu et al., 2019) learn expressive combinations of features from different tasks by deep neural networks. Some approaches (Gao et al., 2020; Sun et al., 2019) flexibly adjust the deep architecture for each individual tasks. Yu et al. (2020) mitigates gradient interference by performing the proposed gradient surgery. However, these deep multi-task learning methods relies on large amounts of training data and therefore tends to overfit with limited data.
Recently, neural processes (Garnelo et al., 2018b) and the related works (Garnelo et al., 2018a; Kim et al., 2019; Wang & Van Hoof, 2020; Requeima et al., 2019; Gordon et al., 2019) combine Gaussian Processes and neural networks, which are not only computationally efficient but also retain a probabilistic interpretation of the model. (Garnelo et al., 2018b; Kim et al., 2019; Wang & Van Hoof, 2020) introduce the latent representation variables to model the function randomness. (Requeima et al., 2019; Gordon et al., 2019) introduce a latent parameter variable to directly model the predictive distribution, which acts as the parameters of the neural network.
Hierarchical modeling in the Bayesian framework has been successful to design the form of the prior (Daumé III, 2009; Zhao et al., 2017; Klushyn et al., 2019; Wang & Van Hoof, 2020) and posterior distributions (Ranganath et al., 2016; Krueger et al., 2017; Zhen et al., 2020) based on many observations. It allows the latent variable to follow a complicated distribution and forms a highly flexible approximation (Krueger et al., 2017).

heading: 5 EXPERIMENTS AND RESULTS

heading: 5.1 1-D MULTI-TASK FUNCTION REGRESSION
text: Our method fully utilizes the transferable knowledge provided by related tasks to improve each task’s performance. To show this, we test the proposed multi-task neural processes on the 1-D multi-task function regression.
Setup. We define several tasks with different data distributions: inputs of each task are sampled from the separated intervals without overlapping, such as [−2π, π), [−π, 0), [0, π), and [π, 2π). Each x-value is drawn uniformly at random in its belonging intervals. We assume that all tasks share the same ground truth function to ensure that there is transferable knowledge between them. The ground truth function is characterized as a sum of sine or cosine functions (Wang & Van Hoof, 2020), such as y = sin(x)+sin(2x)−cos(0.5x)+ , where is the noise drawn fromN (0, 0.00032). At each training step, the hyperparameters of the ground truth function are fixed as set in (Kim et al., 2019; Wang & Van Hoof, 2020).
Results. As shown in Figure 2, we visualize prediction results of different tasks for the comparison of neural processes and multi-task neural processes. The predictions of the proposed multi-task neural processes (the second row) more resemble the ground truth functions than that of neural processes (the first row), especially in the boundary of different tasks. Moreover, we find that when training with fewer context samples, the improvement is more significant. This demonstrates that our method enhances the generalization of each task by fully utilizing the transferable knowledge from related tasks.

heading: 5.2 MULTI-TASK CLASSIFICATION
text: Datasets. We evaluate the performance of our multi-task neural processes on real-world multitask classification, where different tasks are defined as image classification problems in different domains. The tasks are related to each other since they share the same label space. Office-Home (Venkateswara et al., 2017) contains images from four domains/tasks: Artistic (A), Clipart (C), Product (P) and Real-world (R). Each task contains images from 65 categories collected under office and home settings. There are about 15, 500 images in total. Office-Caltech (Gong et al., 2012) contains the ten categories shared between Office-31 (Saenko et al., 2010) and Caltech-256 (Griffin et al., 2007). One task uses data from Caltech-256 (C), and the other three tasks use data from Office-31, whose images were collected from three distinct domains/tasks, namely Amazon (A), Webcam (W) and DSLR (D). There are 8 ∼ 151 samples per category per task, and 2, 533 images in total. ImageCLEF (Long et al., 2017), the benchmark for the ImageCLEF domain adaptation challenge, contains 12 common categories shared by four public datasets/tasks: Caltech-256 (C), ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Bing (B). There are 2, 400 images in total.
Setup. We adopt the standard evaluation protocols (Zhang & Yang, 2021) for multi-task classification datasets. We randomly select 5%, 10% and 20% labeled data for training, which correspond to about 3, 6 and 12 samples per category per task, respectively. In this case, each task has a limited amount of training data, which is insufficient for building the reliable classifier without overfitting. For all three benchmarks, we extract the input features by the pretrained VGGnet as (Long et al., 2017). The architectures of inference networks used in our model are provided in Appendix C. All the results with error bars are obtained based on a 95% confidence interval from five runs.
Compared methods. To show the effectiveness of the proposed multi-tasks neural processes, we conduct a thorough comparison implementing multiple different baseline models. Single task learning (STL) is implemented by task-specific feature extractors and classifiers without knowledge sharing among tasks. Basic multi-task learning (BMTL) shares feature extractors and adds task specific classifiers. We also define variational extensions of the single task learning (VSTL) and basic multitask learning (VBMTL), which cast models as variational Bayesian problems and treat classifiers as latent variables (Shen et al., 2021). We implement neural processes (NPs) and its variant, neural processes with all task context for comparison. NPs with all task context is a straightforward extension of NPs for MTL, which treats context sets from all tasks equally without hierarchical context modeling. For a fair comparison, all the above-mentioned methods share the same architecture of the feature extractor.
Results. We provide comprehensive comparisons on Office-Home in Table 1, which is a more challenging multi-task classification dataset with 65 categories. The results show that our MTNPs outperform other counterpart methods. NPs with all task context performs better than NPs by a large margin when 5% labeled data is available, showing the benefit of exploring shared knowledge from related tasks with limited data. Noticeably compared to NPs with all task context, the proposed multi-task neural processes benefit from hierarchical context modeling and show even better performance.
More comparison results on the Office-Home, Office-Caltech, ImageCLEF datasets are shown in Tables 2, 3 and 4, respectively. The average accuracy of all tasks is used for overall performance measurement. The best results are marked in bold. Our MTNPs achieve competitive and even better performance on such multi-task classification datasets with different train-test splits. Compared with Bayesian baselines, including VSTL, VBMTL, NPs and (Bakker & Heskes, 2003), our MTNPs directly infer the parameters of prediction functions rather than the input representation, which is able to model a broader range of functional distribution. Moreover, in function space the hierarchical context modeling can better explore the task relatedness, which enable the models to capture the relevant knowledge even in presence of distribution shift among tasks. Experimental results on all three benchmarks with 20% labeled data are provided in Appendix D.

heading: 5.3 MULTI-TASK REGRESSION
text: Setup. In order to show the effectiveness of MTNPs for multi-task regression, we conduct experiments on the Rotated MNIST dataset (LeCun et al., 1998). We adopt this dataset to study multi-task regression, where each task is an angle estimation problem for each digit and different tasks corresponding to different digits are related because they share the same rotation angle space. Each image is rotated by 0◦ through 90◦ in intervals of 10◦, where the rotation angle is the regression target. We randomly choose 0.1% samples per task per angle as the training set.
Results. Since we would like to improve the overall performance of all regression tasks, we use the average of normalized mean squared errors of all tasks as the measurement. As shown in Table 5, our MTNPs outperform other counterpart methods by yielding an overall lower mean error.

heading: 5.4 BRAIN IMAGE SEGMENTATION
text: In this section, we demonstrate that multi-task neural processes are also able to explore spatial context information to improve image segmentation. To this end, we adopt a brain image dataset (Buda et al., 2019) with lower-grade gliomas collected from 110 patients. The number of images varies among patients from 20 to 88. The goal is to segment the tumor in each brain image.
Setup. To apply our multi-task neural processes, we reformulate the segmentation task as a pixelwise regression problem, where each pixel corresponds to a regression task to predict the probability of this pixel belonging to the tumor. In doing so, the spatial correlation and dependency among pixels are effectively modeled by capturing the task relatedness. To be specific, we consider the prediction of each pixel to be a regression task. For the task l, we define Ωl as a local region centered at the spatial position l, which provides the local context information. In this case, the region centered at the pixel provides the local context information. Each task incorporates the shared knowledge provided by related tasks into its context of the prediction function. This offers an effective way to model the long-range interdependence of pixels in one image. For implementation, we use the U-Net architecture (Ronneberger et al., 2015) as the backbone and add our model as the final layer.
Results. We compare our method and U-Net on the brain segmentation dataset. The results show that the proposed multi-task neural processes surpass the baseline U-Net by 0.5% in terms of dice similarity coefficients (DSC) for the overall validation set. Figure 3 shows segmentation results of the proposed multi-task neural processes (bottom row) and the U-Net (upper row), where the green outline corresponds to the ground truth and the red to the segmentation output. Our multi-task neural processes predict contours closer to the ground truth. This demonstrates the advantages of exploring context information by multi-task neural processes for segmentation.

heading: 6 CONCLUSION
text: In this paper, we develop multi-task neural processes, a new variant of neural processes for multitask learning. We propose to explore the task relatedness in the function space by specifying the function priors in a hierarchical Bayesian inference framework. The shared knowledge from related tasks is incorporated into the context of each individual task, which serves as the inductive bias for making predictions of this task. The hierarchical architecture allows us to design expressive datadependent prior, enabling the model to explore the complex task relationships in multi-task learning. By leveraging the hierarchical modeling, multi-tasks neural processes are capable of capturing the shared knowledge from other tasks in a principled way by specifying the function prior. We evaluate multi-task neural processes on multi-task regression and classification datasets. Results demonstrate the effectiveness of multi-task neural processes in transferring useful knowledge among tasks for multi-task learning.
ETHICS STATEMENT
Since our method is a general contribution to the multi-task learning field, it has the potential to benefit any societal problem where multiple tasks are performed simultaneously, e.g., medical diagnosis. Accordingly, the method would also potentially face some negative social impacts accompanying with these applications, e.g., lack of fairness with the model trained by incomplete data due to the privacy of patients in medical imaging tasks.
REPRODUCIBILITY STATEMENT
To make our method reproducible, we provide the details of the method in Section 3. We also give the derivations of the evidence lower bound proposed in Section 3 in Appendix A. The datasets used in this paper can be obtained by referencing Section 5. There are also the experimental settings and implementation details for each dataset in Section 5. We also provide an anonymous link for the code of our method, which is in the abstract of this paper.

heading: A DERIVATION OF THE ELBO FOR MULTI-TASK NEURAL PROCESSES
text: We provide a derivation of ELBO of the proposed multi-task processed with hierarchical context modeling. The likelihood of multi-task learning is as follows:
p({Y∗l }|{X∗l }, {Dl}) = L∏ l=1 p(Y∗l |X∗l , {Dl})
= L∏ l=1 ∫ ∫ p(Y∗l |X∗l ,ψl)pθ1(ψl|αl,M)pθ2(αl|Dl)dαldψl
(12)
Based on conditional independence assumption, we introduce the variational joint posterior distribution factorized as (7). By incorporating the variational posteriors in the log likelihood, we can obtain the ELBO as follows:
log p({Y∗l }|{X∗l }, {Dl})
= L∑ l=1 log ∫ ∫ p(Y∗l |X∗l ,ψl)pθ1(ψl|αl,M)pθ2(αl|Dl)dαldψl
= L∑ l=1 log ∫ {∫ p(Y∗l |X∗l ,ψl)pθ1(ψl|αl,M) qϕ1(ψl|D∗l ) qϕ1(ψl|D∗l ) dψl } pθ2(αl|Dl) qϕ2(αl|D∗l ) qϕ2(αl|D∗l ) dαl
≥ L∑ l=1 { Eqϕ2 (αl|D∗l ) {∫ p(Y∗l |X∗l ,ψl)pθ1(ψl|αl,M) qϕ1(ψl|D∗l ) qϕ1(ψl|D∗l ) dψl }
− DKL[qϕ2(αl|D∗l )||pθ2(αl|Dl)] }
≥ L∑ l=1 { Eqϕ2 (αl|D∗l ) { Eqϕ1 (ψl|D∗l )[p(Y ∗ l |X∗l ,ψl)]− DKL[qϕ1(ψl|D∗l )||pθ1(ψl|αl,M)] } − DKL[qϕ2(αl|D∗l )||pθ2(αl|Dl)] } .
(13)

heading: B PROOF OF EXCHANGABILITY AND CONSISTENCY
text: We further provide theoretical proof to show that the proposed multi-task neural processes are valid stochastic processes, which completes the theory of multi-task neural processes. As the statement in Garnelo et al. (2018b): the conditions, including (finite) exchangeability and consistency, are sufficient to define a stochastic process. In our multi-input multi-output setting, we observe L tasks, {X∗l } where X∗l = {x∗l,i} n∗l i=1 and {Y∗l } where Y∗l = {y∗l,i} n∗l i=1. x ∗ l,i denotes the i-th target samples from task l and y∗l,i is its corresponding target or label. Here are the two propositions to state the exchangeability and consistency of the proposed multi-task neural processes. We model the functional posterior distribution of the stochastic process by approximating the joint predictive distribution over each target set p({Y∗l }|{X∗l }, {Dl}), which is conditioned on all context samples {Dl}. Proposition 1 (Exchangability) For finite n = ∑L l=1 n ∗ l , if π = {πl}Ll=1 is a permutation of {1, ...,n} where πl is a permutation of the corresponding order set {1, ..., n∗l }, then:
p(π({Y∗l })|π({X∗l }), {Dl}) = p({Y∗l }|{X∗l }, {Dl}), (14)
where π({Y∗l }) := (π1(Y∗1), ..., πL(Y∗L)) = (y∗π(1), ..., y ∗ π(n)) and π({X ∗ l }) := (π1(X ∗ 1), ..., πL(X ∗ L)) = (x ∗ π(1), ..., x ∗ π(n)).
Proof.
p(π({Y∗l })|π({X∗l }), {Dl})
= ∫ ∫ ( n∏ i=1 p(y∗π(i)|x ∗ π(i),ψ1:L) ) p(ψ1:T |α1:T ,M)p(α1:T |{Dl})dψ1:Ldα1:L
= ∫ ∫ ( L∏ l=1 n∗l∏ j=1 p(y∗π1(j)|x ∗ π1(j) ,ψl) )( L∏ l=1 p(ψl|αl,M) )( L∏ l=1 p(αl|Dl) ) dψ1:Ldα1:L
= ∫ ∫ ( L∏ l=1 n∗l∏ j=1 p(y∗j |x∗j ,ψl) )( L∏ l=1 p(ψl|αl,M) )( L∏ l=1 p(αl|Dl) ) dψ1:Ldα1:L = p({Y∗l }|{X∗l },M)
(15)
Proposition 2 (Consistency) Given m = ∑L l ml=1, if 1 ≤m ≤ n or for each task 1 ≤ ml ≤ n∗l ,
then: ∫ p({Y∗l }|{X∗l }, {Dl})d({Y∗l })m+1:n = p(({Y∗l })1:m|({X∗l })1:m, {Dl}), (16)
where ({Y∗l })1:m = ((Y∗1)1:m1 , ..., (Y∗L)1:mL) = (y∗1 , ..., y∗m) and ({X∗l })1:m = ((X∗1)1:m1 , ..., (X ∗ L)1:mL) = (x ∗ 1, ..., x ∗ m).
Proof.∫ p({Y∗l }|{X∗l }, {Dl})d({Y∗l })m+1:n
= ∫ ∫ ∫ ( n∏ i=1 p(y∗i |x∗i ,ψ1:L) )( L∏ l=1 p(ψl|αl,M) )( L∏ l=1 p(αl|Dl) ) dψ1:Ldα1:Ld({Yl})m+1:n
= ∫ ∫ ( m∏ i=1 p(y∗i |x∗i ,ψ1:L) )(∫ n∏ i=m+1 p(y∗i |x∗i ,ψ1:L)d({Y∗l })m+1:n )
( L∏ l=1 p(ψl|αl,M) )( L∏ l=1 p(αl|Dl) ) dψ1:Ldα1:L
= ∫ ∫ ( m∏ i=1 p(y∗i |x∗i ,ψ1:L) )( L∏ l=1 p(ψl|αl,M) )( L∏ l=1 p(αl|Dl) ) dψ1:Ldα1:L = p(({Y∗l })1:m|({X∗l })1:m,M) (17)

heading: C MORE EXPERIMENTAL DETAILS
text: Details of iteration numbers and batch sizes for different benchmarks are provided in Table 6. In each batch, the number of training samples from each task and category is identical. We train all models and parameters by the Adam optimizer Kingma & Ba (2014) using an NVIDIA Tesla V100 GPU. The learning rate is initially set as 1e − 4 and decreases with a factor of 0.5 every 3, 000 iterations. The network architectures of the proposed multi-task neural processes for multi-task classification are provided as follows.
The architecture of the inference network ϕ1 is provided in Table 7. The architecture of the inference network ϕ2 is provided in Table 8. We note that the inference network θ1 and θ2 share the same architectures with ϕ1 and ϕ2, respectively. The architecture of the neural network h is provided in Table 9. The network is needed because it provides a data-driven way for the model to incorporate the task-specific latent variable αl and the global variable M , which are usually defined in different feature spaces. During inference, we apply the reparameterization trick to generate the samples for the latent variables (Kingma & Welling, 2013).

heading: D MORE EXPERIMENTAL RESULTS
text: D.1 MULTI-TASK REGRESSION AND CLASSIFICATION WITH THE 20% SPLIT
Further, we provide experiments results on the three multi-task classification datasets with 20% training samples in Table 10, 11 and 12. The proposed multi-task neural processes consistently achieve the best performance on all three benchmarks.
D.2 MULTI-TASK REGRESSION WITH LESS DATA
To show the advantages of our model, we compare them on the setting of less data with the 0.05% split. In this case, there are only 20 samples per task during training. As shown in the Table 13, the improvement of our method becomes larger.
D.3 STABILITY OF OUR METHOD
The computational advantage of our method can be illustrated by the training loss as function of iteration on Office-Home. As shown in Fig. 4, our MTNPs converges more stable than NPs under 5%, 10% and 20% train-test splits. Moreover, we have added a new experiment to investigate the training stability by introducing the noise to the input data.
We apply the the fast gradient sign method (Goodfellow et al., 2014) to generate the noise. The results are given in Table 14, where η denotes the noise level. We observe that our MTNPs show better stability than NPs at different noise levels.
D.4 SENSITIVITY OF THE HYPERPARAMETER Nf AND Na
In practice, we set Nf and Na to be 10 and 5, which offer a good balance between performance and efficiency. We determine them by grid search as shown in Table 15 and 16.

Review summaries:
This paper presents a multi-task neural processes approach in which function priors are derived in a hierarchical Bayesian inference framework to incorporate the shared knowledge into its context of the prediction function. Authors introduced a higher-level latent variable derived from the context data of related tasks to control the sharing of common knowledge between the tasks. Previous works used context data from its own task to generate the task-specific latent variables, whereas this work uses data from other tasks as well. The shared knowledge from all the tasks added to context of each individual task acts as inductive bias for predictions. Their experiment results show better performance of the proposed method than compared ones. 

The paper proposes a multi-task learning model with neural processes. The model is based on a hierarchical construction whereby each task is conditioned on global and local information. The paper derives a “hierarchical” ELBO for this model that is evaluated with MC sampling. Experiments are presented to validate the proposed model.

This paper proposes a new multi-task neural process based on the classical neural process. The idea is to introduce additional global variables to share knowledge from different tasks. That is to say, a hierarchical bayesian model is constructed to link single-task neural processes together. The new model is able to handle multi-task applications contract to the classical neural process